Like any TDD proponent (and most cultists, really), the author insists that „you're not doing it right”, despite reading the scriptures. You're always misunderstanding, YOU are the reason TDD doesn't work, no, TDD is itself flawless. There's always a slight misinterpretation of the magic words! But TDD works, you're the sinner for not using it right!
And then you have shocked people when they find out that 100% test coverage doesn't mean that you really have a bug-free codebase.
Another huge problem with TDD is a lot of practitioners do not create negative tests. A negative test verifies that something the system shouldn't do is not being done. For example, if you shouldn't allow unauthenticated users unauthenticated users to delete something, there should be a test to verify that authenticated users can't delete something. But if you only add tests that will fail and then make succeed via implementation, you would probably never think to make those kind of test. You certainly can create negative tests with TDD, but most docs on TDD never discuss it.
This is how I’ve been making sure multitenancy in my app doesn’t leak data. It shouldn’t leak but how can I be sure? Run tests that try to access data on one tenant from another tenant and fail if the data is accessible.
You can certainly sometimes write a test that fails first to have a test that an unauthenticated user can't delete something.
You write the test before have implemented the authentication system. Either generally entirely, or just in the "delete something" subsystem. You write a test that says when unauthenticated user tries to delete something, an error of the appropriate kind happens. The test fails. Then you implement the authentication system to make the test pass.
I have written this exact kind of test.
I do not universally do "TDD" though, I probably write a failing test before I've written code less than 25% of the time. I do write what you call "negative" tests, sometimes TDD sometimes not -- I think I don't write "negative" tests TDD any less often than any other kind of test. I may not write "negative" tests enough, but it's not because of TDD (since I don't mostly write tests first!).
So I'm not here being a universal TDD adherent. I guess I'm just being boring and unhelpful "it depends, and I just use my intuition built on years of developing." (I think this kind of intuition generally grows after years of developing on the same platform, so people switching tech all teh time doesn't help. But that' anotehr topic).
I am aware "you just need to have skilled developers using their judgement" isn't particularly helpful. The attraction of TDD is the idea that you don't need that, a novice can follow this process and still achieve reliabile and robust results, efficiently. I don't really agree... but also think it's important to know about and have experienced TDD so you will have it in your toolbox to apply sometimes. I don't think "negative tests" is a particularly useful category for deciding when -- I have definitely done TDD usefully with "negative tests", you can totally write a failing test for something the system shoudln't do... in many circumstances.
I agree that TDD in the hands of an inexperienced developer isn't a substitute for stepping back and thinking about design/architecture, you still need to think about design/architecture. But also that TDD has definitely sometimes helped me think about design/architecture -- sometimes in a really profoundly useful way. The very unhelpful "it depends".
The amount of code paths that even simply code can traverse can become vastly large quickly. Negative testing is often like shooting in the dark.
Negative tests from what I've seen accrue by encountering unexpected results from code in the wild and writing tests to make sure they are handled. They are a result of code being used, not by proactive or predictive design.
If you would design it why wouldn't you test for it? If I were writing authentication and authorization I would test that unauthenticated and unauthorized activities produced the correct errors.
The big TDD misunderstanding is that like any tool it is to be wielded when appropriate, not followed rigidly for every single piece of code you write.
There are some safety standards and practices in engineering that are matters of life and death. If you're working high above the ground, you need two points of safety clipping, and if you're moving you never disconnect both of them. This is non-negotiable. There is never an appropriate time to violate this. If there is no way to complete the job without constant safety harness restraint, the job doesn't get done. (At least in the developed world.)
This is not the category where TDD or rules like "Never change your code without having a red test." fall in.
I adore TDD. It made me a better engineer, and I followed the process rigorously in my first 5-10 years of my career.
But at some point I realized that what that did is teach me how to write modular, extensible, testable code, and now I can write modules like that upfront, and add tests later. And truthfully, I do go faster. Because I also then focus my tests on JUST the important logic, not the intermediate abstractions in between. I write unit tests for the critical functionality and *integration* tests for the whole system to make sure the whole thing works as expected (because the dirty truth of unit tests is you can have 100% test coverage but a non-functional systems due to frameworks and glue code being incorrectly configured.
High test coverage is the worst metric one can aim for. As soon as frameworks and external APIs are involved, it's basically impossible to achieve without bending over backwards and needlessly creating mocks for half your system.
My biggest pet peeve in this context are test coverage requirements for OO languages that feature properties or getters and setters. 100% test coverage would require writing "tests" for those, which basically amounts to testing whether the compiler works - madness!
> "Originally the unit in “unit test” did not refer to the system under test but the test itself. Meaning the test can be executed as one unit and does not depend on other tests to run upfront (see here and here)."
I find the links provided to support for this proposition underwhelming. In any case it matters zero. No-one interprets unit test that way any more. Everyone interprets it the other way round, like defined on Wikipedia:
> "In computer programming, unit testing is a software testing method by which individual units of source code—sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures—are tested to determine whether they are fit for use. Unit tests are typically automated tests written and run by software developers to ensure that a section of an application (known as the "unit") meets its design and behaves as intended. In procedural programming, a unit could be an entire module, but it is more commonly an individual function or procedure. In object-oriented programming, a unit is often an entire interface, such as a class, or an individual method." -- https://en.wikipedia.org/wiki/Unit_testing
It's generally accepted good practice to not have dependencies between unit tests. Dependencies between unit tests are generally accepted as bad practice.
The central idea of a ‘unit’ test is that its failure should implicate one, and only one, part of the codebase. So while isolating tests is a necessary part of this (stale state from other tests would cloud the reasons for failure), I’ve never heard this isolation proposed as the primary feature of a unit test.
You could also say that no dependencies between unit tests imply no dependencies between modules under test. Hence, the article's claim is incorrect, as their interpretation implies the interpretation they are refuting.
> Instead the pyramid says we should write a lot of unit tests. This leads to an inside out approach testing the structure of the system rather its behaviour.
No it does not. Unit tests should test behavior.
> Only use mocks for truly external systems (e.g. mail service). The database should be part of the tests. Do not stub it.
First, mocks and stubs are not the same thing.
Secondly, stub fragile dependencies in unit tests is a good rule of thumb if you want easy, fast and non-erratic tests.
> Never change your code without having a red test. This is pretty common practice in TDD.
No, In TDD we change our code all the time without a red test, we call it refactoring and we only do that when the test is green.
The biggest misunderstanding about TDD is that you need to do TDD to be good engineer and write good code as promoted by some TDD evangelists. Writing tests is important, writing good tests is important... what is not important is the order you write the tests or if you follow an arcane set of "rules" about how you should write tests as if they are some ancient divine wisdom someone discovered on a stone tablet that's the key to writing good software.
No... forget about writing tests inside out or outside in, top down or bottom up. Forget about the bs about "TDD is actually there to help you design the code so you SHOULD write tests first to expose flaws in your design". It does not matter as long as you write good tests and feel productive while writing code.
Maybe TDD "rules" and rituals are useful when you are a relative beginner and instead of overwhelming them with a ton of information it is nice to have a set of rules and steps you can follow that have a good chance of resulting in a decent final product. But at a certain point you need to what works best for you and ignore the TDD dogma.
I am not anti-TDD, I practice TDD for parts of the code but do it in a way that if a TDD purist saw it they would term it blasphemous. I never use TDD as a design tool, on the contrary I always design the general structure of a class or a package based on intuition and experience without writing a single line of test code. Then when I have the general design and skeleton functionality of the module fleshed out I jump into tests and start doing TDD to start filling out the functionality, addressing edge cases and so on. Coupling design with tests slows me down as it results in a lot of thrashing between tests and real code. I think a large number of developers feel this way and have never really bought into the "TDD is for design" hype. If on the otherhand TDD helps you design, go for it. The point is that TDD should not be treated like religious dogma that it often is.
The biggest advantage of TDD is the dopamine hit it gives you when a test goes from red to green. That hit is so powerful that I continue to do TDD maybe 50% of the time but don't it dogmatically like TDD purists would want people to do it.
I think the thing is, you can absolutely write the code and then write the tests afterwards, but you might end up with something that is hard to test (which is probably an indication your code is not properly modularized).
You should take that as a sign you need to rewrite the code you’ve just written, but a lot of people don’t want to trash all (or part of) their hard work and instead bend the tests so that they deal with the code.
You now have bad code, and you have bad/fragile tests.
I think TDD (when enforced) is mostly meant to prevent people from falling for that.
I don't always write the tests first, I do what seems practical and feasible. Sometimes it feels more natural to write some code to understand the problem I'm solving.
That said - there is a problem with writing tests last: You don't actually know if your tests are going to fail when they should, because you haven't run them on a code-base that isn't fully implemented.
You can deal with this by stashing the implementation changes and running the tests to confirm they fail where expected, but sometimes this is very challenging because you need a partial implementation to reach some of the failed assertions. This requires thoughtfulness and discipline and so it is hard to maintain this consistently across a team. It isn't possible in a code review to confirm if people have done this, and often it isn't practical to do it yourself as part of reviewing code.
TDD is "just" a technique, and being dogmatic about it, is as you say totally pointless. Still I would consider knowing and practicing this technique a "must known" in the industry. When I look back to my 15 years coding :
- the time I lost creating bad designed code and manual testing is huge
- all the companies and projects I have joined always have terrible designed/unmaintainable code, made sometime by 10y+ experienced people and explaining at least how to write proper test and how to design a code that is testable is always the game changer to increase a project velocity and down the defect rate
I will start that I think TDD is very valuable. But reading this article that quickly goes into a numbered list of "rules" is a little disheartening. Already some other commenters are talking about more nuanced views of when to test. I think the real tragedy is that I see a lot of engineers still not writing tests. So when speaking to the broader audience, you have to be really gung-ho about it to get people to do what I think is a very necessary step. People should err on the side of writing the test first, both to get most people in the right mindset, and also to avoid the broken window syndrome of a lot of missing tests and no one bothering to actually start doing what should be obvious: most code should have at least some level of coverage. Most in fact. If there are "reasons", sure, if you're going to pull the senior engineer move and skip the test, then you better be able to back it up clearly. If not, write the damn test. If not you, someone will appreciate it later.
This statement made me think: "The database should be part of the tests. Do not stub it".
Databases are both 1) complex piece of software and 2) hard to stub.
5-10 years ago having a database for testing was expensive.
Today you don't need to pay licenses for most databases, and you can spin a disposable one in a container. Easier than mocking the DB itself.
If, for other reasons, I follow the Hexagonal architecture, replacing the PostgresProjectionsRepository with an MemoryProjectionsRepository is both trivial and easier. Replacing an S3LogStore with a StdOutLogStore just as easy and trivial.
But, if, God forbid, I'm stuck with a tangled mess of Rails ActiveRecord Models that break every SOLID principle with both feet, are highly coupled to the database and often even arbitrarily stick business logic in the DB or in the models, then certainly: it is hard.
And while this sounds like a rant on AR (it is!), ActiveRecord has its positive trade-offs, or, often, simply is there. Pragmatism dictates that sometimes the database is hard or impossible to stub and you're far better off just giving up and treating the entire directory of models+the-fully-seeded-database as a single unit instead.
But, what the OP of the article overlooks entirely - and what many haters of TDD miss completely: the tests are yelling important information at us: the design is a mess, there is too much coupling, we lack abstractions, there's too much going on, we are building God-classes and so on.
But, again, sometimes pragmatism dictates we ignore this and go for the bad design anyway. As long as we pro-actively chose to go for the Bad Design (rather than unknowingly grow into it) this is fine, IMO.
By my personal experience to make TDD work (where you make many small changes, run the tests, make more changes - step by step), tests need to run FAST - as in ideally less than 1 second.
Even if it is easy to spin up a container, seed a database inside that container and drop and recreate that database after every single test (so that tests remain a unit and can't influence each other) - it tends to take a few seconds, which instantly makes TDD tedious to do. Thus you start making more changes at once, run the tests less, and drift away from doing TDD...
The most typical solution is to run databases in-memory rather than in a container.
My personal approach though is to make my code more modular, so that only the "database accessor" class needs to mock the actual database, while the "query builder" needs to mock only that database accessor (much easier) - and the database class needs to only mock the query builder (even easier) - and the actual application code needs to only mock the database class (super easy).
You will, though, end up fighting the system in some ways. CI/CD systems, unit test frameworks, configuration data approaches, etc, aren't equipped to (easily) spin up databases and connect to them. At least not in the more rapid cycle early end of development, versus slower cycling integration tests.
It can still be very expensive after a while. I assume TDD practitioners also practice pruning their test suites to keep them from snowballing indefinitely. If you don't, then eventually running hundreds of thousands of tests against a real database is painful both in terms of waiting time and resource usage.
The GUI example:
1) Make the webdriver based test click a button which does not exists yet. The test will fail, you know that there is no other button which by chance matches this selector.
2) Add the button to the app, the test turns green,
3) you Continue to extend the test to click another botton, etc till it fails again
The Game example:
e2e tests for games probably do not make much sense (I've no experience here). So you will have components/unit tests. Those have a defined interface and behaviour. You want to change this, you change the test first like in any other project.
>Do not isolate code when you test it. If you do so, the tests become fragile and do not help you in case you refactor the software.
I don't agree with this bit. This is how you make a fragile test suites that are difficult to refactor. Why do we go out of our way to hide information in our code? Answer because code that spaghettis in an out of a lot of other code means small changes cause unexpected regressions, and the changes to fix that regression causes other regressions. Suites that don't isolate code have exactly this problem. They are like a scavenger hunt of assumptions and inevitably lead to people fixing the tests instead of the code. In addition because you have introduced a network effect a small change will likely invalidate a huge number of tests. Meaning you have to update them all.
Integration suites have value, but I use them with discretion because I know that they are a challenge to maintain. Tight unit tests that test small pieces of logic thoroughly and make ample use of mocks to ensure deterministic outcomes are a must. These tests are highly durable. The code can change all around it but that function will work as advertised until someone changes it. Then they can update its test suite and you're back in the golden zone. In my experience if you build a complex structure out of a lot of quality components that complex structure will tend to be high quality itself. My pattern usually pushes the complexity into the units and I leave "orchestration" units that tie everything together untested but intentionally simple.
I know the code coverage folks will chime for sure, but I don't aim for 100% code coverage. I aim for a balanced test suite that ensures the best possible outcome for the minimum investment with the least amount of hassle when introducing new changes. Trying to test everything never seems to get me there.
In reality it depends if you should isolate the code or not, are you writing a unit test or an integration test?
If you wrote an algorithmic function crucial to your app, isolate that MF and write a unit test. If you're testing a transformer or API response, you don't need to isolate each layer and run/write your integration tests at this level. If you can't isolate your code for unittest and you're writing OOP your architecture probably needs to be rethought.
That said my goal when writing something new is to have all the critical/core functionality covered on the backend, with a test case for edge cases. The biggest goal is I don't know who will be working on it next, and if they miss-understand what a function is doing because they are just trying to fix a "bug", the test case should catch it. That's why it's import in TTD to have one person write the tests and someone else write the code. More so if your team , is a clusterfuck particularly some of the senior devs are hacks.
> Answer because code that spaghettis in an out of a lot of other code means small changes cause unexpected regressions
That's not a fragile test suite, that's a test suite working well: you want tests to start failing if you introduced a regression to the overall program behaviour.
> the changes to fix that regression causes other regressions
It sounds like what you're really getting at is something different from test fragility: specifically, if a test fails, how easy is it to identify where in the code a fix needs to be applied. That is also an important consideration when writing tests but it's completely separate from test fragility.
> Integration suites have value, but I use them with discretion because I know that they are a challenge to maintain.
This is the opposite of how the test pyramid is supposed to work. Integration tests are supposed to be less fragile because they don't rely on specific implementation details or mocks: they're mimicking a real user or a consumer of your API. If you are getting false positives from your integration tests a lot, then either the tests are written poorly, or you're making too many breaking changes to your library/product/whatever.
> Tight unit tests that test small pieces of logic thoroughly and make ample use of mocks to ensure deterministic outcomes are a must. The code can change all around it but that function will work as advertised until someone changes it. Then they can update its test suite and you're back in the golden zone.
That's completely useless as a test then. Of course if you change code around it the test won't break. The whoe point of a test is that it doesn't break even for (some) changes to the code being tested, and if you always have to update the unit test when you change the code then it is providing no vlaue and giving you a false sense of security that all your tests are green.
Unit tests are great when you have a particular irreducable complexity in a piece of code, or more generally they are good when there is some constraint that should be locally upheld that is likely to be more permanent than the specific implementation.
I'm a bit of a cowboy coder and completely skip all but the very top of the testing pyramid. I strictly run E2E tests and albeit slower than other tests, it tests a much broader spectrum than just a single function. For instance I change the expiration date of a user's subscription, run expire logic, test of this user has received the expected expiration email, this automatically tests the Amazon SES integration, the email template parser and everything in between. Clicking on the expire email call to action button I expect to land on a certain page with elements "missing" and others added, which validates changes in the user's roles. Let's say I swap Amazon SES for Postmark, no new tests have to be written. In my experience this approach works really well for a SaaS like product. It also looks satisfying to see the test suite blast through the site clicking and typing like there is no tomorrow.
This is contrary to most testing tutorials and documentation I've read.
I get frustrated and forego a lot of testing because the idea of mocking out basically the whole system in order to unit test is insane to me. If function A calls function B, you've got to mock function B out. If you refactor function B, you better update the mock as well. Writing an app twice hardly seems like a good use of time.
> If function A calls function B, you've got to mock function B out. If you refactor function B, you better update the mock as well
This is why isolation and encapsulation are important.
Your tests were telling you that A has intricate dependencies on B. If the tests for A start failing when you refactor B, they were (too) tightly coupled. The tests are telling you this (whether or not you act on this is another discussion).
And when the tests for A keep passing even when you broke B -false positives- you should see at least one integration (or even e2e) test failing. If not, the tests are showing a hole in the coverage.
If function A does not much more than calling B, maybe testing A is not strictly necessary.
If A does something important/complex/critical/etc maybe the important/complex/critical/etc part can be refactored into a dedicated function A1 without external dependency and then tested on its own? That way, the tests focuses on where the money is and gets easier to work with at the same time.
That way leads to madness, and I don't follow that style/philosophy of mocking components.
Only mock the things you can't easily set up in your unit tests to get them to work, like database connections. For spring-based components, try to use the real components where possible, but where needed use mock components that do enough to get the system working.
That does mean that if you are testing function A you are also indirectly testing function B, but that is fine.
Clickbait title I would say. The 'big' issue with TDD is whether it's worth the time needed to do it. If it was obviously always worth the time, then I think we wouldn't even talk about it. (Do we worry about whether breathing is worth it? Nope, we never talk about that because the answer is apparent.) Whether it's worth it depends on the context.
I sitll don't understand whether TDD applies outside of some rather specific domains.
In most cases, apart from utility methods, I don't actually know how something will work before I have at least partially implemented it. We have implemented a payments change recently (Paypal to Recurly) and not only are the two very different but we have added additional functionality that we could have done with Paypal but didn't. How do you TDD that?
Even writing a simple test like "Go to this page and click <Order>" doesn't really work since when you come to it, you realise you can't just order but you also need to go to the create account page first etc.
Counterquestion: how do you know, at each release, that the paypal-stuff still works?
If the answer is "I test it by making a purchase at each release", the obvious question is: why not automate this? That is one benefit of TDD.
So why not start of with that automation?
In your specific case, I would write a simple test first, that, says "on the page of the orange Y-combinator swag, order a shirt and receive a timely confirmation.".
Which is subtly, yet importantly different from "go to X and click order". On the level of such acceptence tests, trying to keep away from implementation details like "pages", "buttons" and so on, helps with solving your problem too: you write down the intent&outcome, rather than the implementation. Focus on the business value. Solve that first. And fixate the outcome for all future: regardless of the PSP, pages, exact values (timely vs within 1 sec), actual confirmation-channels and whatnot, the business value is "green" when it works, and "red" when it doesn't.
Then I start implementing it. Maybe just add one of these <iframe> things that Paypal offers, even. I don't really care: the quest thing to make it work. as long as it gets me to have that acceptance test green with least work, I'm happy.
Happy, but certainly not done. Because the most overlooked part of TDD is the Refactoring. Red-Green-Refactor. Once it is green, start refactoring. Replace the iframe with some library. Go a step further and replace that with my own paypal-API-client. Connect that to my event-stream, or not.
A lot of effective functional testing is getting fixtures ready so that your test can reliably hit the important part of the system under test. For external integrations this becomes difficult. Presumably your test uses the api sandbox, so hopefully it gives reproducible results. If not, then you need to test manually and focus on unit testing to assure the software.
The issue with TDD as originally described by Kent Beck (there are as many variations as there are ... ) is the belief that a good enough design emerges from the process.
I know it's good commenting style on HN to avoid quick one-liners, but TDD is one of those topics that we've kicked around forever. Perhaps a one-liner would help clarify things.
TDD is a coding practice to help the programmer understand that their mental model of the code does not reflect the actual code itself.
That's it. There are some follow-ons, of course, like if your mental model is exactly in-line with the code then logical errors are unlikely (although business errors can be quite common!). Another follow-on is that TDD is much more about design than testing. Once you realize that it's a practice or habit that's all about sending information from the code back to the mind of the coder, all of that other stuff sorts itself out.
> The test that hits the endpoint is commonly referred to as an “integration test”, and the other test is commonly referred to as a “unit test”.
> Kent Beck (the originator of test-driven development) defines a unit test as “a test that runs in isolation from other tests”. This is very different from the definition of a unit test as “a test that tests a class/method in isolation from other classes/methods”. The test that hits the endpoint satisfies the original definition.
> But it doesn’t really matter if you want to call a given test an “integration” test or a “unit” test. The point is the test fails when something breaks and passes when something is improved. If it does the opposite, it’s not a good test.
I don't take testing advice from anyone who uses generic terms like "test coverage". Test coverage is a complex subject, but without a basic understanding, tests can create a false sense of security. Saying something like "100% coverage" means almost nothing to anyone that understands testing.
[+] [-] dorinlazar|4 years ago|reply
And then you have shocked people when they find out that 100% test coverage doesn't mean that you really have a bug-free codebase.
[+] [-] d--b|4 years ago|reply
TDD is about designing how a module is going to be used before actually implementing it.
It’s about considering the API as a product, rather than a side effect of the implementation.
It’s about putting oneself in the user’s shoes, instead of making it as easy as possible for the implementer.
Test coverage depends on the implementation, so you can’t design tests that will ensure 100% test coverage before implementing the stuff.
[+] [-] dwheeler|4 years ago|reply
[+] [-] dqv|4 years ago|reply
[+] [-] jrochkind1|4 years ago|reply
You write the test before have implemented the authentication system. Either generally entirely, or just in the "delete something" subsystem. You write a test that says when unauthenticated user tries to delete something, an error of the appropriate kind happens. The test fails. Then you implement the authentication system to make the test pass.
I have written this exact kind of test.
I do not universally do "TDD" though, I probably write a failing test before I've written code less than 25% of the time. I do write what you call "negative" tests, sometimes TDD sometimes not -- I think I don't write "negative" tests TDD any less often than any other kind of test. I may not write "negative" tests enough, but it's not because of TDD (since I don't mostly write tests first!).
So I'm not here being a universal TDD adherent. I guess I'm just being boring and unhelpful "it depends, and I just use my intuition built on years of developing." (I think this kind of intuition generally grows after years of developing on the same platform, so people switching tech all teh time doesn't help. But that' anotehr topic).
I am aware "you just need to have skilled developers using their judgement" isn't particularly helpful. The attraction of TDD is the idea that you don't need that, a novice can follow this process and still achieve reliabile and robust results, efficiently. I don't really agree... but also think it's important to know about and have experienced TDD so you will have it in your toolbox to apply sometimes. I don't think "negative tests" is a particularly useful category for deciding when -- I have definitely done TDD usefully with "negative tests", you can totally write a failing test for something the system shoudln't do... in many circumstances.
I agree that TDD in the hands of an inexperienced developer isn't a substitute for stepping back and thinking about design/architecture, you still need to think about design/architecture. But also that TDD has definitely sometimes helped me think about design/architecture -- sometimes in a really profoundly useful way. The very unhelpful "it depends".
[+] [-] AtlasBarfed|4 years ago|reply
Negative tests from what I've seen accrue by encountering unexpected results from code in the wild and writing tests to make sure they are handled. They are a result of code being used, not by proactive or predictive design.
[+] [-] necovek|4 years ago|reply
What makes you feel it's "a lot of practitioners"?
[+] [-] unknown|4 years ago|reply
[deleted]
[+] [-] projektfu|4 years ago|reply
[+] [-] rmetzler|4 years ago|reply
[+] [-] jay_kyburz|4 years ago|reply
[+] [-] deanCommie|4 years ago|reply
There are some safety standards and practices in engineering that are matters of life and death. If you're working high above the ground, you need two points of safety clipping, and if you're moving you never disconnect both of them. This is non-negotiable. There is never an appropriate time to violate this. If there is no way to complete the job without constant safety harness restraint, the job doesn't get done. (At least in the developed world.)
This is not the category where TDD or rules like "Never change your code without having a red test." fall in. I adore TDD. It made me a better engineer, and I followed the process rigorously in my first 5-10 years of my career.
But at some point I realized that what that did is teach me how to write modular, extensible, testable code, and now I can write modules like that upfront, and add tests later. And truthfully, I do go faster. Because I also then focus my tests on JUST the important logic, not the intermediate abstractions in between. I write unit tests for the critical functionality and *integration* tests for the whole system to make sure the whole thing works as expected (because the dirty truth of unit tests is you can have 100% test coverage but a non-functional systems due to frameworks and glue code being incorrectly configured.
[+] [-] qayxc|4 years ago|reply
My biggest pet peeve in this context are test coverage requirements for OO languages that feature properties or getters and setters. 100% test coverage would require writing "tests" for those, which basically amounts to testing whether the compiler works - madness!
[+] [-] theteapot|4 years ago|reply
> "Originally the unit in “unit test” did not refer to the system under test but the test itself. Meaning the test can be executed as one unit and does not depend on other tests to run upfront (see here and here)."
I find the links provided to support for this proposition underwhelming. In any case it matters zero. No-one interprets unit test that way any more. Everyone interprets it the other way round, like defined on Wikipedia:
> "In computer programming, unit testing is a software testing method by which individual units of source code—sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures—are tested to determine whether they are fit for use. Unit tests are typically automated tests written and run by software developers to ensure that a section of an application (known as the "unit") meets its design and behaves as intended. In procedural programming, a unit could be an entire module, but it is more commonly an individual function or procedure. In object-oriented programming, a unit is often an entire interface, such as a class, or an individual method." -- https://en.wikipedia.org/wiki/Unit_testing
It's generally accepted good practice to not have dependencies between unit tests. Dependencies between unit tests are generally accepted as bad practice.
[+] [-] thom|4 years ago|reply
[+] [-] unknown|4 years ago|reply
[deleted]
[+] [-] js8|4 years ago|reply
[+] [-] cenny|4 years ago|reply
No it does not. Unit tests should test behavior.
> Only use mocks for truly external systems (e.g. mail service). The database should be part of the tests. Do not stub it.
First, mocks and stubs are not the same thing.
Secondly, stub fragile dependencies in unit tests is a good rule of thumb if you want easy, fast and non-erratic tests.
> Never change your code without having a red test. This is pretty common practice in TDD.
No, In TDD we change our code all the time without a red test, we call it refactoring and we only do that when the test is green.
There are so many misunderstandings in testing and TDD. I wished more people would read Kent Becks book [Test-Driven Development: by example](https://www.programmingbooks.dev/#test-driven-development) and then the amazing book by Gerard Meszaros [XUnit Test Patterns](https://www.programmingbooks.dev/#xunit-test-patterns).
In the end I'm just happy if more people practice TDD and understand it's a skill to learn like everything else and not a silver bullet.
[+] [-] WolfOliver|4 years ago|reply
Yes, they should but often they do not. The likelihood to get it right is higher using an outside-in approach.
[+] [-] avl999|4 years ago|reply
No... forget about writing tests inside out or outside in, top down or bottom up. Forget about the bs about "TDD is actually there to help you design the code so you SHOULD write tests first to expose flaws in your design". It does not matter as long as you write good tests and feel productive while writing code.
Maybe TDD "rules" and rituals are useful when you are a relative beginner and instead of overwhelming them with a ton of information it is nice to have a set of rules and steps you can follow that have a good chance of resulting in a decent final product. But at a certain point you need to what works best for you and ignore the TDD dogma.
I am not anti-TDD, I practice TDD for parts of the code but do it in a way that if a TDD purist saw it they would term it blasphemous. I never use TDD as a design tool, on the contrary I always design the general structure of a class or a package based on intuition and experience without writing a single line of test code. Then when I have the general design and skeleton functionality of the module fleshed out I jump into tests and start doing TDD to start filling out the functionality, addressing edge cases and so on. Coupling design with tests slows me down as it results in a lot of thrashing between tests and real code. I think a large number of developers feel this way and have never really bought into the "TDD is for design" hype. If on the otherhand TDD helps you design, go for it. The point is that TDD should not be treated like religious dogma that it often is.
The biggest advantage of TDD is the dopamine hit it gives you when a test goes from red to green. That hit is so powerful that I continue to do TDD maybe 50% of the time but don't it dogmatically like TDD purists would want people to do it.
[+] [-] Aeolun|4 years ago|reply
You should take that as a sign you need to rewrite the code you’ve just written, but a lot of people don’t want to trash all (or part of) their hard work and instead bend the tests so that they deal with the code.
You now have bad code, and you have bad/fragile tests.
I think TDD (when enforced) is mostly meant to prevent people from falling for that.
[+] [-] jeremyjh|4 years ago|reply
That said - there is a problem with writing tests last: You don't actually know if your tests are going to fail when they should, because you haven't run them on a code-base that isn't fully implemented.
You can deal with this by stashing the implementation changes and running the tests to confirm they fail where expected, but sometimes this is very challenging because you need a partial implementation to reach some of the failed assertions. This requires thoughtfulness and discipline and so it is hard to maintain this consistently across a team. It isn't possible in a code review to confirm if people have done this, and often it isn't practical to do it yourself as part of reviewing code.
[+] [-] vmaurin|4 years ago|reply
- the time I lost creating bad designed code and manual testing is huge
- all the companies and projects I have joined always have terrible designed/unmaintainable code, made sometime by 10y+ experienced people and explaining at least how to write proper test and how to design a code that is testable is always the game changer to increase a project velocity and down the defect rate
[+] [-] gleenn|4 years ago|reply
[+] [-] eb0la|4 years ago|reply
Databases are both 1) complex piece of software and 2) hard to stub. 5-10 years ago having a database for testing was expensive. Today you don't need to pay licenses for most databases, and you can spin a disposable one in a container. Easier than mocking the DB itself.
[+] [-] berkes|4 years ago|reply
If, for other reasons, I follow the Hexagonal architecture, replacing the PostgresProjectionsRepository with an MemoryProjectionsRepository is both trivial and easier. Replacing an S3LogStore with a StdOutLogStore just as easy and trivial.
But, if, God forbid, I'm stuck with a tangled mess of Rails ActiveRecord Models that break every SOLID principle with both feet, are highly coupled to the database and often even arbitrarily stick business logic in the DB or in the models, then certainly: it is hard.
And while this sounds like a rant on AR (it is!), ActiveRecord has its positive trade-offs, or, often, simply is there. Pragmatism dictates that sometimes the database is hard or impossible to stub and you're far better off just giving up and treating the entire directory of models+the-fully-seeded-database as a single unit instead.
But, what the OP of the article overlooks entirely - and what many haters of TDD miss completely: the tests are yelling important information at us: the design is a mess, there is too much coupling, we lack abstractions, there's too much going on, we are building God-classes and so on.
But, again, sometimes pragmatism dictates we ignore this and go for the bad design anyway. As long as we pro-actively chose to go for the Bad Design (rather than unknowingly grow into it) this is fine, IMO.
[+] [-] hooby|4 years ago|reply
Even if it is easy to spin up a container, seed a database inside that container and drop and recreate that database after every single test (so that tests remain a unit and can't influence each other) - it tends to take a few seconds, which instantly makes TDD tedious to do. Thus you start making more changes at once, run the tests less, and drift away from doing TDD...
The most typical solution is to run databases in-memory rather than in a container.
My personal approach though is to make my code more modular, so that only the "database accessor" class needs to mock the actual database, while the "query builder" needs to mock only that database accessor (much easier) - and the database class needs to only mock the query builder (even easier) - and the actual application code needs to only mock the database class (super easy).
[+] [-] tyingq|4 years ago|reply
[+] [-] alkonaut|4 years ago|reply
[+] [-] pjmlp|4 years ago|reply
Now go write a native GUI, a game or an audio engine with that approach.
[+] [-] WolfOliver|4 years ago|reply
The Game example: e2e tests for games probably do not make much sense (I've no experience here). So you will have components/unit tests. Those have a defined interface and behaviour. You want to change this, you change the test first like in any other project.
The Audio engine - what is an audio engine?
[+] [-] bckr|4 years ago|reply
[+] [-] SassyGrapefruit|4 years ago|reply
I don't agree with this bit. This is how you make a fragile test suites that are difficult to refactor. Why do we go out of our way to hide information in our code? Answer because code that spaghettis in an out of a lot of other code means small changes cause unexpected regressions, and the changes to fix that regression causes other regressions. Suites that don't isolate code have exactly this problem. They are like a scavenger hunt of assumptions and inevitably lead to people fixing the tests instead of the code. In addition because you have introduced a network effect a small change will likely invalidate a huge number of tests. Meaning you have to update them all.
Integration suites have value, but I use them with discretion because I know that they are a challenge to maintain. Tight unit tests that test small pieces of logic thoroughly and make ample use of mocks to ensure deterministic outcomes are a must. These tests are highly durable. The code can change all around it but that function will work as advertised until someone changes it. Then they can update its test suite and you're back in the golden zone. In my experience if you build a complex structure out of a lot of quality components that complex structure will tend to be high quality itself. My pattern usually pushes the complexity into the units and I leave "orchestration" units that tie everything together untested but intentionally simple.
I know the code coverage folks will chime for sure, but I don't aim for 100% code coverage. I aim for a balanced test suite that ensures the best possible outcome for the minimum investment with the least amount of hassle when introducing new changes. Trying to test everything never seems to get me there.
[+] [-] winddude|4 years ago|reply
If you wrote an algorithmic function crucial to your app, isolate that MF and write a unit test. If you're testing a transformer or API response, you don't need to isolate each layer and run/write your integration tests at this level. If you can't isolate your code for unittest and you're writing OOP your architecture probably needs to be rethought.
That said my goal when writing something new is to have all the critical/core functionality covered on the backend, with a test case for edge cases. The biggest goal is I don't know who will be working on it next, and if they miss-understand what a function is doing because they are just trying to fix a "bug", the test case should catch it. That's why it's import in TTD to have one person write the tests and someone else write the code. More so if your team , is a clusterfuck particularly some of the senior devs are hacks.
[+] [-] Diggsey|4 years ago|reply
That's not a fragile test suite, that's a test suite working well: you want tests to start failing if you introduced a regression to the overall program behaviour.
> the changes to fix that regression causes other regressions
It sounds like what you're really getting at is something different from test fragility: specifically, if a test fails, how easy is it to identify where in the code a fix needs to be applied. That is also an important consideration when writing tests but it's completely separate from test fragility.
> Integration suites have value, but I use them with discretion because I know that they are a challenge to maintain.
This is the opposite of how the test pyramid is supposed to work. Integration tests are supposed to be less fragile because they don't rely on specific implementation details or mocks: they're mimicking a real user or a consumer of your API. If you are getting false positives from your integration tests a lot, then either the tests are written poorly, or you're making too many breaking changes to your library/product/whatever.
> Tight unit tests that test small pieces of logic thoroughly and make ample use of mocks to ensure deterministic outcomes are a must. The code can change all around it but that function will work as advertised until someone changes it. Then they can update its test suite and you're back in the golden zone.
That's completely useless as a test then. Of course if you change code around it the test won't break. The whoe point of a test is that it doesn't break even for (some) changes to the code being tested, and if you always have to update the unit test when you change the code then it is providing no vlaue and giving you a false sense of security that all your tests are green.
Unit tests are great when you have a particular irreducable complexity in a piece of code, or more generally they are good when there is some constraint that should be locally upheld that is likely to be more permanent than the specific implementation.
[+] [-] cersa8|4 years ago|reply
[+] [-] ehutch79|4 years ago|reply
I get frustrated and forego a lot of testing because the idea of mocking out basically the whole system in order to unit test is insane to me. If function A calls function B, you've got to mock function B out. If you refactor function B, you better update the mock as well. Writing an app twice hardly seems like a good use of time.
[+] [-] tra3|4 years ago|reply
[+] [-] berkes|4 years ago|reply
This is why isolation and encapsulation are important.
Your tests were telling you that A has intricate dependencies on B. If the tests for A start failing when you refactor B, they were (too) tightly coupled. The tests are telling you this (whether or not you act on this is another discussion).
And when the tests for A keep passing even when you broke B -false positives- you should see at least one integration (or even e2e) test failing. If not, the tests are showing a hole in the coverage.
[+] [-] pintxo|4 years ago|reply
If A does something important/complex/critical/etc maybe the important/complex/critical/etc part can be refactored into a dedicated function A1 without external dependency and then tested on its own? That way, the tests focuses on where the money is and gets easier to work with at the same time.
[+] [-] rhdunn|4 years ago|reply
Only mock the things you can't easily set up in your unit tests to get them to work, like database connections. For spring-based components, try to use the real components where possible, but where needed use mock components that do enough to get the system working.
That does mean that if you are testing function A you are also indirectly testing function B, but that is fine.
[+] [-] morelish|4 years ago|reply
[+] [-] lbriner|4 years ago|reply
In most cases, apart from utility methods, I don't actually know how something will work before I have at least partially implemented it. We have implemented a payments change recently (Paypal to Recurly) and not only are the two very different but we have added additional functionality that we could have done with Paypal but didn't. How do you TDD that?
Even writing a simple test like "Go to this page and click <Order>" doesn't really work since when you come to it, you realise you can't just order but you also need to go to the create account page first etc.
[+] [-] berkes|4 years ago|reply
Counterquestion: how do you know, at each release, that the paypal-stuff still works?
If the answer is "I test it by making a purchase at each release", the obvious question is: why not automate this? That is one benefit of TDD.
So why not start of with that automation?
In your specific case, I would write a simple test first, that, says "on the page of the orange Y-combinator swag, order a shirt and receive a timely confirmation.".
Which is subtly, yet importantly different from "go to X and click order". On the level of such acceptence tests, trying to keep away from implementation details like "pages", "buttons" and so on, helps with solving your problem too: you write down the intent&outcome, rather than the implementation. Focus on the business value. Solve that first. And fixate the outcome for all future: regardless of the PSP, pages, exact values (timely vs within 1 sec), actual confirmation-channels and whatnot, the business value is "green" when it works, and "red" when it doesn't.
Then I start implementing it. Maybe just add one of these <iframe> things that Paypal offers, even. I don't really care: the quest thing to make it work. as long as it gets me to have that acceptance test green with least work, I'm happy.
Happy, but certainly not done. Because the most overlooked part of TDD is the Refactoring. Red-Green-Refactor. Once it is green, start refactoring. Replace the iframe with some library. Go a step further and replace that with my own paypal-API-client. Connect that to my event-stream, or not.
[+] [-] projektfu|4 years ago|reply
[+] [-] sorokod|4 years ago|reply
[+] [-] DanielBMarkham|4 years ago|reply
TDD is a coding practice to help the programmer understand that their mental model of the code does not reflect the actual code itself.
That's it. There are some follow-ons, of course, like if your mental model is exactly in-line with the code then logical errors are unlikely (although business errors can be quite common!). Another follow-on is that TDD is much more about design than testing. Once you realize that it's a practice or habit that's all about sending information from the code back to the mind of the coder, all of that other stuff sorts itself out.
Shameless plug: this has been on my mind quite a bit lately since I blogged last week on the unreasonable effectiveness of testing as a way out of both cognitive and coding anti-patterns: https://danielbmarkham.com/cognition-versus-programming-the-...
[+] [-] bedobi|4 years ago|reply
> Terminology
> The test that hits the endpoint is commonly referred to as an “integration test”, and the other test is commonly referred to as a “unit test”.
> Kent Beck (the originator of test-driven development) defines a unit test as “a test that runs in isolation from other tests”. This is very different from the definition of a unit test as “a test that tests a class/method in isolation from other classes/methods”. The test that hits the endpoint satisfies the original definition.
> But it doesn’t really matter if you want to call a given test an “integration” test or a “unit” test. The point is the test fails when something breaks and passes when something is improved. If it does the opposite, it’s not a good test.
https://medium.com/expedia-group-tech/test-your-tests-f4e361...
[+] [-] throwaway9870|4 years ago|reply
This is a worthwhile read:
https://en.wikipedia.org/wiki/Code_coverage
Without formal methods, software will realistically never be 100% tested. But don't worry, your hardware isn't either!
[+] [-] furstenheim|4 years ago|reply
So many bugs that only happen on disk. Also so much functionality that cannot be used to fixed them because it's syntax error on the mock database