I like this approach in part because making lots of leaves also adds to the "literate"-ness of the code. With lots of opportunities to name your primitives, the code is much closer to being self documenting.
Depending on the project and its requirements, I also think "lazy" testing has value. Any time you are looking at a block of code, suspicious that it's the source of a bug, write a test for it. If you're in an environment where bugs aren't costly, where attribution goes through few layers of code, and bugs are easily visible when they occur, this can save a lot of time.
All of these basically go the opposite way of the article's philosophy:
Not too many integration tests, mostly unit tests. Clearly define a contract between the boundaries of the code, and stub/mock on the contract. You'll be left with mostly pure functions at the leaves, which you'll unit test.
I’ve been practicing TDD for 6 years and this is exactly what I ended up doing. It’s a fantastic way to program.
My leaves are either pure functions (FP languages) or value objects that init themselves based on other value objects (OOP languages). These value objects have no methods, no computed properties, etc. Just inert data.
No mocks and no “header” interfaces needed.
On top of that I sprinkle a bunch of UI tests to verify it’s all properly wired up.
This is my exact mentality as well! In fact, I like it so much that I apply it to system design as well. Structure the pieces of code into a directed acyclic graph for great success. A tree structure that terminated leaves is a DAG.
Is there any situation where there is integration, but no need to test it?
You seem to be suggesting that if the leaves are thoroughly tested, nothing can go wrong in their integration, but at the same time, I cannot imagine someone believing that.
So much people in this thread is talking about different domains and are not able to see that they need different rules.
It is not the same creating a library that is going to be used to launch a multi-billion rocket to Mars than developing a mostly graphical mobile app where requirements are changing daily as you A/B test your way into better business value.
The article has really good points and the reasons why they work. Apply them wisely.
Take the right decision for your project.
Don't be dogmatic.
I've programmed in many domains. There are only three places where I don't use heavy testing:
1. Very, very graphical programming - like SVG charting with animations. If it were static generation it would be easy, but throwing time into the mix makes the tests really hard plus if things go wrong in the future people can literally see it going wrong and complain, so I don't think it is worth the trouble.
2. Data analysis meant for static reporting. You know, those 2000 line SQL queries that barf out data that you pop into excel to munge through before typing up a 20 pager for upper management.
3. Small personal tools, like a CLI script that spits out equivalent yearly interest rates or what have you.
Everything else I test. Libraries, backend web apps, machine learning shit, compiled, whatever. It is too easy for codebases to turn into a hellscape without tests. You get too afraid to change things.
Should it change the API to the codebase? I usually don't think so, but occasionally I'll, say, make something an instance variable that I'd normally keep as a locally scoped variable. So I model test what I can and I integration test the rest. I think that he's right that integration tests cover a lot of functionality and I think he's right that mocks aren't usually great, but I think he's wrong about how much testing we should be doing. Most things should be tested most of the time.
It's cheaper. Why?
Because it costs money to hire support people and it costs money to validate their tickets and it costs money to fix the bugs and the bugs are harder to fix when the data is already in the system and the data is wrong.
It's also more profitable to write more tests. Why?
Because bugs mean lost customers and even when you keep the customer the feedback that you get from them is which problems you need to fix, not which functionality could be made better.
I think the biggest thing you can do to write more integration tests is to just stop mocking so much stuff.
Okay. The biggest problem I see with people trying to write unit tests is that they don’t want to change how they write code. They just want tests for it. It’s like watching an OO person try their hardest to write OO code in a functional language.
So they try to write E2E tests which work for about 5 or 6 quarters and then fall apart like a cheap bookcase. If you can find a new job before then, you never have to learn to write good tests!
I agree with the author that the trick is to stop using mocks all the time, but you don’t have to write integration tests to get rid of mocks. You have to write better code.
Usually if I have a unit test with more than one mock it’s bcause I’m too invested in the current shape of the code and I need to cleave the function in two, or change the structure of he question asked (eg, remake two methods into two other methods).
Almost always when I accept that the code is wrong, I end up with clearer code and easier tests.
Unit tests run faster, are written faster, and not only can they be fixed faster, they can be deleted and rewritten if the requirements change. The most painful thing to watch by far is someone spending hours trying to recycle an old test because they spent 3 hours on it last time and they’ll be damned if they’re going to just delete it now.
> The biggest problem I see with people trying to write unit tests is that they don’t want to change how they write code. They just want tests for it. It’s like watching an OO person try their hardest to write OO code in a functional language.
The biggest problem I see with people advocating for tests and employing TDD is that they do change how they write code to accommodate tests. This leads to inclusion of lots of unnecessary abstraction and boilerplate patterns that make code less readable and more bug-prone. OO world has spawned numerous non-solutions to turn your code inside-out so that it's easier to mock things, at the expense of code quality itself.
That said, if you go for functional style in OOP, i.e. shoving as much as you can into static helper functions and most of the rest into dumb private stateless functions, you suddenly gain both a clean architecture and lots of test points to use in unit tests. So you can have testable code, but you have to chill out with the OOP thing a bit.
people ... don’t want to change how they write code. They just want tests for it
Have you considered the possibility that those people are right? That's a reasonable conclusion to make if you are seeing lots of otherwise smart people that share an opinion that disagrees with yours.
There are lots of valid reasons to change the style in which you write code. In my mind, fitting somebody's fad testing scheme is not one of them.
Here's a second opinion from a guy who also likes tests, but doesn't think it's a good idea to structure your whole codebase just to accommodate them:
In some situations unit tests with lots of mocks will bring a negative value. Imagine a situation where you want to refactor a big piece of code with many dependencies but you don't want to change its public interface.
If you mock everything, when you refactor, the test will break because the dependency structure will change, and the mocks are no longer relevant to the new implementation. You have to rewrite the tests. You did twice the testing work and more importantly you get absolutely no proctection against regressions because the tests for the 2 versions are not the same.
If you build integration tests, they can remain the same. Less work and actual protection for your refactor.
This sort of discussion often gets confused because people have different ideas about what integration tests are and therefore talk past each other.
I generally avoid the term altogether and recommend testing stable API's (which are often public) and avoiding testing internal API's that are more likely to change. This assumes you have a stable API, but that's true of most libraries.
>Okay. The biggest problem I see with people trying to write unit tests is that they don’t want to change how they write code. They just want tests for it. It’s like watching an OO person try their hardest to write OO code in a functional language.
I've seen what happens when a developer tries to abstract away a database in a database driven app so it can be "better unit tested". It's a goddamn mess.
If your app relies heavily on using a database, your app naturally integrates with a database then it makes no sense to test without it. You are intentionally avoiding testing in a way that will pick up bugs.
>Unit tests run faster, are written faster
Unit tests test less realistically. That means they don't catch bugs integration tests do.
They also often take longer to write and are more tightly coupled.
How coding this way came to be seen as a best practice is beyond me. Tight coupling and premature optimization is usually seen as bad practice in other areas.
Unit tests and integration tests are tools, some tools are better at some tasks then others. The idea that a single tool is the only one you need is preposterous.
If you are writing a library write unit test, if your app mostly binds two libraries together unit tests are meaningless, write integration tests.
Functional, integration, and unit are all different types of tests. He's saying write more integration tests, not more functional tests.
For algorithms, I love how I can refactor the implementation and still have 100% confidence in the result if it tests sample input against expected output.
Because they test that you are actually using some available network ports, have the correct database model in mind, didn't mistake the version of your libraries, got the deployment script right, and isn't just restarting everything in an infinite loop?
Or maybe because E2E tests actually test stuff your company cares about, instead of some made-up rules that you've got from nowhere?
Really, if you have unities, you should unit-test them. But your E2E tests should be the ones you really care about, and they should certainly not break randomly with unrelated development. If yours are lasting for 5 quarters, you may be doing something wrong.
Why does everyone rethink a working strategy. Write lots of unit tests that are fast. Write a good amount of integration tests that are relatively fast. Write fewer system integration tests that are slower. The testing pyramid works. He even talks about it in this post, and then ignores the point of it.
You write lots of unit tests because you can run them inline pre-commit or in a component build. If you integration tests are numerous and interesting enough, that won't work. They are better suited to gating feature branch merges. System integration (the actual name for "end to end") take longer and usually gate progressively more stable branches upstream, or nightlies depending on where you are.
Because for certain kind of project the strategy stops working.
I work as QE on a fairly large, ~7 years old project. Micorservice architecture has been attempted. We always merge to master, which means that everything more-or-less is a feature-branch merge. We have too many repositories to count.
And what we learned is, that most of the components we have are just too thin to allow for useful unit-test coverage. Almost everything is [Gui]--request->[Middleware]--request-->[Proxy]--request-->[Backend]->[Database].
In reality, [Middleware] and [Backend] probably should have been a single component, but devs wanted to do microservices, and be scalable, but they didn't really understand the bounded contexts of their services.
All of this leads us to a place, where unit-tests don't tell us much.
On the other hand, we managed to spawn [Middleware]->[Backend]->[Database], and we can run a useful integration tests-suite in ~2 minutes.
So, on one hand, if we desined this better, the good-old pyramid might be a working strategy. On the other hand, if I can get actual services running in minute, and test them end-to-end, I don't think I will bother with true unit-tests on my next projects. I.e. why mock the database, if I can spawn it seconds :-)
The testing pyramid is built around a lot of assumptions which are often not true.
For example I run our ~10,000 integration tests in under two minutes on our large enterprise codebase. In recent years it has become possible to have fast integration tests.
I've worked on other apps that take 5+ minutes just to start up and integration tests can take hours.
Applying the same testing strategy to both does not make sense.
I've tried various testing strategies over 15~ different companies in all sorts of environments, and unit tests are the only thing that really work (IF you can convince the team to do it...and that's a big IF).
The article starts with a point I agree with: the lower in the pyramid, the cheaper the tests but the lower the confidence level they bring. That's true.
Where I disagree is how much the difference on confidence and cost are.
I can bang out 500 unit tests faster than I can do just a few E2E tests in most large apps. They require almost no trial and error, no real engineering (I feel strongly that abstraction in unit tests is bad), and all around are so easy to write, I don't mind if I have to toss out 150 of time when I make a significant refactor.
E2E tests are amazingly brittle and require a careful understanding of the whole system. They're impossibly expensive to write. They're the only thing that tells you that stuff works though. So you want at least a few of these.
Integration tests are just flat out awkward: you need understanding of a significant portion of code you did not write or touch, they often require complex fixtures (because your test will go through several code paths and might depend on a lot of arguments), they're slower (because a lot of code run), and while you don't throw them away when changing implementation details (unless they involve side effects), you still throw them away when refactoring or changing public interfaces. I've worked with a lot of people who were very vocal about these being so much better, then in the same breath complain that they spent all day writing integration tests.
There's an exception here which is acceptance tests for libraries, especially when doing a full rewrite: the tests that tell you public interfaces used outside of the current context work (as opposed to public interface of objects used in the implementation). Eg: if I was to test lodash or react, that's how I'd do it.
Unit tests to be are about a lot more than "is this change breaking my code". And if that's all you care about, you're missing a big part of the point.
If you have 3 units, A, B and C. A calls B which calls C. If you have a test for A in the context of B, a text for B in the context of C, and a test for C, and they all pass, you know that A + B + C will work. But when writing the tests, you only had to care about itty bitty tiny pieces of code, which made things super cheap.
Then you get other huge benefit: the quality of the entire code base is higher (a side effect of it having testable interfaces all across), the reasoning behind each piece of code is explicit (no one wrote a function that work and they 're not sure why, else the test would be very hard to write), you automatically have a document representing "intentions".
And yes, if you change a module, even if its not expose to your customers, the public interface of that module has tests and the tests will break. But they usually take nothing but a few minutes (often a few seconds) to write. They're cheap enough to be disposable.
And once you have 80%ish unit test coverage, you actually have a very high confidence level. I've gone through major refactoring of multi-million line of code apps with almost no bugs on pure unit tests. You't think the 20% of untested code would be a source of bug, but statistically, that's just not how it happens.
In term of person-hour to ROI, pure unit tests just straight up win out.
The reason software engineers fight back to hard against them is that they're brain dead and repetitive to write, and they can't resist overengineering. "This is such a simple test for such a simple piece of code, why should I test it?!". That's the point. All unit tests should be like this.
The second group I worked with that was earnestly interested in mature testing developed the 5/8ths rule.
To move a test one level down the pyramid, it takes about 5x as many tests. But the tests run 8 times as fast. So moving a test down takes more than 35% off the run time, and it fails the build minutes sooner. If you drop it down two levels it's 60% off the run time.
Interesting enough on its own, but maintaining those tests after one requirements change, plus the cost of rewriting them in the first place, is less work than the cost of maintaining the original tests. We didn't come up with a number for this but the difference was measured in man-days and missed deadlines about once a month, and we were convinced by the evidence.
I also agree with both your 'braindead' comment and your 80% estimate. The big payoffs come between 75% and 85% and above 85% you start getting artifacts. That 'data' distract more than it helps.
One of the most transformative things I've come across for how to structure and test code has been Gary Bernhardt's talk on Boundaries [0]. I've watched it at least ten times. He also has an entire series on testing where he goes deeper into these ideas.
In this video, he talks of a concept called functional core, imperative shell. The functional core is your code that contains your core logic that can be easily unit tested because it just receives plain values from the outside world. The imperative shell is the outside world that talks to disks, databases, APIs, UIs, etc. and builds these values to be used in the core. I'll stop there—Gary's video will do 100x than I can do here :)
I agree with the part that you should write tests, but I definitely disagree with the part that most of your tests should be integration tests.
As you pointed out the testing pyramid suggests that you should write more unit tests. Why? Because if you have ever tried TDD you know that unit tests make you write good (or at least acceptable) code. The reason for this is that testing bad code is hard. By writing mostly integration tests you lose one of the advantages of unit testing and you sidestep the bad code checking part.
The other reason is that unit tests are easy to write. If you have interfaces for your units of code then mocking is also easy. I recommend stubbing though, I think that if you have to use mocks it is a code smell.
Also the .gif with the man in pieces is a straw man. Just because you have to write at least 1 integration test to check whether the man has not fallen apart is not a valid reason to write mostly integration tests! You can't test your codebase reliably with them and they are also very costly to write, run and maintain!
The testing pyramid exists for a reason! It is a product of countless hours of research, testing and head scratching! You should introspect your own methods instead and you might arrive at the conclusion that the codebase you are working on is bad and it is hard to unit test, that's why you chosen writing mostly integration tests.
Sounds good in theory. In practice there is one problem with having integrations tests only. The test are generally simple: they pass or they fail. A unit test tests just a small functionality, so when it fails, it's quite easy to find out the problem.
When an integration test fails, then we can spend hours debugging the whole stack of layers trying to find out the real problem.
I had this situation once. Every failing integration test ended with hours spent on writing unit tests for all the places used by the test.
From my experience an integration test failure that requires significant efforts to investigate can only be covered with unit tests after one knows where the problem comes from. One cannot realistically write a bunch of unit tests and expect them to cover the problem unless one already knows about the problem.
For me, one of the biggest issues with integration tests is the code coverage numbers mean nearly nothing. I've seen an "integration only" tester proudly display his single test with 90% coverage. I asked him to run it again and it was 2% because a condition changed.
So this means that for all the branches your code can take, an integration test is taking one specific one at each point all the way through for that test. All over branches, through the entire call stack are unverified.
I think the answer would be it heavily depends on what you are doing.
if you are creating a library that operates on a protocol, unit tests are necessary / extremly important.
if you are writing a ERP where a lot of your code NEEDS to operate WITH the database you are better of with integration tests, because mocking away the database would lead to so much bugs, especially if your database is extremly important (and not just a dumb datastore)
Edit: having any tests is always better than having none.
The puffing-billy [1] library is awesome, and has changed the way I write integration tests. I also use VCR [2], and now my entire application (both backend and front-end) is wrapped with a proxy that records and replays every request. I can run all my tests once using test Stripe API keys, a test Recaptcha response, or any other external services that I want to test. I don't have to mock anything, which is nice. Then everything is recorded, and I can run all my integration tests offline.
I've also really enjoyed using stripe-ruby-mock when testing specific webhooks, jobs, and controller actions. I don't always aim for 100% test coverage, but I try to write a LOT of tests for any code that deals with billing and subscriptions.
Ooh, I've also been enjoying rswag [4]. It's quite a cool idea - You write rspec tests for your API endpoints, and the tests also serve as a Swagger definition for your API. So when your tests pass, you can use the output to generate documentation or API clients for any language.
I think the testing pyramid reflects a false correlation — it seems to assert that higher up the pyramid tests are more expensive to write/maintain and longer to run.
In reality the execution time of a test says nothing about how hard the test is to write. Sometimes a very fast to execute unit test can be much harder to write/maintain than a longer running test that avoids mocking an api and perhaps utilizes abstractions in the test definition that are already written to support the program’s features.
I think test suite execution speed is the real metric to focus on for most projects — to get the most value, test suites should accelerate the time to useful feedback. Write tests in the simplest way that provides useful feedback into the behavior of the system and runs quickly enough that you can receive that feedback with low latency during development.
I quite like tools like jest and wallabyjs that use code coverage data to figure out which tests to rerun as code changes — means you can have a test suite that includes slow(ish) to execute tests but still get feedback quickly in reasonable time as you make changes to the code.
Integration testing is especially important when talking to a database. People seem to like mocking the data, but that completely misses the subtleties of how databases actually work, including aspects such as concurrency, transaction isolation levels, locking and such.
There should be a few well crafted tests that modify the database from several parallel threads, and afterwards verify that no invariants have been broken. This is pretty much the only opportunity to catch a race condition in a controlled environment.
This is the part I don't get when people preach the gospel of ultra-isolated test suites where nothing talks to an external system. The most nontrivial parts of your code, the most likely to break, are those that talk to external systems.
Building a mock of that system sophisticated enough to sufficiently capture even a significant fraction of the "real" failure modes is just as, if not more error-prone and daunting a task.
Totally isolated tests make it impossible to test the most failure-prone parts of your code in anything approaching a satisfactory manner. Maybe I'm just misunderstanding people's advice but it drives me crazy when I see that.
As always it depends. There are projects where integration is the hard part and others where the business logic is impossible to get right without unit tests.
Unit tests also have the added benefit that they tell you exactly the reason they failed (because there can only be one) whereas integration tests can fail for multiple reasons.
Just reading the comments, I agree with the common sentiment that you should have more unit tests than integration tests, but I have come around to the way of thinking that if I only have time to write a few tests then I would rather write E2E tests. This way, at the very least your entire stack is being exercised, and you have a way of ensuring that the happy path is passing consistently, which is the most important flow for an application (even if I'm personally more interested in keeping other flows sane). While I prefer unit tests due to their simplicity, speed and the speed at which they can aid debugging, these days I will only implement them after I have added a few E2E tests.
Behavior driven development (BDD) and test driven development (TDD) enable much faster coding when done well in my experience, and that includes full coverage fast unit tests, functional tests, benchmark tests, and integration tests.
Unit tests are worth their weight in gold for quickly finding issues, both in our team's code and especially in cases of subtle changes among language releases, or unanticipated input changes, or dependency changes that are supposed to work but don't.
IMHO unit tests lead to better functional approaches, better long range maintainability, better security, and much better handling of corner cases.
Software projects can be extremely large and have wildly different requirements. Software that needs to operate at high scale and require high reliability will have differenet requirements for say - a web application that has low traffic.
I think making rules of thumbs like the title of this article defeats the purpose of one of the essential tasks of being an engineer - making good tradeoffs between different approaches to solving problems. It's not hard to see that some projects will require more unit testing, some may require more integration testing and others may require more of both.
Why are most people using huge hero-graphics (495kb?!) for short blog postings nowadays?
On topic: Testing gets worse if the codebase which needs to be tested is garbage. My experience with other developers who learn testing don't need to learn how testing works - they need to learn basic development rules: Components, loose coupling, dependency injection, ...
Integration tests are very important and in many cases mandatory. I'd argue though unit tests with good coverage are more important. The latter helps you make sure your application has the correct output. Nothing worst than having a piece of software you think is working and silently introducing errors. The former (integration tests) makes sure your application starts and probably works in some scenarios.
With docker and containers, integration testing is easier than ever.
I've written a very simple integration testing tool (a quick hack to be honest) that executes commands, checks their exit code, the output through regexp, etc and produces a nice html report.
Originally written for testing some configuration tools and scripts, we quickly found a use for it for our Kafka connector collection (Stream Reactor). Our connector integration tests are at github as well:
Obviously unit tests make sure that the connectors work as expected. Integration tests catch simpler errors that can be showstoppers. For example an internally renamed configuration option that didn't trickle down to the configuration parser, class shadowing issues between the connectors, unexpected errors in logs that the developers should check out etc. As the tests are written by people who don't do development, they also expose problems in the documentation. In some cases they provide us an easy way to quickly run locally a connector and catch some issues manually, like extensive CPU usage or way too much logging (e.g a function with a log statement was inside a loop that run hundreds or thousands of times per second).
It gives us confidence in what we ship with every release.
[+] [-] moultano|8 years ago|reply
- Structure your code so it is mostly leaves.
- Unit test the leaves.
- Integration test the rest if needed.
I like this approach in part because making lots of leaves also adds to the "literate"-ness of the code. With lots of opportunities to name your primitives, the code is much closer to being self documenting.
Depending on the project and its requirements, I also think "lazy" testing has value. Any time you are looking at a block of code, suspicious that it's the source of a bug, write a test for it. If you're in an environment where bugs aren't costly, where attribution goes through few layers of code, and bugs are easily visible when they occur, this can save a lot of time.
[+] [-] sph|8 years ago|reply
- https://github.com/testdouble/contributing-tests/wiki/London... (and the rest of the Wiki)
- http://blog.testdouble.com/posts/2015-09-10-how-i-use-test-d...
- Most of the screencasts and articles at https://www.destroyallsoftware.com/screencasts (especially this brilliant talk https://www.destroyallsoftware.com/talks/boundaries)
- Integration Tests Are A Scam: https://www.youtube.com/watch?v=VDfX44fZoMc
All of these basically go the opposite way of the article's philosophy:
Not too many integration tests, mostly unit tests. Clearly define a contract between the boundaries of the code, and stub/mock on the contract. You'll be left with mostly pure functions at the leaves, which you'll unit test.
[+] [-] salutis|8 years ago|reply
My leaves are either pure functions (FP languages) or value objects that init themselves based on other value objects (OOP languages). These value objects have no methods, no computed properties, etc. Just inert data.
No mocks and no “header” interfaces needed.
On top of that I sprinkle a bunch of UI tests to verify it’s all properly wired up.
Works great!
[+] [-] didibus|8 years ago|reply
Exactly. You expressed my thoughts very succinctly. Though I feel the post tries to say the same just in a lot more words.
[+] [-] chewbacha|8 years ago|reply
https://en.m.wikipedia.org/wiki/Directed_acyclic_graph
[+] [-] mannykannot|8 years ago|reply
Is there any situation where there is integration, but no need to test it?
You seem to be suggesting that if the leaves are thoroughly tested, nothing can go wrong in their integration, but at the same time, I cannot imagine someone believing that.
[+] [-] edwinyzh|8 years ago|reply
[+] [-] nickthemagicman|8 years ago|reply
[+] [-] kartan|8 years ago|reply
It is not the same creating a library that is going to be used to launch a multi-billion rocket to Mars than developing a mostly graphical mobile app where requirements are changing daily as you A/B test your way into better business value.
The article has really good points and the reasons why they work. Apply them wisely. Take the right decision for your project. Don't be dogmatic.
[+] [-] 3pt14159|8 years ago|reply
1. Very, very graphical programming - like SVG charting with animations. If it were static generation it would be easy, but throwing time into the mix makes the tests really hard plus if things go wrong in the future people can literally see it going wrong and complain, so I don't think it is worth the trouble.
2. Data analysis meant for static reporting. You know, those 2000 line SQL queries that barf out data that you pop into excel to munge through before typing up a 20 pager for upper management.
3. Small personal tools, like a CLI script that spits out equivalent yearly interest rates or what have you.
Everything else I test. Libraries, backend web apps, machine learning shit, compiled, whatever. It is too easy for codebases to turn into a hellscape without tests. You get too afraid to change things.
Should it change the API to the codebase? I usually don't think so, but occasionally I'll, say, make something an instance variable that I'd normally keep as a locally scoped variable. So I model test what I can and I integration test the rest. I think that he's right that integration tests cover a lot of functionality and I think he's right that mocks aren't usually great, but I think he's wrong about how much testing we should be doing. Most things should be tested most of the time.
It's cheaper. Why?
Because it costs money to hire support people and it costs money to validate their tickets and it costs money to fix the bugs and the bugs are harder to fix when the data is already in the system and the data is wrong.
It's also more profitable to write more tests. Why?
Because bugs mean lost customers and even when you keep the customer the feedback that you get from them is which problems you need to fix, not which functionality could be made better.
[+] [-] whipoodle|8 years ago|reply
[deleted]
[+] [-] protonfish|8 years ago|reply
[+] [-] hinkley|8 years ago|reply
So they try to write E2E tests which work for about 5 or 6 quarters and then fall apart like a cheap bookcase. If you can find a new job before then, you never have to learn to write good tests!
I agree with the author that the trick is to stop using mocks all the time, but you don’t have to write integration tests to get rid of mocks. You have to write better code.
Usually if I have a unit test with more than one mock it’s bcause I’m too invested in the current shape of the code and I need to cleave the function in two, or change the structure of he question asked (eg, remake two methods into two other methods).
Almost always when I accept that the code is wrong, I end up with clearer code and easier tests.
Unit tests run faster, are written faster, and not only can they be fixed faster, they can be deleted and rewritten if the requirements change. The most painful thing to watch by far is someone spending hours trying to recycle an old test because they spent 3 hours on it last time and they’ll be damned if they’re going to just delete it now.
[+] [-] TeMPOraL|8 years ago|reply
The biggest problem I see with people advocating for tests and employing TDD is that they do change how they write code to accommodate tests. This leads to inclusion of lots of unnecessary abstraction and boilerplate patterns that make code less readable and more bug-prone. OO world has spawned numerous non-solutions to turn your code inside-out so that it's easier to mock things, at the expense of code quality itself.
That said, if you go for functional style in OOP, i.e. shoving as much as you can into static helper functions and most of the rest into dumb private stateless functions, you suddenly gain both a clean architecture and lots of test points to use in unit tests. So you can have testable code, but you have to chill out with the OOP thing a bit.
[+] [-] jasonkester|8 years ago|reply
Have you considered the possibility that those people are right? That's a reasonable conclusion to make if you are seeing lots of otherwise smart people that share an opinion that disagrees with yours.
There are lots of valid reasons to change the style in which you write code. In my mind, fitting somebody's fad testing scheme is not one of them.
Here's a second opinion from a guy who also likes tests, but doesn't think it's a good idea to structure your whole codebase just to accommodate them:
http://david.heinemeierhansson.com/2014/test-induced-design-...
[+] [-] sbergot|8 years ago|reply
If you mock everything, when you refactor, the test will break because the dependency structure will change, and the mocks are no longer relevant to the new implementation. You have to rewrite the tests. You did twice the testing work and more importantly you get absolutely no proctection against regressions because the tests for the 2 versions are not the same.
If you build integration tests, they can remain the same. Less work and actual protection for your refactor.
[+] [-] skybrian|8 years ago|reply
I generally avoid the term altogether and recommend testing stable API's (which are often public) and avoiding testing internal API's that are more likely to change. This assumes you have a stable API, but that's true of most libraries.
[+] [-] crdoconnor|8 years ago|reply
I've seen what happens when a developer tries to abstract away a database in a database driven app so it can be "better unit tested". It's a goddamn mess.
If your app relies heavily on using a database, your app naturally integrates with a database then it makes no sense to test without it. You are intentionally avoiding testing in a way that will pick up bugs.
>Unit tests run faster, are written faster
Unit tests test less realistically. That means they don't catch bugs integration tests do.
They also often take longer to write and are more tightly coupled.
How coding this way came to be seen as a best practice is beyond me. Tight coupling and premature optimization is usually seen as bad practice in other areas.
[+] [-] josteink|8 years ago|reply
The reason? They don’t “believe” in unit-tests. They don’t think unit-testing “works in the real world”.
They absolutely fail to accept that they need to write their code differently for automated testing to work well.
How do you change such a mindset?
[+] [-] olavk|8 years ago|reply
Because they can find bugs and errors which unit tests cannot.
[+] [-] Illniyar|8 years ago|reply
If you are writing a library write unit test, if your app mostly binds two libraries together unit tests are meaningless, write integration tests.
[+] [-] davidjnelson|8 years ago|reply
For algorithms, I love how I can refactor the implementation and still have 100% confidence in the result if it tests sample input against expected output.
[+] [-] marcosdumay|8 years ago|reply
Because they test that you are actually using some available network ports, have the correct database model in mind, didn't mistake the version of your libraries, got the deployment script right, and isn't just restarting everything in an infinite loop?
Or maybe because E2E tests actually test stuff your company cares about, instead of some made-up rules that you've got from nowhere?
Really, if you have unities, you should unit-test them. But your E2E tests should be the ones you really care about, and they should certainly not break randomly with unrelated development. If yours are lasting for 5 quarters, you may be doing something wrong.
[+] [-] pjmlp|8 years ago|reply
They can only test parts of its behavior, not everything, and are too brittle to any simple UI/UX change.
[+] [-] throwaway5752|8 years ago|reply
You write lots of unit tests because you can run them inline pre-commit or in a component build. If you integration tests are numerous and interesting enough, that won't work. They are better suited to gating feature branch merges. System integration (the actual name for "end to end") take longer and usually gate progressively more stable branches upstream, or nightlies depending on where you are.
[+] [-] a-saleh|8 years ago|reply
I work as QE on a fairly large, ~7 years old project. Micorservice architecture has been attempted. We always merge to master, which means that everything more-or-less is a feature-branch merge. We have too many repositories to count.
And what we learned is, that most of the components we have are just too thin to allow for useful unit-test coverage. Almost everything is [Gui]--request->[Middleware]--request-->[Proxy]--request-->[Backend]->[Database].
In reality, [Middleware] and [Backend] probably should have been a single component, but devs wanted to do microservices, and be scalable, but they didn't really understand the bounded contexts of their services.
All of this leads us to a place, where unit-tests don't tell us much.
On the other hand, we managed to spawn [Middleware]->[Backend]->[Database], and we can run a useful integration tests-suite in ~2 minutes.
So, on one hand, if we desined this better, the good-old pyramid might be a working strategy. On the other hand, if I can get actual services running in minute, and test them end-to-end, I don't think I will bother with true unit-tests on my next projects. I.e. why mock the database, if I can spawn it seconds :-)
[+] [-] crdoconnor|8 years ago|reply
* Because I want to avoid writing more code than necessary.
* Because I want to avoid writing tightly coupled code.
* Because I'd rather have a test that takes 2x longer and catches 5% more bugs. Premature optimization and all that.
[+] [-] saas_co_de|8 years ago|reply
[+] [-] sheepmullet|8 years ago|reply
The testing pyramid is built around a lot of assumptions which are often not true.
For example I run our ~10,000 integration tests in under two minutes on our large enterprise codebase. In recent years it has become possible to have fast integration tests.
I've worked on other apps that take 5+ minutes just to start up and integration tests can take hours.
Applying the same testing strategy to both does not make sense.
[+] [-] shados|8 years ago|reply
I've tried various testing strategies over 15~ different companies in all sorts of environments, and unit tests are the only thing that really work (IF you can convince the team to do it...and that's a big IF).
The article starts with a point I agree with: the lower in the pyramid, the cheaper the tests but the lower the confidence level they bring. That's true.
Where I disagree is how much the difference on confidence and cost are.
I can bang out 500 unit tests faster than I can do just a few E2E tests in most large apps. They require almost no trial and error, no real engineering (I feel strongly that abstraction in unit tests is bad), and all around are so easy to write, I don't mind if I have to toss out 150 of time when I make a significant refactor.
E2E tests are amazingly brittle and require a careful understanding of the whole system. They're impossibly expensive to write. They're the only thing that tells you that stuff works though. So you want at least a few of these.
Integration tests are just flat out awkward: you need understanding of a significant portion of code you did not write or touch, they often require complex fixtures (because your test will go through several code paths and might depend on a lot of arguments), they're slower (because a lot of code run), and while you don't throw them away when changing implementation details (unless they involve side effects), you still throw them away when refactoring or changing public interfaces. I've worked with a lot of people who were very vocal about these being so much better, then in the same breath complain that they spent all day writing integration tests.
There's an exception here which is acceptance tests for libraries, especially when doing a full rewrite: the tests that tell you public interfaces used outside of the current context work (as opposed to public interface of objects used in the implementation). Eg: if I was to test lodash or react, that's how I'd do it.
Unit tests to be are about a lot more than "is this change breaking my code". And if that's all you care about, you're missing a big part of the point.
If you have 3 units, A, B and C. A calls B which calls C. If you have a test for A in the context of B, a text for B in the context of C, and a test for C, and they all pass, you know that A + B + C will work. But when writing the tests, you only had to care about itty bitty tiny pieces of code, which made things super cheap.
Then you get other huge benefit: the quality of the entire code base is higher (a side effect of it having testable interfaces all across), the reasoning behind each piece of code is explicit (no one wrote a function that work and they 're not sure why, else the test would be very hard to write), you automatically have a document representing "intentions".
And yes, if you change a module, even if its not expose to your customers, the public interface of that module has tests and the tests will break. But they usually take nothing but a few minutes (often a few seconds) to write. They're cheap enough to be disposable.
And once you have 80%ish unit test coverage, you actually have a very high confidence level. I've gone through major refactoring of multi-million line of code apps with almost no bugs on pure unit tests. You't think the 20% of untested code would be a source of bug, but statistically, that's just not how it happens.
In term of person-hour to ROI, pure unit tests just straight up win out.
The reason software engineers fight back to hard against them is that they're brain dead and repetitive to write, and they can't resist overengineering. "This is such a simple test for such a simple piece of code, why should I test it?!". That's the point. All unit tests should be like this.
[+] [-] hinkley|8 years ago|reply
To move a test one level down the pyramid, it takes about 5x as many tests. But the tests run 8 times as fast. So moving a test down takes more than 35% off the run time, and it fails the build minutes sooner. If you drop it down two levels it's 60% off the run time.
Interesting enough on its own, but maintaining those tests after one requirements change, plus the cost of rewriting them in the first place, is less work than the cost of maintaining the original tests. We didn't come up with a number for this but the difference was measured in man-days and missed deadlines about once a month, and we were convinced by the evidence.
I also agree with both your 'braindead' comment and your 80% estimate. The big payoffs come between 75% and 85% and above 85% you start getting artifacts. That 'data' distract more than it helps.
[+] [-] greencurry43|8 years ago|reply
In this video, he talks of a concept called functional core, imperative shell. The functional core is your code that contains your core logic that can be easily unit tested because it just receives plain values from the outside world. The imperative shell is the outside world that talks to disks, databases, APIs, UIs, etc. and builds these values to be used in the core. I'll stop there—Gary's video will do 100x than I can do here :)
[0] https://www.destroyallsoftware.com/talks/boundaries
[+] [-] edem|8 years ago|reply
As you pointed out the testing pyramid suggests that you should write more unit tests. Why? Because if you have ever tried TDD you know that unit tests make you write good (or at least acceptable) code. The reason for this is that testing bad code is hard. By writing mostly integration tests you lose one of the advantages of unit testing and you sidestep the bad code checking part.
The other reason is that unit tests are easy to write. If you have interfaces for your units of code then mocking is also easy. I recommend stubbing though, I think that if you have to use mocks it is a code smell.
Also the .gif with the man in pieces is a straw man. Just because you have to write at least 1 integration test to check whether the man has not fallen apart is not a valid reason to write mostly integration tests! You can't test your codebase reliably with them and they are also very costly to write, run and maintain!
The testing pyramid exists for a reason! It is a product of countless hours of research, testing and head scratching! You should introspect your own methods instead and you might arrive at the conclusion that the codebase you are working on is bad and it is hard to unit test, that's why you chosen writing mostly integration tests.
[+] [-] pleasecalllater|8 years ago|reply
I had this situation once. Every failing integration test ended with hours spent on writing unit tests for all the places used by the test.
[+] [-] _0w8t|8 years ago|reply
[+] [-] alien_at_work|8 years ago|reply
So this means that for all the branches your code can take, an integration test is taking one specific one at each point all the way through for that test. All over branches, through the entire call stack are unverified.
[+] [-] methodover|8 years ago|reply
It feels like an area where lots of people have opinions, and there are not much in the way of facts.
[+] [-] merb|8 years ago|reply
if you are writing a ERP where a lot of your code NEEDS to operate WITH the database you are better of with integration tests, because mocking away the database would lead to so much bugs, especially if your database is extremly important (and not just a dumb datastore)
Edit: having any tests is always better than having none.
[+] [-] nathan_f77|8 years ago|reply
I've also really enjoyed using stripe-ruby-mock when testing specific webhooks, jobs, and controller actions. I don't always aim for 100% test coverage, but I try to write a LOT of tests for any code that deals with billing and subscriptions.
Ooh, I've also been enjoying rswag [4]. It's quite a cool idea - You write rspec tests for your API endpoints, and the tests also serve as a Swagger definition for your API. So when your tests pass, you can use the output to generate documentation or API clients for any language.
[1] https://github.com/oesmith/puffing-billy
[2] https://github.com/vcr/vcr
[3] https://github.com/rebelidealist/stripe-ruby-mock
[4] https://github.com/domaindrivendev/rswag
[+] [-] breatheoften|8 years ago|reply
In reality the execution time of a test says nothing about how hard the test is to write. Sometimes a very fast to execute unit test can be much harder to write/maintain than a longer running test that avoids mocking an api and perhaps utilizes abstractions in the test definition that are already written to support the program’s features.
I think test suite execution speed is the real metric to focus on for most projects — to get the most value, test suites should accelerate the time to useful feedback. Write tests in the simplest way that provides useful feedback into the behavior of the system and runs quickly enough that you can receive that feedback with low latency during development.
I quite like tools like jest and wallabyjs that use code coverage data to figure out which tests to rerun as code changes — means you can have a test suite that includes slow(ish) to execute tests but still get feedback quickly in reasonable time as you make changes to the code.
[+] [-] branko_d|8 years ago|reply
There should be a few well crafted tests that modify the database from several parallel threads, and afterwards verify that no invariants have been broken. This is pretty much the only opportunity to catch a race condition in a controlled environment.
[+] [-] gipp|8 years ago|reply
Building a mock of that system sophisticated enough to sufficiently capture even a significant fraction of the "real" failure modes is just as, if not more error-prone and daunting a task.
Totally isolated tests make it impossible to test the most failure-prone parts of your code in anything approaching a satisfactory manner. Maybe I'm just misunderstanding people's advice but it drives me crazy when I see that.
[+] [-] oweiler|8 years ago|reply
Unit tests also have the added benefit that they tell you exactly the reason they failed (because there can only be one) whereas integration tests can fail for multiple reasons.
[+] [-] toasterlovin|8 years ago|reply
[+] [-] ezanmoto|8 years ago|reply
[+] [-] jph|8 years ago|reply
Unit tests are worth their weight in gold for quickly finding issues, both in our team's code and especially in cases of subtle changes among language releases, or unanticipated input changes, or dependency changes that are supposed to work but don't.
IMHO unit tests lead to better functional approaches, better long range maintainability, better security, and much better handling of corner cases.
[+] [-] ivan1931|8 years ago|reply
Software projects can be extremely large and have wildly different requirements. Software that needs to operate at high scale and require high reliability will have differenet requirements for say - a web application that has low traffic.
I think making rules of thumbs like the title of this article defeats the purpose of one of the essential tasks of being an engineer - making good tradeoffs between different approaches to solving problems. It's not hard to see that some projects will require more unit testing, some may require more integration testing and others may require more of both.
[+] [-] MichaelMoser123|8 years ago|reply
[+] [-] waibelp|8 years ago|reply
On topic: Testing gets worse if the codebase which needs to be tested is garbage. My experience with other developers who learn testing don't need to learn how testing works - they need to learn basic development rules: Components, loose coupling, dependency injection, ...
[+] [-] andmarios|8 years ago|reply
With docker and containers, integration testing is easier than ever.
I've written a very simple integration testing tool (a quick hack to be honest) that executes commands, checks their exit code, the output through regexp, etc and produces a nice html report.
https://github.com/landoop/coyote
Originally written for testing some configuration tools and scripts, we quickly found a use for it for our Kafka connector collection (Stream Reactor). Our connector integration tests are at github as well:
https://github.com/Landoop/kafka-connectors-tests
Obviously unit tests make sure that the connectors work as expected. Integration tests catch simpler errors that can be showstoppers. For example an internally renamed configuration option that didn't trickle down to the configuration parser, class shadowing issues between the connectors, unexpected errors in logs that the developers should check out etc. As the tests are written by people who don't do development, they also expose problems in the documentation. In some cases they provide us an easy way to quickly run locally a connector and catch some issues manually, like extensive CPU usage or way too much logging (e.g a function with a log statement was inside a loop that run hundreds or thousands of times per second).
It gives us confidence in what we ship with every release.