A tangent, and somewhat philosophical: is "The primary purpose of software testing" really "to detect any potential defects in a program before it reaches its intended consumers"? I would argue that whilst that is a commonly held belief, and certainly true for some tests of business logic, the main value and reason for testing is to allow developers to confidently make changes to existing code. In my experience user acceptance testing is a more useful tool for detecting user facing defects as it also tests the developers assumptions about functionality.
Let's look at it a different way and start with the assumption that in all cases we're delivering reliable software, it's just a question of how we get there.
The simplest way of doing this is to write code with good practices then do a load of manual system testing to find bugs (system testing including user acceptance testing in this example). This is basically what most people were doing 20 years ago.
This can work, but doing the system testing is time consuming and so expensive and so makes making changes expensive. Instead you automate some or all of your system testing. This makes them cheaper, but these tests are expensive to maintain and still time consuming to run. If the test fails you don't have that much information about the problem.
To complement (or maybe replace) your automated system testing, you add tests for smaller and smaller chunks of software, allowing for tighter feedback loops and more precise diagnosis (there are also some negative tradeoffs). Applied well, this allows you to write features more quickly, refactor more confidently and generally develop more efficiently.
Basically, it's not so much about catching bugs as it is about getting to your desired level of reliability in the most cost and time efficient way.
I was left stupefied by a senior staff member asserting that the point of tests was full code coverage. I thought I knew why he left such a wake behind him but apparently he had whole dimensions of crazy I had yet to discover.
I prefer to express it as the purpose of tests is to ensure something never changes. I don't care how future maintainer touches the code so long as given this set of inputs this is the output.
Even more important, IMO, regardless of whether you're mocking or faking, is to not mock or fake objects that you own. Only mock/fake truly external dependencies. The nice property you get with this is true tests of integration and interoperability between your modules, that are actually calling into each other. Only at the leaf nodes is anything mocked or faked, and that's when it leaves your ownership.
It's a fantastic comparison of the Detroit school of testing (what you and I believe) vs the London school, which is simply a different mental model about code and testing and mocks especially.
Framing the differences this way make it clearer why some people might disagree with us, and it's not because they're stupid or wrong- they just have a different school of thought on the matter. This doesn't excuse, however, the people who are doing a little bit of both and they don't know why, and they're getting the benefits of neither side of that coin.
Yes! Mocking internal dependencies also has the side effect of ossifying the internal interface between the two objects, which is never as well-designed as the external ones IME.
Also, in dynamic languages, you might even have tests that don’t break when you change (or sometimes even delete) the dependency because the mock was correct. That leads to a false sense of confidence.
Mocks for your own code, fakes for external dependencies but in both cases only where you can’t rework your most of your code to consume the outputs of dependencies instead of soliciting them. Fixtures trump mocks and fakes.
Having multiple tests fail for the same reason is a waste of effort, either now or in the future (when new features are added or the constraints otherwise change).
Sometimes you have to depend on transitive properties to keep your test sizes down. If a function received the data, and the function has been tested thoroughly, why test it again while testing the callers?
ShoppingCart depends on an implementation of ICartDiscountCalculator (CartDiscountCalculator)
CartDiscountCalculator depends on a number of implementations of ICartDiscountStrategy (ChristmasCartDiscountStrategy, VIPCustomerDiscountStrategy, BulkPurchaseDiscountStrategy, etc)
Each of those, in turn has their own dependencies, perhaps webservice calls, db calls, accessing caches, etc.
The way I would test ShoppingCart would be to mock out ICartDiscountCalculator, and provide known responses.
Are you suggesting instead that you build up the dependency graph - i.e the actual implementation of the discount calculator, the strategies, etc - and only mock when it goes to external code (eg the DB calls)?
This would seem to make testing very very difficult - since each test needs to know implementation details of everything it depends on.
You're choosing some place in between the integration and unit testing where your test will live. But this choice will not make sense for everyone. If your project includes its own persistence functions, you'll likely want a test to get a "CouldntWriteException" immediately from a mock rather than go through all the layers (that you own) to get an IO error (you don't own the fs dependency) and all layers back. It's a tiny difference for a single test, but once you get to hundreds, it will be a difference between devs getting immediate feedback and "going for coffee, my code is testing" which really impacts productivity.
This only works on small code bases. With anything large you're going to need to mock or fake your own code if you don't want to spend countless unnecessary hours running your tests and finding root causes of failed tests.
Yes, integration tests are important too, but you'll want to minimize the number of scenarios you still need to test at that level.
It seems to me that a Fake is really a Mock, and Mocks became this unholy dynamic framework stuff. I'm not a fan of dynamic mocks where the test code has the mocked implementation because it is super brittle. Instead, a "fake" (i.e. a mock as I understood them) is an alternatively implementation of an interface.
There are a lot of really good "fakes" out there. Fake file system which lets you simulate disk failures. Fake time and randomness to introduce determinism. Every API over network can have a fake. These are all good things to fake out as you can simulate the appropriate failures which are almost impossible to do via integration, acceptance, or E2E tests.
The problem with those alternative implementations is that they turn your unit tests into expensive and complicated integration tests. In a complex system getting a socket into the state you're interested in can be many times harder than the test itself. "Magic" mocks make it cheap and easy.
The fatal flaw is that you’re really introducing a new class which may have its own behaviour problems. Now your test is implicitly testing the test class. I also prefer to see explicit mocking than just trust that the fake does what it says it will. How do I know without looking whether the fake store method REALLY stores the doc, or whether someone set it up to just return OK? And how do you test error handling - make another fake for every error scenario?
This isn't a "fatal flaw." It's a tradeoff. You add a little bit of extra code (which is a liability), and in exchange, you get a test double that you can reuse in many places (which is a benefit). This is in opposition to mocks, where you get an automated framework that can substitute calls anywhere in the system (a benefit), but you're completely replicating the behavior in every single test (a liability).
I argue that [0] the liability of mocks are much higher than the fake. With a fake, you could theoretically write tests that asserts that your class is satisfying the interface. There's no way to do this with a mock. You can just make an interface do anything, regardless of whether it makes sense. This is fragile under modification. With a large codebase, you're inevitably going to change the behavior of a class during a refactoring. Are you going to read every test that mocks that class to decide how the mocks should be rewritten? Probably not - the mocks force-overrode the behavior of the class, so your tests will still pass. Why would you look at them again?
> Make a fake for every error scenario?
By the time you're implementing a fake, you have an interface that the fake implements. One way that I've handled this is to make a FailingBlobStorage that takes an error in its constructor and throws it from any calls. If you need even more detailed behavior than that, you can create a mock for that specific test case. You're not married to the fake. It's not going to magically test every scenario. But fakes handle the common case in so many situations that it's actually surprising when you start trying them out.
Mocks have the same problem. You create something new, that can have uninteneded behavior. Even worse, if you are not perfectly familiar with the mocking framework, you maybe won't even notice it.
The good thing about (simple) fakes is, that you can see all the code in a very small class. No complicated library, that does something funky in a special case.
Mocking can be very useful though, but it can get too complicated very quickly.
> How do I know without looking whether the fake store method REALLY stores the doc, or whether someone set it up to just return OK?
Why would you not look into the fake implementations? If it's just a two liner that takes the parameters and writes them to disk, then that's a LGTM from me. You would not test every getter/setter method in any of your classes, would you?
If it's more complex than this, you can absolutely write tests for your fakes. In many cases fake data-stores use tested backends, and tested client libraries. If the client wrapper is complex enough then you can/should test it as well.
But those are easy to find. This is why double-entry bookkeeping is the accounting standard: it's easy to add incorrectly once. It's much less common to add incorrectly the exact same way twice.
There is another advantage of good fakes: they often can be shipped in production as your demo mode. I've seen cases where when of the most important new features to marketing was just make some part of the fake more useful in production. Sometimes because there is a show coming up and the demo mode is what customers will see. Sometimes because the trainers want a better simulation of some issue.
In some cases the fake is a lot more complex than the real implementation. (I worked on a diagnostic tool, the real implementation "just" needed to send/receive bytes, the fake needed to understand the bytes and return responses)
Fakes are great for objects that store data. Like in this example.
If I use Entity Framework Core I also love the in-memory provider (or SQLite in-memory), which gives you a very well faked database context for unit tests.
In some cases you can create end to end tests, that run and verify the complete business logic, without hitting the database at all. A mixture between unit and integration tests.
> If I use Entity Framework Core I also love the in-memory provider (or SQLite in-memory), which gives you a very well faked database context for unit tests.
This approach easy to setup, however I found that it slows down unit tests especially with data migrations and setup before each test (to discard test data from other tests). I've since limited these to test only migrations and in some well thought out integration testing.
The example is a little contrived, but the whole article fits on one sheet of paper and does yield a better test than mocks would.
Testing against the real system is always better than testing against a fake. That should be your first preference, and if you can modify the system to make testing against it easier, that's even better. Sometimes you can't, and if your interaction is complicated and the parts that you care about are simple, a fake can get you good confidence that your part works. Mocks just feel like writing the code again -- in your code, you write "call a function with an argument, then call another function with an argument" and in the tests you write "pass if a function is called with an argument, and again with an argument". It can be the right thing at the right time, but it's usually writing a test to make some automated linter shut up. And that's a complete waste of time.
> Mocks work well because you can quickly define them in a given unit test and forget about it.
They do, yes. But:
* the mock syntax is complex and verbose. e.g. "repo.Setup(x => x.Method(It.IsAny<string>(), It.IsAny<List<int>>())) .ReturnsAsync(() => ...." is not the easiest syntax.
The syntax for a fake is just regular old "class FakeFoo: IFoo" with trivial method implementations and no library needed to make it work.
* the mock syntax does not scale. Your 5 lines of setup in one test does not help you in a different test class. So it gets repeated.
Soon you might find that the Fake is easier to read, easier to maintain, and fewer lines of code than all the mocks.
As the article says: "their self-contained nature makes them reusable as well, which lends itself to better long-term maintainability." Not all our code is "do it quickly and forget about it", especially where you see that same thing or minor variations of it done over and over again in related tests.
Mocks win here when you do something different, once only. Fakes win in the opposite cases.
> I can see fakes being useful in certain cases,
This is true, but it's also true that Fakes are _underused_ in general in in the .NET world. People reach for mocking frameworks as if they are the only tool.
I question a lot of advice like this, and agree with you. I’d generally far rather use NSubstitute than have to write fake classes, but I can see the utility in some cases.
My honest response is this: Writing tests should be quick and painless. Maintaining fakes is more painful than maintaining substitutes. Substitutes let me interrogate side effects in great detail. I’m not concerned will performance in unit tests. Ergo, substitutes are quicker and less painful, so IMO that’s a win.
But for real though, why should anyone care about someone else’s test architecture, of all things. As long as you neither under- or over-testing, why on earth does it matter?
I have been researching into several testing approaches and so far in my experience yes fakes are better than mocks. But what is even better is not having to fake or mock stuff especially for testing business logic.
One of the problems we get ourselves into is abstracting service io code into functions and calling them from our business logic code. This forces us to mock all the io calls, instead we should be doing the opposite i.e abstracting business logic into discrete functions of pure data in and out and then calling those functions from our io code.
This way we can test the business logic independently of the io code. Testing the business logic now becomes quite simple and one can leverage property testing to further boost our confidence in such tests.
Many times the ramaining io code is just calling other functions and doesn't need to be tested.
Several other ideas that relate to this are hexagonal architecture, Domain Driver Design, clean architecture and functional core with an imperative shell.
I don't think he really sells the point to me. You can use any tool incorrectly. An application with poor composition conmbined with mocks without verifications leads you down the path he describes. What we should do is only use "mocks to create fakes", which if you read the manuals on Mockito, is what they're pushing you towards anyway.
So in conclusion I would say that a lot of people use mocking frameworks incorrectly and the author is hitting home on that.
I'm a little surprised this article and none of the contents here at the time of this writing mentioned inheritance.
The core reason I find OOP to be superior for most classes of business application development in general—and web development in particular—is that external dependencies and other high cost interfaces can be delegated to their own method or instance variable then called. In general, I avoid inheritance if there is a way to approach a problem with composition, but testing via fakes are much more reliable to make by inheriting the original class and overriding a method or setting a default initialization parameter then calling the rest of the constructor. This takes the pain out of so many parts of testing that it's truly hard to understate.[0]
For example, say a library has a class that calls out to a third party service like S3 and an instance of that class is used in a business object in order to persist data. By introducing a fake that inherits from the library's class but overrides it's persistence method to effectively be a noop the rest of the validation, calculation, and other side effects that the instance does is conserved without the downside of waiting on a network call. It's not perfect, for example it could miss validation or side effects that only exist on the external service (e.g., S3 could change a mime type based on some crummy AWS logic) but it's a hell of a lot better than a mock that constantly lies to you about what its done. Furthermore, it's actually extremely simple to code up, unlike pure fakes that need to return hard coded values for certain method calls.
I know that something akin to this is theoretically possible in functional languages, but I've found that in practice it's more difficult. Though I appreciate other areas where functional languages shine, including situations with testing.
It's kinda hard to discuss because it really depends on so much context and good judgement and I've seen truly great development teams use languages like Scala in situations where I would have used Python with Numpy / Cython for speed. So don't take this as some hard and fast pronouncement. Just the general observations of someone that's been coding for decades and is reasonably well paid.
[0] I think the primary reason that I prefer integration testing over unit testing is that I've adapted to its shortcomings by the above approach.
> "However, relying on dynamically-generated mocks can be dangerous, as it typically leads to implementation-aware testing."
I agree, tests should be simple! Dynamically generating stubs, mocks or even arguments for tests hides complexity which leads to test code that's hard to reason about... and is hard to change/fix
Simply saying prefer "this" over "that" without adding any context to the equation is the best recipe to end up doing the wrong thing in the wrong place. Or maybe you mean that you can make the same decisions when testing a Web based app UI than when testing a cryptographic function in the Linux kernel?
IMO, both mocks, fakes and "reals" (to put it in the same language) need to be first class citizens of software testing, each one of them used in the right circumstances.
Just like resorting to unit testing only is as much a bad practice as it is to use acceptance testing only.
A good testing strategy should be adapted to each software and context, and potentially include all types of tests: unit, end-to-end, integration, numerical, acceptance, etc. and all sorts of resources: mocks, dummy data, substitutes, testing engineers, test databases, real databases, real users, etc.
I've tried this approach twice with mixed success. In both cases I wanted to stub out the persistence tier of a Node.js application when test driving the API. I verified the real and fake implementations by running the same tests against them. The first application was quite small, and the process worked well, although it did feel somewhat onerous to implement the fake. However, the second application was more complex, calling for some PostgreSQL specific features which were difficult to implement in the fake. In the end I abandoned the approach for the second application, settling on slower, duplication heavy API tests which used the real persistence tier. I'd love to hear how others solve this problem without mocks, which I agree tend towards brittle, tightly coupled tests with a poorer signal to noise ratio.
The best solution doesn't need anything, e.g. if we have calls to our persistence layer mixed in with calculations, the latter shouldn't be tested with fakes/mocks; instead we should refactor the code so the calculations don't depend on the persistence layer.
If the behaviour of some code depends inextricably on some external system, then our tests should use that system. This avoids unnecessary code/work/duplication, allows tests to exercise more paths through our codebase, exposes problems in our understanding of that system, etc.
If calling an external system in our tests is dangerous, unacceptably slow, costly (if it's some Web service), etc. then we should make a fake implementation to test against. If our code only uses a small part of some large, complicated dependency, we might want to define an interface for the subset that we need, refactor our code to use that, and only have our fake implement that part.
If a fake implementation isn't practical, or would require so much overriding per-test as to be useless on its own, then I might consider using mocks. I would seriously consider whether that code could be refactored to avoid such coupling. I would also never make assertions about irrelevant details, like whether a particular method was called, or what order things are run in.
I'd like to add my own thoughts to this discussion, but feel it's the kind of topic that blows up into a mess/mass of opinion such that adding another comment just adds to the dogpile; after a certain critical mass, no-one really reads the majority of comments before commenting themselves, and a positive feedback of repetition begins.
Maybe if the discussion was broken into narrow sub-topics this could be fixed?
[+] [-] tomgp|5 years ago|reply
[+] [-] rkangel|5 years ago|reply
The simplest way of doing this is to write code with good practices then do a load of manual system testing to find bugs (system testing including user acceptance testing in this example). This is basically what most people were doing 20 years ago.
This can work, but doing the system testing is time consuming and so expensive and so makes making changes expensive. Instead you automate some or all of your system testing. This makes them cheaper, but these tests are expensive to maintain and still time consuming to run. If the test fails you don't have that much information about the problem.
To complement (or maybe replace) your automated system testing, you add tests for smaller and smaller chunks of software, allowing for tighter feedback loops and more precise diagnosis (there are also some negative tradeoffs). Applied well, this allows you to write features more quickly, refactor more confidently and generally develop more efficiently.
Basically, it's not so much about catching bugs as it is about getting to your desired level of reliability in the most cost and time efficient way.
[+] [-] FartyMcFarter|5 years ago|reply
The end goal is still avoiding defects, helping developers is just the "middle-man".
[+] [-] hinkley|5 years ago|reply
[+] [-] rocqua|5 years ago|reply
Bugs can be hard to cause, and thus not be noticed by user acceptance testing. So for catching them, tests are more useful.
[+] [-] bluGill|5 years ago|reply
[+] [-] unknown|5 years ago|reply
[deleted]
[+] [-] mcqueenjordan|5 years ago|reply
[+] [-] mabbo|5 years ago|reply
It's a fantastic comparison of the Detroit school of testing (what you and I believe) vs the London school, which is simply a different mental model about code and testing and mocks especially.
Framing the differences this way make it clearer why some people might disagree with us, and it's not because they're stupid or wrong- they just have a different school of thought on the matter. This doesn't excuse, however, the people who are doing a little bit of both and they don't know why, and they're getting the benefits of neither side of that coin.
[+] [-] ratww|5 years ago|reply
Also, in dynamic languages, you might even have tests that don’t break when you change (or sometimes even delete) the dependency because the mock was correct. That leads to a false sense of confidence.
[+] [-] hinkley|5 years ago|reply
Having multiple tests fail for the same reason is a waste of effort, either now or in the future (when new features are added or the constraints otherwise change).
Sometimes you have to depend on transitive properties to keep your test sizes down. If a function received the data, and the function has been tested thoroughly, why test it again while testing the callers?
[+] [-] paranoidrobot|5 years ago|reply
Say I'm writing tests for lets say ShoppingCart.
ShoppingCart depends on an implementation of ICartDiscountCalculator (CartDiscountCalculator)
CartDiscountCalculator depends on a number of implementations of ICartDiscountStrategy (ChristmasCartDiscountStrategy, VIPCustomerDiscountStrategy, BulkPurchaseDiscountStrategy, etc)
Each of those, in turn has their own dependencies, perhaps webservice calls, db calls, accessing caches, etc.
The way I would test ShoppingCart would be to mock out ICartDiscountCalculator, and provide known responses.
Are you suggesting instead that you build up the dependency graph - i.e the actual implementation of the discount calculator, the strategies, etc - and only mock when it goes to external code (eg the DB calls)?
This would seem to make testing very very difficult - since each test needs to know implementation details of everything it depends on.
[+] [-] viraptor|5 years ago|reply
[+] [-] Griffon26|5 years ago|reply
Yes, integration tests are important too, but you'll want to minimize the number of scenarios you still need to test at that level.
[+] [-] mathgladiator|5 years ago|reply
There are a lot of really good "fakes" out there. Fake file system which lets you simulate disk failures. Fake time and randomness to introduce determinism. Every API over network can have a fake. These are all good things to fake out as you can simulate the appropriate failures which are almost impossible to do via integration, acceptance, or E2E tests.
[+] [-] whalesalad|5 years ago|reply
The lines have all been blurred and words have different meanings across different language ecosystems.
[+] [-] viraptor|5 years ago|reply
[+] [-] silveroriole|5 years ago|reply
[+] [-] jakevoytko|5 years ago|reply
I argue that [0] the liability of mocks are much higher than the fake. With a fake, you could theoretically write tests that asserts that your class is satisfying the interface. There's no way to do this with a mock. You can just make an interface do anything, regardless of whether it makes sense. This is fragile under modification. With a large codebase, you're inevitably going to change the behavior of a class during a refactoring. Are you going to read every test that mocks that class to decide how the mocks should be rewritten? Probably not - the mocks force-overrode the behavior of the class, so your tests will still pass. Why would you look at them again?
> Make a fake for every error scenario?
By the time you're implementing a fake, you have an interface that the fake implements. One way that I've handled this is to make a FailingBlobStorage that takes an error in its constructor and throws it from any calls. If you need even more detailed behavior than that, you can create a mock for that specific test case. You're not married to the fake. It's not going to magically test every scenario. But fakes handle the common case in so many situations that it's actually surprising when you start trying them out.
[0] In fact, I've basically written this same blog post before. https://www.bitlog.com/2019/12/10/simple-software-engineerin...
[+] [-] andix|5 years ago|reply
The good thing about (simple) fakes is, that you can see all the code in a very small class. No complicated library, that does something funky in a special case.
Mocking can be very useful though, but it can get too complicated very quickly.
[+] [-] heinrichhartman|5 years ago|reply
Why would you not look into the fake implementations? If it's just a two liner that takes the parameters and writes them to disk, then that's a LGTM from me. You would not test every getter/setter method in any of your classes, would you?
If it's more complex than this, you can absolutely write tests for your fakes. In many cases fake data-stores use tested backends, and tested client libraries. If the client wrapper is complex enough then you can/should test it as well.
[+] [-] UK-Al05|5 years ago|reply
Test for the relied upon behaviours.
It's essentially contract testing.
It's great objects that perform external actions.
[+] [-] commandlinefan|5 years ago|reply
But those are easy to find. This is why double-entry bookkeeping is the accounting standard: it's easy to add incorrectly once. It's much less common to add incorrectly the exact same way twice.
[+] [-] kelnos|5 years ago|reply
I'm not sure I buy this as something that's desirable to have to do, but it should address your concern.
[+] [-] bluGill|5 years ago|reply
In some cases the fake is a lot more complex than the real implementation. (I worked on a diagnostic tool, the real implementation "just" needed to send/receive bytes, the fake needed to understand the bytes and return responses)
[+] [-] andix|5 years ago|reply
If I use Entity Framework Core I also love the in-memory provider (or SQLite in-memory), which gives you a very well faked database context for unit tests.
In some cases you can create end to end tests, that run and verify the complete business logic, without hitting the database at all. A mixture between unit and integration tests.
https://docs.microsoft.com/en-us/ef/core/miscellaneous/testi...
[+] [-] zabil|5 years ago|reply
This approach easy to setup, however I found that it slows down unit tests especially with data migrations and setup before each test (to discard test data from other tests). I've since limited these to test only migrations and in some well thought out integration testing.
[+] [-] Ronsenshi|5 years ago|reply
I can see fakes being useful in certain cases, but only just in certain cases.
Could be my inexperience with C#, but it didn't do justice describing the point in code samples.
[+] [-] jrockway|5 years ago|reply
The example is a little contrived, but the whole article fits on one sheet of paper and does yield a better test than mocks would.
Testing against the real system is always better than testing against a fake. That should be your first preference, and if you can modify the system to make testing against it easier, that's even better. Sometimes you can't, and if your interaction is complicated and the parts that you care about are simple, a fake can get you good confidence that your part works. Mocks just feel like writing the code again -- in your code, you write "call a function with an argument, then call another function with an argument" and in the tests you write "pass if a function is called with an argument, and again with an argument". It can be the right thing at the right time, but it's usually writing a test to make some automated linter shut up. And that's a complete waste of time.
[+] [-] SideburnsOfDoom|5 years ago|reply
They do, yes. But:
* the mock syntax is complex and verbose. e.g. "repo.Setup(x => x.Method(It.IsAny<string>(), It.IsAny<List<int>>())) .ReturnsAsync(() => ...." is not the easiest syntax.
The syntax for a fake is just regular old "class FakeFoo: IFoo" with trivial method implementations and no library needed to make it work.
* the mock syntax does not scale. Your 5 lines of setup in one test does not help you in a different test class. So it gets repeated.
Soon you might find that the Fake is easier to read, easier to maintain, and fewer lines of code than all the mocks.
As the article says: "their self-contained nature makes them reusable as well, which lends itself to better long-term maintainability." Not all our code is "do it quickly and forget about it", especially where you see that same thing or minor variations of it done over and over again in related tests.
Mocks win here when you do something different, once only. Fakes win in the opposite cases.
> I can see fakes being useful in certain cases,
This is true, but it's also true that Fakes are _underused_ in general in in the .NET world. People reach for mocking frameworks as if they are the only tool.
[+] [-] ddek|5 years ago|reply
My honest response is this: Writing tests should be quick and painless. Maintaining fakes is more painful than maintaining substitutes. Substitutes let me interrogate side effects in great detail. I’m not concerned will performance in unit tests. Ergo, substitutes are quicker and less painful, so IMO that’s a win.
But for real though, why should anyone care about someone else’s test architecture, of all things. As long as you neither under- or over-testing, why on earth does it matter?
[+] [-] srtjstjsj|5 years ago|reply
[+] [-] hackerfromthefu|5 years ago|reply
[+] [-] aszen|5 years ago|reply
One of the problems we get ourselves into is abstracting service io code into functions and calling them from our business logic code. This forces us to mock all the io calls, instead we should be doing the opposite i.e abstracting business logic into discrete functions of pure data in and out and then calling those functions from our io code. This way we can test the business logic independently of the io code. Testing the business logic now becomes quite simple and one can leverage property testing to further boost our confidence in such tests.
Many times the ramaining io code is just calling other functions and doesn't need to be tested.
Several other ideas that relate to this are hexagonal architecture, Domain Driver Design, clean architecture and functional core with an imperative shell.
Some resources to further explore these ideas:
Clean Architecture in Python - https://youtu.be/DJtef410XaM
Why integrated tests are a scam - https://youtu.be/VDfX44fZoMc
Boundaries - https://youtu.be/yTkzNHF6rMs
[+] [-] exabrial|5 years ago|reply
So in conclusion I would say that a lot of people use mocking frameworks incorrectly and the author is hitting home on that.
[+] [-] srtjstjsj|5 years ago|reply
[+] [-] unknown|5 years ago|reply
[deleted]
[+] [-] 3pt14159|5 years ago|reply
The core reason I find OOP to be superior for most classes of business application development in general—and web development in particular—is that external dependencies and other high cost interfaces can be delegated to their own method or instance variable then called. In general, I avoid inheritance if there is a way to approach a problem with composition, but testing via fakes are much more reliable to make by inheriting the original class and overriding a method or setting a default initialization parameter then calling the rest of the constructor. This takes the pain out of so many parts of testing that it's truly hard to understate.[0]
For example, say a library has a class that calls out to a third party service like S3 and an instance of that class is used in a business object in order to persist data. By introducing a fake that inherits from the library's class but overrides it's persistence method to effectively be a noop the rest of the validation, calculation, and other side effects that the instance does is conserved without the downside of waiting on a network call. It's not perfect, for example it could miss validation or side effects that only exist on the external service (e.g., S3 could change a mime type based on some crummy AWS logic) but it's a hell of a lot better than a mock that constantly lies to you about what its done. Furthermore, it's actually extremely simple to code up, unlike pure fakes that need to return hard coded values for certain method calls.
I know that something akin to this is theoretically possible in functional languages, but I've found that in practice it's more difficult. Though I appreciate other areas where functional languages shine, including situations with testing.
It's kinda hard to discuss because it really depends on so much context and good judgement and I've seen truly great development teams use languages like Scala in situations where I would have used Python with Numpy / Cython for speed. So don't take this as some hard and fast pronouncement. Just the general observations of someone that's been coding for decades and is reasonably well paid.
[0] I think the primary reason that I prefer integration testing over unit testing is that I've adapted to its shortcomings by the above approach.
[+] [-] etripe|5 years ago|reply
> I know that something akin to this is theoretically possible in functional languages, but I've found that in practice it's more difficult.
Could you explain what you think is hard about it?
[+] [-] srtjstjsj|5 years ago|reply
[+] [-] unknown|5 years ago|reply
[deleted]
[+] [-] gitgud|5 years ago|reply
I agree, tests should be simple! Dynamically generating stubs, mocks or even arguments for tests hides complexity which leads to test code that's hard to reason about... and is hard to change/fix
[+] [-] csala|5 years ago|reply
IMO, both mocks, fakes and "reals" (to put it in the same language) need to be first class citizens of software testing, each one of them used in the right circumstances. Just like resorting to unit testing only is as much a bad practice as it is to use acceptance testing only.
A good testing strategy should be adapted to each software and context, and potentially include all types of tests: unit, end-to-end, integration, numerical, acceptance, etc. and all sorts of resources: mocks, dummy data, substitutes, testing engineers, test databases, real databases, real users, etc.
[+] [-] scresswell|5 years ago|reply
[+] [-] chriswarbo|5 years ago|reply
The best solution doesn't need anything, e.g. if we have calls to our persistence layer mixed in with calculations, the latter shouldn't be tested with fakes/mocks; instead we should refactor the code so the calculations don't depend on the persistence layer.
If the behaviour of some code depends inextricably on some external system, then our tests should use that system. This avoids unnecessary code/work/duplication, allows tests to exercise more paths through our codebase, exposes problems in our understanding of that system, etc.
If calling an external system in our tests is dangerous, unacceptably slow, costly (if it's some Web service), etc. then we should make a fake implementation to test against. If our code only uses a small part of some large, complicated dependency, we might want to define an interface for the subset that we need, refactor our code to use that, and only have our fake implement that part.
If a fake implementation isn't practical, or would require so much overriding per-test as to be useless on its own, then I might consider using mocks. I would seriously consider whether that code could be refactored to avoid such coupling. I would also never make assertions about irrelevant details, like whether a particular method was called, or what order things are run in.
[+] [-] doubletgl|5 years ago|reply
[+] [-] Chris2048|5 years ago|reply
I'd like to add my own thoughts to this discussion, but feel it's the kind of topic that blows up into a mess/mass of opinion such that adding another comment just adds to the dogpile; after a certain critical mass, no-one really reads the majority of comments before commenting themselves, and a positive feedback of repetition begins.
Maybe if the discussion was broken into narrow sub-topics this could be fixed?
[+] [-] unknown|5 years ago|reply
[deleted]
[+] [-] pjmlp|5 years ago|reply
https://docs.microsoft.com/en-us/visualstudio/test/isolating...
Which I initially thought was the topic being discussed.
[+] [-] sethammons|5 years ago|reply
So has Martin Fowler: https://martinfowler.com/articles/mocksArentStubs.html
The TL;DR: Mocks increase coupling and make systems harder to change. Fakes are more versatile.