The problem with mocks is that they test your assumptions, not reality...
When you mock a CRM client to return one account, you're assuming it always returns one account, that IDs have a particular format, that there's no pagination, that all fields are populated. Each assumption is a place where production could behave differently whilst your tests stay green
Your contract tests use cached JSON fixtures. Salesforce changes a field type, your contract test still passes (old fixture), your mocks return the wrong type, production breaks. You've now got three test layers (contract, mock scenarios, E2E) where two can lie to you. All your contract and mock tests won't save you. Production will still go down
I have zero confidence in these types of tests. Integration tests and E2E tests against real infrastructure give me actual confidence. They're slower, but they tell you the truth. Want to test rate limiting? Use real rate limits. Want to test missing data? Delete the data.
Slow tests that tell the truth beat fast tests that lie. That said, fast tests are valuable for developer productivity. The trade-off is whether you want speed or confidence
I really dislike this idea of testing in go: only ever use an interface, never the real implementation + mockgen the mocks based on this interface + use the mocks to assert that a function is called, with exactly this parameters and in this exact order.
I find this types of tests incredibly coupled with the implementation, since any chance require you to chance your interfaces + mocks + tests, also very brittle and many times it ends up not even testing the thing that actually matters.
I try to make integration test whenever possible now, even if they are costly I find that the flexibility of being able to change my implementation and not break a thousand tests for no reason much better to work with.
I'm a fan of writing tests that can be either. Write your tests first such that the real dependencies can be run against. Snapshot the results to feed into integration test mocks for those dependencies so that you can maintain the speed benefit of limited test scope. Re-run against the real dependencies at intervals you feel is right to ensure that your contracts remain satisfied, or just dedicate a test per external endpoint on top of this to validate the response shape hasn't changed.
The fundamental point of tests should be to check that your assumptions about a system's behavior hold true over time. If your tests break that is a good thing. Your tests breaking should mean that your users will have a degraded experience at best if you try to deploy your changes. If your tests break for any other reason then what the hell are they even doing?
> I really dislike this idea of testing in go: only ever use an interface, never the real implementation + mockgen the mocks based on this interface + use the mocks to assert that a function is called, with exactly this parameters and in this exact order.
Same I have zero confidence in these tests and the article even states that the tests will fail if a contract for a external service/system changes
If you're testing the interface, changing the implementation internals won't create any churn (as the mocks and tests don't change).
If you are changing the interface, though, that would mean a contract change. And if you're changing the contract, surely you wouldn't be able to even use the old tests?
This isn't really a go problem at all. Any contract change means changing tests.
My DB heavy app, when I run a Go unit test, it spins up a DB instance, populates it, runs the tests, and drops the DB. Never ever any mocks. The best part about unit tests isn't testing the Go code. It is testing the SQL.
We do the same thing. The core of our service is ingesting data and applying transformations to it. There’s so many permutations and complex interactions in here that the only way to ensure a refactor hasn’t broken one of these interactions is to document those edge cases by piping data through the system and reading it back out. We have thousands of these tests and it’s all tidy controlled via docker compose. It takes about 15 minutes to run the test suite. Sure I wish it was faster - but the real unlock is that we can make big sweeping refactors without breaking behaviors of the system. The organizational speed unlock this kind of safety net is well worth a bit of slowness in CI.
We also have mocks/stubs/spies in our unit tests. Those are great for producing hard-to-trigger edge cases. But contract testing? The contract is the data flow. In the end it’s all about using the right tool for the right test. There is no one-size-fits-all.
If your SQL is isolated to one place then you only need to test that single package with a real DB and can use mocks everywhere else. Your mocks can be tested with the exact same test suite as the SQL package, so you know it conforms to the same contract.
If you have SQL scattered all over the place... Leave the spaghetti for dinner.
I learned “test your mocks” long ago from Sandi Metz, and that advice has paid off well for me. Have some set of behavioral conformance tests for the kind of thing you expect (e.g. any database worth its salt should be able write and read back the same record). Then stick your mock right under that same battery of tests alongside your implementation(s). If either deviate from the behavior you depend on, you’ll know about it.
Bang on Sandi! I hadn’t heard that quote but she’s my favorite speaker on testing and OO.
Another way of looking at this advice is that every time there’s a mock there needs to be a test that shows that the real code can be used in the same way that the mock is used.
I'm of the opinion that mocks should be provided by the thing that you're mocking. That is, if you are wanting to mock out a service, the mock should be owned by the service that is being mocked.
And then it should be part of that service's test suite, to verify it's own mock.
You update your service? Then you must update the mock.
I guess that's more of a fake, but the naming doesn't matter as much as the behavior.
Testing in Go is great, for unit testing. But anything above that should be done manually. "Mocking" never made sense to me and I never used it in my entire life. If I want to test a database/repository, I will spin up a real database with real data and data persistence instead of relying on in-memory storage and try to abstract away things that make no sense to do in the real world, as we're not living in a theoretical world of interfaces and adapters and whatnot.
Interesting point about mocks being seen as a bad word. I've been in situations where relying solely on integration tests led to some really frustrating moments. It's like every time we thought we had everything covered, some edge case would pop up out of nowhere due to an API behaving differently than we expected. I remember one time we spent hours debugging a production incident, only to realize a mock that hadn’t been updated was the culprit—definitely felt like we'd fallen into that "mock drift" trap.
I've also started to appreciate the idea of contract tests more and more, especially as our system scales. It kind of feels like setting a solid foundation before building on top. I haven’t used Pact or anything similar yet, but it’s been on my mind.
I wonder if there’s a way to combine the benefits of mocks and contracts more seamlessly, maybe some hybrid approach where you can get the speed of mocks but with the assurance of contracts... What do you think?
aranw|2 months ago
When you mock a CRM client to return one account, you're assuming it always returns one account, that IDs have a particular format, that there's no pagination, that all fields are populated. Each assumption is a place where production could behave differently whilst your tests stay green
Your contract tests use cached JSON fixtures. Salesforce changes a field type, your contract test still passes (old fixture), your mocks return the wrong type, production breaks. You've now got three test layers (contract, mock scenarios, E2E) where two can lie to you. All your contract and mock tests won't save you. Production will still go down
I have zero confidence in these types of tests. Integration tests and E2E tests against real infrastructure give me actual confidence. They're slower, but they tell you the truth. Want to test rate limiting? Use real rate limits. Want to test missing data? Delete the data.
Slow tests that tell the truth beat fast tests that lie. That said, fast tests are valuable for developer productivity. The trade-off is whether you want speed or confidence
unknown|2 months ago
[deleted]
Thaxll|2 months ago
You make a lot of assumption about contract change which in reality should rarely happen.
SPascareli13|2 months ago
I find this types of tests incredibly coupled with the implementation, since any chance require you to chance your interfaces + mocks + tests, also very brittle and many times it ends up not even testing the thing that actually matters.
I try to make integration test whenever possible now, even if they are costly I find that the flexibility of being able to change my implementation and not break a thousand tests for no reason much better to work with.
Phlebsy|2 months ago
The fundamental point of tests should be to check that your assumptions about a system's behavior hold true over time. If your tests break that is a good thing. Your tests breaking should mean that your users will have a degraded experience at best if you try to deploy your changes. If your tests break for any other reason then what the hell are they even doing?
aranw|2 months ago
Same I have zero confidence in these tests and the article even states that the tests will fail if a contract for a external service/system changes
tonyhb|2 months ago
If you are changing the interface, though, that would mean a contract change. And if you're changing the contract, surely you wouldn't be able to even use the old tests?
This isn't really a go problem at all. Any contract change means changing tests.
unknown|2 months ago
[deleted]
esafak|2 months ago
kardianos|2 months ago
Yes, it takes longer to run your tests. So be it.
bikelang|2 months ago
We also have mocks/stubs/spies in our unit tests. Those are great for producing hard-to-trigger edge cases. But contract testing? The contract is the data flow. In the end it’s all about using the right tool for the right test. There is no one-size-fits-all.
preetamjinka|2 months ago
We also have mocks. It’s not one way or the other. This post is talking about the mocking side of things.
9rx|2 months ago
If you have SQL scattered all over the place... Leave the spaghetti for dinner.
teeray|2 months ago
I learned “test your mocks” long ago from Sandi Metz, and that advice has paid off well for me. Have some set of behavioral conformance tests for the kind of thing you expect (e.g. any database worth its salt should be able write and read back the same record). Then stick your mock right under that same battery of tests alongside your implementation(s). If either deviate from the behavior you depend on, you’ll know about it.
zingar|2 months ago
Another way of looking at this advice is that every time there’s a mock there needs to be a test that shows that the real code can be used in the same way that the mock is used.
mzi|2 months ago
This excludes a lot of cases, like just a simple postgres where reads are done from a replica.
fireflash38|2 months ago
And then it should be part of that service's test suite, to verify it's own mock.
You update your service? Then you must update the mock.
I guess that's more of a fake, but the naming doesn't matter as much as the behavior.
gethly|2 months ago
isuckatcoding|2 months ago
https://pactflow.io/
fleahunter|2 months ago
I've also started to appreciate the idea of contract tests more and more, especially as our system scales. It kind of feels like setting a solid foundation before building on top. I haven’t used Pact or anything similar yet, but it’s been on my mind.
I wonder if there’s a way to combine the benefits of mocks and contracts more seamlessly, maybe some hybrid approach where you can get the speed of mocks but with the assurance of contracts... What do you think?
konart|2 months ago