I just listened to the DHH/Kent Beck/Martin Fowler discussion about TDD "damage" and both sides still seemed unconvinced by the end of it, but this exact example came up. It seems like SOA (whether it's DDD or Hexagonal or Clean or w/e) and TDD really push you towards this kind of layer bloat for one reason or another.
I'm (maybe obviously) on the SOA-skeptic side, my arguments generally are:
- Most apps aren't that big and don't need multiple layers of abstraction (i.e. the ORM and its models are totally fine). If the app starts getting too big for its britches, probably the best thing to do is make it 2 apps (too big: 2 apps is a good slogan here).
- Dependency injection and mocks are pretty bad ideas that are only occasionally useful (DHH uses the example of a payments gateway), but mostly push IoC through your whole app and make control flow confusingly backwards. Mocks are always in disrepair, and almost never accurately reflect what they're trying to mock, and thus ironically are big vectors for bugs that make it through testing.
- Having tons of unit tests tends to slow eng velocity to a crawl, because they test the parts of the application that aren't the requirements (were these functions called, what's the call signature of this function, was this class instantiated, etc.). Unit tests create a super-fine-grained shadow spec about the lowest level details of your application, and mostly I think they shouldn't ever be committed to a repo. They help during individual development, but then the whole team is stuck maintaining them forever whenever they make changes. They also tend to slow down CI because they're slow and always flaky.
- You almost certainly will never need to switch databases, let alone abstract across a database, a message queue, and a web api. It's not worth doing a "repo" abstraction and encapsulating those details.
- There are (now) really good libraries for almost anything you want to do. ORMs literally map database entities to domain entities--they just abstract the persistence for you. Sounds like a repo to me! We also have good validation, logging, monitoring, auth/auth etc. built into frameworks and 3rd party services. A lot of the things you might put into other layers or even other services are now neatly packaged into libraries/frameworks you can just use and SaaS things you can just buy, leaving you free to just implement your business logic.
I generally agree with the position that unit tests should be used with discretion, and that full coverage via unit tests often leads to thousands of low-ulitility or redundant tests, and so on. However I cannot agree with this:
>They also tend to slow down CI because they're slow and always flaky.
In my experience, unit tests are the most stable, the least flaky, because they touch the least code and often have very simple setup. An integration test might rely on four database tables being just-so, and go on to connect with two external services (and whether mocked, replayed, or live, flakiness may arise). That integration test is twenty times more valuable, but it is equally more likely to break for reasons tangential to its core assertions.
Agree on the points that you should never need to abstract over your database, orm, message queue, etc.
Disagree on dependency injection. I came from the globals/patch everything school of python, to the Fastapi/Pytest DI flavor, and it's a breath of fresh air. It's just so much easier to abstract the IO providers and swap them out with objects tailored to the test suite - eg for database, I create db objects which roll back any transactions between tests.
Hard disagree on unit tests. Maybe in other languages, but in Python, trying to develop even a moderately complex app without unit tests is a nightmare. I know, I've lived it. Even in an app with >85% unit test coverage, there was still a ton of friction around development on any of the interfaces which had low coverage.
Any gains in velocity of development almost always cost far more in debugging down the road.
I love python, but it is really prone to dumb footguns at runtime, NoneType errors in particular. You need to impose a lot of discipline to make large python apps enjoyable to develop on.
I mostly agree with you and DHH on that topic, however in my experience reasonably applied SOA/DDD actually shields me from this layering nonsense.
When your apps live as a service on the network or as a nicely isolated module in your repo, you no longer have a reason to over-engineer them. You don't need a grandiose architecture that solves every problem, instead you can make local decisions that are good enough in the specific context. Though, admittedly, I found it hard to sell such "inconsistencies" to other tech leaders, most folks aspire to those grandiosities.
> If the app starts getting too big for its britches, probably the best thing to do is make it 2 apps
Thaxll|3 years ago
Storing the API model in your DB is really a bad idea.
camgunz|3 years ago
I'm (maybe obviously) on the SOA-skeptic side, my arguments generally are:
- Most apps aren't that big and don't need multiple layers of abstraction (i.e. the ORM and its models are totally fine). If the app starts getting too big for its britches, probably the best thing to do is make it 2 apps (too big: 2 apps is a good slogan here).
- Dependency injection and mocks are pretty bad ideas that are only occasionally useful (DHH uses the example of a payments gateway), but mostly push IoC through your whole app and make control flow confusingly backwards. Mocks are always in disrepair, and almost never accurately reflect what they're trying to mock, and thus ironically are big vectors for bugs that make it through testing.
- Having tons of unit tests tends to slow eng velocity to a crawl, because they test the parts of the application that aren't the requirements (were these functions called, what's the call signature of this function, was this class instantiated, etc.). Unit tests create a super-fine-grained shadow spec about the lowest level details of your application, and mostly I think they shouldn't ever be committed to a repo. They help during individual development, but then the whole team is stuck maintaining them forever whenever they make changes. They also tend to slow down CI because they're slow and always flaky.
- You almost certainly will never need to switch databases, let alone abstract across a database, a message queue, and a web api. It's not worth doing a "repo" abstraction and encapsulating those details.
- There are (now) really good libraries for almost anything you want to do. ORMs literally map database entities to domain entities--they just abstract the persistence for you. Sounds like a repo to me! We also have good validation, logging, monitoring, auth/auth etc. built into frameworks and 3rd party services. A lot of the things you might put into other layers or even other services are now neatly packaged into libraries/frameworks you can just use and SaaS things you can just buy, leaving you free to just implement your business logic.
chao-|3 years ago
>They also tend to slow down CI because they're slow and always flaky.
In my experience, unit tests are the most stable, the least flaky, because they touch the least code and often have very simple setup. An integration test might rely on four database tables being just-so, and go on to connect with two external services (and whether mocked, replayed, or live, flakiness may arise). That integration test is twenty times more valuable, but it is equally more likely to break for reasons tangential to its core assertions.
kortex|3 years ago
Disagree on dependency injection. I came from the globals/patch everything school of python, to the Fastapi/Pytest DI flavor, and it's a breath of fresh air. It's just so much easier to abstract the IO providers and swap them out with objects tailored to the test suite - eg for database, I create db objects which roll back any transactions between tests.
Hard disagree on unit tests. Maybe in other languages, but in Python, trying to develop even a moderately complex app without unit tests is a nightmare. I know, I've lived it. Even in an app with >85% unit test coverage, there was still a ton of friction around development on any of the interfaces which had low coverage.
Any gains in velocity of development almost always cost far more in debugging down the road.
I love python, but it is really prone to dumb footguns at runtime, NoneType errors in particular. You need to impose a lot of discipline to make large python apps enjoyable to develop on.
yunohn|3 years ago
Often, engineers (and HN) forget that code is a means to an end - not an artistic expression that provides value by its pure existence.
hbrn|3 years ago
When your apps live as a service on the network or as a nicely isolated module in your repo, you no longer have a reason to over-engineer them. You don't need a grandiose architecture that solves every problem, instead you can make local decisions that are good enough in the specific context. Though, admittedly, I found it hard to sell such "inconsistencies" to other tech leaders, most folks aspire to those grandiosities.
> If the app starts getting too big for its britches, probably the best thing to do is make it 2 apps
That's the argument in favor of SOA, isn't it?