top | item 43501989

Architecture Patterns with Python

488 points| asicsp | 1 year ago |cosmicpython.com | reply

140 comments

order
[+] DanielVZ|1 year ago|reply
Wow this book is a goldmine for architecture patterns. I love how easy it is to get into a topic and quickly grasp it.

Having said that, from a practical and experience standpoint, using some of these patterns can really spiral out into an increased complexity and performance issues in Python, specially when you use already opinionated frameworks like Django which already uses the ActiveRecord pattern.

I’ve been in companies big and small using Python, both using and ignoring architectural patterns. Turns out all the big ones with strict architectural (n=3) pattern usage, although “clean”, the code is waaaay to complex and unnecessarily slow in tasks that at first glance should had been simple.

Whereas the big companies that didn’t care for these although the code was REALLY ugly in some places (huge if-else files/functions, huge Django models with all business logic implemented in them), I was most productive because although the code was ugly I could read it, understand it, and modify the 1000 lines of if-else statements.

Maybe this says something about me more than the code but I hate to admit I was more productive in the non clean code companies. And don’t get me started on the huge amount of discussions they avoided on what’s clean or not.

[+] Jcampuzano2|1 year ago|reply
I think one of the biggest problems I encounter whenever I hear that a project follows strict architectural patterns essentially boils down to too many obfuscated abstractions that hide what is going on, or force you to jump through too many layers to accomplish tasks.

Many files/functions/classes need to be updated to accomplish even simple tasks because somebody made a decision that you aren't allowed to do X or Y thing without creating N other things.

But in those companies that didn't care about architectural patterns its very likely that while there was more ugly code in certain places, it resulted in code with less indirection and more contained to a single area/unit or the task at hand making it easier for people to jump in and understand. I see so many people who create function after function in file after file to abstract away functionality when I'd honestly rather have a 100 line function or method that I can easily jump around and edit/debug vs many tiny functions all in separate areas.

Not to say having some abstractions are bad but the more I work in this field the more I realize the less abstractions there are, the easier it is to reason about singular units/features in code. I've basically landed on just abstract away the really hard stuff, but stop abstracting out things that simple.

[+] lijok|1 year ago|reply
Strict architectural pattern usage requires understanding the domain, and understanding the patterns. If you have both, navigating the codebase will be intuitive. If you don't, you'll find 1000 LOC functions easier to parse.
[+] directevolve|1 year ago|reply
I found the book's use of modeling how to pilot an alien starship to be a little misleading, because a starship is a highly engineered product that functions in large part as a control mechanism for software. It comes with a clean design model already available for you to discover and copy.

Domain modeling should not be about copying the existing model -- it should be about improving on it using all the advantages software has over the physical and social technologies the new software product is meant to replace. People are smart, and in most projects, there are key aspects of the existing domain model that are excellent abstractions that can and should be part of the new model. It's important to understand what stakeholders are trying to achieve with their current system before attempting to replace it.

But the models used in the business and cultural world are often messy, outdated and unoptimized for code. They rely on a human to interpret the edge cases and underspecified parts. We should treat that as inspiration, not the end goal.

[+] frankc|1 year ago|reply
I love this book but yes, you really need to understand when it makes sense to apply these patterns and when not to. I think of these kinds of architectural patterns like I think of project management. They both add an overhead, and both get a bad rap because if they are used indiscriminately, you will have many cases where the overhead completely dominates any value you get from applying them. However, when used judiciously they are critical to the success of the project.

For example, if I am standing up a straight-forward calendar rest api, I am not going to have a complicated architecture. However, these kinds of patterns, especially an adherence to a ports and adapters architecture, has been critical for me in building trading systems that are easy to switch between simulation and production modes seamlessly. In those cases I am really sure I will need to easily unplug simulators with real trading engines, or historical event feeds with real-time feeds, and its necessary that the business logic have not dual implementations to keep in sync.

[+] zahlman|1 year ago|reply
>I’ve been in companies big and small using Python, both using and ignoring architectural patterns. Turns out all the big ones with strict architectural (n=3) pattern usage, although “clean”, the code is waaaay to complex and unnecessarily slow in tasks that at first glance should had been simple.

The problem with "strict architectural pattern usage" is that people think that a specific implementation, as listed in the reference, is "the pattern".

"The pattern" is the thought process behind what you're doing, and the plan for working with it, and the highest-level design of the API you want to offer to the rest of the code.

A state machine in Python, thanks to functions being objects, can often just be a group of functions that return each other, and an iteration of "f = f(x)". Sometimes people suggest using a Borg pattern in Python rather than a Singleton, but often what you really want is to just use the module. `sys` is making it a singleton for you already. "Dependency injection" is often just a fancy term for passing an argument (possibly another function) to a function. A Flyweight isn't a thing; it's just the technique of interning. The Command pattern described in TFA was half the point of Jack Diederich's famous rant (https://www.youtube.com/watch?v=o9pEzgHorH0); `functools.partial` is your friend.

> Maybe this says something about me more than the code but I hate to admit I was more productive in the non clean code companies.

I think you've come to draw a false dichotomy because you just haven't seen anything better. Short functions don't require complex class hierarchies to exist. They don't require classes to exist at all.

Object-oriented programming is about objects, not classes. If it were about classes, it would be called class-oriented programming.

[+] porridgeraisin|1 year ago|reply
My experience matches this. It's so liberating as well. I find it easier to internalise such code in my head compared to abstraction-soup. As you can imagine, I like golang.
[+] tzcnt|1 year ago|reply
This has been my experience in working with any kind of dogmatic structure or pattern in any language. It seems that the architecture astronauts have missed the point: making the code easier to understand for future developers without context, and provide some certainty that modifications behave as expected.

Here's an example of how things can go off the rails very quickly: Rule 1: Functions should be short (no longer than 50 lines). Rule 2: Public functions should be implemented with an interface (so they can be mocked).

Now as a developer who wants to follow the logic of the program, you have to constantly "go to definition" on function calls on interfaces, then "go to implementation" to find the behavior. This breaks your train of thought / flow state very quickly.

Now let's amp it up to another level of suck: replace the interface with a microservice API (gRPC). Now you have to tab between multiple completely different repos to follow the logic of the program. And when opening a new repo, which has its own architectural layers, you have to browse around just to find the implementation of the function you're looking for.

These aren't strawmen either... I've seen these patterns in place at multiple companies, and at this point I yearn for a 1000 line function with all of the behavior in 1 place.

[+] dkarl|1 year ago|reply
> Turns out all the big ones with strict architectural (n=3) pattern usage, although “clean”, the code is waaaay to complex and unnecessarily slow in tasks that at first glance should had been simple.

My last job had a Python codebase just like this. Lots of patterns, implemented by people who wanted to do things "right," and it was a big slow mess. You can't get away with nearly as much in Python (pre-JIT, anyway) as you can in a natively compiled language or a JVM language. Every layer of indirection gets executed in the interpreter every single time.

What bothers me about this book and other books that are prescriptive about application architecture is that it pushes people towards baking in all the complexity right at the start, regardless of requirements, instead of adding complexity in response to real demands. You end up implementing both the complexity you need now and the complexity you don't need. You implement the complexity you'll need in two years if the product grows, and you place that complexity on the backs of the small team you have now, at the cost of functionality you need to make the product successful.

To me, that's architectural malpractice. Even worse, it affects how the programmers on your team think. They start thinking that it's always a good idea to make code more abstract. Your code gets bloated with ghosts of dreamed-of future functionality, layers that could hypothetically support future needs if those needs emerged. A culture of "more is better" can really take off with junior programmers who are eager to do good work, and they start implementing general frameworks on top of everything they do, making the codebase progressively more complex and harder to work in. And when a need they anticipated emerges in reality, the code they wrote to prepare for it usually turns out to be a liability.

Looking back on the large codebases I've worked with, they all have had areas where demands were simple and very little complexity was needed. The ones where the developers accepted their good luck and left those parts of the codebase simple were the ones that were relatively trouble-free and could evolve to meet new demands. The ones where the developers did things "right" and made every part of the codebase equally complex were overengineered messes that struggled under their own weight.

My preferred definition of architecture is the subset of design decisions that will be costly to change in the future. It follows that a goal of good design is minimizing architecture, avoiding choices that are costly to walk back. In software, the decision to ignore a problem you don't have is very rarely an expensive decision to undo. When a problem arises, it is almost always cheaper and easier to start from scratch than to adapt a solution that was created when the problem existed only in your head. The rare exceptions to this are extremely important, and from the point of view of optics, it always looks smarter and more responsible to have solved a problem incorrectly than not to have solved it at all, but we shouldn't make the mistake of identifying our worth and responsibility solely with those exceptions.

[+] BerislavLopac|1 year ago|reply
Some parts of this book are extremely useful, especially when it's talking about concepts that are more general than Python or any other specific language -- such as event-driven architecture, commands, CQRS etc.

That being said, I have a number issues with other parts of it, and I have seen how dangerous it can be when inexperienced developers take it as a gospel and try to implement everything at once (which is a common problem with any collection of design patterns like this.

For example, repository is a helpful pattern in general; but in many cases, including the examples in the book itself, it is a huge overkill that adds complexity with very little benefit. Even more so as they're using SQLAlchemy, which is a "repository" in its own right (or, more precisely, a relational database abstraction layer with an ORM added on top).

Similarly, service layers and unit of work are useful when you have complex applications that cover multiple complex use cases; but in a system consisting of small services with narrow responsibilities they quickly become overly bloated using this pattern. And don't even get me started with dependency injection in Python.

The essential thing about design patterns is that they're tools like any other, and the developers should understand when to use them, and even more importantly when not to use them. This book has some advice in that direction, but in my opinion it should be more prominent and placed upfront rather at the end of each chapter.

[+] kelafoja|1 year ago|reply
Could you explain how repository pattern is a "huge overkill that adds complexity with very little benefit"? I find it a very light-weight pattern and would recommend to always use it when database access is needed, to clearly separate concerns.

In the end, it's just making sure that all database access for a specific entity all goes through one point (the repository for that entity). Inside the repository, you can do whatever you want (run queries yourself, use ORM, etc).

A lot of the stuff written in the article under the section Repository pattern has very little to do with the pattern, and much more to do with all sorts of Python, Django, and SQLAlchemy details.

[+] ryan-duve|1 year ago|reply
> And don't even get me started with dependency injection in Python.

Could I get you started? Or could you point me to a place to get myself started? I primarily code in Python and I've found dependency injection, by which I mean giving a function all the inputs it needs to calculate via parameters, is a principle worth designing projects around.

[+] wormlord|1 year ago|reply
> I have seen how dangerous it can be when inexperienced developers take it as a gospel and try to implement everything at once

This book explicitly tells you not to do this.

> Similarly, service layers and unit of work are useful when you have complex applications that cover multiple complex use cases; but in a system consisting of small services with narrow responsibilities they quickly become overly bloated using this pattern. And don't even get me started with dependency injection in Python.

I have found service layers and DI really helpful for writing functional programs. I have some complex image-processing scripts in Python that I can use as plug-ins with a distributed image processing service in Celery. Service layer and DI just takes code from:

```python

dependency.do_thing(params)

```

To:

```python

do_thing(dependency, params)

```

Which ends up being a lot more testable. I can run image processing tasks in a live deployment with all of their I/O mocked, or I can run real image processing tasks on a mocked version of Celery. This lets me test all my different functions end-to-end before I ever do a full deploy. Also using the Result type with service layer has helped me propagate relevant error information back to the web client without crashing the program, since the failure modes are all handled in their specific service layer function.

[+] jbs789|1 year ago|reply
This was my takeaway too. It’s interesting to see the patterns. It would be helpful for some guidance upfront around when the situations in which they are most useful to implement. If a pattern is a tool, then steering me towards when it’s used or best avoided would be helpful. I do appreciate that the pros and cons sections get to this point, so perhaps it’s just ordering and emphasis.

That said, having built a small web app to enable a new business, and learning python along the way to get there, this provided me with some ideas for patterns I could implement to simplify things (but others I think I’ll avoid).

[+] SaturnIC|1 year ago|reply
> That being said, I have a number issues with other parts of it, and I have seen how dangerous it can be when inexperienced developers take it as a gospel and try to implement everything at once (which is a common problem with any collection of design patterns like this.

Robert Martin is one of those examples, he did billions in damages by brainwashing inexperienced developers with his gaslighting garbage like "Clean Code".

Software engineering is not a hard science so there is almost never a silver bullet, everything is trade-offs, so people that claim to know the one true way are subcriminal psychopaths or noobs

[+] seveibar|1 year ago|reply
I’m a Typescript dev but this book is one of my favorite architecture books, I reference it all the time. My favorite pattern is the fake unit of work/service patterns for testing, I use this religiously in all my projects for faking (not mocking!!) third party services. It also helped me with dilemmas around naming, eg it recommends naming events in a very domain specific way rather than infrastructure or pattern specific way (eg CART_ITEM_BECAME_UNAVAILABLE is better than USER_NOTIFICATION). Some of these things are obvious but tedious to explain to teammates, so the fact that cosmic python is fully online makes it really easy to link to. Overall, a fantastic and formative resource for me!
[+] serial_dev|1 year ago|reply
I haven't seen this book before, but I noticed that one of the authors, Harry J. W Percival, is the author of the TDD "goat" book.

https://www.obeythetestinggoat.com/pages/book.html

That book is in a similar place in my heart, I barely used Python in my professional life, yet it's a book I often come back to even if I'm using a different language. It's also great that book is available both online and in paper form.

I'll definitely give this book a chance!

[+] DeathArrow|1 year ago|reply
I see Python at a nice glue language.

I grew tired from the forced OOP mindset, where you have to enforce encapsulation and inheritance on everything, where you only have private fields which are set through methods.

I grew tired of SOLID, clean coding, clean architecture, GoF patterns and Uncle Bob.

I grew tired of the Kingdom of Nouns and of FizzBuzz Enterprise Editions.

I now follow imperative or functional flows with least OOP as possible.

In the rare cases I use Python (not because I don't want to, but because I mainly use .NET at work) I want the experience to be free of objects and patterns.

I am not trying to say that this book doesn't have a value. It does. It's useful to learn some patterns. But don't try to fit everything in real life programming. Don't make everything about patterns, objects and SOLID.

[+] exe34|1 year ago|reply
my favourite model is to write as many pure functions as possible, and then as many functions of 1-4 parameters that interact with the outside world, and only then create domain objects to wrap those - it keeps the unrelated complexity out of the domain and then I can also reuse those functions without having to create the entire object that I don't always need.
[+] throw1222212121|1 year ago|reply
Hmm let's see we're going to

- Reimplement SQLAlchemy models (we'll call it a "repository")

- Reimplement SQLAlchemy sessions ("unit of work")

- Add a "service layer" that doesn't even use the models -- we unroll all the model attributes into separate function parameters because that's less coupled somehow

- Scatter everything across a message bus to remove any hope of debugging it

- AND THIS IS JUST FOR WRITES!

- For reads, we have a separate fucking denormalized table that we query using raw SQL. (Seriously, see Chapter 12)

Hey, let's see how much traffic MADE.com serves. 500k total visits from desktop + mobile last month works out to... 12 views per MINUTE.

Gee, I wish my job was cushy enough that I could spend all day writing about "DDD" with my thumb up my ass.

[+] florbo|1 year ago|reply
I've made it through about 75% of the book and have never gotten the sense that they think everything discussed in the book is something you should always do. Each pattern discussed has a summary of pros and cons. While they may be a bit lacking, they clearly articulate the fact that you should be thinking whether or not the pattern matches the application's needs.

I don't think there's many applications that will require everything in the book but there are certaintly many applications that could apply one or more patterns discussed.

[+] globular-toast|1 year ago|reply
OK so show us how to write software for a complex business properly. Oh, I see, it's a throwaway account. This is just drive-by negativity with zero value.
[+] localghost3000|1 year ago|reply
I started writing python professionally a few years ago. Coming from Kotlin and TypeScript, I found the language approachable but I was struggling to build things in an idiomatic fashion that achieved the loose coupling and testability that I was used to. I bought this book after a colleague recommended it and read it cover to cover. It really helped me get my head around ways to manage complexity in non trivial Python codebases. I don’t follow every pattern it recommends, but it opened my eyes to what’s possible and how to apply my experience in other paradigms to Python without it becoming “Java guy does Python”.

I cannot recommend it enough. Worth every penny.

[+] Pandabob|1 year ago|reply
Truly one of the great python programming books. The one thing that I found missing was the lack of static typing in the code, but that was a deliberate decision by the authors.
[+] sevensor|1 year ago|reply
Haven’t read the book, so I don’t know exactly what position they’re taking there, but type checking has done more to improve my Python than any amount of architectural advice. How hard it is to type hint your code is a very good gauge of how hard it will be to understand it later.
[+] zahlman|1 year ago|reply
Some examples use dataclasses, which force type annotations.

Python does not support static typing. Tooling based on type annotations doesn't affect the compilation process (unless you use metaprogramming, like dataclasses do) and cannot force Python to reject the code; it only offers diagnostics.

[+] globular-toast|1 year ago|reply
I have this on my shelf. It's a small volume, similar to K&R, and like that book mine is showing visible signs of wear as I've thumbed through it a lot.
[+] chr1ss_code|1 year ago|reply
This was a great read and summary! About three years ago, I worked in a C#/.NET DDD environment, and now revisiting these concepts in Python really distills the essential parts. As I said, great read — highly recommend it if you're also into this kind of stuff.
[+] jjice|1 year ago|reply
Oh neat, I read the paper back of this book maybe two and a half or three years or so ago. I enjoyed it quite a bit. They do a good job at keeping tests a first class topic and consistently updating them with each addition. Some older architecture books don't treat testing as being as high up in their priorities. I've just found that having tests ready, easy to write, and easy to update, makes the development process more enjoyable for me since it's less manual for running the code to check for issue - tighter feedback look I guess.

I will say that some of the event oriented parts of this book were very interesting, but didn't seem as practical to implement in my current work.

[+] shesstillamodel|1 year ago|reply
Even though most people might think of web architectures when it comes to this book, we used this to design an architecture for an AI that optimises energy efficiency in a manufacturing factory.

Great book!

[+] fastasucan|1 year ago|reply
Is it easy to transpose to other types of architectures, or is it leaning heavily against web development? Thank you for sharing, your project sounds really interesting by the way! :)
[+] eterps|1 year ago|reply
I would have expected the book mentioning something about the concept of DTOs at some point. What could be the reason it doesn't?
[+] throwawaysjskdk|1 year ago|reply
I don’t understand the need for most of the patterns described in this book. Why abstract away SQLalchemy using a Repository when it is already an abstraction over the database? What’s the purpose of the unit of work? To me hand rolling SQL is much more maintainable than this abstraction soup
[+] barrenko|1 year ago|reply
Excellent sequel to the goat book (TDD with Python), that got me to deploy my first real web application.
[+] SaturnIC|1 year ago|reply
TDD is dysfunctional crap pushed by the lying scammer Robert Martin on inexperienced devs
[+] floppydiscen|1 year ago|reply
Great source for understanding how DDD works in a larger context. I love how concrete it is
[+] moi2388|1 year ago|reply
“ If you’re reading this book, we probably don’t need to convince you that Python is great”

I actually do. It’s slow, buggy and not type safe.

Everything good about Python is actually C, namely the good packages. They’re not written in Python, because Python is shit.

[+] slt2021|1 year ago|reply
Great stuff, thank you for sharing