sbellware's comments

As an entrepreneur and engineer and programmer having spent 30 years in software without washing out and still accelerating, I'm shocked every time I encounter the well-conditioned presumption that work cells in knowledge work should be decomposed to the single point of absolute irreducibility, which is inevitably the 1-person/1-work-item presumption. It implies the assertion that parallelization _should_ be maximized to the ultimate, which implies that the nature of the work is largely just implementation details and typing out code on a keyboard. But that's not where delays and inefficiencies are in software. And decomposing work cells to the point of total parallelization only exacerbates the factors that do indeed worsen the delays and inefficiencies.

Software development has been so underperforming for so long that we presume that the natural, unavoidable state of software development is what the average team experiences today. That's only true of average teams with average productivity and average per-person yield. It's true in the middle of the productivity/performance bell curve, but not to the right of it.

The 1-person/1-work-item division of labor sub-optimizes the momentum transfer that high-performance teams harness to get the high-performance results that they're after. Doing so is predicated on a number of factors, but principally it requires a vigorous attention to structural design fundamentals, rigorous norms, the ease of provability of changes and additions made to the design and implementation, and a solid grasp of organizational mechanics and process mechanics with a fluency in the effects of batching and queueing in systems-of-work in knowledge work and product development.

In my work, and in the work in our organization, the 1-person/1-work-item system is the exception and collective work is the norm. That's because of breakthrough levels of productivity that can be unlocked at the upper echelons of quality. In fact, at that level, the terms "productivity" and "quality" don't have any meaningful difference.

In the middle of the software development curve, there is already such middling productivity/quality that any particular strategy is unlikely to have much observable effect at all on outcomes. It's not until quality is increased to a point well over the present average that the effects of the numerous individual virtuous systems amplifying each other in the overall production system that the sub-optimization of the 1-person/1-work-item division of labor is obvious.

At that level, management of software development operates like an accounting system that only records gains and disregards losses from the final bottom line. Every iota of spent effort is counted as a gain without regard for whether the result of the work creates loss-free assets or not.

The field of software development has largely lost track of structural design fundamentals, both in the software itself and the design of the processes and procedures wherein the software is built. The vast majority of software developers operating today don't know what a mistake is in software development. They don't even recognize the design mistakes inherent in the tools they choose, and they tend to live in software ghettos where productivity poverty is the norm. They've got good slum survival street smarts, but have never been outside of the slum, so they couldn't orient themselves toward a healthier approach and healthier work life if they wanted to. They don't recognize the mistakes that are painfully obvious when seen through the lens of design fundamentals, so they don't know what to avoid. Software developers at large don't know where the "true north" heading is.

To me, it's this current state of ghettoized software development that's the scam. It's the productivity balance sheet that fails to recognize losses and setbacks and that books "technical debt" as an asset that is the real scam in software work and software work management. But again, the vast majority of software workers and managers have never known any better.

So, yeah... putting advanced, interconnected systems of work in the hands of teams that labor under the conditions that can't benefit of those systems of work is obviously ill-advised. But that doesn't mean that those teams can't induce productivity breakthroughs and realizations by practicing the techniques, even if in an uncoordinated and indefinite way. Though it will look like inefficiency to a manager who also has never had the opportunity to work outside the limits of software poverty.

For a team to be able to go beyond single-person work cell division of labor and parallelization, it'd have an enormous amount of historically-cumulative deficits to overcome. And a team that's operating like this is operating with such relatively-low throughput that they're likely already to behind on their deliverables even before they get started on work that it would be impractical to suggest taking a more deliberate approach. In short, this kind of team is always operating under what are effectively emergency measures. That team's principal management system is Crisis Management. And when every work item is managed as a matter of crisis management, there's no point in hoping for much more than mere survival and coping mechanisms.

I wouldn't say that it's necessarily a "scam" to fail to manage the endless emergencies and crisis intentionally, but it should raise some eyebrows for a team to be so far behind the eight ball to be experimenting with systems of work that aren't supported by the counter-balancing systems of work that interact to create emergent productivity. But then again, if they don't start some time, they'll never really get ahead of the software poverty line.

The real scam in software development is "technical debt". The normalized predatory lending and borrowing that the term "technical debt" is a proxy for is the real racket in software development. Being a "post technical debt" team is something that few developers who started their careers since the term was coined can even imagine. It's like a mythical Lost City of Atlantis to developers who've never known any difference. The technical debt that keeps software works and software workers in technical poverty is the issue that should be hauled out into the cold light of day and exercised like a malevolent demon.

But that can't happen until software developers, designers, managers, and operators invest the time to come to an understanding of what structural design decisions makes software work harder and increases the level of poverty that developers live in, and keeps increasing it until the developers themselves are utterly desensitized to their own unfortunate circumstances.

But if they can make it out of the slum, getting past the grasping at 1-person/1-work-item management is part of it. That person will inevitably realize that it's an entirely arbitrary structure born out of the desperation of a project that embraced its own decline in the very first moments of its life. It's a work system that is an effect of the same conditions that make software work harder to do than necessary. At that point, every developer and every manager will have an intimate grasp of the difference between efficiency and productivity, and will know not to forsake long-term productivity for mere short-term efficiencies, and managers will know better than to manage for utilization rather than continuity.

But it can't be readily done in a team that's already implemented itself into a technical deficit for not having known what to avoid in the first place.

The scam is the low-yield spend of most software efforts. "Cooking the books" is today's software development's stock and trade. In a very real sense, software development is largely operated as negligence at best, and fraud at worst. And a good deal of the negligence is the failing to recognize that the sub-optimizations that we submit ourselves to every day are self-inflicted.

The most heartbreaking thing about the whole sordid software affair is that to solutions to the problems have been well-known for decades in industries that have more maturity than software, but software's industrial immaturity is the very thing that keeps it from seeing the need to make a study of industry and production systems.

Taylorism is the sickness that doomed so many industries that came before software, and it remains the mindset that continues to suppress software work today. And nothing will change until developers start to really, really make an effort to understand the subtle reasons why things are the way they are, and to start working for a life beyond the self-inflicted poverty of the ghetto and the slum. But generational poverty is a much more pernicious and endemic issue in software work than the appearance of the petty crimes and scams that impoverished people give into just to make their lives a little more survivable.

I've always found it odd (at least since 2001-ish) that developers see the 1-work-for-1-person as some kind of desirable norm, rather than some arbitrary fixation that has been burned into them since the start of their careers.

It's wishful thinking at best to expect effectiveness and productivity come from the isolation of developers to their own isolated work items. It's hopes and dreams of promoted-beyond-their-competency managers to parallelize the workforce to maximum capacity, rather than to focus on the ability to make progress without slowing down, and then optimize from that baseline.

It's odd to me when someone has to call out "pairing", rather than just assume that work items are taken on by work cells and work groups, and that the more important objective is continuity of knowledge and understanding amongst the workers so that momentum can always be transferred immediately without having to reset and relearn.

The 1-work-for-1-person ethos is so obviously counterproductive that I struggle to understand how it's still so prevalent. Except that human history is rife with popular mythology, and humanity's efforts have been undermined by it for as long as there have been humans.

It speaks to the dark ages period of software that we live in that even the production systems in software work are so antiquated that they still try to grasp at the most implausible straws, not to mention ones that have already been debunked and disproven by industry at large.

It speaks to the reality that software development largely still operates as a craft rather than an industry. Software developers barely pay attention to industry as a body of knowledge, and the sorry state of both software products and software jobs reflects it.

When the day comes when "pairing" never even has to be stated, software work will have entered its modern age of industry. It should be presumed that the processes of momentum transfer between knowledge workers is the singular element that dictates whether a team can continue to build momentum in perpetuity, or sputters out prematurely early in a project's lifetime.

We always have to mention "pairing" when talking to people from that dark ages, and those people will never have a good frame of reference for it due to seeing it through the lens of a more primitive grasp of industry and the body of knowledge that surrounds it. It's almost impossible to explain a high-functioning production system to anyone who has made no time in their professional lives to become as qualified in industrial theory, organizational theory, and production systems as they have to learn the latest convoluted front-end toolkit.

We're like cavemen working with tools gifted to us by some advanced civilization. We've been trained to use high-tech, but we barely understand the systems of work that produce it, and we don't allow ourselves to benefit from the things we can learn from more advanced and more mature fields.

But then, as long as the software tooling available to us is a bad is the things that we make with it, we'll never have the buffer of spare time needed to make the study. And down and down we go.

Erl and Yourdon's books are definitely still relevant, but they have to be put into historical context and historical perspective at this point in order to extract the enduring value that they have.

I wouldn't literally build a SOA system the way that it was done back in the mid-2000s, so I know what to disregard in a legacy SOA book, and what to give attention to. Same for an OO book from the 90s. But without having that sense of history, and a fluency in fundamentals and principles, the necessary background that puts these books into context and makes them useful today would simply be missing.

It's not uncommon for developers to pick up a "classic" software book and come away with all of the wrong ideas.

Pull requests are a relatively recent addition to software development. What did you do before pull requests? If you haven't been around for that long, and have had a career that always had pull requests, then pull requests seem like an inevitable and irreducible inherent feature of software work. But they're not inherent to software work.

I'm going to generalize "pull request" into other terminology so that I can describe the deeper issue:

A pull request is an asynchronous handoff, and asynchronous handoffs are a known sub-optimizer of production processes of all kinds, not just software. It causes batching and queueing, and the harmful effects of those on processes are also well-understood.

Why would I want to introduce asynchronous handoffs into a process that we've worked so hard to optimize at every level and in every turn?

Our process is what it was before GitHub popularized pull requests, and popularized workflows that increased the engagement with their digital product.

The problem is that developers aren't asking whether pull requests are even a good thing to build a development process around. Given the detrimental effects of instituting asynchrony as a desirable target, I find surprising that software workers have gone all-in on something that is an obvious production system anti-pattern.

But to a degree, I'm also not surprised. There's a prominence of a certain kind of psychology in software development that wants to work in isolation without having to be bothered by things like realtime communication and coordination. It's seen as a kind of privilege in that cohort of software developers, but it's one of the most harmful things that can be done to software work (or any work where more than one person is involved).

Software developers tend to be antisocial, and there are enough antisocial developers gathered in this industry that they can enable and reinforce the idea that being antisocial is desirable.

We have a zero-tolerance posture when it comes to antisocial tendencies amongst developers. Our processes are all realtime and synchronous. The level of productivity that we're after requires zero-tolerance posture with regards to batching and queueing and handoffs. We collapse the whole rotten mess of software development down to the fundamentals of production systems and reap many multiples of productivity relative to industry norms specifically by harnessing the momentum that comes from always moving forward, rather than contending with the pervasive start-stop interrupted processes that are common in software development due to failure to recognize the effects of batch handoffs.

The specific details of the work flows aren't as important to disregarding pull requests as the character and behavior of the people in the process. We're not antisocial, and we don't accept antisocial behavior in the team. We work very closely with each other in tightly coordinated and highly-communicative ways in everything we do. At any given time, a very small number of active work items are being worked by a single developer.

In the end, we know more about production systems itself as a field of study and body of knowledge than the average team of devs with 5-to-10-years of experience. And we know a lot about structural design. We marry these two concerns into a process that would be quite recognizable to someone who's studies Lean and the work methods that Edwards Deming championed that changed the face of human productivity decades ago - in every field of human endeavor EXCEPT software development.

So, for a team that is irretrievably mired in asynchrony, and operating on a belief system that has never questioned the workflow fundamentals and the detrimental effect that asynchrony has on the upper potential of productivity, I can see the utility of pull requests. But it's not something that I'd willfully volunteer for. I won't trade the luxuries of working in asynchronously isolation for the vastly-improved lifestyle of rejecting the myth that technical debt is inevitable. This is a lesson that American production industries learned when Japanese producers started beating western producers at their own game by exploiting the virtuous relationship of quality and productivity.

The lesson is that productivity and quality are indistinguishable at the the upper limits of high-performance work and work teams. The level of quality that can generate multiples of end-to-end productivity of an entire process and production system doesn't survive in the presence of a tolerance of asynchronous handoffs.

We don't use pull requests because we want to have a more higher quality of software development life, and we can't have that if we indulge our endemic sense of entitlement to working in the dark all by ourselves.

There's an old saying that was oft-repeated in the days before Scrum executed its ethnic cleansing campaign on XP: If code reviews are good, do them constantly and continuously.

We don't do pull requests because every moment of our work is a code review simultaneously with code production. When you learn how to put together an entire production system with these fundamentals as the non-negotiable parts of the work, the productivity that will emerge will be revolutionary.

Most software teams can't imagine how good and how easy software work can be when the target is the upper echelons of quality.

That said, software work has been removed from this body of knowledge for so long (largely due to the rapid increase in the software development labor pool that started around 2010) that software development culture at present at large no longer has possesses the knowledge of what mistakes are in software work.

Until software developers gain a fluency in software structural design, no amount of fiddling with team workflows will result in anything my negligible, middling improvements.

Right. We don't name the methodology. If we did, people who claim to be "doing it" by mentioning the name of the thing. It's the individual practices that are talked about, and that's on-purpose. We can't afford to have the system of practices be obscured by the kind of grasping at status that can come from being able to claim something by name. It's a means of protecting the methodology itself from decay, and from protecting the people on the team from the shallowness that inevitably making something "a thing".

Some additional context on this:

- There are additional organizational and process mechanics in-play that are out-of-scope for just a write-up on commit messages

- The team is made up of around 15 people supporting around 400 repos and about 8 live products

- The team DOES NOT use pull requests. Contributors are expected to be socially capable and fully engaged in the inter-personal, realtime communication that supports high-performance, low-handoff work.

- The ticketing system is NOT the primary means of communication between team members. It's merely a catalog of work items with only high-level supporting notes. A work item, as the old saying goes, is a "placeholder for a conversation", rather than a work order that can be executed upon by a contributor operating in isolation.

- The original author is describing an exceptional software product organization. He's not debating the merits of various subjective perspectives on what kind of personal flair might drive the content of commit messages. Instead, he's describing one small piece of a methodology that explicitly discourages personal flair and explicitly encourages (and provides supports for) direct communication, and rigorous attention to the team's well-considered, proven, and communicated norms.

- Contributors on the team are expected to maintain their own work logs in the spirit of engineer's logs from engineering industries. If the "why" of a change is important, it will be recorded in the work log. Senior technical leaders (at a minimum) keep up with the individual work logs looking for hazards and maintaining an understanding of the learning that's taking place in the team.

- Contributors on the team are expected to maintain equipment logs for the components that they're making changes to. This isn't the same thing as a SCM log, as an SCM log is for the SCM, not the product under development.

- The reason that the "subject-first" approach is used is to get contributors to talk specifically to the impacts made to the software itself, rather than talk about themselves and their labors. It's about training developers (and designers, and operators, etc) to be more objective about the work they do, and to communicate it in an objective way. Mentions of "changed", "fixed", etc, are records from the perspective of the worker, not the product. When scanning the list of changes made to a work product, the team members are primarily interested in the impacts to that piece of equipment first and foremost. The work product change logs are voiced in terms of the work product, and not the worker. Again, the reflections, learning, questions, observations, etc, of the humans involved in the work go into the human's own logs, and are supported by the person-to-person processes that are beyond the scope of the article on commit messages.

- These 15 people with their 400 repos and 8 live products was winning industry awards within a couple months of go-live within the first year of the team's charter. There's nothing average about the team, the product, and the methods in-play. The commit messages are just one detail of the finely-honed process and culture that were intentionally designed, socialized, and painstakingly-supported to achieve its objectives.

- The author of the article is describing one facet of an overall product development system for a high-performance software organization. He's not trying to spark a dialog about the myriad opinions of how commit messages are written across the breadth of software development within processes and cultures that may not go much beyond the cross-referencing of items between ticketing systems with an SCM system, which is arguably a higher-ceremony approach from an artifacts perspective, but also a "bare minimum" from a process and culture perspective.

It's not entirely pertinent, but just in case there's some question as to whether this work is being done in a business of some scale or other at some particular time or other in a business's lifecycle that accounts for how the development system works, and whether it's a special case: the business context of team's work is a $1B multi-national with about 2,000 employees working in the regulatory side of corporate law, finance, and venture capital.

The business context isn't the enabling factor or the deciding factor. The product development system in-effect is the key factor, and it was shaped explicitly for its outcomes irrespective of the business context. The development system has been used at various companies up-and-down the axis of business scale and scope.

The single greatest contributing factor of the team and its work is that they carry very little technical debt, and the methodology required to work this way touches every molecule of process, organization, and culture. It would not be recognizable from the perspective of typical mid-curve software development, and it would not be understood by examining any single process element in isolation.

If the question "are there any case studies where microservices went well" is a valid question, then so must be, "are there any case studies where monolithic architecture went well".

My point being that if and only if we have a track record as an entire field of making decisions based on case studies, AND if case studies have a track record of being objective rather than proffered as a result of a marketing agenda, then the question is ultimately legitimate.

There are more shops by total count that fail with monolithic architecture. That's inevitable just based on the infinitesimal number of projects executed as service architectures rather than monoliths at large in the wild. But still, we carry on with monolithic architectural style as if its outcomes were assured.

It's far easier for the vast majority of developers to build a monolith because it allows development to proceed without having to have any knowledge of or practice with the tricks and traps of distributed systems - an entire body of knowledge that a developer might never get meaningful and practical exposure to for an entire career.

The trouble starts when microservices are attempted by developers who can't imagine that there are entire bodies of software development knowledge that developers aren't presently in possession of.

Microservices is just, as Adrian Cockroft used to say, "Service-Oriented Architecture with bounded contexts".

Web development is absolutely not a preparatory course in service oriented architecture. But the vast majority of web developers who attempt to take on SOA while simultaneously presuming an omniscience in all things software development due to their experiences only with monolithic web development will often fail to build a SOA. They usually end up with something that isn't quite SOA and isn't quite a monolith. And that's where the failures largely come from.

I work exclusively in microservices and SOA, and have since 2015. Before that, I worked principally as a web app developer, and did some work off-and-on in SOA implementations. And before that I spent years becoming oriented to the architecture. I don't make the mistakes that web developers typically do when they presume that web development knowledge is a sufficient prerequisite for working in SOA.

So, it's not a question of whether an architectural style works or doesn't. The majority of failures in microservices can be attributed to ignorance and to the narcissism that is permissive of it.

So, I would ask this question instead: Are there any cases where developer over-confidence and over-simplification went well?

These qualities don't tend to serve any architectural style well.

I've never heard of a well-designed SOA not going well. Every single case of microservice project remediation that I've participated in had as the most significant contributing factor an utter disregard for the body of knowledge that the microservices architectural style is built upon.

There are a lot of things that developers can get away with when doing the kinds of tinkering and wandering that typifies typical web development work. But those things don't work once we cross the line into SOA. And unfortunately, the incessant chasing after trivial resumé candy hasn't prepared the average developer for the rigorous mindset needed for SOA work.

As "microservices" became to next fad for perennial fad chasers of the software development world, they finally encountered a kind of work that they could not get away with by faking it. And so, we see a lot of failures. But the vast majority of the failures are personal failures and character failures, rather than failures of an architectural style.

The fat part of the developer bell curve was simply overreaching when it presumed to try to get away with building service architectures with the same level of disinterest in architecture and process that we can get away with in typical web development. Like a kid with copious experience building kites presuming to strap themselves to a hang glider and just "going for it". The outcomes are mostly predictable.

In the end, if a little time is invested in learning the fundamentals that have so far been eschewed for the sake of expediency, anyone can succeed with SOA and microservices. It's not that the realities of the architectural style are unlearnable, but it can't be arrived at by the level of tinkering and wandering that we can just get away with in monolithic web development.

What happens when the write of the current position fails?

It's the same problem as presuming that ACK messages in message brokers/queues are guaranteed to not fail.

Since the message transport and other durable resources are rarely able to be enlisted in the same atomic transaction, and since distributed transactions would largely be an antipattern, it would seem that a reactor that records its current position can't be presumed to be an infallible way of ensuring that messages aren't processed more than once.

In the end, it always comes back to ensuring that handlers are idempotent, having something that can be used as a stable and consistent idempotence key, and accepting that the idempotence logic is the responsibility of the handler coders rather than something we can count on generalized infrastructure for.

> most I experimented with would lose data in a catastrophic event and cause inconsistencies

Fair enough. Those are probably message buses or message queues that are ephemeral transports. Since event sourcing is predicated upon permanent storage of events, there's no way to lose events that have already been committed (unless someone actually physically deletes the events).

What happens when the server that holds the thing that holds that state is restarted?

> Pub/sub in Event Sourcing is a bad idea

I find this point surprising. I would say the exact opposite. I would say that pub/sub and event sourcing are two sides of the same coin: events.

> what to do if sub happens after pub

That should only ever be a problem with a non-durable transport that doesn't have serialized writes per topic. Which, admittedly, can be pretty common. But it's not so much an event sourcing or pub/sub issue as much as a choice of message transport issue.

> Concurrency. Ensuring aggregates are essentially single-threaded entities is a must. Having the same aggregate id running in multiple places can cause some really fun bugs. This usually requires a distributed lock of some sort.

Or it requires partitioning the queues and using an optimistic lock when writing (just to be on the safe side).

The only telltale metric of software work that I know is the ratio of time spent making progress to time spent not being able to make progress because of past mistakes made and shortcuts taken.

It's reasonable to consider that the reason that software developers are saddled with such shallow process and tooling metrics is that they themselves don't have any better alternatives to offer.

If we roll over and accept the lowest possible maturity organization and process methodologies like Scrum, then we ultimately deserve what we get.

We have a roll to play, and it ultimately requires stepping up and seeing our jobs as more than mere tool-wielding. Or said otherwise, in the rush to call ourselves "engineers" we've stopped being engineers and stopped even pursuing an understanding of the body of knowledge of engineering (which leads to better grasp of the subject area that comprises process and measurement).

Velocity is a standard we're held to because we didn't step up to provide better standards. And we didn't do that because we didn't believe it was our responsibility to learn about the rest of software development. And in that vacuum, people who know even less than us are left to dictate pseudo science to people who should already be in command of the science.

We have a role to play if we wish to play it. It's not obligatory, but there are real costs to pay when we don't.

There's no requirement or expectation to go above and beyond in an effort to catch up on all the knowledge and understanding we've shed from our wheelhouse. It's totally fine and understandable if we don't. But we also have to accept that nature abhors a vacuum.

The problem here is that an unmitigated monolith with no domain partitioning in its data model can't be transformed into a service architecture by carving off pieces.

For pieces to be able to be carved off, they already have to be autonomous.

What usually happens is that devs without experience in service architectures presume that service architectures are probably just the things they are used to seeing, ie: monolithic entities.

It's usually then that we hear things like "extract the product service". The problem is that product is an entity, and entities are the last thing to build services around. That's how we end up with distributed monoliths, and subsequently failed service architecture projects.

In order for an app to be able to transition to services, the app has to be designed this way from the beginning.

And yes, it's definitely possible to know what the model partitions should be up front. They're very natural divisions. But they can't be arrived at by looking through an entity-centric lens. And unfortunately, forms-over-data apps very rarely provide us with an opportunity to learn about the "other" way to do it.

Indeed. And having been around in both the SOA period and the Microservices period, and having witnessed the transition, I'm comfortable with maintaining the assertion that "Microservices" as a term was specifically introduced to demarcate the period characterized by big-vendor smart message transports and the period characterized by dumb transports we settled on as a reflection of all that we'd learned by trying to rely on messaging "magic".

It's a very similar transition to the shift from EJB to Hibernate (or EJB to Rails) or from SOAP to REST.

In the mean time, a lot of folks who don't have that background and perspective picked up on Microservices and made a lot of presumptions based on a lot of experience with web apps and web APIs. It's this that made "Microservices" a largely meaningless, muddied term. This is ironic because "Microservices" was introduced as a means to disambiguate the many competing meanings of "Service-Oriented Architecture" that had come into existence as many and vendors wanted to be seen as being involved with it without actually having been involved with it.

So, there's two meanings of "Microservices": One that comes from a background of service architectures and one that comes from a background of web development. My background is in both of them, but I hew to the meaning of "Microservices" which is closer to "SOA without big vendor smart pipes of the mid-to-late 2000s" than "The same old HTTP APIs we see the world through as web developers".

That's largely a lost battle now, just as SOA was lost to the competing interests of message tech vendors.

The term "Autonomous Services" is a much more precise and unambiguous in its intent, and conveys more specifically what's intended by "SOA done right", a.k.a.: "Microservices". And even more specifically, "Evented Autonomous Services" does an even better job of conveying the intent and the implications or the architecture than what "Microservices" can do now.

Or as Adrian Cockroft originally put it: "Loosely-coupled service oriented architecture with bounded contexts".

Knowing what the implications of all those words are, it's quite impossible to look at what is commonly asserted about the meaning of "Microservices" in 2019 and see much left of the great value of what was originally intended. Much of it still remains unlearned.

> Are you saying that this is bad? It doesn't seem so.

It's not "bad" per se, but it's not what it seems on the surface.

ACKing a message doesn't mean that the message will not be received more than one time. As long as the smarts for recognizing recycled messages are embedded in the application logic, all will be fine.

Here's a good explanation written about SQS, but applies to all message technologies that work based on ACKs: https://sookocheff.com/post/messaging/dissecting-sqs-fifo-qu...

So, while it's possible to add more consumers, depending on the implementation of the technology's internals for tracking consumer state, you may loose messages or, more typically, receive messages more than once or receive them out of order.

This often happens without the developers' and operators' knowledge. Without foreknowledge of the causes and effects of these things, developers and operators may know there there's some kind of strange intermittent glitch, but don't presume that it's because message processors are processing messages that had already previously been processed (or are not processing messages that had been skipped).

What you can't do is add a new consumer that is interested in processing historical messages beyond the retention window of typical cloud-hosted message transports.

But that's ok because message queues are typically just transports, not message stores, and they serve the needs of moving messages from one place to another. A message store can do that as well, but has different semantics and serves other needs.

So, as long as ACK-based message processors work within the retention window, everything's good. It's when that's not the case that the problems arise. Lost messages and recycled messages are pretty much the only guarantee over the lifetime of a messaging app. As long as that's within tolerances, then it's fine.

Specifically, like all "dumb pipe" technology that came about in the post-SOA age, a message store doesn't use a protocol based on ACKs. Instead, a consumer tracks its own state and does not defer this responsibility to the transport.

And inevitably, if any consumer of any technology wants to guarantee that it never processes a recycled message, this tracking logic has to also be implemented in the application logic of even SQS, RabbitMQ, etc consumers.

There's no such thing as a messaging technology that can guarantee only-once delivery, as you alluded to. There's a good examination of this aspect of messaging and distributed systems here: https://bravenewgeek.com/you-cannot-have-exactly-once-delive...

The best guarantee from message transports that we can hope for amounts to a "maybe". And that's ok. The conditions and countermeasures are well-known. As long as we're not blind-sided and haven't taken "only once" literally, things will be fine.

If we don't realize that application logic is always responsible for making the ultimate decision as to whether to reject a recycled message, we're going to have problems that can be difficult-to-impossible to detect and correct.

Message stores and event stores aren't alternatives to message queues. They're technologies that support architectures that are themselves alternatives to each other.

A message store is really not a comparable pattern to a message queue. It's unlikely that the usages would be similar enough to engender a meaningful comparison. They're not as dissimilar as an apples-to-oranges comparison, but maybe an apples-to-applesauce comparison.

The right store is the one that has the features needed to implement the targeted patterns, has client libraries for most programming languages, can be tuned and scaled, has a large ecosystem, has numerous managed hosting solutions offered by popular cloud providers, and is approachable by the largest segment of the potential audience.

Postgres checks those boxes, as do others.

Technically, though, Postgres has been an "object-relational" database for some time.

In the end, Message DB uses a table as an append-only log, and leverages Postgres indexes, advisory locks, and JSON documents (and indexes) to implement some of the critical messaging features and patterns. It's not using the "relational" aspects of Postgres. There are no relational tables.

Here's the table schema: https://github.com/message-db/message-db/blob/master/databas...

Given the paucity of Postgres features that the message store leverages, it could have been implemented against the raw Postgres storage engine. Had we done that, though, few people could have understood it and been able to specialize it to their purposes using plain old SQL. And any performance improvements induced by skipping Postgres's tabular data abstractions would have been so negligible for the schema in question that it wouldn't have offered much return on the effort.

There are too many "it depends" issues, especially the hardware, and not least the application architecture sitting on top of it.

There's a script in the Message DB codebase that will do some fairly rudimentary read and write performance measures for your Postgres deployment:

https://github.com/message-db/message-db/blob/master/databas...

It's run right on the database, so no network I/O or client implementation in the loop - which is inevitably non-representative of "real world" scenarios.

Or managed Postgres on AWS Aurora, AWS RDS, Google Cloud SQL, Heroku, etc :)

SNS, Kenesis (Kafka), Google Pub/Sub are awesome. Not the same problem/solution fit as an event store, but awesome for the scenarios and architectures they're targeted at.

> The almost-but-not-quite unstated major premise of the whole argument is that you're putting a key piece of smarts into the queue: Tracking whether a message has been processed. One could argue that that is the real antipattern.

Indeed.

Lots of good writing on the tradeoffs, for example: https://sookocheff.com/post/messaging/dissecting-sqs-fifo-qu...

At the risk of beating the point to death, the transition from SOA to Microservices is marked by the transition of smart pipes to dumb pipes, and dumb pipes have no knowledge of whether a message has been processed. In the microservices era, the basic presumption is that the transport has no knowledge of the state of the message processors' progress through a queue.