That’s not my experience. In fact I’d say fat events add coupling because they create an invisible (from the emitter) dependency on the event body, which becomes ossified.
So I’d say the opposite: thin events reduce coupling. Sure, the receiver might call an API and that creates coupling with the API. But receivers are also free to call or not call any other API they want. What if they don’t care about the body of the object?
So I’m on team thin. Every time I’ve been tempted by the other team, I’ve regretted it. It’s also in my experience a lot more difficult to version events than it is to version APIs, so reducing their surface area also solves other problems.
> thin events reduce coupling. Sure, the receiver might call an API and that creates coupling with the API.
You make a statement in the first sentence, and in the next sentence produce evidence ... that the statement is wrong. And, YMMV.
It is my experience that thin events add coupling. If service B receives an event, and wants to process it ASAP (i.e. near real time) and so calls back over http to Service A for the details, then
a) there is additional latency for a http call. And time variance - Even if the average latency of a http request round-trip is fine, the P99 might be bad.
b) You're asking for occasional "eventual consistency" trouble when A's state lags or has moved on ahead of the event
c) Worst of all: When service A is down or unreachable, Service B is unable to do work: Service B uptime must be <= Service A uptime. You have coupled their reliability, and if Service B is identified as mission-critical, then you have the choice of either making Service A equally critical, or decoupling them e.g. with "fat events".
I don't believe that it's accurate to say "receivers are also free to call or not call..." it's not choosing a flavor of ice-cream, you do the calls that the work at hand _needs_.
If you find that you never need to call back to service A then yes, "thin events" would suit your case better. That has not been my experience.
It's fair that event data format versioning is a lot of work with fat events - nothing is without downside. But in your case, do you have "dependency on the event body" ? All of it? If a thin event is all that you need, then you depend on a couple of ids in the event body, and not the rest. Json reading is very forgiving of added / removed fields, you can ignore the parts of a fat event that you don't care about.
I also disagree with the article - thin events don't always result in more coupling, and I'll add that thin events can remove temporal or state coupling as illustrated below. However, the caveat is: as with many things I think choosing one team or the other has nuance and depends on the specific scenario.
An example: I'm using thin events in a master data application integration scenario to send a 'sync this record' type of command message into a queue. The message body does not have the record details, only the basic information to uniquely identify the record. It also doesn't identify the type of change except for a flag to identify deletes. The 'sync' message is generalized to work for all entities and systems, so routing, logging, and other functions preceding the mapping and target operation have no coupling to any system or entity and can expect a fixed message format that will probably never change. Thus versioning isn't a concern.
Choosing team 'thin event' does result in an extra read of the target system, but that is a feature for this scenario and what I want to enforce. I can't assume a target system is in any particular state, and the operation to be performed will be determined from the target system at whatever point in time a message is processed, which could be more than once. If the message ended up in a dead letter queue, it can be reprocessed later without issue. If one production system's data is cloned down to a lower environment, the integrations continue to work even if the source and target environment data is mismatched. No state is stored or depended upon from either system and the design is idempotent (ignoring a target system's business rules that may constrain valid operations over time).
In contrast, other scenarios may benefit from or require a fat event. I've never used event sourcing, but as others mention, if current state can be built from all previous events 'rolled forward' or 'replayed', then each event must be a stand-alone immutable record with all information - thin events cannot be used. Or, if a scenario requires high performance we might need to use a fat event to eliminate the extra read, and then compensate for the other consequences that arise.
assume the data format changes, it would change in the called api as well. as long as the fat event sends data that it's in the same format that the api would return, you'd have the same level of coupling.
I think fat vs thin is more about how much other services the event have to travel, because thin event would multiply reads by a fair factor, with the tradeoff being the performance hit for the queue system to store and ship large events
When I've seen this fat event pattern it's been because different services' responsibilities were not fully separated. And that's tight coupling. Fat events imply tight coupling.
The "thin" pattern described in the article goes like this:
1) service FOO gets an event
2) FOO then has to query BAR (and maybe BAZ and QUUX) to determine the overall state of everything to determine what to do next
And #2 means all of that is kind of "thin" is tightly coupled, too.
I've also personally seen thin events that are not the article's thin strawman.
I sometimes wonder if people understand coupling or design.
When the "state" is large, or changes often, obviously you can't send full state every time - that would be too much for end-nodes to process on every event. Both cpu - deserialization, and bandwidth. Delta is the answer.
Delta though is hard, since there always is an inherent race between getting the first full snapshot, and subscribing to updates.
On the other hand doing delta is hard. Therefore, for simple small updated not-often things, fat events carrying all state might be okay.
There is a linear tradeoff on the "data delivery" component:
- worse latency saves cpu and bandwidth (think: batching updates)
- better latency burns more cpu and bandwidth
Finally, the receiver system always requires some domain specific API. In some cases passing delta to application is fine, in some cases passing a full object is better. For example, sometimes you can save a re-draw by just updating some value, in other cases the receiver will need to redraw everything so changing the full object is totally fine.
I would like to see a pub/sub messaging system that solves these issues. That you can "publish" and object, select latency goal, and "subscribe" to the event on the receiver and allow the system to choose the correct delivery method. For example, the system might choose pull vs push, or appropriate delta algorithm. As a programmer, I really just want to get access to the "synchronized" object on multiple systems.
You send the entire state of the entire object that changed. Irrelevant fields and all.
This makes business logic and migrations easier in dependent services. You can easily roll back to earlier points in time without diffing objects to determine what state changed. You don't have to replay an entire history of events to repopulate caches and databases. You can even send "synthetic" events to reset the state of everything that is listening from a central point of control.
I've dealt with all three types of system, and this is by far the easiest one to work with.
>Delta though is hard, since there always is an inherent race between getting the first full snapshot, and subscribing to updates.
Since the deltas include a version identifier for what they should be applied on top of, then you should always be able to safely start by requesting the deltas, then ask for the object. Buffer the deltas till your full copy is received, then discard deltas for previous versions until the stream applies to yours, applying them thereafter to keep it up to date.
This omits the issues with "thin events" - it may be fine most of the time, but as it usually involves a "get more details" call over http or of some other kind, it has more moving parts, is therefor more prone to failures and slowdowns due to the extra coupling. This can kick in when load goes up or some other issue affects reliability, and cause cascading failure.
I‘d pick neither and just let the system in possession of data send with the event only the part of data it owns (i.e. something in between fat and thin). Saves API call back, the body doesn’t have to be fully deserialized, so no format coupling, the rest can be fetched from other services on demand (coherent state though is not guaranteed, but that’s usually not critical with well designed bounded contexts).
I'm a fan of fat events, and let the receiver decide if they want to trust the event or not, or go ahead and make a call to the service to get the data.
for example:
if one receiver wants to know if you have read a book, then there is no reason to make a call to the service.
but if a service wants to know the last book you read, and doesn't trust the events to be in order, then it would make sense to just call the service.
> if a service wants to know the last book you read, and doesn't trust the events to be in order, then it would make sense to just call the service.
It would make more sense to me if the events had an increasing sequence number, version number or accurate timestamp, so that if I record that "'sithlord' last read 'The Godfather' at event '123456'" I can record that, and ignore any event related to "sithlord last read" with event < 123456.
This is not a new problem, there are existing solutions to it.
As always, it depends. Yay engineering and trade-offs.
Hey just remember: both is always an option if you're consumers disagree. Thin stream from the consumers who don't trust the fat data, fat stream for the event log and other consumers that prefer it.
Fat events once overloaded our message broker with OOM under high load and the broker's default behavior was to block all publishers until the queue was emptied (to release memory) - downtime as a result. Another issue was that under high load, if the event queue was too large, handlers would end up processing very stale data resulting in all kinds of broken behavior (from the point of view of the user).
Thin events resulted in DDoS of our service a few times because handlers would call our APIs too frequently to retrieve object state (which was partially mitigated by having separate machines serve incoming traffic and process events).
(A trick we used which worked for both fat and thin events was to add versioning to objects to avoid unnecessary processing).
We also used delta events as well but they had same issues as thin events because handlers usually have to retrieve full object state anyway to have meaningful processing (not always, depends on business logic and the architecture).
There are so many ways to shoot yourself in the foot with all three approaches and I still hesitate a lot when choosing what kind of events to use for the next project.
For me, this depends on the semantics of the system. Is the sender commanding the receiver to carry out the rest of the process, or is the sender broadcasting information to a dynamic set of interested parties? In other words, are you building a pipeline or a pub-sub?
If the former, there is inherently tight coupling between sender and receiver, and the sender should send all necessary context to simplify the system design.
If the latter, then we talking about a decoupled system, where the sender cannot make assumptions about what info the receiver does or doesn't need to take further action. A thin event is called for to keep the contract simple.
One of my frustrations with the event-driven trend is that people don't always seem to think through what they're designing. It's easy to end up with a much more complex system than a transactional architecture.
Generally, I favor modeling as much of my system as possible as pipelines, and use pub-subs sparingly, as places where you have fan out to parallel pipelines.
Raw events are like GOTOs. They are extremely powerful, but also very difficult to reason about.
Thin events have the benefit of easy retry/resend logic. Depending on your message queue solution you might need to sometimes resend events.
If the event is 'user account changed', receiving it a few times too many causes only performance issues, but not correctness problems.
Sometimes this is the better tradeoff.
It is easier to send an event 'user account changed' than to analyze in detail what exactly changed, which also allows you to decouple the event logic from everything else.
Of course not every system benefits from such solutions, but sometimes simplicity wins.
If you send to multiple receivers, some of the messages may not make it through. Then you're in the position of deciding whether 'the message was sent' or not.
Because the very real downsides of thin events are not simple or obvious. It may be fine most of the time, but as it usually involves a "get more details" call over http or of some other kind, it has more moving parts, is therefor more prone to failures and slowdowns due to the extra coupling. This can kick in when load goes up or some other issue affects reliability, and cause cascading failure.
It seems rare that every operation in a business is atomic across a single aggregate, so I’ve always been wary of events that are pure CRUD, whatever you choose to call them.
I agree in principle, but I also to need action commands when crossing boundaries in and out of the event system (e.g. to the user, or to an external provider.) Of course I still phrase them in the past tense:
doctor_eval|3 years ago
That’s not my experience. In fact I’d say fat events add coupling because they create an invisible (from the emitter) dependency on the event body, which becomes ossified.
So I’d say the opposite: thin events reduce coupling. Sure, the receiver might call an API and that creates coupling with the API. But receivers are also free to call or not call any other API they want. What if they don’t care about the body of the object?
So I’m on team thin. Every time I’ve been tempted by the other team, I’ve regretted it. It’s also in my experience a lot more difficult to version events than it is to version APIs, so reducing their surface area also solves other problems.
SideburnsOfDoom|3 years ago
You make a statement in the first sentence, and in the next sentence produce evidence ... that the statement is wrong. And, YMMV.
It is my experience that thin events add coupling. If service B receives an event, and wants to process it ASAP (i.e. near real time) and so calls back over http to Service A for the details, then
a) there is additional latency for a http call. And time variance - Even if the average latency of a http request round-trip is fine, the P99 might be bad.
b) You're asking for occasional "eventual consistency" trouble when A's state lags or has moved on ahead of the event
c) Worst of all: When service A is down or unreachable, Service B is unable to do work: Service B uptime must be <= Service A uptime. You have coupled their reliability, and if Service B is identified as mission-critical, then you have the choice of either making Service A equally critical, or decoupling them e.g. with "fat events".
I don't believe that it's accurate to say "receivers are also free to call or not call..." it's not choosing a flavor of ice-cream, you do the calls that the work at hand _needs_.
If you find that you never need to call back to service A then yes, "thin events" would suit your case better. That has not been my experience.
It's fair that event data format versioning is a lot of work with fat events - nothing is without downside. But in your case, do you have "dependency on the event body" ? All of it? If a thin event is all that you need, then you depend on a couple of ids in the event body, and not the rest. Json reading is very forgiving of added / removed fields, you can ignore the parts of a fat event that you don't care about.
paphillips|3 years ago
An example: I'm using thin events in a master data application integration scenario to send a 'sync this record' type of command message into a queue. The message body does not have the record details, only the basic information to uniquely identify the record. It also doesn't identify the type of change except for a flag to identify deletes. The 'sync' message is generalized to work for all entities and systems, so routing, logging, and other functions preceding the mapping and target operation have no coupling to any system or entity and can expect a fixed message format that will probably never change. Thus versioning isn't a concern.
Choosing team 'thin event' does result in an extra read of the target system, but that is a feature for this scenario and what I want to enforce. I can't assume a target system is in any particular state, and the operation to be performed will be determined from the target system at whatever point in time a message is processed, which could be more than once. If the message ended up in a dead letter queue, it can be reprocessed later without issue. If one production system's data is cloned down to a lower environment, the integrations continue to work even if the source and target environment data is mismatched. No state is stored or depended upon from either system and the design is idempotent (ignoring a target system's business rules that may constrain valid operations over time).
In contrast, other scenarios may benefit from or require a fat event. I've never used event sourcing, but as others mention, if current state can be built from all previous events 'rolled forward' or 'replayed', then each event must be a stand-alone immutable record with all information - thin events cannot be used. Or, if a scenario requires high performance we might need to use a fat event to eliminate the extra read, and then compensate for the other consequences that arise.
avereveard|3 years ago
I think fat vs thin is more about how much other services the event have to travel, because thin event would multiply reads by a fair factor, with the tradeoff being the performance hit for the queue system to store and ship large events
drewcoo|3 years ago
When I've seen this fat event pattern it's been because different services' responsibilities were not fully separated. And that's tight coupling. Fat events imply tight coupling.
The "thin" pattern described in the article goes like this:
1) service FOO gets an event
2) FOO then has to query BAR (and maybe BAZ and QUUX) to determine the overall state of everything to determine what to do next
And #2 means all of that is kind of "thin" is tightly coupled, too.
I've also personally seen thin events that are not the article's thin strawman.
I sometimes wonder if people understand coupling or design.
majke|3 years ago
When the "state" is large, or changes often, obviously you can't send full state every time - that would be too much for end-nodes to process on every event. Both cpu - deserialization, and bandwidth. Delta is the answer.
Delta though is hard, since there always is an inherent race between getting the first full snapshot, and subscribing to updates.
On the other hand doing delta is hard. Therefore, for simple small updated not-often things, fat events carrying all state might be okay.
There is a linear tradeoff on the "data delivery" component:
- worse latency saves cpu and bandwidth (think: batching updates)
- better latency burns more cpu and bandwidth
Finally, the receiver system always requires some domain specific API. In some cases passing delta to application is fine, in some cases passing a full object is better. For example, sometimes you can save a re-draw by just updating some value, in other cases the receiver will need to redraw everything so changing the full object is totally fine.
I would like to see a pub/sub messaging system that solves these issues. That you can "publish" and object, select latency goal, and "subscribe" to the event on the receiver and allow the system to choose the correct delivery method. For example, the system might choose pull vs push, or appropriate delta algorithm. As a programmer, I really just want to get access to the "synchronized" object on multiple systems.
echelon|3 years ago
- Entire Object.
You send the entire state of the entire object that changed. Irrelevant fields and all.
This makes business logic and migrations easier in dependent services. You can easily roll back to earlier points in time without diffing objects to determine what state changed. You don't have to replay an entire history of events to repopulate caches and databases. You can even send "synthetic" events to reset the state of everything that is listening from a central point of control.
I've dealt with all three types of system, and this is by far the easiest one to work with.
knome|3 years ago
Since the deltas include a version identifier for what they should be applied on top of, then you should always be able to safely start by requesting the deltas, then ask for the object. Buffer the deltas till your full copy is received, then discard deltas for previous versions until the stream applies to yours, applying them thereafter to keep it up to date.
SideburnsOfDoom|3 years ago
ivan_gammel|3 years ago
SideburnsOfDoom|3 years ago
Is that really different from a fat event?
sithlord|3 years ago
for example:
if one receiver wants to know if you have read a book, then there is no reason to make a call to the service.
but if a service wants to know the last book you read, and doesn't trust the events to be in order, then it would make sense to just call the service.
SideburnsOfDoom|3 years ago
It would make more sense to me if the events had an increasing sequence number, version number or accurate timestamp, so that if I record that "'sithlord' last read 'The Godfather' at event '123456'" I can record that, and ignore any event related to "sithlord last read" with event < 123456.
This is not a new problem, there are existing solutions to it.
mabbo|3 years ago
Hey just remember: both is always an option if you're consumers disagree. Thin stream from the consumers who don't trust the fat data, fat stream for the event log and other consumers that prefer it.
kgeist|3 years ago
Thin events resulted in DDoS of our service a few times because handlers would call our APIs too frequently to retrieve object state (which was partially mitigated by having separate machines serve incoming traffic and process events).
(A trick we used which worked for both fat and thin events was to add versioning to objects to avoid unnecessary processing).
We also used delta events as well but they had same issues as thin events because handlers usually have to retrieve full object state anyway to have meaningful processing (not always, depends on business logic and the architecture).
There are so many ways to shoot yourself in the foot with all three approaches and I still hesitate a lot when choosing what kind of events to use for the next project.
ulam2|3 years ago
acjohnson55|3 years ago
If the former, there is inherently tight coupling between sender and receiver, and the sender should send all necessary context to simplify the system design.
If the latter, then we talking about a decoupled system, where the sender cannot make assumptions about what info the receiver does or doesn't need to take further action. A thin event is called for to keep the contract simple.
One of my frustrations with the event-driven trend is that people don't always seem to think through what they're designing. It's easy to end up with a much more complex system than a transactional architecture.
Generally, I favor modeling as much of my system as possible as pipelines, and use pub-subs sparingly, as places where you have fan out to parallel pipelines.
Raw events are like GOTOs. They are extremely powerful, but also very difficult to reason about.
aerffrea|3 years ago
It is easier to send an event 'user account changed' than to analyze in detail what exactly changed, which also allows you to decouple the event logic from everything else.
Of course not every system benefits from such solutions, but sometimes simplicity wins.
unknown|3 years ago
[deleted]
didip|3 years ago
Then the payload is guaranteed to be small but still able to handle complex operations.
mrkeen|3 years ago
andsbf|3 years ago
SideburnsOfDoom|3 years ago
eurasiantiger|3 years ago
thom|3 years ago
nfgivivu|3 years ago
Nothing is worse than an event driven system polluted with action commands.
mrkeen|3 years ago