It still boggles my mind that, in an age where K8s is huge and K8s's reconciliation loop pattern is well recognized, Temporal has yet to put "reconciler" or "state-machine" in their documentation or descriptions.
I say this as someone who loves using Temporal and see a big future for them.
Temporal is a managed state-machine. If you're familiar with K8s, you can think of it as a dedicated reconciler. Plain and simple.
Do you want a resource to go from one state to another (i.e. user orders food -> food is delivered) regardless of transient errors and other hiccups? Temporal handles the plumbing of driving your resource to it's desired end state.
Do you have a workflow that has multiples steps? Can anyone of those steps fail for whatever reason and you'll need to figure out how to do retrying/cleaning up? Temporal handles that.
Essentially you can write your business logic without the worry of including the (often) boilerplate retry, cleanup, scheduling, etc logic. If you're familiar with K8s reconciliation and how it retries failed operations again and again until it reaches the goal, it's much like that.
Temporal can do much more than that, but that's its bread and butter and I'm trying to describe it simply so that people can understand. It is a bit of a paradigm shift and has a bit of a learning curve when you first start using it.
ctrl+f "state machine" on this page :) but yeah this is a very good perspective on temporal. we should talk, if you're interested in helping us tell our story better.
i'm actually interested in a longer form "Temporal vs K8s" comparison - can make a case for Temporal atop k8s, or Temporal replacing k8s (already had a couple customers talk to us about this). It's a nuanced story to tell and I'm not sure if it has a conclusion.
(EDIT: links got fixed) For clickable links, remove the leading spaces so it doesn't get formatted as plaintext, and make newlines double to avoid them all being on one line
Love Temporal. Hope you solve the infrastructure and cloud story with this.
The biggest pain point is still deploying and configuring a production environment. I started suspecting it was so awful and poorly documented to push us to using the cloud offering, but then I couldn’t even find a link to sign up for cloud.
We've been using Temporal for the past year and have found it to be an absolute joy.
Being able to model business processes--some of which may take multiple _years_, and require human approval/rejection steps--using procedural go/java/js, and knowing that once that function starts executing, it is basically indestructible (barring the complete destruction of temporal's backend database), is just really cool. Being able to outsource a lot of the complexity involved in using event sourcing to manage distributed transactions frees up our developers to worry about just the high-end business logic.
Congrats to the team! I'm excited to see Temporal gaining some more traction.
Temporal is such a powerful tool, really don't rollout your own job system using pub/sub, just use Temporal.
At first it looks easy to use any pub/sub or DB to create a job system, but there are so many edge cases, I'm not even talking about scaling etc ... just having a solid job system that properly handle retries, error, durability, timeout etc ...
IMO, this is the value of Temporal. You certainly can write your own system—but wouldn't you rather focus on writing code that directly addresses the unique value that you or your company offers?
Temporal is a great system for jobs, workflows, etc. (we use it at my current job), but it isn’t really a replacement for PubSub per se, as there is only one consumer per task
Congrats to the folks involved. Building a long running (hours to days) workflow pipeline on top of Cadence was some of the most fun I had writing code at Uber. Stellar tooling, stellar team.
Temporal provides a unified backend for automatically managing implicit application state that is normally stored in transient queues, databases etc. Furthermore, Temporal does this without explicitly requiring the developer to think about and manage the state themselves. This means developers spend way more time building stuff that actually matters, and less time writing buggy reliability code.
I personally find the best way to explain it is with an analogy. Back in the late 90s many developers built applications with C and therefore had to manage their own memory. For a long time, this was not wasted effort as it was the only real option. But then, Java came around and offered an experience where developers didn't have to manage their memory. And for the majority of apps, the performance and capabilities of Java were more than sufficient. At this point, writing the average application in C meant you were doing a serious amount of undifferentiated work. Furthermore, most developers weren't that great at memory management so choosing to do it by hand meant more work for a worse result.
The value proposition of Temporal is nearly identical, but instead of manually managing memory with C, developers are manually managing state using queues, CRON services, databases and more. The result is a bunch of time spent doing undifferentiated things that a computer would have done better anyway.
lots of answers already here, just throwing in mine :)
Temporal is a workflow engine for managing distributed state. It abstracts away queues, databases, schedulers, state machines, load balancers/gateways and makes it so you don't have to be a distributed systems expert to get this right.
To me there are four levels of appeal:
1. standardized, declarative system for timeouts and retries <-- most users start here
2. event sourced, highly fault tolerant internal architecture ---> we even like the term "fault oblivious" - when you write workflow code you can assume that it is robust to downtime in external APIs, or Temporal Server or Temporal Workers, the thing just keeps going until it hits a timeout or an actual application failure!
3. idiomatic SDKs for "workflows as code" -> no need to learn some JSON or YML based DSL, use all familiar software tooling
4. horizontally scalable architecture for every part of the system (key differentiator vs handrolled systems... although i'm not saying you should prematurely scale of course, just saying this architecture lets you scale without replatforming)
Some projects that we get compared to (although of course we're not 1:1 competitors): Apache Airflow, AWS Step Functions, Argo Workflows, Sidekiq, BullMQ
Basically, it's a distributed code-first workflow manager (as opposed to configuration-centric solutions that use BPML). It's a fork of Cadence, which was the author's project at Uber to create a single workflow manager for their platform.
The use case is basically any kind of temporally-distributed workflow that you might use something like Airflow for, or else otherwise hack together with some combination of event buses, temp tables, and cronjobs. Its main charm is that you define the logic in code rather than templates, so you can intermix business logic directly into the workflow.
ctrl + F to here "Temporal is a new kind of platform. It’s not a database, cache, queue, or a means to run your code. It’s a runtime for distributed applications that transparently manages their state at any scope and scale. By externalizing state to a generalized platform like Temporal, application teams get to outsource the majority of cross-cutting concerns presented by cloud applications. In doing so, applications become" if you dont want to read the whole thing
It's like computer hibernation, but for a process that runs on a whole distributed system instead of on a single computer. As programmer of that process, you just focus on writing your business logic with the assumption that each step will eventually complete, and not have to worry about power loss/network loss/machine loss while your process is running. This way, it's much easier to reason about and focus on the thing you care the most: your business.
What is interesting is that they portray as if entire Netflix uses this in the home page.
From what I know two people at Netflix tried to pitch and even implemented it in their projects. The rest of the company still uses other things and they have their own similar tool. Seems like false marketing to me.
Thats interesting, I went to their Conductor meetup last week and someone asked this question on whether Conductor is being replaced. They confirmed that one team is trying Temporal and Conductor is the primary orchestration tool in Netflix. There is a lot of new buzz around Conductor now given recent meetup.
I reached out to the guy trying Temporal at Netflix, apparently he never tried Conductor first. Big company problems I guess.
It will be interesting to see the future of Conductor given the Temporal efforts.
Yup. I expected a shutdown, a response to widespread harassment allegations, or some sort of open source license change. But it's none of those things, they got funded!
So... what actually is it? I've seen the other comments, I've clicked the links and scanned the blog posts. I get that it is a "resilient workflow engine" or whatever.
If my SQL & PHP backend serving a basic web API to a React storefront is struggling to handle customer's states (carts, lists, orders, etc) how is this going to help?
For me, the compelling part was at my last company, we'd build these complicated distributed systems with a whole lot moving pieces—any of which could go down at a moment's notice. For a complex enough system, there was sometimes more work to be done to deal with any part of the system failing than to build the system itself. Basically, we were implementing our own bespoke version of Temporal over and over again.
To your point, the complexity of a SQL/PHP application is significantly smaller, but, the value proposition of Temporal is probably still there: If any part of the checkout process fails (e.g. charging the credit card, successfully triggering the order after they've charged the credit card, timing an email if there is an abandoned cart, handling the moving pieces of a return). These pieces are either handled manually by a human or you've got a decent amount of business logic in place to handle each of these edge cases individually.
The value-add of Temporal—in this case—would be that it would keep track of where the customer/order was in the overall flow of ordering something and pick up where it left off in the event that something went wrong.
Would really love to see movement on integrating Temporal better with Scala on the JVM, just like it supports Java and Kotlin now. Would love to collaborate or help making this thing happen if you have many things on your plate.
Temporal JavaSDK may get a temporal-scala module the same way it has temporal-kotlin with extension functions and method pointer "disassembling" . Reach out to @Dmitry at temporal support Slack if you want to contribute.
How is this offering different from AWS Step Functions? Looking over their docs briefly, see lots of similarities like Workflows, state management, concurrent executions, retires etc.
The main difference is that workflows are written as code in a general purpose programming language. Java, Go, Javascript/Typescript and PHP are already supported. Python and .NET are under development. AWS Step Functions are using JSON to specify workflow logic. JSON is OK for very simple scenarios, but is not adequate for the majority real business use cases. The fun fact is that Step Functions are a thin layer on top of AWS SWF which is based on the same idea as Temporal.
The technology is interesting but honestly I don't recommend it to any early stage startup as it has a very high learning curve, high maintainability, limited visibility and so on. Maybe if you got to a stage that you have a problem that temporal can solve and a separate team that can keep it alive and debug it.
I get where you are coming from. However with their cloud offering you get to forget about some of the not easy parts of Temporal and just build something awesome while benefiting from all that temporal has to offer :)
Im reading the description of this service via the multiple questions asking what Temporal is. So now I am curious, could this be pivoted to work in game development? I understand that is not your target audience, but state management is a necessary part game dev.
It certainly could, but not a fit for a real-time game. In order to make Temporal general-purpose, fault-tolerant, and scalable, it had to do more database writes / have higher latency than real-time game backends.
Pardon me, I haven’t done more research. But how does Temporal save/manage state for Go? Are there any API which let you access state of a Goroutine (like thread state)?
Can someone point me to a simple example use case and why temporal is beneficial? The website has a lot of temporal-specific jargon and not a lot of straightforward examples.
[+] [-] sunny--tech|4 years ago|reply
I say this as someone who loves using Temporal and see a big future for them.
Temporal is a managed state-machine. If you're familiar with K8s, you can think of it as a dedicated reconciler. Plain and simple.
Do you want a resource to go from one state to another (i.e. user orders food -> food is delivered) regardless of transient errors and other hiccups? Temporal handles the plumbing of driving your resource to it's desired end state.
Do you have a workflow that has multiples steps? Can anyone of those steps fail for whatever reason and you'll need to figure out how to do retrying/cleaning up? Temporal handles that.
Essentially you can write your business logic without the worry of including the (often) boilerplate retry, cleanup, scheduling, etc logic. If you're familiar with K8s reconciliation and how it retries failed operations again and again until it reaches the goal, it's much like that.
Temporal can do much more than that, but that's its bread and butter and I'm trying to describe it simply so that people can understand. It is a bit of a paradigm shift and has a bit of a learning curve when you first start using it.
But like others have said, it's very powerful.
[+] [-] swyx|4 years ago|reply
i'm actually interested in a longer form "Temporal vs K8s" comparison - can make a case for Temporal atop k8s, or Temporal replacing k8s (already had a couple customers talk to us about this). It's a nuanced story to tell and I'm not sure if it has a conclusion.
[+] [-] swyx|4 years ago|reply
Additional coverage for more info:
- https://www.geekwire.com/2022/temporal-is-a-unicorn-develope...
- https://www.theregister.com/2022/02/16/temporal_103m_funding...
- https://siliconangle.com/2022/02/16/temporal-raises-103-mill...
- https://dailyuknews.com/tech/temporal-raises-103m-in-series-...
- https://finance.yahoo.com/news/temporal-io-raises-103-millio...
- https://www.computerweekly.com/blog/CW-Developer-Network/Tem...
- https://www.indexventures.com/perspectives/everything-is-a-w...
[+] [-] jffry|4 years ago|reply
(EDIT: links got fixed) For clickable links, remove the leading spaces so it doesn't get formatted as plaintext, and make newlines double to avoid them all being on one line
[+] [-] dang|4 years ago|reply
[+] [-] ctvo|4 years ago|reply
The biggest pain point is still deploying and configuring a production environment. I started suspecting it was so awful and poorly documented to push us to using the cloud offering, but then I couldn’t even find a link to sign up for cloud.
[+] [-] jmcnevin|4 years ago|reply
Being able to model business processes--some of which may take multiple _years_, and require human approval/rejection steps--using procedural go/java/js, and knowing that once that function starts executing, it is basically indestructible (barring the complete destruction of temporal's backend database), is just really cool. Being able to outsource a lot of the complexity involved in using event sourcing to manage distributed transactions frees up our developers to worry about just the high-end business logic.
Congrats to the team! I'm excited to see Temporal gaining some more traction.
[+] [-] swyx|4 years ago|reply
[+] [-] MaknMoreGtnLess|4 years ago|reply
[deleted]
[+] [-] Thaxll|4 years ago|reply
At first it looks easy to use any pub/sub or DB to create a job system, but there are so many edge cases, I'm not even talking about scaling etc ... just having a solid job system that properly handle retries, error, durability, timeout etc ...
[+] [-] djmagee|4 years ago|reply
[+] [-] suchire|4 years ago|reply
[+] [-] rrix2|4 years ago|reply
[+] [-] swyx|4 years ago|reply
[+] [-] josephg|4 years ago|reply
[+] [-] rylandgold|4 years ago|reply
Temporal provides a unified backend for automatically managing implicit application state that is normally stored in transient queues, databases etc. Furthermore, Temporal does this without explicitly requiring the developer to think about and manage the state themselves. This means developers spend way more time building stuff that actually matters, and less time writing buggy reliability code.
I personally find the best way to explain it is with an analogy. Back in the late 90s many developers built applications with C and therefore had to manage their own memory. For a long time, this was not wasted effort as it was the only real option. But then, Java came around and offered an experience where developers didn't have to manage their memory. And for the majority of apps, the performance and capabilities of Java were more than sufficient. At this point, writing the average application in C meant you were doing a serious amount of undifferentiated work. Furthermore, most developers weren't that great at memory management so choosing to do it by hand meant more work for a worse result.
The value proposition of Temporal is nearly identical, but instead of manually managing memory with C, developers are manually managing state using queues, CRON services, databases and more. The result is a bunch of time spent doing undifferentiated things that a computer would have done better anyway.
[+] [-] swyx|4 years ago|reply
Temporal is a workflow engine for managing distributed state. It abstracts away queues, databases, schedulers, state machines, load balancers/gateways and makes it so you don't have to be a distributed systems expert to get this right.
To me there are four levels of appeal:
1. standardized, declarative system for timeouts and retries <-- most users start here
2. event sourced, highly fault tolerant internal architecture ---> we even like the term "fault oblivious" - when you write workflow code you can assume that it is robust to downtime in external APIs, or Temporal Server or Temporal Workers, the thing just keeps going until it hits a timeout or an actual application failure!
3. idiomatic SDKs for "workflows as code" -> no need to learn some JSON or YML based DSL, use all familiar software tooling
4. horizontally scalable architecture for every part of the system (key differentiator vs handrolled systems... although i'm not saying you should prematurely scale of course, just saying this architecture lets you scale without replatforming)
Some projects that we get compared to (although of course we're not 1:1 competitors): Apache Airflow, AWS Step Functions, Argo Workflows, Sidekiq, BullMQ
My 7 min intro: https://www.youtube.com/watch?v=CQhUL5RogXI
Architecture principles (23mins) https://docs.temporal.io/blog/workflow-engine-principles
Full blogpost: https://www.swyx.io/why-temporal/
[+] [-] HillRat|4 years ago|reply
The use case is basically any kind of temporally-distributed workflow that you might use something like Airflow for, or else otherwise hack together with some combination of event buses, temp tables, and cronjobs. Its main charm is that you define the logic in code rather than templates, so you can intermix business logic directly into the workflow.
[+] [-] chugh09|4 years ago|reply
ctrl + F to here "Temporal is a new kind of platform. It’s not a database, cache, queue, or a means to run your code. It’s a runtime for distributed applications that transparently manages their state at any scope and scale. By externalizing state to a generalized platform like Temporal, application teams get to outsource the majority of cross-cutting concerns presented by cloud applications. In doing so, applications become" if you dont want to read the whole thing
[+] [-] dustrider|4 years ago|reply
From my take, an interesting way of dealing with scalable workflows and batch jobs. But I need to read more into it.
[+] [-] tjchear|4 years ago|reply
It's like computer hibernation, but for a process that runs on a whole distributed system instead of on a single computer. As programmer of that process, you just focus on writing your business logic with the assumption that each step will eventually complete, and not have to worry about power loss/network loss/machine loss while your process is running. This way, it's much easier to reason about and focus on the thing you care the most: your business.
[+] [-] djmagee|4 years ago|reply
[+] [-] engprod|4 years ago|reply
From what I know two people at Netflix tried to pitch and even implemented it in their projects. The rest of the company still uses other things and they have their own similar tool. Seems like false marketing to me.
[+] [-] jusonchan81|4 years ago|reply
I reached out to the guy trying Temporal at Netflix, apparently he never tried Conductor first. Big company problems I guess.
It will be interesting to see the future of Conductor given the Temporal efforts.
[+] [-] wmf|4 years ago|reply
[+] [-] jrockway|4 years ago|reply
[+] [-] swyx|4 years ago|reply
[+] [-] atarian|4 years ago|reply
EDIT: According to https://blog.amplifypartners.com/our-investment-in-temporal/, Temporal was founded by 2 engineers at Uber who created an open-source orchestration engine called Cadence and then later decided to found Temporal.
[+] [-] swyx|4 years ago|reply
[+] [-] unknown|4 years ago|reply
[deleted]
[+] [-] shuntress|4 years ago|reply
If my SQL & PHP backend serving a basic web API to a React storefront is struggling to handle customer's states (carts, lists, orders, etc) how is this going to help?
What are my engineers going to actually do?
[+] [-] stevekinney|4 years ago|reply
To your point, the complexity of a SQL/PHP application is significantly smaller, but, the value proposition of Temporal is probably still there: If any part of the checkout process fails (e.g. charging the credit card, successfully triggering the order after they've charged the credit card, timing an email if there is an abandoned cart, handling the moving pieces of a return). These pieces are either handled manually by a human or you've got a decent amount of business logic in place to handle each of these edge cases individually.
The value-add of Temporal—in this case—would be that it would keep track of where the customer/order was in the overall flow of ordering something and pick up where it left off in the event that something went wrong.
[+] [-] jvican|4 years ago|reply
[+] [-] spikhalskiy|4 years ago|reply
[+] [-] stock_toaster|4 years ago|reply
[+] [-] firstfewshells|4 years ago|reply
[+] [-] mfateev|4 years ago|reply
Here is a more detailed answer from Temporal forum: https://community.temporal.io/t/why-use-temporal-over-a-comb...
[+] [-] yevpats|4 years ago|reply
[+] [-] swyx|4 years ago|reply
but yeah I would never say that we're for everyone. depends on your knowledge, needs, usecases!
[+] [-] roopakv|4 years ago|reply
[+] [-] jusonchan81|4 years ago|reply
[+] [-] eezurr|4 years ago|reply
[+] [-] lorendsr|4 years ago|reply
[+] [-] avinassh|4 years ago|reply
[+] [-] daveguy|4 years ago|reply
[+] [-] unknown|4 years ago|reply
[deleted]