Amazon States Language – A JSON-based language to describe state machines

[termie]

"The Step Functions free tier includes 4,000 free state transitions per month." .. "$0.025 per 1,000 state transitions thereafter"

So Amazon has created a new calling convention, where conditional logic now also requires a context switch and JSON serialization. Then they charge you for the call .. $tdcall?

This will certainly have some useful applications, maybe someone will build an inexpensive data processing pipeline on top of it. Having seen many visual workflow tools through the years, most are simplifying complex underlying process, but these step functions and state transitions are modeling basic internal control flow with complex abstractions and little benefit other than retries with backoff.

The pure functional immutable nature of the data flow is ideally nice but tainted by the JSON. The parallelism is interesting, but it seems bolted on instead of a more powerful central part of the design.

[mSparks]

wait, what..!!

is that entire system state to state or per state?

i.e. if i have 100 states each transitioning 365 times bootstrapped a thousand times

they'll charge nearly a million dollars or nearly a thousand?

noting that takes like a minute on my desktop.

[DigitalJack]

Why do something like this in JSON? I don't work in javascript, but surely it would have been far simpler to use. Restrict it to a subset if you have to, but this is absurd:

  "ChoiceStateX": {
  "Type" : "Choice",
  "Choices": [
    {
        "Not": {
          "Variable": "$.type",
          "StringEquals": "Private"
        },
        "Next": "Public"
    },
    {
      "And": [
        {
          "Variable": "$.value",
          "NumericGreaterThanEquals": 20
        },
        {
          "Variable": "$.value",
          "NumericLessThan": 30
        }
      ],
      "Next": "ValueInTwenties"
    }
  ],
  "Default": "DefaultState"
  }

[murukesh_s]

It looks like they want to make a User interface on top of it. It is very difficult to make a user interface that reads and writes from a regular programming language as there are multiple ways to write the same expression (with space, without space, with new line etc).

[mSparks]

competely agree.

its one thing to serialise your states to json. quite another to try and recreate javascript or any other low permission language in it.

I know _why_ they did it, and i bet a fair few pure managerial types fall for the bait and commission a project or thousand on it, locking themselves in to amazons absurd pricing model for production systems. But everyones got to make a buck eh.

[junke]

Because Javascript doesn't have macros.

[ChrisRus]

I have been working on a moon-shot project for more than a decade that seeks to model distributed systems as if they were digital logic circuits. One of the natural areas of interest is of course FSM's - in particular declarative specifications of FSM's. I continue to believe (although it's a wildly unpopular notion among some of my machine learning friends) that this obsession with the mechanics of declaring design intent with models and then synthesizing runtime code will fundamentally transform software engineering over time. But, declarative FSM's aren't enough to make it practical IMHO.

I wrote a short essay about this work in which I argue that the software engineering community needs to embrace the design methodologies and rigour of hardware designers: https://medium.com/@alpinelakes/on-monday-i-learned-i-got-ac...

See also: http://blog.encapsule.org/early-encapsule-project-history/20... (old code but same ideas as what I'm building now in Node.js/HTML5 @Encapsule).

Several things I believe are actually essential to make use of any of these ideas at scale:

- There needs to be an ad-hoc extensible standard for notating serialized data with markers, tags, semantics, metadata (whatever you care to call it). It is not practical to do unsupervised feature extraction on internal message streams. And, it's _insanity_ to write/test/maintain custom validation/normalization logic.

- Given the above, FSM declarations must be encoded with labels (as above) so that generic code can easily affect interop.

- Small FSM's are reasonably easy to comprehend. But, very few systems can be modeled with simple FSM's. Rather, real systems can be modeled as complex directed graphs where edges represent the flow of observable state from one FSM to another (vertices represent individual FSM).

- Given that real systems can be modeled using non-trivial graph models of FSM (as above), building reusable components by splicing and dicing the graph up is logically possible. But, this is not something that mortals can do by hand. Considerable tooling is required to make it practical to design systems like this.

If you're interested in these topics, and want to help, look me up @Encapsule.

[jnwatson]

You're about 30 years late. The programming with visual state machines started with Harel and Booch and led to UML (see https://en.wikipedia.org/wiki/Shlaer%E2%80%93Mellor_method#/...). Executable UML (xUML) is a thing. It just never caught on.

[parasubvert]

I tend to agree with you on this, my focus has been on the application of declarative Hierarchical FSMs, Behaviour Trees, and/or Hierarchical Task Networks (using automated planning) as a way of describing FSMs declaratively and yet handling the state explosion problem. The intent is to enabling better integration and interoperability on the Web - basically, getting rid of Twitter, Facebook, etc. centralized monopolies of "write" functionality on the Web. ("Read" functionality, i.e. web crawling / HTTP GET / Google is also monopolized but that's less due to architectural problems and more economic)

I have given a couple of keynotes on this topic at the W3C and RESTfest over the years, but just haven't done a lot of the grunt work since I have a day job.

See - http://www.slideshare.net/StuC/ill-see-you-on-the-write-side...

Also - http://www.slideshare.net/StuC/linking-data-and-actions-on-t...

And per your point about how you need serialized data notation + FSMs, see http://web.archive.org/web/20160410102032/http://www.stuchar...

I have felt this train of thought could be useful for a general purpose approach to software engineering beyond distributed systems interop. Unfortunately this has been a hobby horse of mine for about 10 years that I don't have a lot of time to dedicate to....

[lacampbell]

A new market to disrupt - undergraduate computer science projects as a service.

Wait until you guys check out my hashtable implementation in the cloud.

[delbel]

By chance you want to buy some bubble sort for $5/month? that's less then .005 per sort. I also have some 24 bit IEEE floating points as a service (IEEEFPaSS), on sale as well.

[automatwon]

Is a KeyValue database not a hashtable in the cloud?

[paulddraper]

So... DynamoDB?

[michaelsbradley]

See also SCXML and SCION:

https://www.w3.org/TR/scxml/

https://github.com/jbeard4/SCION

And for background, see the pioneering work of Dr. David Harel:

A list of all of his papers

http://www.wisdom.weizmann.ac.il/~harel/papers.html

A few on Statecharts

http://www.wisdom.weizmann.ac.il/~harel/SCANNED.PAPERS/Seman...

http://www.wisdom.weizmann.ac.il/~harel/reactive_systems.htm...

http://www.wisdom.weizmann.ac.il/~harel/papers/RhapsodySeman...

http://www.wisdom.weizmann.ac.il/~harel/papers/Statecharts.H...

http://www.wisdom.weizmann.ac.il/~harel/papers/Statecharts.H...

Prof. Harel the dreamer...

http://www.wisdom.weizmann.ac.il/~harel/papers/LiberatingPro...

[lobster_johnson]

Interesting that they're using JSONPath, which isn't even specified formally anywhere. The only other major implementor that I know about is Kubernetes, which has some odd extensions for templating. (JSONPath itself, of course, isn't very well designed in the first place.)

We're working on a new variant of JSONPath that we're hoping to publish as a formal, comprehensive specification. It's essentially a superset of JSONPath with some syntax warts fixed (like the need to start with $). I wrote a little about it on HN a week ago [1].

[1] https://news.ycombinator.com/item?id=13032391

[Marazan]

Remember when we used XML, and then people started making DSLs in XML and XML was the worst and too 'heavy' and having to write schemas was enterprise and awful and we are totes using lightweight schema free sexy JSON now?

I wonder what the new thing to replace enterprise JSON will be.

[aindhaden]

It's still going to be XML. JSON is fairly limited, it's literally the serialization format for JavaScript variables, and that one size does not fit all.

[supergreg]

Yaml looks like the perfect candidate

[tonylucas]

This is interesting, has a lot of similarities (not surprisingly) with how the state machine we build workflows on our bot platform, although of course ours is specifically fixed around chat/messages as a key interaction point.

Major differences that I can see are we enable multiple functions to be sent per state, and that the output data from any state is referenceable by any other state, not just passing it down in turn through the states.

We support fallback states but in a different way, and don't support the retry concept directly within the state language itself, has to be built as a set of states to perform a loop to attempt a retry.

We don't support parallel stages, but do support branches, and remerging of those branches.

Probably the final difference I can see, is one of our options when running a function allows you to actually append additional states to the machine during the runtime process.

[Swizec]

I did it better 5 years ago. https://swizec.com/blog/a-turing-machine-in-133-bytes-of-jav...

133 byte interpreter in JavaScript. Input is JSON specifying state name, write, move direction, and next state. Turing machines, basically.

Mine was for fun, but why is Amazon doing this?

[slowmovintarget]

People needed a state engine for sewing Lambda functions together. For example, try doing retry with exponential back-off in Lambda. You quickly run into a number of problems that are difficult without an execution context outside the Lambda itself.

Step Functions give you this external context for doing retry, conditional trigger of downstream functions, parallel trigger of additional functions and more. Execution time of a state machine can last for up to a year, so this also gives you a way to do more than 5 minutes of work at a time.

[automatwon]

⊆ {"Money", "Power", "Respect"}

[nighthawk454]

Some more info on Tim Bray's site: https://www.tbray.org/ongoing/When/201x/2016/12/01/J2119-Val...

[timbray]

Glad to hear that so many people already did this (only better), years ago. Nice to have company.

The only thing new or interesting about States is it has a product behind it that implements it at scale, available now; give it a try.

I think it's quite likely that this syntax is state-machine assembler, and smart people will find nicer expressions of this and compile them down.

In particular, some people prefer dependency graphs to explicit state machines for this sort of thing.

[trickyager]

Amazon has also created a ruby gem to lint JSON state machines. https://github.com/awslabs/statelint

[TeMPOraL]

Wait, did they just made a Lisp in JSON?

[c3534l]

You wait, Google is going to create one in XML, then Apple is going to invent a cool minimalistic pseudo-Lisp that doesn't require any brackets or colons. Then Windows is going to try to create one that only runs on .NET. Finally, someone will create a format that to be read by all of them, and someone is going to implement another LISP in that format.

[absrnd]

Greenspun's tenth rule?

[tree_of_item]

No, what makes you say that?

[hacker_9]

"This document describes a JSON-based language used to describe state machines declaratively. The state machines thus defined may be executed by software. In this document, the software is referred to as “the interpreter”."

Oh no.

[snoman]

Everything old is new again.

So, now that it is happening to me, am I allowed to apologize to the numerous old-coders that tried to tell me this when I was coming up?

[slowmovintarget]

In explaining the choice of JSON as the description language for state machines Tim Bray briefly said, "I couldn't find a good reason not to express it as JSON and in this day and age, you need a good reason for it to not be JSON."

He also mentioned that because it was a formally specified syntax, you could, should you choose to, build other more convenient syntaxes that reduce to it. It won't surprise me to see that happen fairly quickly.

[haimez]

Everything at Amazon is like this. You should try to use cloud formation.

[ape4]

So JSON has officially Jumped The Shark (like XML)

[plandis]

Seems like a decent way to glue AWS Lambda functions together?

Why: "oh no"?

[ajkjk]

Why?

[devj]

Wondering why they aren't using their own data format(Amazon Ion - https://amznlabs.github.io/ion-docs/spec.html) instead of JSON.

[flaviuspopan]

This strikes me as the underlying spec for the new Batch service mentioned at re:invent.

[slowmovintarget]

It's the underlying spec for the JSON description of state machines in AWS Step Functions.

https://aws.amazon.com/step-functions/

[neurostimulant]

This could be useful. Currently there is no easy way to add delay when invoking lambdas from an event. Perhaps this could be used to create a delayed execution triggers (e.g. Invoking a lambda 10 minutes after a dynamodb entries updated).

[avodonosov]

JSON? Somebody forbade s-expressions?

[TeMPOraL]

For some weird reason people these days believe curly brackets are the the best brackets.

[rhizome]

JATEOAS

[bunderbunder]

Context: https://spring.io/understanding/HATEOAS