scottdw2's comments

This seems to be, quite obviously, an attempt at free marketing. They want to be able to point at the corpus of products influenced by SO, and say "see, we're pretty cool".

Seems misguided to me.

There are several connections between Lambda and functional programming.

1. The snippets of code you run in Lambda are called "cloud functions".

2. Lambda applications are generally event driven and stateless, which is a very functional idiom.

As other folks mentioned, however, you can use Lambda with non functional languages (like Java). So even though a Lambda application is inherently functional (small stateless functions communicating via events), you don't have to write in a functional style if you don't want to.

Your code looks correct. I would expect something closer to 50ms in the warm path (300ms in the cold path seems about right).

I'll take a look tomorrow and see if I can reproduce what you are seeing. I'm not super familiar with API gateway, so there could be some config issues over there.

If you want to discuss this more offline, feel free to contact me at "scottwis AT amazon".

Lambda has a fairly generous free tier.

Here are the details on pricing:

https://aws.amazon.com/lambda/pricing/

You get up to 1M requests / month and 400,000 GB-seconds of compute time per month.

A default lambda function uses 128 MB of ram (0.125 GB), which gives you about 3.2 M seconds of compute time (time actually spent executing requests) every month for free.

Thus if you have functions that take 500 ms on average, and use the default amount of RAM, you can process about 6.4 M requests in a month for a total bill of $1.20.

Above the free tier limits you pay $0.20 per million requests, and $0.1667 per million GB-seconds.

The pricing is fairly attractive.

Are you using Node.js or Java for your Lambda function?

If you are using "Node.js", you may be seeing slow times if you are not calling "context.done" in the correct place, or if you have code paths that don't call it.

Not calling context.done could either cause Node.js to exit, either because the node event loop is empty, or because the code times out, and Lambda kills it.

When node exits, the container shuts down, which means Lambda can't re-use the container for the next invoke and needs to create a new one. The "cold-start" path is much slower than the "warm-start" path. When Lambda is able to re-use containers, invokes will be much faster than when it can't.

Also, how are you initializing your connection to DDB? Is it happening inside your Lambda function? If you either move it to the module initializer (for Node), or to a static constructor (for Java), you may also see speed improvements.

If you initiate a connection to DDB inside the Lambda function, that code will run on every invoke. However, if you create it outside the Lambda function (again in the Module initializer or in a static constructor) then that code will only run once, when the container is spun up. Subsequent invokes that use the same container will be able to re-use the HTTP connection to DDB, which will improve invoke times.

Also, you may want to consider increasing the amount of RAM allocated to your Lambda function. The "memory size" option is badly named, and it controls more than just the maximum ram you are allowed to use. It also controls the proportion of CPU that your container is allowed to use. Increasing the memory size will result in a corresponding (linear) increase in CPU power.

One final thing to keep in mind when scaling a Lambda function is that Lambda mainly throttles on the number of concurrent requests, not on the transactions per second (TPS). By default Lambda will allow up to 100 concurrent requests.

If you want a maximum limit greater than that you do have to call us, but we can set the limit fairly high for you.

The scaling is still dynamic, even if we have to raise your upper limit. Lambda will spin up and spin down servers for you, as you invoke, depending on actual traffic.

The default limit of 100 is mainly meant as safety limit. For example, unbounded recursion is a mistake we see frequently. Having default throttles in place is good protection, both for us and for you. We generally want to make sure that you really want to use a large number of servers before we go ahead and allocate them to you.

For example, a lot of folks use Lambda with S3 for generating thumbnail images, sometimes in several different sizes. A common mistake some folks make when implementing this for the first time is to write back to the same s3 bucket they are triggering off of, without filtering the generated files from the trigger. The end result is an exponential explosion of Lambda requests. Having a safety limit in place helps with that.

In any case, if you are having trouble getting Lambda to scale, I'm happy to try and help.

In a leader election, a node won't vote for a candidate that doesn't have a record it believes to be committed. Thus, writing to a majority ensures that when the leader dies, the new leader will have all the committed dats. Thus, up to n/2 - 1 nodes can be lost with a guarantee that no committed data is lost.

If you don't need strong consistency (after a write commits, all future reads will see the write), you can use simpler replication strategies.

Redis, for example, using async replication, that is not guaranteed to succeed. A redis master may ack a write, and a subsequent read may see it, but a loss of the master before replication occurs can result in data loss. The failover is not guaranteed to contain all writes. Some times weak consistency is ok. Sometimes it's not, but eventual consistency is ok. Other times strong consistency is needed.

Raft is useful when you need strong consistency.

Raft does support membership changes (adding and removing nodes from the cluster), so it can support losing more than n / 2 - 1 nodes, just not simultaneously.

A few things:

1. You don't have to round trip to every member before acking a write request, only to a majority of them.

2. If you use the right data structures for your state machine (like persistent data structures), you don't have to fully commit a write before you start the next write. That is, as long as you can quickly revert to a previously committed state (which you can with persistent data structures, it just becomes a CAS on a pointer) you can pipeline requests and batch commits. That lets you start on write n+1 using the state machine results of write n, even though write n hasn't been committed yet. You can then commit a bunch of results (n, n-1, n-2, ...) as a single batch. That does create a dependency that some write j may be rolled-back because a write i < j failed to commit, but you would have had that dependency anyways (you wouldn't have even started write i unless write j had committed). So, although latency of a single request is bounded by the time to replicate a log entry to n/2 + 1 hosts, the throughput is not bounded by the latency of a single write.

The raft paper doesn't actually say you can do this (it says things like "don't apply state machines until a transaction has been committed), but that's because it assumes arbitrary side effecting state machines. However, with some restrictions (like using persistent data structures), you can relax that requirement. Relaxing that requirement can give quite good throughput.

3. You can horizontally scale raft the same way you horizontally scale any other distributed data store: using consistent hashing. You would use a hash ring of raft clusters.

4. You can loose up to n/2 hosts from a raft cluster and it will still work.

5. Raft is "multiple servers with failover and a version clock". In fact it's "multiple servers with failover, a version clock, and strong sequential consistency".

This is a nitpick, but it is lazy.

It's not represented in head normal form, but it's still lazy.

The limitations you mentioned are not an artifact of the implementation being eager(because it's not eager), but instead are an artifact of the representation used.

Something like that may work for digital goods.

You deliver some digital assets, the key to which can only be derived if a particular crypto currency transaction is performed.

But if you are delivering physical goods, there's going to be some human aspect to it.

Some human, somewhere, is going to deliver the good to some other human.

I'm not discounting the idea of automated contract enforcement. I think contract enforcement is really inefficient. Finding a way to automate it seems like it could be useful. That's the niche paypal originally filled (providing escrow services for ebay transactions). If you could find a way to automate such things, so that everywhere you currently see a "standard service" contract nowadays you had someone clicking on a button (and maybe entering a credit card number), it seems like it could be disruptive. Kind of like a Google App engine for executing contract logic.

I'm just not sure that bit coin is central to the whole thing.

The whole "free from government interference" thing is a bit of a fallacy. If the government wants to seize your bit coins, they will find a way. It may be difficult for them to manipulate the price of bit coin, and to implement wholesale theft (stealing everyone's bit coins), but seizure of a specific individual's bit coins is definitely possible. They'll find a way.

Maybe some form of contracts, like options trading, might work. Equity funding of startups might work too. Basically anything that's purely digital.

But I don't think that any contract involving physical goods can be rendered mathematically perfect.

There's an interesting split here: between contract validation, contract enforcement.

Both have typically been handled by courts.

Contract validation is typically very "human centric". It centers on questions such as "what is a bargain?" or "is this result acceptable to the conscience (I.e. is morally acceptable)".

Contract enforcement, however, is much more formulaic. Given a valid contract, and a series of events (leading to a breach), quantify what relief would look like.

I don't think validity can be automated. It's analogous to Dijkstra's obsession with "formal verification".

However, most contracts are valid. They typically involve genuine negotiations on reasonable terms.

Assuming validation, enforcement is easily automated.

I don't think bit coin is relevant though. For automated contract enforcement any payment API will work. Wether it's bit coin, or paypal in Estonian Kroon, the important part is automation.

There are already 1M apps in the App Store.

There's already too many apps, so preventing "over choice" isn't their goal.

A shelf space analogy doesn't make sense. With a real shelf, there is scarcity. Only so many shelves can fit in the store, and only so many products on a shelf.

The marginal cost of a virtual good is zero. Apple's ability to support apps is effectively infinite.

There are over 1M apps in the App Store.

This is not about "shelf space".

In war, identity is a defined "negatively". That is, not by describing positive attributes to you, but by describing negative attributes to the enemy. "We are the opposite of x", hence we fight. If you neutralize the ability of the enemy to define you (because he is like you) he can't define himself, and so he cannot fight you. Thus, there is no enemy, and hence no war.

Is there any better way to change someone's mind, to change their fundamental mode of thought, to alter their perceptions and manner of reason than to educate them?

Here's an even better question: if your enemy thinks like you, talks like you, acts like you, shares your values and views the world the way you do, is he your enemy? Can he be your enemy? Why would he be your enemy?

Let's consider a central question.You espouse a belief in freedom, a nation built on liberty. Your enemy denies this, proclaims you a tyrant, and endeavors to make war against you. How do you respond?

One method undercuts your enemy's message, reaffirms your core values, and has the potential to alter and modify your enemy so that he is predisposed to be united with you.

The other affirms his message, strengths his position, and emboldens those who follow him.

This leads to the last question: what is your objective? Is it victory, or something else?

What better path to victory then to alter the mind of your enemy until he is incapable of fighting you?

Our educational assets ought to therefore be shared. To sensor them in the name of security is foolish.

That's the whole point.

It's disruptive technology: not as good, way more accessible.

This bothers me.

I know this is a bit of a 'libertarian fantasy', but I think the constitution ought to be amended to contain something like the following:

1) No person shall be subject to any criminal penalty exceeding one year of incarceration or $300 in fines for any non violent offense relating solely to the possession, sales, distribution, manufacture, or purchase of an intoxicating substance.

2) The purpose of this article is to limit the scope of criminal penalties that may be applied to "non violent drug offenders". It's provisions shall be interpreted with such intent in mind.

3) This article applies to all jurisdictions with the Several States, the United States, and any territories or possessions thereof

4) Any forfeiture of assets resulting from the conviction of a "non violent drug crime" must be limited to:

a) The intoxicating substances constituting the "core element of the crime"

b) Any asset materially and predominantly used for the manufacture, production, and possession of such substances.

Provided that such seized assets do not also have reasonable, fundamental, predominant, and legally authorized uses. In such case any seizure must be subject to the provisions of "eminent domain".

5) Congress, or the states, acting within the provisions otherwise authorized by this constitution, may adopt measures to ensure assets actually used in the commission of a "non violent drug crime", when not seized in accordance with this constitution, are only used in accordance to lawfully authorized purchases.

The key take away from the article SHOULD be: don't rely on ntp if you don't have to.

There are people who have to. They run their own atomic clocks, and worry about things such as precision delivery of nuclear ordanance.

Then there's you. You should use vector clocks, with a builtin conflict resolution mechanism based on domain knowledge.

That's the point of the article.

I won't argue that constraints can lead to innovation. All art exists in some medium. Constraints are therefore inherit.

Here's a simple question: what spurred Picaso's decision to paint in Blue? Was it adherence to a protocol describing blue as the answer to his expressive woes, or was it his own internal experimentation with the color blue?

Did Picaso paint in blue because he felt like Blue, or because someone said "any work 'submitted' to this gallery must be blue"?

Sigh. Not about the original article, but about the top comment in the original article.

It presumes that Picaso, Bansky, and Monet can't paint like each other.

Probably not true if you actually have Picaso, Bansky and Monet.

In any case, style guides are for hacks, not artists.

Actually he has admitted he was wrong about the iPhone, thinking of it as a high end phone instead of a low end (but very convienient) computer.

The author of the article, I think, makes a "phalous leap" from "He was wrong about the iPhone" to "disruption doesn't apply to consumer markets".

Primarily because the iPhone is a perfect case of this. It (and it's successor, the iPad) is disrupting the PC industry.

You have to look at the phone as a computer, not a phone. Then disruption theory REALLY makes sense.