I’ve had a chance to use the service for a couple of weeks. My quick summary review is that it’s a little tricky setting up the IAM roles and security groups, but once you have that going, it works great! I see a ton of potential here in transforming the way people use AWS.
I also put together the Netflix use cases in the keynote so if you have any questions I’ll try to answer them!
One thing I am struggling to understand is how much of my applications can be replaced with Lambda.
I think it would be really awesome to make applications that do not have any dedicated instances associated with them, but are entirely based on the various AWS platforms.
For some things, it seems like Lambda could be a drop-in replacement, whereas others, it would not quite fit the bill. Do you think it will be possible to build a viable a "server-free" application with Lambda and the other services (S3, DynamoDB)? Or is it probably going to be limited to the types of use cases listed on the landing page[1]?
Could you use it more generically, meaning not "just" within an app but also as part of an overall control plane, sending commands to other APIs like a notification system?
From hearing the details it seems that there is no real limitation to what you can execute as long as it runs on the instance. I get the feeling that if you zipped up all of the parts of python or ruby you need to ru you could execute a script using either of them as long as you initiated it from your node script. The key word here is probably "support" and that may be more of an issue if you want to edit your scripts with their web IDE.
as MBlume mentionned[1], they seem to support NodeJS.
From the FAQ :
"At launch AWS Lambda supports code written in Node.js (JavaScript). Your code can include existing Node.js libraries, even native ones."
so it seems that you can avoid lock-in if most of your javascript code is not too coupled to Amazon's API
And they claim on the detail page:
"With AWS Lambda, there are no new languages, tools, or frameworks to learn. You can use any third party library, even native ones. At launch, AWS Lambda will support Node.js code, with support for other languages coming in the future."
that's a REALLY good point! this feature seems really solid, but they seem to be putting a sort-of "golden handcuff" on you. If you have a distributed system on AWS which implements "lambda functions," you can forget about switching to Azure, or your own bare metal servers without having to go through some pain. "vendor lock-in" definitely seems to have been their motto here, very clever & great observation!
Well, yes. That's the thing about AWS. It's a very feature rich & sophisticated platform, which is hard to replicate elsewhere with the same harmony without spending a great deal of investment.
I won't be surprised if someone writes a generic lambda library which given a cloud provider generates relevant mappings/code. Will take care of some of the vendor lock-in concerns ?
As someone who is non-technical, how much code/infrastructure would this save? I'm trying to understand how locked in this would really make you, and how difficult it would be to replace this on some other IaaS provider?
I'm guessing that this Lambda functionality is built on the new ECS (EC2 Container Service) which is built on Docker. Therefor, it should be reasonably doable to recreate it on-premises and escape the AWS lock-in. "should" being the operative word there. :)
It's no different from stored procedures and triggers in a database, or a shell language on an operating system. At some point you will have to invest time creating logic, for your purpose, on somebody else's system.
I wish they hadn't named the units of computation "Lambda functions". Cause, you know there's already something known as a "Lambda function" in computer science.
But kudos for Amazon for furthering the datacenter-is-the-computer approach. It is simply the right thing to do.
Really douche to use a computer science term as your product name, are they going to trade mark that or are the just happy with the confusion it will cause.
I'm just the opposite - when I heard they were releasing a new service called Lamda, I immediately wondered whether it was like the inline lambda function in Python - and when I read through the description - they were very similar in nature.
Amazon has a LOT of services, and the better the naming scheme they use, the easier it will be to remember what they each are. I'm genuinely happy that they used a name that will make it trivial to map the function (heh) it serves in my head.
Holy mind blowing awesomeness, this changes everything, yet I feel like this was such an obvious thing to do. So obvious that I can't believe it is real. This, ladies and gentlemen, is why AWS defines the cloud, they are so far beyond everyone else and still innovating, even and the most basic level.
Seems like a much more interesting announcement than the container service. Can see the architectures of dozens of our systms collapsing down with this.
At a guess, Container Service was built in order to build this. I.e. Lambda is probably an ECS cluster creating and running Node.js-based Docker containers. (A bit like Heroku's original Ruby stack, actually.)
It's a bit amusing to note that they could have built Lambda on top of their own previous Elastic Beanstalk Docker offering instead, but chose to build ECS presumably for cost reasons (which everyone who has evaluated Elastic Beanstalk's Docker support will understand.)
Should I think of this essentially as an abstraction that can replace my asynchronous task queue + workers (e.g. RabbitMQ + Celery workers, obviously assuming you aren't using MQ for additional messaging)? I hate managing those pieces and would be happy to hand that infrastructure to Lambda, but are there additional benefits or use cases that are opened up?
I guess I would have expected others to describe this the same way ("replaces your distributed task queue"), but since I'm not seeing that description I wonder if I've misunderstood.
I guess maybe part of what you might be missing is that these "Lambda functions" can (will be able to?) be kicked off by all sorts of different AWS events, so it can replace queue + workers where you're generating tasks from your own code, but it might also be able to replace the code that generates those tasks too.
Rabbit and celery is the backend system I have now. I was doing a lot of things on demand, event driven, but this resulted in overloading the database ( currently my system constraint ) or in elongating the queue too much. Some event would flood the queue with thousands of tasks. Now I have a periodic task that on each iteration chooses the most pressing task. And the system is stable but I have a bad backlog.
So I'd love to have a reactive task system where I didn't have to worry about the resource usage. Think how much precious developer time I've wasted just managing resource expenditures. Much of my coding this week has been just trying to optimize for this stupid constraint. How many processors do I have, how much disk io, is the memory getting close to swapping ?
Is it me or AWS is releasing too many services... there is a service for everything. I wounder if they are just throwing stuff out there see what sticks... kinda like landing pages.
To be fair, unlike google they are not cancelling any services. I think Amazon first builds these services for internal use, and then releases them to the public.
If you listen to Amazon at their shin-digs, they tell it like all of these new services and features are in response to requests from existing customers.
Claiming that your code is a 'lambda function' makes it sound sexy, but.. isn't it really just a procedure? Unless I'm missing something and there is some higher-ordered capability for composing AWS lambda functions together in a way that permits the platform to perform lambda reductions or optimize resource allocation...
Lambda function, not lambda function. Capitalization is key (the former is just a brand name).
Having reductions and composition would be really cool, but from what I could glean from the demos each function is containerized. Stateless and idempotent like lambda functions, but right now chaining more than one together requires an intermediary event.
I'm excited for this. This replaces what I wanted to use SQS for.
SQS always felt like too much vendor lock-in to me to justify not using something like RabbitMQ or Beanstalkd.
With Lambda, the resource consuming the queue is managed for me - that is huge. Also, the pay by time model is perfect for this - instead of managing when to stop/start resource intensive instances, I don't even have to think about the problem. I only get charged what I used, which can be crucial during growth stages of a business or prototype.
The big penalty is the vendor lock-in, but this tips the scales for me for certain things.
> You can use any third party library, even native ones. [1]
I realize they are starting with node.js, but I wonder how this will work? It sounds like they plan to support arbitrary dependencies. Will you upload a Docker container with your necessary dependencies? They talk about milliseconds until your code is running, and containers may be (?) slower than that. Or am I hoping for too much.
It's definitely possible to do this if you're not set on containers.
We do it at http://wym.io/ by putting our runtime library in the environment that the code runs in and keeping the runtimes around, keeping the environment clean along the way.
We're working towards doing all that (and more than) Lambda does in a deploy-your-own fashion...
One difficult part of doing event triggered processing is in the progress reporting and keeping the code related to it simple. I wonder how they deal with that.
I'd pretty much given up on AWS for compute and moved most everything to Linode and some bare metal servers, but this service looks very compelling for discrete compute tasks.
The ability to pay only for the fractions of seconds actually used, and the ability to scale quickly without provisioning (or over-provisioning) EC2 instances, is awfully attractive.
Plus, Amazon has priced this pretty aggressively — i.e., it looks shockingly cheap.
Good stuff. Basically it seems to do the bind+listen for you if you are the trigger subscriber. If you are the trigger generator, then it does socket.write for you. But the big deal is that you dont pay for 'listen', just pay for the function execution.
The one thing that will surely happen with this is that the code written will be 'locked in' to run only on aws territory.
Agreed. However, Amazon's pricing and rather complicated infrastructure means that there will always be a market for people like the DigitalOcean's of the world (I just want a cheap VM, now).
This looks quite interesting, and a lot more fun to work with than maintaining a pool of servers ready to handle events and spinning up new ones based on capacity.
Anyone know of any similar mechanisms for the OpenStack world, or more generally for any cloud infrastructure other than AWS?
Similar, yes. We're excited that developers are seeing the power of the task/worker/lambda being the scalable unit of work rather than server or VM or even container.
Haven't used Lambda much yet but off hand we see many IronWorker advantages:
- Supports all languages
- Easily scale to thousands of parallel workers
- Integrated logging, error reporting, alerts, auto-retries on failure
- Integration with IronMQ our queue service to achieve stream processing and pipelining
- A great dashboard to manage your workers/tasks
And many innovations/announcements coming soon that'll make using IronWorker a great choice for developers both on AWS and inside the enterprise firewall.
Yes, seems like a direct competitor to IronWorker to me.
IronWorker has a few benefits such as being able to run any docker container and react to a HTTP Post (webhook) directly, but overall very similar to Lambda.
You can also configure your IronWorker more so than you can a lambda. For instance, I have an IronWorker that requires a library not installed by default, so I use the "deb" command to install it during the build process.
And of course, it supports more than just Node.js.
It sounds like this is essentially what Joyent's Manta is, which we've been using in production for the last year and have found to be absolutely fantastic. Are there differences that I'm not seeing?
How is it different from Google App Engine? Conceptually the two seem very similar to me, that is, developers do not have to worry about the underling infrastructure at all---just write code and deploy.
As a tldr, not-quite-right distinction, GAE is more of a platform whereas Lambda is more of a service.
I feel that GAE is a much more generalized solution for building/deploying applications, e.g. it's very integrated with BigTable and has caching/cron/queuing support. To a point you do not have to worry about the infrastructure, but write any GAE app at scale and soon you'll run into questions of caching some content and optimizing DB writes.
Whereas Lambda, at least right now, is a more specific use case: respond to an event by running some code. It's too early to tell whether managing Lambda functions will require fine-tuning the "advanced settings" (in all honesty it probably will) but it seems like it's much simpler.
Haven't had much time to read the docs. Sorry if it's already evident, but does it allow for running Lambda code on cron as opposed to listening to some event?
AWS Lambda functions can be triggered by external event timers, so functions can be run during regularly scheduled maintenance times or non-peak hours. For example, you can trigger an AWS Lambda function to perform nightly archive cleanups during non-busy hours.
havent looked too much into it but I dont see why it couldnt. use s3 website, event notification to notify a user accessed a file (web page) and generates a new custom file in your bucket. generated file has some sort of preshared known info between the two in the file name (eg client ip address or cookie). accessed web page either sends back a page reload to the new generated page or just an async page that will render the custom content.
edit: oh also delete the generated file once accessed using lambda by either delete on get notification or just a lambda scheduled event.
Using Lambda for the slaves is probably not the best idea because, as @saynay mentioned, you couldn't run builds any longer than a few minutes.
You could "wrap" a Jenkins fleet running on EC2 with Lambda functions to set up on a build request event and tear down on a build finish event, and theoretically this would reduce the amount of resources you consume if your builds are few and far between. Think of GitHub webhooks triggering Travis CI as an analogy.
Whether or not that would actually be useful is debatable.
It looks like Lambda functions are limited in complexity and runtime duration.
The way I see it is pairing it with the container service (ECS) and spawning containers in response to events, possibly spawning EC2 instances if you need more computer.
It looks like Lambdas can also be set to execute at a scheduled time, so you could make a delayed Lambda that would shut down any EC2 instances you spawned as well.
Huh? Shared server infrastructure? That's really what this sounds like. Welcome to web hosting in 1999 guys. Most of the point of AWS was that you have your own dedicated resources. Sure, this is a scaling solution, but revolutionary?
What about dependencies. What if you need a specific environment setup first in order to process. Would you end up paying 1 minute for each request just so that it can start installing bunch of stuff? Is it possible to just setup a VM of some sort and use that environment each time?
If thats possible Lambda would be like PiCloud but without Python, and will stick around (hopefully).
jedberg|11 years ago
I also put together the Netflix use cases in the keynote so if you have any questions I’ll try to answer them!
joshuahutt|11 years ago
I think it would be really awesome to make applications that do not have any dedicated instances associated with them, but are entirely based on the various AWS platforms.
For some things, it seems like Lambda could be a drop-in replacement, whereas others, it would not quite fit the bill. Do you think it will be possible to build a viable a "server-free" application with Lambda and the other services (S3, DynamoDB)? Or is it probably going to be limited to the types of use cases listed on the landing page[1]?
[1] http://aws.amazon.com/lambda/
turingbook|11 years ago
namuol|11 years ago
epowell101|11 years ago
jvandyke|11 years ago
MBlume|11 years ago
(figured people would want to know this, and you have to scroll a ways to find out, so)
netcraft|11 years ago
pkpp1233|11 years ago
carson|11 years ago
jakozaur|11 years ago
The jokes ppl make are starting to get a bit more real: https://www.destroyallsoftware.com/talks/the-birth-and-death...
notastartup|11 years ago
makmanalp|11 years ago
vmarsy|11 years ago
From the FAQ :
"At launch AWS Lambda supports code written in Node.js (JavaScript). Your code can include existing Node.js libraries, even native ones."
so it seems that you can avoid lock-in if most of your javascript code is not too coupled to Amazon's API
And they claim on the detail page:
"With AWS Lambda, there are no new languages, tools, or frameworks to learn. You can use any third party library, even native ones. At launch, AWS Lambda will support Node.js code, with support for other languages coming in the future."
[1] : https://news.ycombinator.com/item?id=8603059
orange_sharpie|11 years ago
pnathan|11 years ago
hatred|11 years ago
abakker|11 years ago
logicallee|11 years ago
Corrado|11 years ago
mathnode|11 years ago
HowardMei|11 years ago
spitfire|11 years ago
But kudos for Amazon for furthering the datacenter-is-the-computer approach. It is simply the right thing to do.
yzzxy|11 years ago
mehh|11 years ago
erichmond|11 years ago
duaneb|11 years ago
ghshephard|11 years ago
Amazon has a LOT of services, and the better the naming scheme they use, the easier it will be to remember what they each are. I'm genuinely happy that they used a name that will make it trivial to map the function (heh) it serves in my head.
marknadal|11 years ago
erichmond|11 years ago
cperciva|11 years ago
ncallaway|11 years ago
tomcart|11 years ago
derefr|11 years ago
It's a bit amusing to note that they could have built Lambda on top of their own previous Elastic Beanstalk Docker offering instead, but chose to build ECS presumably for cost reasons (which everyone who has evaluated Elastic Beanstalk's Docker support will understand.)
mathgladiator|11 years ago
numlocked|11 years ago
I guess I would have expected others to describe this the same way ("replaces your distributed task queue"), but since I'm not seeing that description I wonder if I've misunderstood.
jberryman|11 years ago
crucialfelix|11 years ago
So I'd love to have a reactive task system where I didn't have to worry about the resource usage. Think how much precious developer time I've wasted just managing resource expenditures. Much of my coding this week has been just trying to optimize for this stupid constraint. How many processors do I have, how much disk io, is the memory getting close to swapping ?
I for one welcome our new cloud overlords.
hackerews|11 years ago
Will definitely be interesting to see how Lambda actually works.
Try out Blockspring, you can play around with your usecase now (and no vendor lock-in).
dk8996|11 years ago
poxrud|11 years ago
jrmcauliffe|11 years ago
jameshart|11 years ago
willyg302|11 years ago
Having reductions and composition would be really cool, but from what I could glean from the demos each function is containerized. Stateless and idempotent like lambda functions, but right now chaining more than one together requires an intermediary event.
gbrits|11 years ago
nightpool|11 years ago
debaserab2|11 years ago
With Lambda, the resource consuming the queue is managed for me - that is huge. Also, the pay by time model is perfect for this - instead of managing when to stop/start resource intensive instances, I don't even have to think about the problem. I only get charged what I used, which can be crucial during growth stages of a business or prototype.
The big penalty is the vendor lock-in, but this tips the scales for me for certain things.
crescentfresh|11 years ago
Are you referring to kinesis as the queue here? Sqs/sns are not (yet?) supported as event generators for lambda.
turingbook|11 years ago
cschmidt|11 years ago
I realize they are starting with node.js, but I wonder how this will work? It sounds like they plan to support arbitrary dependencies. Will you upload a Docker container with your necessary dependencies? They talk about milliseconds until your code is running, and containers may be (?) slower than that. Or am I hoping for too much.
[1] http://aws.amazon.com/lambda/details/
tekacs|11 years ago
We do it at http://wym.io/ by putting our runtime library in the environment that the code runs in and keeping the runtimes around, keeping the environment clean along the way.
We're working towards doing all that (and more than) Lambda does in a deploy-your-own fashion...
amelius|11 years ago
ColinCera|11 years ago
The ability to pay only for the fractions of seconds actually used, and the ability to scale quickly without provisioning (or over-provisioning) EC2 instances, is awfully attractive.
Plus, Amazon has priced this pretty aggressively — i.e., it looks shockingly cheap.
jlrubin|11 years ago
rajatchopra|11 years ago
unknown|11 years ago
[deleted]
dkarapetyan|11 years ago
Karunamon|11 years ago
rodgerd|11 years ago
tekacs|11 years ago
JoshTriplett|11 years ago
Anyone know of any similar mechanisms for the OpenStack world, or more generally for any cloud infrastructure other than AWS?
hendry|11 years ago
I HAVE THE POWER
https://twitter.com/kaihendry/status/532783437225025537
hcarvalhoalves|11 years ago
unknown|11 years ago
[deleted]
adelevie|11 years ago
carimura|11 years ago
Similar, yes. We're excited that developers are seeing the power of the task/worker/lambda being the scalable unit of work rather than server or VM or even container.
Haven't used Lambda much yet but off hand we see many IronWorker advantages:
- Supports all languages - Easily scale to thousands of parallel workers - Integrated logging, error reporting, alerts, auto-retries on failure - Integration with IronMQ our queue service to achieve stream processing and pipelining - A great dashboard to manage your workers/tasks
And many innovations/announcements coming soon that'll make using IronWorker a great choice for developers both on AWS and inside the enterprise firewall.
sync|11 years ago
IronWorker has a few benefits such as being able to run any docker container and react to a HTTP Post (webhook) directly, but overall very similar to Lambda.
impostervt|11 years ago
And of course, it supports more than just Node.js.
unknown|11 years ago
[deleted]
james33|11 years ago
nivertech|11 years ago
riobard|11 years ago
willyg302|11 years ago
I feel that GAE is a much more generalized solution for building/deploying applications, e.g. it's very integrated with BigTable and has caching/cron/queuing support. To a point you do not have to worry about the infrastructure, but write any GAE app at scale and soon you'll run into questions of caching some content and optimizing DB writes.
Whereas Lambda, at least right now, is a more specific use case: respond to an event by running some code. It's too early to tell whether managing Lambda functions will require fine-tuning the "advanced settings" (in all honesty it probably will) but it seems like it's much simpler.
hyperliner|11 years ago
luminati|11 years ago
numlocked|11 years ago
Scheduled Tasks
AWS Lambda functions can be triggered by external event timers, so functions can be run during regularly scheduled maintenance times or non-peak hours. For example, you can trigger an AWS Lambda function to perform nightly archive cleanups during non-busy hours.
impostervt|11 years ago
impostervt|11 years ago
Now, how to deal with node packages that require libraries not on the default amazon linux ami?
motoboi|11 years ago
nivertech|11 years ago
epowell101|11 years ago
Zaheer|11 years ago
Similar single event -> function service
jbaudanza|11 years ago
tekacs|11 years ago
http://wym.io/
Incidentally I almost had a heart attack when I saw the Lambda announcement. :P
[1]: We also provide the ability to respond to other kinds of events as does Lambda.
saranagati|11 years ago
edit: oh also delete the generated file once accessed using lambda by either delete on get notification or just a lambda scheduled event.
dominotw|11 years ago
willyg302|11 years ago
You could "wrap" a Jenkins fleet running on EC2 with Lambda functions to set up on a build request event and tear down on a build finish event, and theoretically this would reduce the amount of resources you consume if your builds are few and far between. Think of GitHub webhooks triggering Travis CI as an analogy.
Whether or not that would actually be useful is debatable.
saynay|11 years ago
The way I see it is pairing it with the container service (ECS) and spawning containers in response to events, possibly spawning EC2 instances if you need more computer.
It looks like Lambdas can also be set to execute at a scheduled time, so you could make a delayed Lambda that would shut down any EC2 instances you spawned as well.
davidw|11 years ago
Apoplectic|11 years ago
ceejayoz|11 years ago
Sorry, where were you getting servers in 1999 that existed for fractions of a second?
salimmadjd|11 years ago
GIFtheory|11 years ago
yohanatan|11 years ago
squidcactus28|11 years ago
sly010|11 years ago
waitingkuo|11 years ago
epowell101|11 years ago
notastartup|11 years ago
If thats possible Lambda would be like PiCloud but without Python, and will stick around (hopefully).
unknown|11 years ago
[deleted]