Ask HN: Does anyone else find the AWS Lambda developer experience frustrating?

[tdfirth]

Hey HN,

I've been using AWS lambda a bit recently, mostly as a way to glue together various bits and pieces.

Maybe I'm doing this wrong but does anyone else find the experience to be really frustrating?

I can unit test bits of the code just fine, but at some point I always end up stuck in a slow feedback loop where I deploy the code, do some manual invoking, go and dig through the logs in CloudWatch, add another print statement in my lambda... and so on.

What I want is to run the lambdas locally, ideally more than one, and then exercise them with streams of test events (perhaps captured from a real environment). It would be quite cool if I could define BDD style tests around them too.

Anyone have any suggestions or share my frustrations?

I have heard localstack is quite good although I haven't given it a go yet. Would that work for me? I did try SAM but I was a bit underwhelmed and I don't want to use a separate IaC tool for these.

Alternatively, do other FaaS providers solve this problem?

Thanks for any help.

[jiggawatts]

You've discovered what many other people have: The cloud is the new time-share mainframe.

Programming in the 1960s to 80s was like this too. You'd develop some program in isolation, unable to properly run it. You "submit" it to the system, and it would be scheduled to run along with other workloads. You'd get a printout of the results back hours later, or even tomorrow. Rinse and repeat.

This work loop is incredibly inefficient, and was replaced by development that happened entirely locally on a workstation. This dramatically tightened the edit-compile-debug loop, down to seconds or at most minutes. Productivity skyrocketed, and most enterprises shifted the majority of their workload away from mainframes.

Now, in the 2020s, mainframes are back! They're just called "the cloud" now, but not much of their essential nature has changed other than the vendor name.

The cloud, just like mainframes:

- Does not provide all-local workstations. The only full-fidelity platform is the shared server.

- Is closed source. Only Amazon provides AWS. Only Microsoft provides Azure. Only Google provides GCP. You can't peer into their source code, it is all proprietary and even secret.

- Has a poor debugging experience. Shared platforms can't generally allow "invasive" debugging for security reasons. Their sheer size and complexity will mean that your visibility will always be limited. You'll never been able to get a stack trace that crosses into the internal calls of the platform services like S3 or Lambda. Contrast this with typical debugging where you can even trace into the OS kernel if you so choose.

- Are generally based on the "print the logs out" feedback mechanism, with all the usual issues of mainframes such as hours-long delays.

[andreineculau]

I've been in the AWS world for 4+ years now and my immediate feedback is don't run any local emulators. None! Write unit tests for the internals and test your system end-to-end. I say that both because AWS services can have unpredictable behavior that you need to account for but also because local emulators are at best functional, but in reality far from emulating the AWS world on a 1:1 scale (especially the unpredictable behaviors i mentioned). So instead optimize for many local unit tests and many live end-to-end tests (implies many deployments and parallel environments prod, staging, dev, etc)

When it comes to Lambdas, given the reasons above, there's only one thing that can improve the experience: PROXY. Before i went on parental leave i had the idea is creating a proxy lambda which can be configured with an IP and port number. That IP and port is for your local dev environment. This way, when developing you can instruct a live system to short-circuit and to proxy calls to a local lambda available om the respective port. Trigger end-to-end tests by invoking AWS services that will eventually call the proxy lambda, and then your local lambda with the same environment/context/input, reply with output which will reach the proxy lambda, which will output the same content forward.

[iends]

"Think of the history of data access strategies to come out of Microsoft. ODBC, RDO, DAO, ADO, OLEDB, now ADO.NET – All New! Are these technological imperatives? The result of an incompetent design group that needs to reinvent data access every goddamn year? (That’s probably it, actually.) But the end result is just cover fire. The competition has no choice but to spend all their time porting and keeping up, time that they can’t spend writing new features. Look closely at the software landscape. The companies that do well are the ones who rely least on big companies and don’t have to spend all their cycles catching up and reimplementing and fixing bugs that crop up only on Windows XP." - Fire And Motion, Joel on Software

In my own work with serverless I think about this a lot. In my day job, we built a new service and we pay AWS a few dollars per month to run. It would have cost us around $100 a month in AWS costs. However, the operational complexity is extremely high and we are also coupled to another service via Kinesis. For a billion dollar business, the trade off doesn't seem worth it.

I wonder how much of serverless is just AWS firing at Google (and Heroku, DO, etc) and other competitors. It certainly hasn't made my life as a developer easier. It's certainly much cheaper for some use cases but the complexity of the system goes up very quickly, and you're end up having to manage a lot of that complexity using an inferior tool like Cloud Formation (or Terraform).

[orf]

> I wonder how much of serverless is just AWS firing at Google (and Heroku, DO, etc) and other competitors.

It depends on what you mean by "serverless". Serverless webapps? Sure, maybe a fair bit. But the real killer feature of the Lambda service is that it's a universal extension mechanism for other AWS services. Want to run some shoddy virus scanning whenever an object is added to a bucket for compliance reasons? That's a lambda. Want to manipulate Firehose records before they are sent to the sink? That's a lambda. Want to execute some crazy Python function from an Athena SQL query? You guessed it. That's a lambda.

So when you say "It certainly hasn't made my life as a developer easier" it means you haven't needed to do these bespoke gluing together of stuff, because "being able to do this somehow" is infinitely easier than "not being able to do it at all because AWS hasn't exposed an API for this use case".

[mattmanser]

It's like the SPA craze, microservices craze, NoSQL craze or similar things.

Appropriate for a small amount of apps, but used inappropriately by a lot more for a while because it's the hot new thing.

[rualca]

> It's certainly much cheaper for some use cases but the complexity of the system goes up very quickly, and you're end up having to manage a lot of that complexity using an inferior tool like Cloud Formation (or Terraform).

The complexity bogeyman is a red herring. What is software development if not a constant fight between increasing complexity and adding features, including making systems more robust and resilient and reliable?

Sure, you are free to claim that function-as-a-service paradigm is very complex because the code runs somewhere else, and you have to use a separate workflow than the one you're used to, and that you actually have to write unit tests to have some assurances that stuff does work as expected.

But what would be the alternative approach?

Would it be simpler to write a full-blown service that polls events and listens to message queues and launches background tasks? Obviously, no.

AWS Lambda is a fancy way to let developers develop event-driven systems by just writing tiny event handlers and plug in events. It's no rocket science. In terms of complexity, they pale in comparison with even the tiniest hello world windows GUI app, or even Qt. I'm terms of web services, they are a service where you just have to write the request controller bit after that runs after all auth things passed and request messages were parsed and validated. Considering this, isn't it unreasonable to talk about increasing complexity, when everything was made.far simpler than what it would otherwise be?

[anfrank]

SST is an Open-Source framework worth checking out if you are looking for a lighting fast local dev environment for serverless. https://github.com/serverless-stack/serverless-stack

I often see ppl adopt one of two development patterns:

1. Locally mock all the services that your Lambda function uses. Like API Gateway, SNS, SQS, etc. This is hard to do. If you are using a tool that mocks a specific service (like API Gateway), you won't be able to test a Lambda that's invoked by a different service (like SNS). On the other hand a service like LocalStack, that tries to mock a whole suite of services, is slow and the mocked services can be out of date.

2. Or, you'll need to deploy your changes to test them. Each deployment can take at least a minute. And repeatedly deploying to test a change really slows down the feedback loop.

SST lets you develop your Lambda functions locally while connecting to others AWS services without needing to mock them locally. Behind the scene, the lambda requests are streamed to ur local, runs ur local function, response streamed back to Lambda. So you can iterate on ur function code without redeploying.

Think of your Lambda functions are Live Reloaded. You can see it in action here — https://youtu.be/hnTSTm5n11g

I’m the core maintainer of the project. And folks in our community are saying using SST feels like "working in the future" ;)

I'd love for you to give it a try. We are also a part of the YC W21.

[davmar]

I use this companies services (seed.run as well). Frank and Jay are doing something truly special in this space.

Also, this guide they put together is definitive: https://serverless-stack.com/#guide

Had I found that years ago, I would have saved SO MANY headaches. I've implemented local debugging (hell, and doesn't actually replicate AWS). I've followed the CloudWatch logs, just like you, painful.

The SST local debugging with lambda is the best way forward. Deploying micro services on seed.run is also the way forward.

[tdfirth]

Hey, thanks for the link. It does look quite good - there was another comment where someone had a home-brewed set up that did something fairly similar.

I'll check it out!

[superchink]

I second this. SST is like magic.

[windowshopping]

I find all of AWS frustrating. Their interfaces feel like they were designed by an 80s IBM analyst.

[ufmace]

I use them heavily for work. IMO, they're pretty well designed for managing large numbers of servers with sophisticated coordination and management systems. If you just wanna run a single server, it's kind of ridiculously over-complex. IMO, Digital Ocean, Linode, etc have much better workflows for just spin up a single Linux server attached to this DNS and let me set everything up on it.

[dragonwriter]

> Their interfaces feel like they were designed by an 80s IBM analyst.

The purpose of the AWS Console, AFAICT, is:

(1) To be used by semi-technical ops and management types, and people doing tutorials, and

(2) To drive everyone else as quickly as possible to the APIs/CLIs/SDKs.

[tdfirth]

This made me laugh, thanks.

If I'm honest, I do find AWS in general fairly high friction at times but it's way better than renting servers directly I think.

It's the only cloud provider I've really worked with to be honest though (other than a few experiments), maybe some others are much better in this regard?

[picardo]

There is some truth to that. They are designed to serve a number of personas in an organization, so if you just need to do one thing or another, they will seem overly complex, but I think you could say the same thing about a cockpit or any interface for a complex system.

[cpach]

Haha! Funny, and there is probably a kernel of truth in there.

I’ve been thinking a lot about AWS lately and it definitely seems to me like a modern take on what IBM did in the 80’s.

[vangelis]

I'm amazed that people use the console for anything other than discovery. Any heavy lifting you want the AWS CLI or CloudFormation.

[bamboozled]

Ultra slow too, logging into the console is one of my least favourite things to do.

[StratusBen]

Time to check out https://vantage.sh/ :)

[fookyong]

I was completely lost with AWS lambda until I discovered https://serverless.com - the fact that you haven't found them yourself suggests they need to do a much better job of marketing themselves!

I run my lambdas locally with a single command: serverless offline

If the lambda has an http endpoint, it creates the endpoint at localhost/<endpoint> and I'm good to go, it even does live reloading as I edit my code.

If the lambda runs off AWS events, I can invoke the lambda locally with a command, and point it to a JSON file of a simulated AWS event. I get my local rails app to create these AWS event JSON files, so that I can test end to end locally. Works well for my purposes.

To deploy I just run: serverless deploy --stage production

Which sets up all the necessary additional services like API Gateway, cloudwatch etc.

I can't imagine using AWS lambda any other way.

[innomatics]

I can also recommend serverless.com framework for local development and running lambdas offline.

AWS provides a docker image for emulating DynamoDB which works great for local dev and will commonly be paired with lambdas.

Another option I have used recently for implementing node web services is https://github.com/apex/up which also has a nice local dev experience.

[tdfirth]

Thanks for the recommendation, that does sound good and I'll give it a go.

[wharfjumper]

+1 for serverless offline

[astashov]

I recently migrated from Cloudflare Workers to AWS Lambda + Dynamo for my relatively large pet project.

It was surprisingly hard - the local development of lambdas is still very raw, documentation is scarce, and there are various small issues appearing here and there. I should probably write a blogpost about how to setup decent developer environment for AWS CDK and Lambdas, because there's not much on the Internet about it.

I set up the whole AWS infrastructure via AWS CDK. I have one TypeScript file, that creates a stack with lambdas, dynamodb tables, API gateways, S3 buckets, wires up the secrets manager, etc - all of that in 2 environments - dev and prod.

AWS CLI also can generate a Cloudformation YAML file from the CDK file (via `cdk synth`), which could be fed into SAM. So, I generate a `template.yaml` Cloudformation file this way, then run SAM like `sam local start-api`, and it runs exactly the same lambda locally, as in AWS, using that Cloudformation YAML file. SAM also supports live reload, so if you change any source files, it will automatically get those changes.

So, okay-ish developer experience for lambdas is possible. There're caveats though:

* I couldn't figure out how to use "nameless" Dynamodb tables (i.e. when CDK assigns the name to them automatically), because if I omit a table name in CDK template, then the local lambda and lambda in AWS assume different names for some reason.

* Locally, the binary outputs don't work. It ignores `isBase64Encoded` by some reason, and lambda just returns Base64 output, instead of binary.

* And the main problem - local lambdas are SLOW. It seems like it restarts some container or something under the hood on each API call, which adds 2-3 seconds to each API call. So, the calls that should be like 30ms, are actually 2 seconds now. This is super frustrating.

[vinnymac]

I could have written your message word for word. I recently had a very similar experience moving to AWS Lambda using AWS CDK and SAM. It is pretty shocking how unhelpful the AWS documentation can be at times.

Each use case is so unique, that it is likely difficult for them to account for every single situation where Lambdas are being used. Instead they opt for a generalized and non-specific documentation site, which results in docs I find practically useless.

I frequently find solutions to problems for AWS on someone's obscure blog, who after many days of suffering stumbled onto the answer, rather than finding anything of use directly from those who wrote the software.

Does a site exist where programmers can submit useful findings? Similar to stackoverflow, but rather than asking for questions, you just submit answers. I would think being able to search across such a crowdsourced utility would save programmers a ton of time.

[mkl]

What made you switch? How do Cloudflare workers and their development compare?

[css]

I have not had difficulty writing code for AWS Lambda. I just write everything locally and run the code by invoking the entry point with the proper event and context dictionaries. For debugging, I just attach a debugger and run the entry point like any normal script.

I don't know why you need to deploy to test Lambda code; you can hit remote AWS stuff from local. The AWS SDK picks up your local config's IAM role the same way as it picks up the one granted to the Lambda itself. You don't need localstack for this, just an account in your AWS organization with the right role attached.

Packaging dependencies was a little weird to figure out, but the docs [0] are very good. A simple shell script can do the packaging work; its just a few lines to make the zip file.

[0]: https://docs.aws.amazon.com/lambda/latest/dg/python-package-...

[msluyter]

Not a full solution, but when I was doing this I really got to love the awslogs utility:

https://github.com/jorgebastida/awslogs

It allows you to stream Cloudwatch logs from the command line, so you can grep them, save them to files, etc... (The web based Cloudwatch interface is terrible.)

Another suggestion is to try to modularize the core business logic in your lambda such that you separate the lambda-centric stuff from the rest of it. Obviously, though, if "the rest of it" is hitting other AWS services, you're going to hit the same testing roadblock.

Or you can try mocking, which may or may not provide much value for you. There's a python library for that, (moto), but it's not 100% up to date wrt AWS services/interfaces, last I had checked. Might be worth a try though.

https://github.com/spulec/moto

[wharfjumper]

awslogs was a massive productivity improvement for me - highly recommended

[endgame]

Suggestions:

1. If you are building APIs and using Lambda functions as targets from an API Gateway API, look into libraries like serverless-wsgi (Python) or wai-handler-hal (Haskell) that translate between API Gateway request/response payloads and some kind of ecosystem-native representation. Then as long as you're writing code where all state gets persisted outside of the request/response cycle, you can develop locally as if you were writing for a more normal deploy environment.

2. Look into the lambda runtime interface emulator ( https://github.com/aws/aws-lambda-runtime-interface-emulator... ). This lets you send invoke requests to a fake listener and locally test the lambda more easily. While the emulator is provided in the AWS container base images, you don't need to run it inside a container if you're deploying with zip files. (AWS-provided container images automatically enable the emulator if not running in a lambda runtime environment, and using docker for port remapping, which is nice but not at all required.)

3. Get really good at capturing all requests to external services, and mocking them out for local testing. Whether this is done with free monads, effect systems, or by routing everything through gateway classes will depend on your language and library choices.

[dmlittle]

If all you need is the ability to run a lambda function's code locally you might interested in docker-lambda[1]. I haven't really used localstack or SAM but a couple of years ago when we needed to run some lambda functions locally for development docker-lambda worked well enough.

[1] https://github.com/lambci/docker-lambda

[tdfirth]

Oh that does look quite interesting, I hadn't heard of that. Thanks for the tip.

As you say, it only does the 'host lambdas locally' bit though. If you wanted to do stuff like test the interactions between lambdas, or test them over HTTP (for lambdas that act as components in REST APIs), you'd need to build something yourself.

[sonthonax]

Don’t bother with Localstack, it’s a bunch of Docker images combined with Python’s Moto library.

AWS has Step Functions, which is a vaguely functional and declarative language for linking your lambdas together. It’s a bit verbose and annoying to write, but at least a it makes your lambada setup repeatable.

The state machine which actually runs your step functions is publicly available.

https://docs.aws.amazon.com/step-functions/latest/dg/sfn-loc...

[coredog64]

Seconded that Localstack is pointless. So much of what I want to do isn't supported and/or throws weird errors that are nothing like the AWS service.

[_6pvr]

I've found both AWS Lambda and all Azure serverless tooling to be extremely frustrating. "Just don't do local development" is the feedback I see a lot, which just seems bonkers to me. Like the orgs that don't run tests locally and just use CI to validate builds. If your workflow involves long periods of waiting to test even very basic functionality, it is (to me) totally broken.

We have containers which unify local environment and production environment. In my opinion (and experience), there aren't any more efficient shared mediums to work with.

[kaishiro]

It sounds like you're committed to Lambda, but we've spent extensive time across both Lambdas and Google Cloud Functions, and my off the cuff, somewhat biased opinion is that GCF gives a significantly better development experience the Lambdas. I note biased because a majority of the work we do lives in RTDB/Firestore.

However, in response to your specific pain point - after initializing a new functions project you can immediately run an 'npm run serve' and have local execution firing for any scaffolded endpoints. These - by default - will connect to a project associated RTDB/Firestore instance without the need for additional service account/IAM configuration.

I've always enjoyed that this was a part of the mainline dev flow - and it removes a huge barrier of entry for me (mostly mental) to spin up little toy projects or test endpoints whenever the thought crosses my mind.

[wussboy]

This response surprises me. I find Google Cloud Functions to be orders of magnitude worse than Lambdas. Everything that you deal with in AWS with the added bonus that instead of taking 5 seconds to deploy it takes 5 minutes. I would love to know better options. How do you deal with that horrible deployment time?

[sharms]

I share these frustrations and don't have a great answer. When integrating N cloud services, at some point it gets incredibly hard to build out fast feedback / local environments. The best way I know how to deal with it is use less services and ensure the ones you stick with are simple.

[tdfirth]

I am not alone then at least! The particularly annoying thing is that I feel like N isn't even that high. Even 2 or 3 things interacting is a pain (unless it's stuff like RDS or just a plain old container of course).

[mozey]

I write Lambda functions in golang. In dev I run the function as local http server. In prod I map a custom domain to the lambda, routing is done internally to the function. API Gateway is only used for the top level route. This workflow is enabled by https://github.com/apex/gateway

In principle I try to avoid Amazon Web Services that lock me into the platform. So, for my dev stack I run a few other containers to give me the equivalent of the prod environment, RDS (MySQL or PostgreSQL), ES (OpenDistro), S3 (Minio), and SES https://github.com/mozey/aws-local

Dev is easy because I can test everything locally, and the resulting code should be portable to any other cloud.

Obviously this approach might not be feasible if you choose to use AWS that do not have equivalent self-hosted options.

[i_v]

The best solution I've found for local testing is the Lambda Runtime Interface Emulator (RIE)[1]. It's basically a little Go wrapper[2] that runs your handler and makes it callable through an HTTP API. I considered writing a test harness around this for convenient integration testing but in my recent evaluation, I ended up avoiding Lambda (it's been really nice for small one-off tasks but was too expensive for the most recent and somewhat fringe use-case I considered using it for).

[1]: https://docs.aws.amazon.com/lambda/latest/dg/images-test.htm... [2]: https://github.com/aws/aws-lambda-runtime-interface-emulator

[jmb12686]

That can be true if you are using tightly coupled event sources (like s3, SNS, etc) where you need to inspect the incoming object. If you are doing a HTTP / REST API, try to decouple the code as much as possible from the API Gateway invocation by using ExpressJS / aws-serverless-express to aid in local testing and debugging. Then testing and debugging locally becomes much easier.

[rezonant]

This is my preferred style, and since an express app is technically just a function that accepts req/res, you can pass a whole express app in as the function when running in the cloud, so you have multiple path routes, middleware, the whole nine. You can actually host a fairly complex API in a single function subject to the package size/dependency count restrictions of your FaaS of choice

[reilly3000]

There are a few ways to improve the feedback loop, like SAM, Stackery, and Serverless Framework. That said, containers on Lambda (or GCP Cloud Run) are where it’s at. Just code your code, run local unit tests, and push the container up. The most important thing is to clearly define a mock request from your invocation source as your first step of the project. There is a vast difference between SQS, ALB and API Gateway events that need to be accounted for, and trip up people on deployment. I usually have those in my tests folder for every event the code will handle, and build from there.

‘aws lambda invoke -e myevent.JSON’ goes a long ways, because you get the full response object back.

[fifthofeight]

Have you tried using serverless/serverless-offline? https://www.serverless.com/

[tdfirth]

I haven't tried it yet, no. It always struck me as being focused on making deployment quick/easy rather than on creating a good local dev workflow (i.e. the same workflow I currently have but a quicker cycle time). This never really appealed to me.

Cosmotic's comment kind of confirms my suspicions to be honest.

[anfrank]

It's quite limiting on: 1. It only mocks API Gateway, and not accurately (ie. IAM permissions). You also need to use a bunch of other similar plugins to mock DynamoDB, SNS, SQS, etc. 2. You need to change your code (wrapping your aws-sdk to connect to localhost) to use these mocking plugins.

I posted elsewhere in this thread but checkout SST (https://github.com/serverless-stack/serverless-stack), it lets’ you work on your Lambda functions live without having to mock AWS services locally. Here’s a short demo (https://youtu.be/hnTSTm5n11g)

[cosmotic]

Our team has; it's been very difficult. Our jenkins build agent (running as a container) runs integration tests that start a node+serverless container, that starts a localstack container, that starts a lambda contianer.