Running FFmpeg on AWS Lambda for 1.9% the Cost of AWS Elastic Transcoder

[mncharity]

Similarly, you can compile ffmpeg on Lambda, in 0.5 minutes, for 9 cents.[1] Versus 10 min on one core, for ~free. And while -j200 of ffmpeg is nice, -j1000 of the linux kernel is... wow, like seeing the future.

[1] demo in a talk: https://www.youtube.com/watch?v=O9qqSZAny3I&t=55m15s (the actual run (sans uploading) is at https://www.youtube.com/watch?v=O9qqSZAny3I&t=1h2m58s ); code: https://github.com/StanfordSNR/gg ; some slides (page 24): http://www.serverlesscomputing.org/wosc2/presentations/s2-wo...

[adtac]

This is impossibly amazing to me! Thank you so much, what an excellent lecture.

[benmanns]

Very cool. I hope more of the "value add" stuff from cloud providers ($$$) can be replaced with open source running on their cloud functions. My suggestions:

* FFmpeg supports http/https as input protocols if compiled with the options enabled. See `ffmpeg -protocols`

* You can parallelize or chunk FFmpeg to enable longer inputs, e.g. I found https://github.com/nergdron/dve/blob/9f1ca516b18f50d1d99d15e...

* Try with larger memory sizes. Larger memory = more CPU for Lambda which may result in shorter transcodes. You might even pay the same amount if the transcodes are CPU bound and finish in roughly linear time wrt CPUs

[akvadrako]

This is not "cool" - this is either doing less than Amazon's encoding service or just exposing the pricing model. If it's cheaper they could charge less, unless they are terrible programmers who can't even use their own lambda functions.

[kwillets]

Next installment: "Running FFmpeg on a tower under my desk for 1% the cost of AWS Lambda".

Is there any effort to on-prem lambda stuff yet? I know it's a moving target, but I wouldn't recommend getting into cloud stuff you can't migrate out of.

[hangtwenty]

Yes! If one uses something like Kubeless[1], you have something like AWS Lambda, but where the backend is Kubernetes rather than AWS. Yes you're still on a framework, but it's a vendor-agnostic, open-source one. There are some other attempts at similar things, too. I am partial to this one for now.

[1]: https://github.com/kubeless/kubeless

[checker]

At a pure level, Lambda "integration" is essentially an interface for passing in runtime arguments+environment to your application. If you're concerned about lock-in, you can write or integrate another entry point that executes your function on on-prem Tomcat or whatever other environment (Cloud Functions?) you want to run in.

The much harder challenge would be provisioning thousands of on-prem servers to handle the load, but I wouldn't necessarily qualify a dependency on cloud-like autoscaling as lock-in.

I guess the lock-in might come as you unwittingly couple yourself to the intricacies of the rest of AWS's offerings as your app architecture grows more complex.

[ty_a]

Well there's iron.io (which just got bought by Oracle) and there are several FaaS projects for Kubernetes such as: OpenFaaS and Kubeless.

Edit: OpenFaaS can run on Nomad as well apparently: https://www.hashicorp.com/blog/functions-as-a-service-with-n...

[polskibus]

https://openwhisk.apache.org/ backed by IBM AFAIK, but not only.

[2RTZZSro]

Next installment: "Running a private cloud under my desk for 1% the cost of AWS".

[yaronhaviv]

check: https://github.com/nuclio/nuclio high-perf serverless runs over (standalone) docker or Kubernetes or Cloud services

[buildbuildbuild]

Their tool which facilitates the packaging and relocation of dynamically linked binaries is interesting: https://github.com/intoli/exodus

"Painless relocation of Linux binaries–and all of their dependencies–without containers."

[therein]

Yeah, seriously. This sounds great.

Also, if you found exodus interesting, you may find the following interesting too.

https://github.com/endrazine/wcc

[athrun]

Wow! Thanks for flagging this. This tool has so many useful applications; mind blown.

[pjc50]

Yes, I think this is the buried lede - that could be incredibly useful.

[maxk42]

Very misleading title. Elastic Transcoder pricing applies primarily to video. This tutorial only covers the audio transcription which is much, much less resource-intensive.

[Johnny555]

Nothing misleading here, they compared their project's cost to the Elastic Transcoder audio pricing.

Elastic Transcoder Audio is $0.00450 per minute [1], this article says that with Lambda it cost "$0.00008273 per minute of audio, a full factor of 54 times less than Elastic Transcoder.".

0.00450 / 0.00008273 = 54

[1] https://aws.amazon.com/elastictranscoder/pricing/

[jwildeboer]

Misleading title. Article is about audio encoding, not video. Better: “Using FFmpeg on AWS Lambda for audio encoding at 1.9% of cost for AWS Elastic Transcoder”

[taion]

What's the benefit of using Exodus over just using the official-ish static builds of FFmpeg? https://johnvansickle.com/ffmpeg/

This works just fine on AWS Lambda. The `ffmpeg` binary there weighs in at 46 MB. Unless you need something not bundled with that build, it seems like this is sufficient and is easier to set up.

[kwillets]

It looks like the youtube download has to complete before ffmpeg can start; is there a way to start processing the head while the tail is still being written?

This problem comes up a lot with storage blobs. The bigger they are the worse it is to serialize write/reads.

[RJIb8RBYxzAMX9u]

I don't see why not with I/O redirection, or named pipes. It probably wasn't done for simplicity.

[throwaway2016a]

This is great. I wouldn't have thought to run FFmpeg on Lambda.

I'm going to stick with Elastic Transcoder for now though. I like that I have no upgrades to maintain and very little code. I feel like if I did this, it would take me years to recoup the cost even with a 99% savings.

But that is only because I only have a few videos a month. Roughly $1.00 on Elastic Transcoder. If I had thousands or even hundreds of videos this seems like a great and worthwhile project. Especially since this article appears to take a lot of the trial and error and proof of concept out of the mix.

I worked for a large Internet company that had a Netflix like product back in 2007. The transcoders were literally just plugged in underneath people's cubicles. Kept things nice and warm in the winter and I'm sure the costs were pretty low.

[jordan314]

Cool but only for up to 8 minute videos. Unless you found a way to parallelize the lambda tasks.

[pfg]

I'm far from an FFmpeg expert, but I believe it's possible to segment the input video, transcode the segments one by one, and then concatenate them. Not sure how the segmentation and concatenation steps perform, but if that's fast, this might even improve your overall transcoding speed due to the parallelization.

[wmf]

http://ex.camera/nsdi17/

https://www.usenix.org/conference/nsdi17/technical-sessions/...

[lozenge]

Doubling the memory allocation for a lambda task also doubles the CPU allocation. It's possible it could go much faster than this.

[lostmsu]

Don't you also have to pay for traffic going to/from lambda? In that case raw audio and video would be very expensive!

[celerity]

Traffic to Lambda from the internet and between Lambda and S3 is free. The only thing you pay for are the transfer costs from S3 (at cents per GB).

[chatmasta]

What if you use an S3 upload hook? Are you charged for bandwidth S3 -> lambda? Also, wouldn't you use the same bandwidth with elastic transcoder anyway?

[tyrankh]

When I worked at Panasonic we did this exact thing. It's remarkably easy for the cost savings.

[moonbread]

I used FFmpeg static build to transcode WAV to mp3, but the latest 64-bit build gave me corrupt files, so I had to hunt down an archived version. Works well though!

[sp332]

Is it downloading the whole Youtube video just to pull out the audio? Why not just download the audio to begin with?

[archgoon]

What is the API to download only the audio from a youtube video? How would you do your proposed solution?

[rhizome]

Because it's in the video container.

[marta_moreno]

Yes well, now you just have to pay the fee for licensing the codecs FFmpeg gives you ;). What was it? One million dollars for MP4? Good luck with that.