Ask HN: Managing tons of parallel uploads on upload heavy site

[mrvir]

How to handle lots of parallel file uploads? Is Apache up to the task or are the better options? Any recommendations about multi-file vs single file at time uploading from transfer reliability point of view.

Html forms are cross-platform solution, but seeing many sites to use Flash, ActiveX, or Java for more functionality. Tips or articles about client side also very much appreciated.

[devmonk]

I'm not an expert on scaling uploads, but here's what I'd say off-the-cuff:

1. Try to save time on the UI if possible by reusing existing solutions for multi-file uploads: https://encrypted.google.com/search?q=upload+javascript+plug...

2. Queue up uploads so that users don't start uploading until the server(s) can take them.

3. Delay any processing of the uploaded files.

4. Throw some more cheap servers, additional VMs at the problem, or more dynos/workers if using Heroku, etc.

5. Ensure your pipes are large enough.

6. Or just use Amazon S3, CloudFiles, etc.:

- http://aws.amazon.com/articles/1434

- http://www.rackspacecloud.com/cloud_hosting_products/files

but this may be much higher burn rate, depending on your needs. See what people have written: https://encrypted.google.com/search?q=amazon+s3+alternatives

Scaling is difficult. Don't scale until you need to, because it gets expensive (higher burn rate is bad).

If you really have that many uploads, you may be ready to hire someone that has some proven experience in it, not just helping develop an application that was being scaled by someone else, but someone who was hands on, and preferably someone with recent experience working with S3, etc.