WingNews logo WingNews GitHub [2]
top | item 24351111

Ask HN: What is your favorite method of sending large files?

319 points| mettamage | 5 years ago

I just opened up a simple HTTP server to send someone a large file. Then I figured, I never gave this question proper thought.

But some of you have, and I figured they make for fun and interesting stories ;-)

So what's your favorite method to send large files, of at least 5GB or bigger? Though, I'm also curious on how you'd send 10TB or more.

354 comments

order
[+] livueta|5 years ago|reply
Bittorrent. No, really. Lots of nice behaviors when transferring large amounts of data between arbitrary endpoints.

Transferring the torrent metadata is pretty trivial and can be done via a wide range of methods, and having that flexibility can be nice.

Unlike HTTP, you get reasonable retry behavior on network hiccups. Also, more robust data integrity guarantees, though a manual hash test is probably a good idea either way.

At least among people I'm throwing TBs of data around with, torrent infra is common and it's nice to not have to deal with some special-purpose tool that, in practice, is probably a pain in the ass to get compatible versions deployed across a range of OSes. Basically every platform known to man can run a torrent client of some sort.

And obviously, no dependency on an intermediary. This is good if you're trying to avoid Google et al. That does, however, bring a potential con: if my side is fast and your side is slow, I'm seeing until you're done. If I'm uploading something to gdrive or whatever, I can disconnect one r the upload is done. If you control an intermediary like a seedbox, that's less of a problem.

In general, though, torrents are pretty great for this sort of thing. Just encrypt your stuff beforehand.

[+] arsome|5 years ago|reply
Just be careful - unless you mark that torrent private it'll get posted to the DHT and crawlers like BtDig will pick it up and list it publicly.

For this reason I prefer using something like Syncthing which is designed more with this purpose in mind.

[+] libertine|5 years ago|reply
Sometimes I wonder if the bad connotation due to piracy made this protocol not standard (in a sense, that's not widely adopted, and people need dedicated clients, instead of it working for example in browsers).

I don't know enough, maybe it has security issues, or some limits... but the fact is that it's still widely used, by tech literate people, so it still stands the test of time and it must be doing something right.

[+] myself248|5 years ago|reply
> Unlike HTTP, you get reasonable retry behavior on network hiccups

Seriously. Zmodem/resume was doing better in 1986.

[+] cryo|5 years ago|reply
I've written a cross platform gui tool to transfer arbitrary large files via p2p with end-to-end encryption.

https://cryonet.io

- Fully p2p, even the signalling server part which is done via Tor hidden services (later on there will be an alternative using a DHT).

- So unless the Internet or Tor goes offline it can never be down.

- Works with proxies and in corporate environments when UDP might be blocked.

[+] efreak|5 years ago|reply
I had to transfer 2tb of data over a wireless connection that kept dropping packets. BitTorrent was the only method I found that worked; everything else either broke down when it disconnected, or had too low of a retry limit to be useful, or wouldn't resume properly.
[+] mderazon|5 years ago|reply
Can you recommend easy software to do the creation? Don't you need a tracker as well?
[+] StavrosK|5 years ago|reply
BitTorrent is the best for sending large files. I wish there existed a tool to make torrent creation easier, and I also wish my computer could seed. Every time I create a torrent, nobody can discover my server, even though the port is open and even if I add the server IP as a peer directly onto the client.

I have no trouble seeding in swarms, only when I create torrents. Other than that, BT is ideal.

[+] hoorayimhelping|5 years ago|reply
>Just encrypt your stuff beforehand.

This is such a facepalm moment. "This service is awesome, it does everything you need, it's reliable, it's safe, it's practical, and all you have to do is encrypt your files, share encryption keys, validate identities, and establish trust with the person you're sharing with. Also if you screw it up in any way, your file is public forever. It's so rad!"

[+] geocrasher|5 years ago|reply
5GB on my local network: Windows File Share, Rsync, or HTTP depending on source/destination

5GB on Internet: Upload to my OVH VPS and HTTP or Rsync it to its destination

10TB, local or Internet: Physical Hard Drive.

Never underestimate the bandwidth of a station wagon full of backup tapes!

https://www.tidbitsfortechs.com/2013/09/never-underestimate-...

And since you now buy 1TB micro SD cards, so perhaps I'd split the file 11 ways (no way it'll fit exactly) and send them via carrier pigeon. Or heck, I could just tape them to a drone and hope they aren't damaged in the crash. There's lots of ways to move data around. Maybe you want to UUENCODE it and submit bad URL's to a servers log so that it can be exfiltrated later? It would probably take a very, very long time, but could be done. I call it "CURLyTP"

https://miscdotgeek.com/curlytp-every-web-server-is-a-dead-d...

[+] terramex|5 years ago|reply
10TB, local or Internet: Physical Hard Drive.

Never underestimate the bandwidth of a station wagon full of backup tapes!

In Poland there is an option to send package via train for 8$. Just go to station and give package to conductor/train guard, few hours later receiver can get it at destination station. Sending HDDs full of raw video footage this way is very popular among video editors here.

[+] austinheap|5 years ago|reply
OVH is great if it’s not business critical! As the semi-official safe haven of piracy in Europe, OVH’s network has uhh...seen better days.
[+] simonebrunozzi|5 years ago|reply
> Never underestimate the bandwidth of a station wagon full of backup tapes!

The exact quote [0] by Andrew Tanenbaum is: "Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.".

Obligatory xkcd [1].

I met Andy Tanenbaum in 2008 at Vrije Universiteit in Amsterdam. I was the Technology Evangelist for AWS in Europe, giving a seminar there on cloud computing.

I loved him. Funny, smart. I didn't even know he was American until I met him in person (having studied computer science, you're not exactly sure of someone's nationality solely based on his last name).

[0]: https://en.wikiquote.org/wiki/Andrew_S._Tanenbaum

[1]: https://what-if.xkcd.com/31/

[+] brippalcharrid|5 years ago|reply
You wouldn't want to keep all of your proverbial eggs in one basket. Split the file into a multi-volume archive and create a par2/par3 parity volume set. Distribute parts of the parity set and parts of the data archive amongst m pigeons such that you only need n pigeons to arrive on time without having gotten lost, shot or eviscerated by hawks, and you can reliably rebuild the original data set with fault tolerance that also handles read/write errors on the MicroSD cards.
[+] peternicky|5 years ago|reply
Would you share details of the design of your “ohs vps” facility?
[+] k__|5 years ago|reply
Just thought about AWS Snowmobile.
[+] mullen|5 years ago|reply
If the data is below 50G in size and on my personal computer, then I just drop it into the Google Drive folder and it syncs to Google over night. When it is done, I export it and then send a link to the person(s). I pay $2 a month for the 100G account and I usually have about 50G disk space unused, so this is not an issue for me.

If it is above 50G and on my personal computer, I encrypt the data and then physically mail a USB stick with the data on it. Trying to arrange the downloading/uploading of 50G of data from my personal computer to another personal computer is a real pain. The people that I would send/receive that much data to/from, that is stored on my personal computer are usually people who don't know much about ftp, scp or easily sharing files over the Internet. Sending a USB stick is just so much easier and in many cases, faster. I make sure the recipients know how to decrypt the data before sending the data.

If it is on a server (For example, a side project instance in AWS), then I drop it into S3 bucket, export it and send the URL to the recipient. I just eat the cost of them downloading it. Usually I am making money off the work anyways, so it is the cost of doing business.

[+] lowwave|5 years ago|reply
>If it is on a server (For example, a side project instance in AWS), then I drop it into S3 bucket, export it and send the URL to the recipient. I just eat the cost of them downloading it. Usually I am making money off the work anyways, so it is the cost of doing business.

Consider BunnyCDN way cheaper than s3.

[+] quietbritishjim|5 years ago|reply
> If the data is below 50G in size and on my personal computer, then I just drop it into the Google Drive folder

I do the same thing with OneDrive. Of course that's the same thing and I'm just agreeing with your answer, there are any number of DropBox-like services (including DropBox!). OneDrive is nice if you have Office 365 because you can password protect a link and send the password via a separate channel, which makes the process pretty secure (if you trust Microsoft with your data).

[+] kvn_95|5 years ago|reply
Magic Wormhole (https://magic-wormhole.readthedocs.io/en/latest/) if it's between OSes.

Between OSX, AirDrop works very well. I have sent >10GB files between Macs, quite quick as well.

I have never send a 10TB file so I wouldn't know. None of my drives are that large yet :)

[+] mettamage|5 years ago|reply
I use the magic of AirDrop as well :) Unfortunately, sometimes I'm on my Linux machine (in which case python3 -m http.server works just fine), or need to send a file to someone else and then I'd like to have at least some form of monitoring/progress.

Edit: magic wormhole is awesome! Thanks! :D

[+] kilroy123|5 years ago|reply
I once sent a huge file over airdrop and was shocked how much faster it was then sending through the local LAN.
[+] emit_time|5 years ago|reply
Just had a 41GB set of files sent to me in one transfer using magic-wormhole. Worked first try without a hitch. Pretty awesome from the receiving end.
[+] Zenbit_UX|5 years ago|reply
This is local lan only right? Or are you port forwarding to expose it?
[+] ashton314|5 years ago|reply
Netcat:

    $ nc -l 4242 > dest
And then on the sending end:

    $ nc hostname 4242 < file_to_send
This works great when you just need to get a file of any size from one machine to another and you’re both on the same network. Are used this a lot at one of my offices to schlep the files around between a few of the machines we had.
[+] unilynx|5 years ago|reply
This is especially fun when combined with piping through tar, and adding pv in the mix for a transfer speed "progress" bar.

Fastest way to transfer a collection of files on a local network and doesn't require temporary storage for the archive

[+] Matthias247|5 years ago|reply
and if you are not worried about corruption. You might at least want to run a checksum on both ends and compare them.
[+] rags2riches|5 years ago|reply
I've done that many times, often in combination with tar and pv as mentioned in another reply. But I prefer to do the listening on the sending side. Feels more like serving than shoving I guess. Also, I remember needing a -q flag on one of the ends for graceful shutdowns.
[+] BrandonM|5 years ago|reply
I remember using AIM (AOL (America Online) Instant Messenger) and other instant messaging applications for direct peer-to-peer file transfer back in 2002. On fast university connections, you could transfer movies in a few minutes. It's crazy to me that direct, fast file sharing was easier and more ubiquitous almost 20 years ago than it seems to be now. More context for my oft-misunderstood early feedback to dhouston about Dropbox.
[+] jve|5 years ago|reply
Private Nextcloud. Or some free tier OneDrive/alternative is enought for me.

However no-one mentioned a super simple service: https://wetransfer.com/ - Simple as drag & drop, enter recipient address, SEND. Pretty simple if you want non techie to send you something.

[+] anderspitman|5 years ago|reply
I've been working on solutions in this space for a couple years now. IMO making data on your local device available to others via HTTP range requests is the sweet spot between storing your data in the cloud and going full p2p.

Here's a couple of my projects:

https://patchbay.pub/

code[0]

Sender:

  curl https://patchbay.pub/random-channel/filename.bin --data-binary @filename.bin
Receiver:

  curl -O https://patchbay.pub/random-channel/filename.bin
https://fbrg.xyz

code[1]

This one works in the browser. You select a file and it gives you a link you can share with others. The underlying tech[2] is more complicated (relies on WebSockets) than patchbay, but the server is currently more reliable. I'm in the process of improving the patchbay server to be more robust, and I think it will eventually replace fibridge completely.

My current project is building something along the lines of Google Drive, but much simpler and designed to be self-hosted from within your home.

[0]: https://github.com/patchbay-pub/

[1]: https://github.com/anderspitman/fibridge-proxy-rs

[2]: https://github.com/omnistreams/omnistreams-spec

[+] BenjiWiebe|5 years ago|reply
Sending a 10TB file on my internet connection would take 2.5 years of constant uploading. Shipping a hard drive is cheaper and quicker.
[+] tbronchain|5 years ago|reply
It doesn't have to be large files, file sharing sucks. Even copy/past across devices, even your own devices, is a pain. How many times did I email myself some links to share them across devices. And I'm not even speaking about sharing pictures with people not very accustomed to technology (in other words, don't try to get them install a cloud storage app).

I gave that question a few tries but I feel part of the problem is the market being saturated by giants doing half the job. Another part of it is the lack of interoperability regarding file sharing between operating systems. I mean - a native right click>send to>user - native notification to the user with a solution to direct p2p download from that person. No software needed, seemless integration. Why is that so hard?

I really wish the big boys would give that a try rather than giving us toy-like features and price bump.

[+] StavrosK|5 years ago|reply
You're gonna love magic-wormhole.
[+] akerro|5 years ago|reply
>I'm also curious on how you'd send 10TB or more.

For my 12TiB of data I use Syncthing when I need to sync them more often, rsync.

I used rsync several times for billions of smaller files totalling to 300GiB, but really all depending on how I connect nodes. I prefer syncthing, but when only ssh is available, then rsync is good too.

Currently largest synced directory by syncthing (that shares usage stats) is over 61384 GiB :)) https://data.syncthing.net/

[+] superkuh|5 years ago|reply
I have run a internet reachable whatever.com webserver from my home desktop computer for 20 years. I just copy or softlink the file to a place in ~/www/ and give out the corresponding web link. I have a couple nginx locations prepared with pre-existing bandwith throttles so it's a matter of soft linking to the appropriate ~/www/dirwithassociatedrate/.

If it's for more than 1 person I upload it to a VPS if it's small (<20 GB) or make a torrent if it's not.

[+] itake|5 years ago|reply
You can create a simple python http server to expose files in a directory to the local web:

$ python -m http.server 8000

and then you can start ngrok to expose the file

$ ngrok http 8000

that will give you an URL to share with whoever wants it.

[+] adzm|5 years ago|reply
Note that the built-in web server does not support resuming downloads, unless that changed recently. twistd works well and just as simple.

ngrok is an amazing utility though!

[+] anderspitman|5 years ago|reply
Does ngrok limit data at all or just connections?
[+] GRBurst|5 years ago|reply
It depends in whether I want to send it to someone in the local network or through the internet.

For local network:

I use miniserve ( https://github.com/svenstaro/miniserve ) which is just a simple http server. There are similar tools for when I want to share it from the smartphone.

Through the internet it really varies:

Sometimes it is Firefox send ( https://send.firefox.com/ )

For photos, I use a self hosted photo solution piwigo ( https://github.com/Piwigo/Piwigo )

In earlier days it has been a self hosted nextcloud ( https://github.com/nextcloud/server ) instance. I still use it when the files are getting too large for Firefox send.

I also tried the already mentioned wormhole but this works only with tec ppl.

[+] Paul-ish|5 years ago|reply
I use Dropbox to send and receive files that over the email/slack/etc attachment limit. Dropbox has a cool "request files" feature that let's people upload files directly to your Dropbox.

I've never needed to send TBs of data.

[+] LinuxBender|5 years ago|reply
SFTP Chroot server. lftp client using mirror subsystem + sftp. It is multi-threaded, even for a single file and supports rsync like behavior even in sftp chroot. I can max out any internet link using lftp (client) + sftp (protocol).