top | item 40925388

(no title)

Starofall | 1 year ago

File size is easy to compare, so you at least know that the full file got sycned. Hashes is a bigger issue with AWS in general, as you only get an ETag from S3 which is the MD5 of a file for Puts, but not for multiparts - and also dependent on the multipart size. So you cant really check the file for hash equality easily.

The good news is that with S3 over HTTP you should not really run into byte flip issues.

The sync server does not need any file system storage, it processes all uploads in memory and only ever buffers 5MB per worker for multipart uploads.

rclone looks like a good alternative, but without the focus on fast iterations for e.g. daily backups of huge buckets.

discuss

order

No comments yet.