top | item 40924184

(no title)

kapilvt | 1 year ago

For large buckets key space enumeration is a significant portion of most bulk operations, especially on a potentially non optimized key space (aka hotspots), there’s a few heuristics that can be utilized, but doing an s3 inventory allows skipping that and focusing on transfer with significantly less api calls, albeit requires bucket preparation.

discuss

order

Starofall|1 year ago

For a big sync this is definitly the way to go. But for a simple daily sync we found the tool more usefull. Especially when syncing to a non AWS S3 provider that does not offer a inventory feature.