[jayzalowitz]

Anyone have time to dig in and say what tricks its using most?

[Starofall]

The code is really extremly simple, just these few files: https://github.com/Starofall/S3HyperSync/tree/main/src/main/...

1) The underlying S3 Framework is already super fast https://pekko.apache.org/docs/pekko-connectors/current/s3.ht...

2) Lots of multithreading, stream buffering and pipelineing

3) For the fast iteration speed the "read,parse,ask for next" loop is the main bottleneck - so if you e.g. know that your sync source prefix contains uuids - the tool creates a file iterator for each known subfolder prefix. And with 16 iterators, its mainly the CPU that bottlenecks the XML parsing :)