[droffel]

Merkle trees! You (simplified) hash the data at the leaves of the tree, merge adjacent leaves (concatenation for example), and hash the new leaves, until you're left with just the Merkle Root, a single hash representing the entirety of your data. Verifying the root is easier than loading the data itself into memory and hashing it there since you can verify it piecemeal and without loading it all into memory at once.

[krackers]

But regardless of the ordering which you read, don't you ultimately still need to read every bit once?

[CJefferson]

Each level of the tree contains a hash of the hashes from the level below, so to verify the top hash you need only hash all its children (which are hashes themselves). Then you can explore a child, continuing recursively. At the bottom you find a hash of an actual file (or part of a file), then hash the file to check it’s validity.

This allows you to only hash those parts of the tree you actually want to read.

[nomel]

That's around 3s, for their slowest offering.