[ylow]

The dedupe is optimistic and is designed to scale to 1-10 PB range. There is a more complicated architecture blog post we are working on. We can dedupe across repositories but we do not right now largely for privacy reasons so that blocks are not shared across different people as that can cause information leakage.