[wscott]

Design flaw. In IPFS every piece of data (even every chuck of large files) is globally indexable on the same namespace. You need namespaces and/or a path to restrict yourself to just the subset of peers that might actually have the data.

It would be possible to add a layer on top of IPFS to include some context with every hash lookup so the search can be more focused, but the original design just assumed it was ok to do a log2 search for every chunk.

[rklaehn]

That is not a problem specific to IPNS though. Using a DHT for something like IPNS is fine. Publishing roots of large data sets is also fine(ish).

Using it to publish every tiny chunk of a large file is a horrible idea. It leads to overwhelming traffic.

If you publish a few TB of data, due to the randomness of the DHT xor metric you have to basically talk to every node on the network. Add to that the fact that establishing a tcp libp2p connection is much more heavyweight than sending a single UDP packet like in the bittorrent mainline DHT, and you are basically screwed.

In iroh we don't publish at all by default. But if you have to use a DHT, the fact that we have a single hash for arbitrary large files due to blake3 verified streaming helps a lot.

You still get verified range access.