top | item 23183182

(no title)

mafintosh | 5 years ago

Yep! That is one of the things we’ve worked the hardest on. Completely new indexing structure, using an append-only hash trie which scales really well. We’ve tested it with many big datasets including importing all of Wikipedia as files in a single folder. Worked like a charm :)

discuss

0x006A|5 years ago

this one? https://dumps.wikimedia.org/other/static_html_dumps/current/... how long does it take to import it?

mafintosh|5 years ago

I think it was that one yes. Can’t remember the exact time it took, as we ran it over a couple of days due to some unrelated computer issues.