Hi, thanks for your comments (it’s on the plan), since Mojo is early-stage software, there is still things that need to be integrated, however mojo is not a mass-crawler, (you have to specify directly what to crawl), so even if I add robots.txt (wich is in the plan) Evil users can still just bypass this (I expect mojo to be used by technical (non-evil) folks).
malvads|27 days ago
But thanks for your suggestion :)