This. I'm glad these guys are running their own crawler, but their story doesn't add up.
Most likely this is an outfit that does a lot of webscraping and wants to minimize the amount of manual banning they incur. So they set up a plausible indie search engine as a front. Of course the search engine has to actually work when the scrapees notice and try it out.
octoberfranklin|3 years ago
Most likely this is an outfit that does a lot of webscraping and wants to minimize the amount of manual banning they incur. So they set up a plausible indie search engine as a front. Of course the search engine has to actually work when the scrapees notice and try it out.