[jd20]

I didn't see it mentioned, but why not just use robots.txt? Does Bytespider ignore it?

[hajimuz]

Yes. It ignores robots.txt.

[chptung]

Not super sure so this felt like a faster way to plug it ASAP.