(no title)
hyperhopper | 3 months ago
This is not banning you for following <h1><a>Today's Weather</a></h1>
If you are a robot that's so poorly coded that it is following links it clearly shouldn't that's are explicitly numerated as not to be followed, that's a problem. From an operator's perspective, how is this different than a case you described.
If a googler kicked off the googlebot manually from a session every morning, should they not respect robots.txt either?
varenc|3 months ago
hyperhopper|3 months ago