(no title)
dredge | 13 years ago
User-agent: *
Disallow: /
While the (new) digg.com/robots.txt contains: User-agent: *
Disallow:
Those are very different. The former essentially disallows bots from crawling the entire site, while the latter disallows nothing - effectively allowing everything. The syntax is unusual, granted, for historical reasons.
No comments yet.