top | item 5408630

(no title)

dredge | 13 years ago

No, it doesn't. Your linked example contains:

  User-agent: *
  Disallow: /
While the (new) digg.com/robots.txt contains:

  User-agent: *
  Disallow:
Those are very different. The former essentially disallows bots from crawling the entire site, while the latter disallows nothing - effectively allowing everything. The syntax is unusual, granted, for historical reasons.

discuss

order

No comments yet.