top | item 43496856 (no title) Epskampie | 11 months ago In line 1 of of the linked page: "waste the resources of AI Crawlers and other bots that don’t respect “no crawl” directives". discuss order hn newest ccgreg|11 months ago Does that indicate the robots.txt is how "no crawl" is indicated? robots.txt doesn't have "no crawl", it has allow and disallow.
ccgreg|11 months ago Does that indicate the robots.txt is how "no crawl" is indicated? robots.txt doesn't have "no crawl", it has allow and disallow.
ccgreg|11 months ago