top | item 43839692

(no title)

abalaji | 10 months ago

faustian bargain: you trade scrape ability for your website for one click buy of your product.

discuss

gcanyon|10 months ago

It's hardly faustian if your goal is to sell your product. If you have some weird hybrid sales/content-based ad revenue setup, then yeah, maybe it's faustian.

simonw|10 months ago

The document suggests unblocking OAI-SearchBot in robots.txt, and then clarifies:

> OAI-SearchBot is for search. OAI-SearchBot is used to link to and surface websites in search results in ChatGPT's search features. It is not used to crawl content to train OpenAI’s generative AI foundation models. Learn more (opens in a new window).

carlosjobim|10 months ago

If you sell a product you want everybody to scrape as much as possible from your website.