Open's GPTBot Will Scrape Your Website to Train Its AI, Unless You Opt Out A new web crawler dramatically expands the amount of data OpenAI's models will train on, but the company says you can opt out to prevent it from visiting your website. Emily Dreibelbis By Emily Dreibelbis August 8, 2023 global data(Credit: DKosig / Getty Images) With its new GPTBot, AI models from OpenAI can crawl the web for new information, meaning your website and its content can be scraped to train artificial intelligence—unless you opt out. "Web pages crawled with the GPTBot user agent may potentially be used to improve future models," OpenAI says.

Allowing GPTBot to access your site can help AI models become more accurate and improve their general capabilities and safety." OpenAI notes that GPTBot will not breach sites that USA Telegram Number Data require paywall access, a nod to a recent controversy where ChatGPT Plus members using "Browse with Bing" were able to bypass paywalls to read articles. GPTBot will also filter out sources "known to gather personally identifiable information, or have text that violates our policies." To prevent GPTBot from mining your website, OpenAI provides two lines of code you can copy and paste into your own site's code that will tell it to buzz off.



Another code snippet will give GPTBot access to "only parts of your site," a middle-ground option between fully blocking it and keeping the gates wide open. This likely only applies to sites you own and operate, meaning anything you publish website, or a blogging site like Substack or Medium, is still fair game. The experience of using ChatGPT does not seem to have been immediately affected by the change. In the past, ChatGPT has operated on a fixed dataset that only goes up to 2021. As of this writing, it still cannot answer questions regarding current events. RECOMMENDED BY OUR EDITORS Hands typing.