JASON REDMOND/AFP via Getty Images; Jaap Arriens/NurPhoto via Getty Images
- Hundreds of major companies and websites are now blocking ChatGPT's web crawler.
- Dozens more are also now blocking the crawler of Common Crawl, a major source of AI training data.
- Unique, high quality data, mainly scraped from the web, is vital to the performance of AI models.
More and more companies are trying to avoid having their data freely scraped and saved by web crawlers working for the benefit of AI models.