OpenAI just admitted it has a bot that crawls the web to collect AI training data. If you don't block GPTbot, that's self-sabotage.

• Spiderbots have been crawling the web for years collecting data.
• Some of these bots have been helpful because they send users to sources of original content online.
• The rise of generative AI and LLMs is undermining this grand internet bargain.

I hate spiders. When I traveled around the world in 2003, the thought of chunky, hairy arachnids creeping beneath my mosquito net kept me awake on many a tropical night.