Cloudflare explains Tuesday’s outage that temporarily took down ChatGPT
Cloudflare’s bot controls are supposed to help solve problems like crawlers scraping information to train generative AI. It also recently announced a system that uses generative AI to build the “AI Labyrinth,” a new mitigation approach that uses AI-generated content to slow down, confuse, and waste the resources of AI Crawlers and other robots that fail to follow “no crawl” guidelines.
However, he claims that today’s issues were due to changes to a database’s permissions system, not generative AI technology or DNS, and not what Cloudflare initially suspected, which was a cyberattack or malicious activity such as a “large-scale DDoS attack.”
According to Prince, the machine learning model behind Bot Management that generates bot scores for requests that pass through its network has a frequently updated configuration file that helps identify automated requests; However, “A change in our underlying ClickHouse query behavior that generates this file has resulted in a large number of duplicate ‘feature’ lines.”
There are more details in the article about what happened next, but the query change caused its ClickHouse database to generate duplicate information. As the configuration file quickly exceeded predefined memory limits, it removed “the primary proxy system that handles traffic processing for our customers, for any traffic that depended on the bot module.”
As a result, companies that used Cloudflare rules to block certain bots returned false positives and disrupted real traffic, while Cloudflare customers that did not use the bot score generated in their rules remained online.
