AI Bots Are Now a Signifigant Source of Web Traffic
The viral virtual The OpenClaw assistant, formerly known as Moltbot, and before that Clawdbot, is a symbol of a broader revolution underway that could fundamentally change how the Internet works. Instead of being a place primarily inhabited by humans, the web could very soon be dominated by autonomous AI robots.
A new report measuring bot activity across the web, along with associated data shared with WIRED by internet infrastructure company Akamai, shows that AI bots already account for a significant portion of web traffic. The findings also shed light on an increasingly sophisticated arms race unfolding as bots deploy clever tactics to bypass website defenses meant to keep them out.
“The majority of the Internet will be bot traffic in the future,” says Toshit Pangrahi, co-founder and CEO of TollBit, a company that tracks web-scraping activity and published the new report. “It’s not just a copyright issue, there is a new visitor emerging on the Internet.”
Most large websites try to limit the content that bots can retrieve and transmit to AI systems for training purposes. (WIRED parent company Condé Nast, along with other publishers, is currently suing several AI companies for alleged copyright infringement related to AI training.)
But another type of AI-related website scraping is also on the rise. Many chatbots and other AI tools can now retrieve real-time information from the web and use it to augment and improve their results. This could include updated product prices, movie times, or summaries of the latest news.
According to Akamai data, training-related bot traffic has been steadily increasing since last July. Meanwhile, global activity of bots scraping web content for AI agents is also on the rise.
“AI is changing the web as we know it,” Robert Blumofe, Akamai’s chief technology officer, told WIRED. “The ensuing arms race will determine the future look, feel, and functionality of the Web, as well as the basis for doing business.”
As of Q4 2025, TollBit estimates that on average one in 50 visits to its customers’ websites came from an AI scraping bot. In the first three months of 2025, that figure was just one in 200. The company says that in the fourth quarter, more than 13% of bot requests bypassed the robots.txt file, a file that some websites use to indicate which pages bots should avoid. TollBit reports that the share of AI bots ignoring robots.txt increased by 400% between the second and fourth quarters of last year.
TollBit also reported a 336% increase in the number of websites attempting to block AI bots over the past year. Pangrahi says scraping techniques are becoming more sophisticated as sites attempt to assert control over how bots access their content. Some bots disguise themselves by making their traffic appear to come from a normal web browser or send requests designed to mimic the way humans normally interact with websites. The TollBit study notes that the behavior of some AI agents is now almost indistinguishable from human web traffic.
TollBit sells tools that website owners can use to charge AI scrapers for access to their content. Other companies, including Cloudflare, offer similar tools. “Anyone who relies on human web traffic – starting with publishers, but pretty much everyone – will be impacted,” Pangrahi says. “There has to be a faster way to have this machine-to-machine programmatic exchange of value.”
