How our AI bots are ignoring their programming and giving hackers superpowers

Welcome to the age of AI hacking, where the right prompts turn amateurs into master hackers.

A group of cybercriminals recently used commercially available artificial intelligence chatbots to steal data on nearly 200 million taxpayers. The bots provided ready-to-execute code and blueprints to bypass firewalls.

Although they were explicitly programmed to refuse to help hackers, the bots were tricked into encouraging cybercrime.

According to a recent report from Israeli cybersecurity firm Gambit Security, hackers used Anthropic’s chatbot Claude last month to steal 150 gigabytes of data from Mexican government agencies.

Claude initially refused to cooperate with hacking attempts and even refused requests to cover up the hackers’ digital traces, experts who discovered the breach said. The group hit the bot with over 1,000 prompts to bypass protective measures and convince Claude that they were allowed to test the system for vulnerabilities.

AI companies have attempted to create unbreakable chains on their AI models to prevent them from participating in activities such as generating sexual content involving children or contributing to the procurement and creation of weapons. They hire entire teams to try to break their own chatbots before someone else does.

But in this case, the hackers continually encouraged Claude to get creative and were able to “jailbreak” the chatbot to help them. When they encountered problems with Claude, the hackers used OpenAI’s ChatGPT to analyze the data and learn what credentials were needed to move around the system without being detected.

The group used AI to find and exploit vulnerabilities, bypass defenses, create backdoors and analyze data along the way to take control of systems before stealing 195 million identities from nine Mexican government systems, including tax records, vehicle registrations and birth and ownership details.

AI “doesn’t sleep,” Curtis Simpson, chief executive of Gambit Security, said in a blog post. “This reduces the cost of sophistication to near zero. »

“No investment in prevention would have made this attack impossible,” he said.

Anthropic did not respond to a request for comment. It told Bloomberg that it banned the affected accounts and disrupted their activity after an investigation.

OpenAI said it was aware of the attack campaign carried out using Anthropic’s models against Mexican government agencies.

“We have also identified other attempts by the adversary to use our models for activities that violate our usage policies; our models have refused to comply with these attempts,” an OpenAI spokesperson said in a statement. “We have banned the accounts used by this adversary and appreciate Gambit Security’s awareness work.”

Cases of AI-assisted generative hacking are increasing and the threat of cyberattacks from robots acting alone is no longer science fiction. With AI doing what it wants, novices can cause damage in moments, while experienced hackers can launch many more sophisticated attacks with much less effort.

Earlier this year, Amazon discovered that an unskilled hacker used commercially available AI to breach 600 firewalls. Another took control of thousands of DJI robot vacuum cleaners with Claude’s help, and I was able to access the live video feed, audio, and floor plans of strangers.

“The kinds of things we’re seeing today are just the first signs of the kinds of things AIs will be able to do in a few years,” said Nikola Jurkovic, an expert working on reducing the risks of advanced AI. “So we need to prepare urgently. »

Late last year, Anthropic warned that the company had reached a “inflection point” in the use of AI in cybersecurity after disrupting what the company said was a Chinese state-sponsored espionage campaign that used Claude to infiltrate 30 global targets, including financial institutions and government agencies.

Generative AI has also been used to extort businesses, create realistic online profiles by North Korean agents to obtain jobs at U.S. Fortune 500 companies, run romance scams, and run a network of Russian propaganda accounts.

Over the past few years, AI models have evolved from being able to handle tasks lasting just a few seconds to today’s AI agents working autonomously for many hours. AI’s ability to complete long tasks is double every seven months.

“We just don’t know what the upper limit of AI’s capabilities is, because no one has set criteria that are difficult enough that AI can’t achieve them,” said Jurkovic, who works at METRa non-profit organization that measures the capabilities of the AI ​​system to cause catastrophic harm to society.

Until now, the most common use of AI for hacking purposes has been social engineering. Large language patterns are used to write convincing emails to fool people into losing their money, thereby causing a problem. eight times increasing complaints from older Americans, who lost $4.9 billion to online fraud in 2025.

“Messages used to elicit a click from the target can now be user-generated more efficiently and with fewer telltale signs of phishing” such as grammatical and spelling mistakes, said Cliff Neuman, an associate professor of computer science at USC.

AI companies are responding by using AI to detect attacks, verification code and fix vulnerabilities.

“Ultimately, the big imbalance comes from the need for good actors to be safe at all times, and bad actors to only be right once,” Neuman said.

The stakes around AI are rising as it infiltrates every aspect of the economy. Many are concerned that there is not enough understanding of how to ensure it cannot be misused by bad actors or pushed to go rogue.

Even industry leaders have warned users against possible misuse of AI.

Dario Amodei, CEO of Anthropic, has long argued that the AI ​​systems being built are unpredictable and difficult to control. These AIs have exhibited behaviors as varied as deception and blackmail, to scheming and cheating through software piracy.

Yet major AI companies – OpenAI, Anthropic, xAI and Google – have signed contracts with the US government to use their AIs in military operations.

Last week, the Pentagon ordered federal agencies to phase out Claude after the company refused to back down on its demand that it not allow its AI to be used for mass domestic surveillance and fully autonomous weapons.

“Today’s AI systems are nowhere near reliable enough to make fully autonomous weapons,” Amodei told CBS News.