Pentagon dispute bolsters Anthropic reputation but raises questions about AI readiness in military

Anthropic’s moral stance on U.S. military use of artificial intelligence reshapes competition among leading AI companies, but also reveals a growing realization that chatbots may simply not be capable enough for acts of war.

Anthropic’s Claude chatbot, for the first time, overtook rival ChatGPT in phone app downloads in the United States this week, a sign of growing interest among consumers who are siding with Anthropic in its clash with the Pentagon, according to market research firm Sensor Tower.

The Trump administration on Friday ordered government agencies to stop using Claude and designated it a supply chain risk after Anthropic CEO Dario Amodei refused to bypass his company’s ethical safeguards preventing the technology’s application to autonomous weapons and domestic mass surveillance. Anthropic said it would challenge the Pentagon in court once it received a sanctions notice.

And while many military and human rights experts have praised Amodei for upholding ethical principles, some are also frustrated by years of AI industry marketing that has persuaded the government to apply the technology to high-stakes tasks.

“He caused this mess,” said Missy Cummings, a former Navy fighter pilot who now directs the robotics and automation center at George Mason University. “They were the first company to promote ridiculous hype about the capabilities of these technologies. And now, all of a sudden, they want to be real. They want to tell people, ‘Oh, wait a minute. We really shouldn’t be using these technologies in weapons.'”

Anthropic did not immediately respond to a request for comment. The Defense Department declined to say whether it was still using Claude, including in the Iran war, citing operational security.

Cummings published a paper at a major AI conference in December, saying government agencies should prohibit the use of generative AI “to control, direct, guide or govern any weapon.” Not because AI is so intelligent that it could become malicious, but because the large language models behind chatbots like Claude make too many mistakes – called hallucinations or confabulations – and are “inherently unreliable and inappropriate in environments that could result in the loss of human life.”

“You’re going to kill noncombatants,” Cummings said Tuesday in an interview with The Associated Press. “You’re going to kill your own troops. I don’t know if the military really understands the limits.”

Amodei sought to highlight these limitations when defending Anthropic’s ethical stance last week, arguing that “cutting-edge AI systems are simply not reliable enough to power fully autonomous weapons. We will not knowingly deliver a product that would endanger America’s warfighters and civilians.”

Anthropic, until recently, was the only one of its peers cleared for use in classified military systems, where it partnered with data analytics firm Palantir and other defense contractors. President Donald Trump said Friday, around the same time he approved Saturday’s military strikes against Iran, that the Pentagon would have six months to phase out Anthropic’s military applications.

Cummings, a former Palantir adviser, said it was possible that Claude had already been used in planning military attacks.

“I just hope there are humans in the loop,” she said. “A human has to monitor these technologies very closely. You can use them to do these things, but you have to check, check, check.”

She said this contrasts with messages from AI companies that suggest their technology is evolving to the point where it is “almost sentient.”

“If there is culpability here, I would say half of it is Anthropic’s for fueling the hype and the other half is the War Department’s fault for firing all the people who otherwise would have advised them against using the technology in stupid ways,” Cummings said.

One social media commentator this week described Anthropic’s government woes as a “fad tax” — a message that was reposted by President Donald Trump’s top AI adviser, David Sacks, a frequent critic of the company.

And while it has caused legal problems that could jeopardize Anthropic’s business partnerships with other military contractors, it has also cemented its reputation as a security-minded AI developer.

“It’s applaudable that a company stood up to the government to maintain what it saw as its business ethics and choices, even in the face of these potentially crippling policy responses,” said Jennifer Huddleston, a senior fellow at the libertarian-leaning Cato Institute.

Consumers have already spoken out, leading to a surge in Claude downloads that made it the most popular iPhone app as of Saturday and for all phone systems in the United States on Monday, according to Sensor Tower. This came at the expense of OpenAI’s ChatGPT, which saw its consumer reputation damaged when it announced a deal with the Pentagon on Friday to effectively replace Anthropic with ChatGPT in classified environments.

In the Apple Store, the number of 1-star reviews – the worst rating – for ChatGPT increased by 775% on Saturday and continued to grow earlier this week, reflecting a backlash that forced OpenAI to do damage control.

“We shouldn’t have rushed to release this on Friday,” OpenAI CEO Sam Altman said in a social media post on Monday. “The issues are extremely complex and require clear communication. We were genuinely trying to de-escalate things and avoid a much worse outcome, but I think it just seemed opportunistic and sloppy.”

Altman gathered employees for an “all-hands” meeting Tuesday to discuss next steps.

“There are a lot of things that the technology is just not ready for, and in many areas we don’t yet understand the security tradeoffs required,” Altman said on X. “We will work on that, slowly, with the (Pentagon), with technical safeguards and other methods.”