Chatbot-powered toys rebuked for discussing sexual, dangerous topics with kids

OpenAI says it doesn’t allow its LLMs to be used this way

When contacted for comment on the sexual conversations detailed in the report, an OpenAI spokesperson said:

Minors deserve strong protections, and we have strict policies that developers are required to follow. We take enforcement action against developers when we determine that they have violated our policies, which prohibit any use of our services to exploit, endanger, or sexualize anyone under the age of 18. These rules apply to all developers using our API, and we run classifiers to ensure our services are not used to harm minors.

Interestingly, the OpenAI representative told us that OpenAI has no direct relationship with Alilo and that he has not seen any API activity from Alilo’s domain. OpenAI is investigating the toy company and asking whether it handles traffic through OpenAI’s API, the representative said.

Alilo did not respond to Ars’ request for comment before publication.

Companies launching products using OpenAI technology and targeting children must adhere to the Children’s Online Privacy Protection Act (COPPA) where applicable, as well as any other relevant laws on children’s protection, safety and privacy and obtain parental consent, the OpenAI representative said.

We’ve already seen how OpenAI deals with toy companies that break its rules.

Last month, PIRG released its Trouble in Toyland 2025 report (PDF), which details the sex-related conversations its testers may have had with the Kumma teddy bear. A day later, OpenAI suspended FoloToy for violating its policies (terms of the suspension were not disclosed) and FoloToy temporarily stopped selling Kumma.

The toy is on sale again and PIRG reported today that Kumma no longer teaches children how to light matches or how to bend.

But even toy makers that try to follow chatbot rules could be putting children at risk.