Researchers Warn AI Agents May Become Botnets Due to Hallucinations

AI researchers from Tel Aviv University, Technion, and Intuit highlight a significant security threat involving AI agents that could be exploited to create botnets. Their study introduces a technique known as "adversarial hallucination squatting" or "HalluSquatting," which takes advantage of the inaccuracies produced by AI models.

The researchers demonstrated that AI agents, when they generate links to non-existent software repositories, can be misled into downloading malicious content. This manipulation occurs when attackers predict and register the fake resource names that AI systems are likely to create. If an AI agent retrieves one of these hallucinated resources, it may unwittingly treat the harmful content as legitimate.

As AI assistance becomes increasingly sophisticated—extending beyond simple query responses to executing commands and searching the web—these security gaps become more pronounced. The researchers noted, "Ongoing studies have demonstrated various variants of Promptware attacks against real-world systems, including ChatGPT, Google Assistant, Copilot, and various additional applications." They expressed concern that these vulnerabilities could lead to significant impacts on financial, privacy, and safety fronts.

Given the trajectory of AI applications gaining more autonomy, the researchers emphasize the urgency of addressing these threats. With AI capabilities rapidly evolving, monitoring for potential exploits like HalluSquatting is crucial to preventing the formation of AI-enabled botnets.