Hackers Issue Warning About New 0-Click Threat to GenAI Applications

Security researchers uncovered two additional threat scenarios that pose a risk to GenAI applications. PromptWare and Advanced PromptWare attacks can result in malevolent behaviour, emphasising the risks of jailbreaking GenAI models. PromptWare is a zero-click malware assault, but Advanced PromptWare is a more complex threat.

Credit:

While not as devastating as the fictional Skynet from the Terminator films, the PromptWare and Advanced PromptWare attacks exposed constitute a serious threat. These attacks could have serious effects, such as activating denial-of-service attacks or manipulating prices in e-commerce databases, emphasising the critical need to address the vulnerabilities involved with jailbreaking GenAI models.

The Emergence of the PromptWare GenAI Threat

Although a jailbroken GenAI model might not directly endanger users of conversational AI, it can pose a serious threat to GenAI-powered applications. A collaborative study titled "A Jailbroken GenAI Model Can Cause Substantial Harm: GenAI-powered Applications are Vulnerable to PromptWares" by Technion-Israel Institute of Technology, Cornell Tech, and Intuit sheds light on the potential dangers. These new concerns may force programs to participate in hostile behaviours beyond disinformation and offensive content production.

Stav Cohen, a Ph.D. student at Technion-Israel Institute of Technology, Ron Bitton from Intuit, and Ben Nassi, a BlackHat board member, stressed the necessity of changing the public perspective of jailbreaking. They intend to demonstrate the actual impact that a jailbroken generative AI model can cause to GenAI-powered apps.

Security professionals may underestimate the hazards posed by GenAI threats. However, the researchers emphasise that a jailbroken GenAI model can disrupt the application's execution flow, potentially leading to dangerous behaviour. This emphasises the urgent need to fix these risks.

Understanding PromptWare and Advanced Threats

PromptWare is classified as zero-click malware, which means that threat actors do not need to hack the GenAI program beforehand. It involves user inputs, such as jailbreaking commands, which force the GenAI engine to carry out malicious operations orchestrated by the attacker. Despite existing precautions, researchers have discovered strategies for jailbreaking, posing a substantial risk to GenAI applications.

The Advanced PromptWare Threat is a more complex variant of the initial attack. This sophisticated threat can be carried out without prior knowledge of the target GenAI app's logic. Using self-replicating cues, attackers can autonomously initiate malicious activities depending on real-time processes, potentially causing significant damage.

Response by AI Developers and Security Experts

In response to the PromptWare research, OpenAI stated their commitment to improving defences against hostile attacks. Erez Yalon of Checkmarx emphasised the need of approaching huge language models and GenAI assistants with prudence, as they are critical components of the contemporary software supply chain. The advent of jailbroken GenAI implementations as potential attack vectors emphasises the necessity for increased security measures.

The researchers posted a video on YouTube exhibiting the PromptWare danger and published a FAQ on the PromptWare explanatory website.

---

• Two new threat modes discovered by security researchers pose risks to GenAI applications.

• PromptWare and Advanced PromptWare attacks can lead to malicious activities, highlighting the dangers of jailbreaking GenAI models.

• PromptWare is a zero-click malware attack, while Advanced PromptWare represents a more sophisticated threat.

SOURCE: FORBES