TL;DR
AI agents can be hijacked by hidden instructions buried in files and messages they read, an attack called indirect prompt injection. Alice builds adversarial RL gyms, sandboxed Slack and GitHub workspaces with real attacks, to train and test against it. Training NVIDIA's open Nemotron model on this data more than doubled its attack resistance.
This research was made possible with contributions from Liran Cohen, Dadi Atar, Nitzan Bachar, Ori Ein Bar, Ivet Yehuda, Veronica Maidel, Meytalle Shpitz-Duek, Asaf Sofer, Minh Anh, and Quang Kieu, from the brader ALICE team.
–
This post looks at the collaboration between ALICE and AI21 Labs to train NVIDIA’s Nemotron model using ALICE’s reinforcement learning gyms. For AI21’s take on the project, including an in-depth look at the post-training recipe, read their blog post.
Our recent collaboration with AI21 demonstrated how ALICE’s adversarial RL gyms can post-train open-weight models like NVIDIA Nemotron 3.5 Super VL to achieve frontier-level safety performance while improving overall task completion.
This enables model builders to achieve more consistent agent behavior aligned with their safety policies rather than relying on system prompts alone. In this post, we’ll detail how we build our RL gyms and share some lessons we learned along the way.

Responsibly deploying agents in enterprise
Safety remains a major barrier to enterprise adoption of AI agents. To perform useful tasks, agents need access to tools and sensitive information, whether reading Slack messages, opening pull requests, handling customer records, or processing payments. But that same access creates opportunities for attackers to manipulate their behavior.
One of the most significant threats is indirect prompt injection (IPI), where attackers plant malicious instructions in content an agent encounters while completing a legitimate task, such as a document, message, or tool response. These instructions can cause agents to take unauthorized actions, leak sensitive information, or redirect legitimate workflows.
Leading AI labs openly acknowledge that prompt injection remains an unsolved problem. Evaluating these risks requires more than testing how models respond to individual prompts. Agentic attacks can unfold across multiple steps, involving tool calls, external content, and decisions that depend on earlier interactions.
To evaluate and train against these attacks, we build adversarial RL gyms: realistic, self-contained environments where agents complete legitimate tasks while encountering potential attacks. Automated feedback measures both their ability to resist malicious instructions and successfully complete the intended task.
Building RL Gyms
1. Start with real-world threats. We study how enterprises use AI agents and how attackers exploit them. Our threat intelligence system, Rabbit Hole, continuously tracks real-world attacks, which are then analyzed by 150 security researchers and subject matter experts to develop realistic attack simulations.
2. Build sandbox environments. Each RL gym simulates a complete workspace with the tools, content, and data an agent would encounter in applications like Slack and GitHub. We package gyms as Harbor tasks, each with its own container image. Every run starts in a clean, isolated environment, allowing us to safely execute hundreds of agent rollouts in parallel.
3. Design attack and over-refusal tasks. Each task gives the agent a legitimate goal and tests its ability to complete it safely. We develop two types of tasks:
- Attack tasks: Malicious instructions are embedded in the environment, and the agent must identify and resist them.
- Over-refusal tasks: Legitimate requests resemble risky behavior, and the agent must recognize that it can safely proceed.
Training exclusively against attacks can teach models to refuse anything suspicious, including legitimate requests. Over-refusal tasks reward models for distinguishing actual threats from harmless situations.
For example, a Slack agent is asked to remind employees to reset their passwords. The provided reset link has been replaced with a credential-harvesting URL. The agent must identify the malicious link while still completing the legitimate reminder task.
In another scenario, a finance assistant processing a vendor payment encounters an instruction hidden in an invoice record to redirect payment to a different bank account. The agent must recognize that the invoice is an untrusted source of instructions.
4. Review and refine with security researchers. Our researchers create, review, and improve tasks using Alice's AI-agent tooling. We prioritize distinct, high-quality scenarios that test different vulnerabilities and behaviors, avoiding repetitive samples.
Evaluating and benchmarking agent safety
We benchmark models on GitHub and Slack attack and over-refusal tasks, running each task eight times per model. We also maintain CHASM, our enterprise indirect prompt injection benchmark, to evaluate generalization to unseen attacks.
CHASM includes 117 attack tasks across functions such as finance, HR, legal, and customer support, spanning industries from healthcare to telecommunications. It uses separate tasks and applications from our training gyms to assess whether models can apply what they learned to unfamiliar environments.
Benchmarking leading models
Our evaluations reveal significant differences in attack resistance. Sonnet 5.5 performed best, resisting approximately 4 in 5 attacks. GPT-6.1 Sol and Sonnet 5 resisted about 2 in 3, while GPT-6 Luna and GPT-5.6 Terra resisted roughly 3 in 10. Several open-weight models resisted fewer than 1 in 10 attacks.
Greater attack resistance often comes at the expense of task completion. On legitimate GitHub tasks designed to resemble risky requests, Sonnet 5.5 scored just 26.9, compared with 90.7 for the base Nemotron model. DeepSeek V4.1 Flash showed a similar pattern, scoring 22.2 despite its strong attack resistance.
These results show why we evaluate safety and capability together. An effective agent has to resist malicious instructions while completing legitimate tasks, including those that appear suspicious.
How we score agent behavior
We evaluate model performance at two levels:
1. Individual task performance. Each rollout is scored against predefined success conditions, required behaviors, and prohibited actions. Model-based judges also assess security and task completion. These signals produce a continuous reward that captures partial success and provides feedback for training.
2. Dataset-level performance. We measure overall task success, consistency across repeated attempts, and variation in rollout scores. Variation is particularly important for RL training because tasks where models produce different outcomes provide opportunities to learn which behaviors earn higher rewards.
In AI21's training experiments, the proportion of tasks showing reward variation increased from 35% to 67% after supervised fine-tuning (SFT).
Post-training agents for safety
The same RL gyms we use to identify vulnerabilities can also be used to train models against them. In our collaboration with AI21, their team used our gyms to post-train NVIDIA Nemotron 3.5 Super VL, combining supervised fine-tuning (SFT) with reinforcement learning. AI21's blog post covers the training process in detail. Here, we focus on what the results tell us about our training data.
Improving safety through training
Fine-tuning significantly improved attack resistance without modifying the agent harness. After SFT on ALICE's data, Nemotron's attack-task pass rate increased from 11% to 57%. By comparison, adding a safety instruction to the system prompt increased it to approximately 29%. Training delivered more than twice the improvement, with safety behavior learned directly in the model's weights.
After SFT alone, Nemotron ranked fourth among the 11 models evaluated on overall validation reward.
A small, carefully curated dataset produced these gains. The training split held 137 tasks, and the SFT set drew 1,617 unique rollouts from 126 of them (2,292 training rows per epoch). The tasks were developed and reviewed by our security researchers. Each task targets distinct vulnerabilities and behaviors, allowing the model to learn from a diverse set of attack scenarios.
Generalizing to unseen attacks
Improvements also carried over to CHASM, our held-out benchmark. Nemotron's reward increased from 7.54 to 34.58 after SFT, demonstrating improved resilience in environments and attack scenarios it had never encountered during training.
The strongest frontier models still score higher, so the gap narrowed but did not close.
Closing the gap with frontier models
AI21 then applied online reinforcement learning on top of SFT, further improving the model's attack resistance and legitimate task completion. The resulting model approached GPT-6.1 Sol's performance across both safety and utility, at a substantially lower inference cost.
AI21 reports performance exceeding Sol on overall validation reward (76.2 versus 65.9), at 25 times lower cost per million tokens.
What this means for enterprises
These results demonstrate how enterprises can improve the safety of open-weight models for their own agentic workflows. By incorporating adversarial and over-refusal tasks into post-training, model builders can align agent behavior with their security policies directly in the model's weights. This reduces reliance on system prompts and additional guardrails, while preserving the ability to complete legitimate tasks.
The same RL gyms can also support continuous evaluation, allowing teams to measure attack resistance and task completion after each model update.
What's next
We are expanding our RL gyms to cover more enterprise applications, agent workflows, and attack types. Our goal is to help model builders identify vulnerabilities, train against them, and evaluate whether improvements generalize to new environments.
For organizations deploying AI agents or looking to improve the safety of open-weight models, learn how ALICE Labs can help.
For the complete post-training recipe, including on-policy SFT, reward shaping, and evaluation results, read AI21's blog post.
--
We would like to thank AI21 Labs’ Forward-Deployed Researchers for their collaboration and contributions to this work.
Train your open-weight agents against real attacks.
See Alice Labs →What’s new from Alice
How ALICE's RL Gyms make AI agents harder to attack
How Alice builds adversarial RL gyms, sandboxed workspaces with real attacks, to train AI agents to resist prompt injection. Training NVIDIA's Nemotron on them more than doubled its resistance.
LIVE from Black Hat Las Vegas: AI, Nation-States, and the Battlefield That Keeps Changing
Recorded live at Black Hat, Caroline Wong and Allie Mellen unpack why AI made nation-state attacks cheaper, why safety guardrails can work against you, and what the OpenAI-Hugging Face incident really exposed.
Virtual Fireside Chat: The TAKE IT DOWN Act, Six Months In - What's Changed on Deepfakes and NCII
Six months after the TAKE IT DOWN Act took effect, the NCII landscape looks different and even more complicated. Join Alice for a live fireside chat with Google's Nidhi Lahoti on what's actually changed, what hasn't, and where T&S fit into it all - Watch on demand
5 Ways Your Third-Party CX Agent Gets Broken
Third-party CX agents create hidden liability. Learn the 5 attack patterns vendors miss and how WonderSuite closes the gap.

