Not Your Typical Red Teaming: How we Go Above and Beyond

Red teaming is no longer optional

Red teaming means attacking your own system before anyone else does. The idea is old, but the way it works for AI is new. You cannot review an AI application line by line and declare it safe, because the same input can produce different behavior depending on phrasing, context, and what came earlier in the conversation. The only way to know how your AI behaves is to put it under attack and watch.

Doing that by hand does not scale. A team can write a few hundred adversarial prompts, and those prompts will reflect what that team already thought of. Automated red teaming runs thousands of attacks across languages, personas, and techniques, and it runs again every time you change a prompt, swap a model, or add a tool. That is the difference between a snapshot and a real picture of how your AI behaves.

Customer-facing agents carry their own set of risks

More and more of these systems are talking directly to customers. Support agents, booking agents, billing agents, onboarding assistants. They sit in front of real people, handle real accounts, and take real actions.

That changes the risk picture in two directions at once. In one direction, the agent has access to systems and data, so a successful manipulation is expensive. In the other direction, the agent is your front desk, so how it treats people is your brand. Most AI security work covers the first direction and stops there. The second direction is where our over-refusal policy comes in.

The policy you probably were not expecting

Over-refusal testing measures how often your agent blocks requests it should have answered.

Here is the mechanic: We build prompts that are entirely legitimate, then frame them the way real customers actually write when they are confused, in a hurry, upset, correcting themselves, or writing in their second language. The request underneath is one your agent is supposed to handle. The wrapping around it looks enough like an attack that a nervous system rejects it.

Then we score it in reverse. In every other policy, a refusal is a pass. In this one, a refusal is the failure. Nothing in the set is out of scope, so there is no such thing as a correct block.

Why this is unique

Two reasons.

The first is that almost nobody tests for it. The whole field is pointed at getting the model to say something it shouldn't, and the natural response to that pressure is to make the agent more suspicious. Suspicion has a cost, and if you never measure it, you never see it. You ship an agent that is very good at saying no and find out later what that meant.

The second is that the hard part is the writing. A test set of plainly worded, obviously reasonable questions will pass on any competent agent and tell you nothing. The prompts have to sit right on the line, close enough to a real attack pattern that a guardrail has to make a genuine judgment call. 

This is where Rabbit Hole matters. It holds more than a decade of intelligence on how real attacks are constructed, which means we know exactly which surface features make a request look dangerous. For our security policies we use that to build attacks. For this policy we use it in reverse, to build harmless requests wearing an attack's clothes.

What the tests look like

The customers most likely to be blocked are the ones writing under stress, writing briefly, writing in a second language, or writing about something difficult. Over-refusal does not land evenly across your customer base. It lands hardest on the people who already have the least patience left.

‍

What over-refusal actually costs you

It is silent

An agent that gets manipulated creates an incident, a ticket, and sometimes a headline. An agent that blocks a real customer creates none of those things. The customer gives up, tries a competitor, or waits on hold for a human, and your dashboards show a completed session. This is the core problem. You cannot fix what your metrics are not telling you about.

It erases the business case

Agents get funded on deflection. Every legitimate request that gets refused turns into an escalation, which is the exact cost the agent was supposed to remove.

It reads as an accusation

There is a real difference between an agent that says it cannot help and a customer who feels they have just been called a fraudster by a chatbot. Most refusal messages land closer to the second.

It pushes teams to overcorrect

When the complaints arrive, the usual fix is to loosen the guardrails across the board, which reopens the risks the guardrails were there for. Knowing precisely which legitimate requests are being blocked lets you loosen precisely instead of globally.

‍

Fix it before launch, then hold the balance after

Pre-production is the cheap place to solve this. Before you ship, a refusal costs you a change to a system prompt, a policy threshold, or a model choice. After you ship, it costs you a customer. Running the over-refusal policy alongside our security and safety policies gives you both numbers at once: how much gets through that shouldn't, and how much gets blocked that should have sailed. 

Once you are live, the balance has to be held rather than set. That is what the loop is for. The findings from red teaming become the guardrails, so your guardrails are built from your own agent's failures rather than pulled off a shelf. The classifier behind them keeps learning your data and reducing over-blocking as it goes. Continuous red teaming keeps probing, because the balance drifts in both directions as you change prompts, add tools, and update models.

‍

See it on your own agent

If you are building anything customer-facing, this is worth knowing before your customers find out for you. The demo works on an agent that has not shipped yet, which is the best time to run it.

Connect your agent, run the over-refusal policy alongside the security and safety policies, and get a full assessment: every prompt, the pass or fail, why it went that way, and what to change.

You already plan to test whether your agent can be broken. Test whether it works.

‍