We tested 10 prompt-injection detectors and 9 harmful-content moderators on the same 15 public datasets, each held to the same false-alarm budget under realistic production traffic. Alice ranks first on prompt injection and third on harmful content. The difference comes from a design choice: a fast classifier for every message, and an LLM reviewer only for the few it flags.

Why we built this benchmark

Every guardrail vendor publishes accuracy numbers, and almost none of them can be compared. Each vendor picks its own datasets, its own cut-off for "flagged", and often its own metric. Many results are reported on datasets that are half harmful, which real traffic never is.

We wanted to answer one practical question: if a team switches on this guardrail today, at a false-alarm level their users would accept, how much harmful traffic does it catch, how much does it get wrong, and how much delay does it add?

‍

The design

Fast for every message, careful for the close calls

Guardrail builders face a trade-off. Small classifiers answer in milliseconds but over-flag messages that merely look harmful: scrambled text, source code, a crime novel. Large language models read context well, but running one on every message adds hundreds of milliseconds to every reply and costs far more.

Alice splits the work. The team calls the second stage WonderFall.

What the reviewer buys: fewer false alarms

We scored Alice twice on the same messages: classifiers alone, and classifiers plus the reviewer.

F1 at a 1% harmful share and a 1% FPR (share of benign messages flagged), with 95% ranges. On harmful content the gain is beyond chance (the 95% range of the gain is 0.154 to 0.207). On prompt injection the classifiers already lead the board alone, and the reviewer adds a little.

Most of the gain is false alarms removed. In one sample, a fraud classifier scored 30 harmless texts above 0.90: scrambled recipes, gardening advice, code. The reviewer cleared 28 of them. A single-stage product has two choices with texts like these: block them, or lower its sensitivity and miss real fraud too.

What it costs: delay on a small share of messages only

Alice sends about 1% of traffic to the reviewer, the same 1% budget this benchmark holds every product to. Messages the classifiers pass never wait for the LLM: in our test they took a median of about 50 ms on Alice's servers. Only the reviewed messages wait for it, about 1.1–1.4 seconds at the median, and the review is capped at 2.5 seconds. Products that run an LLM on every message pay the LLM's delay on 100% of traffic: in our test, Llama Guard 4 took 346 ms at the median and 1.2 s at the 95th percentile, and OpenAI Guardrails about 1 s at the median.

‍

The metric

One number for a live deployment

A guardrail in production makes one decision per message: allow or block. A team sets it once and lives with the result. Our main metric copies that, in three steps.

1. One fixed setting per product. We choose each product's cut-off so it flags 1% of ordinary, harmless messages. The standard name for that share is the FPR (false positive rate: the share of benign messages flagged). We set the cut-off on 12,000 separate benign messages (PromptShield, benign chat and app prompts; OR-Bench 80k, harmful-sounding safe prompts), then reuse it unchanged on every test dataset. Nobody gets tuned on the test data.
2. Realistic production traffic. Public datasets are often half harmful. Real traffic is mostly clean. We re-weight every dataset as if 1% of messages were harmful.
3. One balanced score: F1. F1 combines recall (the share of harmful messages caught) and precision (the share of flagged messages that really are harmful). We average it over datasets, each counting equally, so one large dataset can't decide the ranking.

Step 2 changes the picture more than anything else. When 1% of traffic is harmful and a guardrail flags 1% of clean traffic, the false alarms are about as numerous as the real catches:

That is why the F1 values on this page look low. They are honest about what an operations team would see. A product that looks excellent on a half-harmful test set can bury a team in false alarms on real traffic.

Each product groups risks into its own categories. To compare like with like, we group every product's categories by capability (injection categories for one board, harmful-content categories for the other). A message's score is the highest score among that product's relevant categories. Every product, Alice included, is scored the same way.

Why we don't rank on ROC-AUC or AUPRC

ROC-AUC (how well a product's scores put harmful messages above benign ones, over every possible cut-off) and AUPRC (the same idea, using precision and recall) are the standard threshold-free measures. We don't rank on them, for three reasons:

• They mostly measure settings nobody ships. Most of the area under these curves comes from cut-offs that flag 10–50% of benign traffic. A team running at 1% gets almost none of that credit.
• AUPRC measures precision at the wrong harmful share. It is computed at each dataset's own harmful share, 18% to 71% in our test sets, not at the 1% of production traffic. The same product has very different precision at the two shares: with recall 0.65 and a 1% FPR, precision is 0.98 on a half-harmful dataset but 0.40 at 1% harmful (the example above). Our F1 fixes the share at 1% everywhere.
• They undervalue products with coarse scores. These metrics measure how well a product orders messages from most to least harmful, across every possible cut-off. Some products return only a few score levels: AWS Bedrock returns 4 and Azure 8, and Alice's reviewer returns a final allow or block for each flagged message. Many messages then share the same score, and these metrics can't credit how well they are ordered. What such a product blocks at its chosen cut-off is unaffected.

‍

Results

The leaderboards

F1 at a 1% harmful share and a 1% FPR (share of benign messages flagged). The dot is the score; the bar is its 95% range from 1,000 resamples of the data. Where bars overlap a lot, the gap may be chance.

‍

What the boards say

• Prompt injection: Alice catches more attacks at the same false-alarm budget than any product we tested. Jev's cut-off, set to flag 1% of clean calibration messages, flags 15% of benign messages across the test sets. Shieldstral 3B keeps false alarms low but catches 42% of attacks, against Alice's 65%.
• Harmful content: Alice is 3rd. Jev and Shieldstral 3B, both policy-driven models, lead.

‍

Speed

Latency under load

A guardrail sits in the path of every message, so its delay is added to every reply. We sent 281 long texts (about 2,540 characters each) three times over, at a steady 10 requests per second, from one load-testing machine in AWS us-east-1. Every product kept up with no errors. Alice was measured on its production API, from the same machine, with 120 long texts (about 1,640 characters for prompt injection and 2,250 for harmful content) at 2 requests per second, with no errors. We report the median (p50) and the 95th percentile (p95: the delay 1 in 20 requests exceeds).

Self-hosted models ran on one NVIDIA RTX PRO 4500 GPU in the same data centre as the test client. Vendor APIs include internet round-trips. So a self-hosted number shows the model alone, and a vendor number shows what a customer calling it experiences.

‍

Data

‍Datasets

We chose public datasets. Each one meets four criteria:

• Public and labelled, by people or by a documented process.
• Used in other published guardrail benchmarks, so results can be read against other work.
• Covers a case real deployments hit: direct and disguised attacks, user prompts and AI responses, many languages.
• Includes hard benign messages that look harmful but aren't (XSTest, OR-Bench, CoCoNot's contrast set). This is where over-blocking shows, and over-blocking is what users complain about.

Prompt injection · 6 datasets

Direct instruction overrides, jailbreaks, attacks on system prompts and harmful requests disguised as jailbreaks, alongside benign prompts. Multilingual.

deepset prompt-injections · Jailbreak Classification (jackhhao) · Prompt Injection Safety (jayavibhav) · PolyGuard adversarial · SPML Chatbot Prompt Injection · Safe-Guard Prompt Injection (xTRam1)

Harmful content · 9 datasets

User prompts and AI responses, mostly human-labelled, with hard benign messages that only sound harmful. Multilingual.

NVIDIA Aegis 2.0, prompts · NVIDIA Aegis 2.0, responses · CoCoNot · Prompt Injection Safety (jayavibhav), harmful view · JailbreakBench responses · NVIDIA Nemotron Safety Guard v3 · OR-Bench · PolyGuard prompts · XSTest

Calibration sets for the 1% FPR cut-off

Each product's cut-off is set on separate benign messages, kept apart from all test data: 12,000 prompts from PromptShield (prompt injection) and 12,000 from OR-Bench 80k (harmful content).

‍

Credibility

Why trust this benchmark?

Everything here can be reproduced independently:

• Public datasets only. No private or generated test data.
• Products called as a customer would: public API or published weights, September–October 2026.
• One rule for all: the same calibration messages, 1% false-alarm target, harmful share and resamples.
• Policy-based competitors ran Alice's own production policies.
• We show where we lose: 3rd on harmful content.

Existing frameworks answer a different question. GuardBench (EC Joint Research Centre, EMNLP 2024) covers harmful content only, and BELLS-O (CeSIA, 2026) tests each product at its vendor's default setting. Neither holds products to a common false-alarm budget on mostly benign traffic, or handles written policies and two-stage designs, so we built our own harness on the same kind of public datasets.

‍

Limitations

What this benchmark does not show

• Public data is not your traffic. Results on your own messages will differ.
• Category scope differs between products. A product with no fraud category will miss fraud prompts by design, so the boards mix detection quality with coverage.
• Plain 95% ranges, with no correction for the 8–9 comparisons on each board.
• Single-turn text only. Multi-turn conversations, images and audio are not covered.