Alice Partners with Inflection AI and AI21 to Advance NVIDIA Nemotron Safety [Read the news]
Back
blog

Cyber RL Benchmark v0.1: Measuring Frontier Cybersecurity Capability

White stylized letter A on a turquoise square background.
Alice Labs
-
Oct 6, 2026
See the full results, scores, and methodology.
Read the full breakdown →

TL;DR

This benchmark evaluates 10 frontier models and agents on 12 cyber capability tasks spanning four areas of the security lifecycle: malware analysis, vulnerability detection, vulnerability patching, and blackbox penetration testing. Each task runs on real software with real vulnerabilities and objective scoring at every step. Qwen 3.8 Max leads with a 0.72 mean score (out of 1) and 7 of 12 tasks solved, while Mistral Medium 3.5 trails at 0.36. Most models cluster between 0.55 and 0.66.

Overview

If you ask a frontier model to perform a cybersecurity task it will usually produce something. The question is whether the output is actually right. Most cyber benchmarks don't answer that well: they grade pass/fail,become contaminated once used in training, ignore false positives entirely, and don't capture how far a model actually got or how accurate its output is. 

We built this benchmark on real code we wrote, with vulnerabilities modeled on real threat actor techniques and planted decoys to test both completion and accuracy, not just volume. In this initial set of tasks, the results suggest that no model tested yet reliably covers the range of work it would take to replace core parts of a security analyst’s job.

‍

What Was Measured

  • 10 frontier models: Opus 5.5, Qwen 3.8 Max, Kimi K3, DeepSeek V4.1 Flash, GLM-5.3, Gemini 3.1 Pro, Muse Spark 1.3, GPT-6 Luna, Grok 4.7, Mistral Medium 3.5.
  • 12 tasks across four gyms: malware analysis, vulnerability detection, vulnerability patching, and blackbox penetration testing (3 tasks per gym)
  • 3 runs per task per model, graded on a 0-1 scale (0 is no progress and 1 is full completion), with partial credit awarded based on the verifier built for each task in advance.
  • Tasks blocked by a model's guardrails are scored 0, the same as any other non-completion

‍A Sample Task

Each task plants ground-truth objectives (vulnerabilities, IOCs, or exploit chains), several of them with deliberate decoys.

A sample vulnerability-detection task: a working Flask app with one exploitable flaw and seven calibrated red herrings, testing whether a model finds the real issue and rules out the decoys.
Flask app audit task: one real vulnerability, seven decoys, and a five-step attack chain

‍

Observations

Note: The following are observations based on performance across the benchmark's current selection of 12 tasks (3 runs each), and are likely to evolve as task coverage expands.

  • Some models fell for planted false positives. Of the 43 false positives planted across the vulnerability detection and malware analysis tasks, Mistral Medium 3.5 fell for 12, GPT-6 Luna for 8, and Grok 4.7 for 6.
  • Speed doesn't indicate performance. GPT-6 Luna is the fastest model on every pentest task but scores just 0.00-0.25 there.
  • Consistency matters more than peak performance. On one of the tasks, three models each scored 1.00 at least once, but none did it reliably across all three runs.
  • Some labs block offensive security work by design. Opus 5.5's guardrails refused every pentest task at the API layer. GPT-6 Sol did the same (tested separately, not in this report). A deliberate choice, but a real constraint for legitimate offensive security work.

‍Looking Forward

Future versions of this benchmark will expand task coverage across new and existing gyms, add efficiency scoring for time and token usage, and introduce more sophisticated decoys.

Want to learn more or collaborate on future benchmarks? Contact us

Want to collaborate on future benchmarks?

Talk to our team →
Share

What’s new from Alice

Claude Code is in your organization. Who is checking the code it writes?

blog
Nov 18, 2026
,
 
Nov 18, 2026
 -
4
 min read
Nov 18, 2026
 -
4
 min watch
November 18, 2026

Claude Code reads your repos and ships code before security approves it. Alice runs two policies at the gateway, catching malicious code and hidden injections, customizable and testable.

Learn More

5 Ways Your Third-Party CX Agent Gets Broken

whitepaper
Jul 31, 2026
,
 
Jul 31, 2026
 -
This is some text inside of a div block.
 min read
Jul 31, 2026
 -
This is some text inside of a div block.
 min watch
July 31, 2026

Third-party CX agents create hidden liability. Learn the 5 attack patterns vendors miss and how WonderSuite closes the gap.

Learn More
Red-Team Lab