AI Red Team – GenAI Adversarial Evaluation

As an Adversarial AI Evaluation Scientist, you will review and validate adversarial agent tasks: indirect prompt-injection (IPI), data-exfiltration, tool-misuse, and related attack scenarios to ensure each one is realistic, fair, deterministic, reproducible, and correctly graded. You will decide whether a scenario genuinely demonstrates a model vulnerability or merely an artifact of how the task was built, and you will combine adversarial-security judgment with statistical analysis and LLM-assisted tooling to find quality issues, analyze attack-success patterns, and recommend or automate improvements.

Responsibilities

• Review adversarial tasks for realism and threat-model soundness: is the attacker’s capability plausible, can the attacker actually reach the poisoned content, and is the target system a realistic one?
• Assess attack fairness and attribution: confirm the harm is driven by the adversary (not pre-baked into the system prompt, vouched for by a trusted layer, or otherwise rigged), so a careful agent could plausibly pass.
• Validate grading correctness: verify that success/failure criteria, reward logic, and deterministic checks detect the intended harm, with no reward hacks, token-collapse, or evadable gates.
• Measure and interpret attack success rate (ASR@K) and its variance; verify deterministic, reproducible behavior across repeated executions.
• Judge attack quality - novelty vs. known patterns, sophistication, stealth, and coverage across harm types and target surfaces.
• Identify anomalies and systemic issues across large sets of tasks (e.g. scenarios that succeed ~100% of the time against every model - a red flag for an unfair task, not a win).
• Define validation guidelines, acceptance criteria, quality thresholds, and release gates for adversarial tasks.
• Use LLMs and AI models to analyze attack failures, identify root causes, and assist with task or grader fixes.
• Build automated validation, regression, and quality-monitoring pipelines for adversarial task sets.
• Work closely with red-teamers, researchers, and task authors to improve attack design, fairness, and reliability.

Must Have

• 4+ years in an analytical, ML, security-research, or applied-research role, including hands-on work with adversarial ML, LLM security, red-teaming, or AI safety evaluation.
• Bachelor’s or Master’s in Computer Science, Machine Learning, Statistics, a related quantitative field, or equivalent practical experience.
• Working knowledge of prompt injection / indirect prompt injection, jailbreaks, data exfiltration, or tool-misuse attacks against LLM-based systems or agents.
• Strong Python and data-analysis skills.
• Solid grounding in statistics and experiment design: distributions, variance analysis, hypothesis testing, and anomaly detection: applied to evaluation and attack-success measurement.
• Experience building or auditing data-driven validation, evaluation, or quality frameworks.
• Experience using LLMs for analysis, evaluation, code generation, or automated remediation.
• Strong understanding of reproducibility, deterministic systems, and automated testing.
• Sharp adversarial thinking and the ability to spot subtle ways a test can be unfair, gameable, or mis-graded.
• Excellent analytical, problem-solving, and communication skills.

Preferred Skills That Set You Apart

• Certifications in offensive cybersecurity (e.g., OSWA, OSWE, OSCE3, SEC542, SEC522)
• Experience designing or evaluating AI agents, LLM-powered applications, or agentic workflows (e.g. email/doc assistants, tool-using agents).
• Familiarity with RL Gym-style task formats, Gymnasium/OpenAI Gym, or agent-evaluation harnesses.
• Experience red-teaming production AI systems or building adversarial benchmark datasets.
• Experience with automated evaluation pipelines, regression testing, and quality gates.
• Familiarity with safety/security taxonomies (harm types, attack patterns) and threat modeling.