Benchmarks

Bug-bounty evaluation of 40 historical bounties across 25 real systems with separate Detect, Exploit, and Patch tasks and executable graders.

Forty containerized critical real-world web CVEs with deterministic/executable graders for testing verified completion.

Java white-box discovery/exploitation/patching benchmark from the IRIS-SAST project.

CTF benchmark for long-horizon planning and recovery.

Large-scale benchmark for source analysis and PoC generation.

Common evaluation harness/plumbing (from NIST CAISI) for running CVE-Bench, Cybench, and related environments rather than new training content.

NYU LLM CTF benchmark test set for long-horizon planning and recovery.

Benchmark for white-box vulnerability discovery, exploitation, and patching.

XBOW-engineering validation benchmark set.

An OpenSSF benchmark containing code and metadata for over 200 real-life CVEs, with tooling to evaluate Static Analysis Security Testing (SAST) solutions by measuring vulnerability detection rates and false positives against vulnerable/patched open-source code.

MIT-licensed one-notebook benchmark that sends 116 labeled prompts to TypeSafe's hosted system_one API and reports per-call latency plus a confusion matrix of its prompt-injection verdicts; needs a TypeSafe account.

MIT-licensed Go benchmark that scores TypeSafe's hosted Jev model on 662 labelled messages (263 injections) and 200 matched vulnerable/secure code pairs; needs a TypeSafe API key, and the documented runner is not in the tree.