Benchmark
Vulkro is measured against six other scanners on one ground-truth corpus, on the same machine, on the same day, with the same scoring code. Every bug in the corpus is pinned to published provenance in a real project (the provenance rule below): a CVE or GHSA advisory where one exists, and otherwise the upstream project's own in-source vulnerability marker or documented challenge, which is how deliberately-vulnerable corpora such as Juice Shop, NodeGoat and DVGA identify their planted bugs. A finding counts only within five lines of the catalogued bug. Every table on this page, including the comparisons we lose, is scored that way. The methodology is published in full; you can check any number against the advisory it cites.
On naming. We report where Vulkro sits against the industry baseline, by category, without naming other vendors' products. The categories below describe what each lane actually is - a rules-based open-source engine, a broad multi-language open-source scanner, a single-language specialist - and the scoring, the corpus and the method are identical whatever the lane is called.
Headline results
The lanes that scan more than one language, rolled up over the
scored corpus: 83 catalogued bugs (one repo whose primary language
Vulkro does not yet support is excluded from the roll-up; the scorecard
states the exclusion).
Vulkro is scored at --min-confidence high, the recommended CI
setting; the other lanes are never confidence-filtered (see
Methodology for why).
Read the precision row as exactly this and nothing more: precision against the catalogued-bug set of this curated corpus, at high confidence. It is not a claim about the share of all findings that are real on an arbitrary repo; the "what this precision is measured over" note below the table states the difference explicitly, and every figure regenerates from the committed scorecard.
| Vulkro | Open-source rules baseline | Same ruleset, alternate engine | Multi-language OSS scanner | |
|---|---|---|---|---|
| Precision | 0.78 | 0.53 | 0.53 | 0.37 |
| Recall | 0.57 | 0.20 | 0.20 | 0.52 |
| F1 | 0.66 | 0.30 | 0.30 | 0.43 |
| Catalogued bugs found (of 83) | 47 | 17 | 17 | 43 |
| False positives | 13 | 15 | 15 | 72 |
| TP / FP / FN | 47 / 13 / 36 | 17 / 15 / 66 | 17 / 15 / 66 | 43 / 72 / 40 |
How to read it:
- Precision first. When Vulkro fires at high confidence, it is right 0.78 of the time. That is the highest precision of any multi-language lane in the run.
- Against the free baseline. Vulkro finds 2.8x the catalogued bugs the open-source rules baseline finds (47 vs 17) at higher precision (0.78 vs 0.53). The first two baseline lanes score identically because both run the same widely-used community security ruleset; the point of the second lane is comparing two engines on identical rules.
- Against the closest lane. The broad multi-language open-source scanner is the only lane close on recall (43 vs 47) and pays for it with 72 false positives to Vulkro's 13.
- What this precision is measured over. These figures count a finding against the catalogued-bug set. Vulkro emitted 565 total findings in this run, so the share of ALL findings that map to a catalogued bug is far lower than 0.78. That is expected for a scanner that also reports IaC, secrets and dependency issues the corpus does not catalogue, but it is not the same statistic and we do not present it as one.
- The PHP and Java repos are not hidden. All four lanes also scan DVWA (PHP), WebGoat, and java-sec-code (Java); the per-repo rows are published in the scorecard. They are excluded from these totals for two different reasons. PHP is out of Vulkro's language scope. Java is now analysed - see Java and Spring - but that support shipped after this corpus was scored (these numbers come from a Vulkro 0.15.0 run), so the Java repos stay out of the roll-up pending a re-score. The multi-language lane's wins on those repos are called out in Where Vulkro misses.
The corpus, version 2026-07-05
The Tier 1 corpus is 15 public repositories, pinned by SHA: 12 deliberately-vulnerable apps with 90 catalogued bugs, plus 3 clean framework controls (express, fastapi, flask) that must produce zero findings. 76 of the 90 bugs are in the supported-language roll-up above; the other 14 live in the PHP and Java repos.
We grew our own ground truth from 69 to 90 bugs in July 2026 and published the recall drop that came with it. The growth batch added two repos chosen to be hard for us (DVGA, a Python GraphQL app, and govwa, a Go app - both score 0 Vulkro recall today) and 21 previously-missed bugs in existing repos. The corpus is versioned; every change is in the changelog.
The provenance rule. Every catalogued bug must be independently
verifiable and must cite the repo's own vulnerability documentation as
the primary source: its README or solutions doc, an in-source marker
(juice-shop's vuln-code-snippet comments, NodeGoat's tutorial
pages), or a published CVE / GHSA pinned to the commit. Each entry is
then confirmed by reading the sink at the pinned SHA. Selection is
not biased toward what Vulkro detects: catalogued bugs Vulkro misses
stay in the corpus as legitimate false negatives. That honesty is the
point of the benchmark.
[[vuln]]
file = "app/routes/contributions.js"
line = 32
class = "command-injection"
cwe = "CWE-94"
note = "eval(req.body.preTax) - A1 SSJS injection (source comment line 31)"
Methodology
Ground truth and matching
Each catalogued bug is a (file, line, class) tuple. A finding
matches a ground-truth entry iff the files are equal and the finding
line is within 5 lines of the catalogued line (+/-5). Each
ground-truth entry can be claimed by at most one finding (greedy
nearest), so over-firing on the same bug does not inflate the score.
The four buckets
| bucket | meaning |
|---|---|
| TP | Tool fired on a catalogued bug (same file, line within +/-5). |
| FP | Tool fired in a file we catalogued, but at no catalogued bug line. |
| FN | Catalogued bug, no tool finding fired against it. |
| OOS | Tool fired in a file we did not catalogue. Neither rewarded nor penalized: we do not claim to know whether those files are clean or just unannotated. |
The OOS bucket is what separates "noisy" from "broad": a tool that fires thousands of times outside the catalogued files is neither punished for it nor credited with it.
Confidence filtering: Vulkro only, and why
Vulkro is scored at --min-confidence high for the headline (its
recommended CI setting) and at low for the labeled max-recall run.
The other lanes are never confidence-filtered, because their
confidence ladders are not directly comparable to ours: dropping
another tool's "low confidence" findings using our idea of what "low"
means would silently rig the comparison. Every other lane runs at its
own defaults instead.
Metrics
- precision = TP / (TP + FP): when the tool fires, how often is it right?
- recall = TP / (TP + FN): of the catalogued bugs, how many did it find?
- F1 = 2 x (precision x recall) / (precision + recall): the standard combined score.
Per-language specialists: the fair-fight rule
Some scanners target exactly one language. Rolling a single-language tool up over a mixed corpus would manufacture zeros for it, so the specialists never enter the roll-up. Instead, each specialist is scored only over the repos matching its language, and Vulkro is re-scored over the same repo subset. Every comparison below is a fair fight. Vulkro wins two of the three; the one it loses is printed just as large.
Python specialist vs Vulkro, 7 Python repos (42 catalogued bugs)
| Python-only specialist | Vulkro (same repos) | |
|---|---|---|
| Precision | 0.56 | 0.87 |
| Recall | 0.43 | 0.64 |
| TP / FP / FN | 18 / 14 / 24 | 27 / 4 / 15 |
On its home language, the Python specialist finds 18 of the 42 catalogued Python bugs with 14 false positives; Vulkro finds 27 with 4.
Go specialist vs Vulkro, 1 Go repo (4 catalogued bugs): the specialist wins
| Go-only specialist | Vulkro (same repo) | |
|---|---|---|
| Precision | 0.20 | 0.00 |
| Recall | 0.75 | 0.00 |
| TP / FP / FN | 3 / 12 / 1 | 0 / 1 / 4 |
On our one Go repo, the Go specialist finds 3 of 4 bugs; Vulkro
finds 0. Go detection depth is a known gap we publish. The four
misses (a fmt.Sprintf SQL interpolation, an html/template XSS, and
two HttpOnly: false insecure cookies in govwa) stay catalogued as
false negatives rather than being quietly dropped from the corpus. If
Go is your primary language, a dedicated free Go specialist beats us
there today, and you should run one.
JavaScript specialist vs Vulkro, 4 JS/TS repos (30 catalogued bugs)
| JavaScript-only specialist | Vulkro (same repos) | |
|---|---|---|
| Precision | 0.50 | 0.77 |
| Recall | 0.33 | 0.57 |
| TP / FP / FN | 10 / 10 / 20 | 17 / 5 / 13 |
One caveat that the per-repo rows keep visible: this specialist
targets plain JavaScript, so on the TypeScript repo (juice-shop) it
sees almost nothing in .ts files and scores 0 of 8 catalogued bugs
there.
Max-recall mode
Drop the confidence threshold (--min-confidence low, the default)
and Vulkro trades precision for recall. The published scorecard
labels this run as max-recall:
| metric | Vulkro max-recall | Vulkro headline (high) |
|---|---|---|
| Precision | 0.15 | 0.78 |
| Recall | 0.76 | 0.57 |
| TP / FP / FN | 63 / 347 / 20 | 47 / 13 / 36 |
Max-recall finds 63 of the 83 roll-up bugs at the cost of 347 false
positives. It is the right setting for a one-shot audit where a human
will triage everything once; --min-confidence high is the right
setting for CI. We publish both so the trade-off is yours to make,
not ours to hide.
Where Vulkro misses
The corpus is built to keep our failures on the record:
- GraphQL resolver taint. DVGA (Damn Vulnerable GraphQL Application) has 6 catalogued bugs reached through GraphQL resolvers. Vulkro finds 0 of them. Tracing taint through resolver dispatch is an open detection gap.
- Go. govwa has 4 catalogued bugs; Vulkro finds 0, the Go specialist finds 3 (see the fair-fight table above).
- PHP and Java. DVWA (7 bugs, PHP) is outside Vulkro's language scope entirely. WebGoat (4) and java-sec-code (3) are Java: this scorecard was generated with Vulkro 0.15.0, before Java/Spring support shipped, so those repos score 0 here and stay excluded from the roll-up until the corpus is re-scored with Java analysis on. The broad multi-language lane finds 9 of those 14 catalogued bugs. Java coverage today is documented on the Java and Spring page.
By bug class, the 32 headline-run misses across the supported-language repos concentrate in: SQL injection (11), command injection (8), XSS (6), open redirect (4), plus single-digit counts of insecure deserialization, insecure cookies, SSRF, JWT verification, path traversal, and LFI. The full FN-by-class table is regenerated in every scorecard.
Speed
Wall clock for the whole 15-repo corpus, one full end-to-end process invocation per tool per repo. Measured 2026-07-04 on an Apple M5 Pro, 24 GB RAM, macOS 26.5.1.
| lane | whole-corpus scan time |
|---|---|
| Go-only specialist | 0.6s (1 Go repo only) |
| Python-only specialist | 3.2s (7 Python repos only) |
| JavaScript-only specialist | 24.1s (4 JS/TS repos only) |
| Vulkro 0.15.0 (high confidence) | 38.0s |
| Vulkro 0.15.0 (default) | 39.3s |
| Same ruleset, alternate engine | 46.1s |
| Open-source rules baseline | 48.9s |
| Multi-language OSS scanner | 363.0s |
The honesty notes that come with that table:
- This reference is a single cold run per tool per repo
(
--timing-runs 1), because the 7-lane matrix makes median-of-3 a very long session. For publishable single-tool comparisons the harness supports--timing-runs 3and we recommend it; the aggregation line embedded in the scorecard records exactly what each run did. - Repos are pre-cloned to local disk before timing; clone time is excluded. Every sample includes process start, scan, and report serialization.
- Tool-internal result caches are disabled where the tool supports it, so every sample is a cold scan. The multi-language lane has an incremental cache that makes its repeat scans on unchanged code much faster than shown here; the table just does not mix cold and warm numbers.
- The specialists only scan their matching-language repos, so their totals cover only that subset. A 0.6s single-language total is not comparable to a 38.0s whole-corpus total and the table says so.
- Wall clock is machine-dependent. Compare the ratios between lanes on the same machine, not absolute seconds across machines.
How the numbers are produced
You do not have to take any number on this page on faith. Two things make it checkable. First, every catalogued bug cites a public CVE or GHSA advisory in a real project (the provenance rule above), so you can verify any single entry against its source. Second, the exact method is written down. The harness runs one command per configuration:
# the headline setting (min-confidence high, the recommended CI setting)
bench/comparison/run.sh --tier1 --min-confidence high
# speed numbers (median-of-3)
bench/comparison/run.sh --tier1 --timing-runs 3
Each run writes raw findings, normalized JSON, the markdown scorecard,
runtime.json, and timing_meta.json (hardware, run count, cache
state, and the tool versions actually used).
One honest limitation: the harness and the pinned corpus live in our private repository today, so you cannot run these commands yourself yet. We publish the method, the per-bug advisories, and the comparisons we lose, not a downloadable corpus. If we publish the corpus, this section will say so and link it.
What "reproducible" means for the run itself: same Vulkro binary + same corpus SHAs + same scoring code = same TP/FP/FN counts, every run, on any machine. Known sources of drift:
- The baseline ruleset is not pinned. The community security ruleset behind the two baseline lanes is served live by its registry and changes over time, so those counts can move without any change on our side. Our corpus SHAs and scoring code are pinned; someone else's rule feed is not ours to pin.
- Third-party versions. Newer releases of any lane change its rules. The scorecard records the versions it ran; compare like with like.
- Hardware moves speed numbers. Compare ratios, not absolute seconds.
The lanes in the published run
| lane | what it is | note |
|---|---|---|
| Vulkro 0.15.0 | this product | scored at --min-confidence high (headline) and low (max-recall) |
| Open-source rules baseline | free rules-based engine on a widely-used community security ruleset | ruleset fetched live, not pinned upstream |
| Same ruleset, alternate engine | a second engine running the identical ruleset | isolates engine differences from rule differences |
| Multi-language OSS scanner | broad open-source scanner with the widest language reach in the run | data-flow lane, cache disabled |
| Python-only specialist | single-language scanner | Python repos only (specialists section) |
| Go-only specialist | single-language scanner | Go repos only (specialists section) |
| JavaScript-only specialist | single-language scanner | JS/TS repos only; bundles its own rules engine |
The exact products and version strings behind each lane are recorded in the internal scorecard that generates this page, so every figure stays traceable on our side.
Lanes not included
Authenticated cloud SAST products are deliberately not part of the
published runs: they require an authenticated account, and publishing
benchmark results may be restricted by their terms of service (a check
we have not completed). The harness keeps working runners for them:
add the runner to --tools and authenticate to produce the numbers
locally for yourself, after checking the current terms. The scorecards
carry a standing "not included" note so this gap is a documented
decision, not a silent one.
DAST tools are excluded by shape, not by policy: they cannot scan a static code directory at all, and exist only as documented stub lanes in the harness.
Caveats: what these numbers do not tell you
- The corpus is deliberately-vulnerable code. Real-world performance varies; deliberately-vulnerable apps are denser in bugs and lighter in framework noise than production code. The 3 clean framework controls (express, fastapi, flask) are the noise-floor check inside the corpus.
- Class matching matters. A finding under the wrong class does not count as a TP even when the line is right. This penalizes every lane roughly equally.
- Ground truth is a floor, not a census. The 90 catalogued bugs are the bugs we could verify by provenance; the OOS bucket exists precisely because we do not pretend the rest of every file is clean.
- A benchmark is one input. Run the tools on your own code. The harness exists so that "run it yourself" is a one-command suggestion rather than a slogan.
Related
- Compare page - the same numbers next to the qualitative capability table and the switch guides.
- Confidence model - how
HighvsMediumvsLowis computed from evidence weights. - OWASP API Top 10 - per-class detector inventory.
- Reproducing in CI - running the benchmark as part of every PR.