Benchmark
Vulkro is measured against six other scanners on one ground-truth corpus, on the same machine, on the same day, with the same scoring code. Every bug in the corpus is pinned to published provenance in a real project (the provenance rule below): a CVE or GHSA advisory where one exists, and otherwise the upstream project's own in-source vulnerability marker or documented challenge, which is how deliberately-vulnerable corpora such as Juice Shop, NodeGoat and DVGA identify their planted bugs. A finding counts only within five lines of the catalogued bug. Every table on this page, including the comparisons we lose, is scored that way. The methodology is published in full; you can check any number against the advisory it cites.
Headline results
The four tools that scan more than one language, rolled up over the
scored corpus: 83 catalogued bugs (one repo whose primary language
Vulkro does not yet support is excluded from the roll-up; the scorecard
states the exclusion).
Vulkro is scored at --min-confidence high, the recommended CI
setting; the other tools' findings are never confidence-filtered (see
Methodology for why).
Read the precision row as exactly this and nothing more: precision against the catalogued-bug set of this curated corpus, at high confidence. It is not a claim about the share of all findings that are real on an arbitrary repo; the "what this precision is measured over" note below the table states the difference explicitly, and every figure regenerates from the committed scorecard.
| Vulkro | Tool A 2.0.2 | Tool B 1.136.0 | Tool E 1.25.0 | |
|---|---|---|---|---|
| Precision | 0.62 | 0.37 | 0.53 | 0.53 |
| Recall | 0.40 | 0.52 | 0.20 | 0.20 |
| F1 | 0.49 | 0.43 | 0.30 | 0.30 |
| Catalogued bugs found (of 83) | 33 | 43 | 17 | 17 |
| False positives | 20 | 72 | 15 | 15 |
| TP / FP / FN | 33 / 20 / 50 | 43 / 72 / 40 | 17 / 15 / 66 | 17 / 15 / 66 |
Why the other tools are not named. They appear as Tool A through Tool F, here and in every table below. That is site policy: we do not put other vendors' names on our own pages. The labels are stable across this page, and each tool keeps the pinned version number it ran at. State the cost plainly: with the names withheld you cannot take a row from this page and go check it against a named product yourself. What is not withheld is everything that decides what the numbers mean. The corpus and its provenance rule, the matching rule, the four buckets, the scoring formulas and the exact commands the harness runs are all published below, and the rows Vulkro loses are printed at the same size as the rows it wins. We do not publish the mapping, and the order of the labels carries no information.
How to read it:
- Precision first. When Vulkro fires at high confidence, it is right 0.62 of the time. That is the highest precision of any multi-language tool in the run.
- Against the free baseline. Vulkro finds 1.9x the catalogued bugs Tool B finds (33 vs 17) at higher precision (0.62 vs 0.53). Tool B and Tool E score identically because both lanes run the same live community security ruleset; the point of the Tool E lane is comparing engines on identical rules.
- Where a rival scores higher. Tool A finds more of the catalogued bugs (43 vs 33) and pays for it with 72 false positives to Vulkro's 20, and about eleven times the runtime. Higher recall, far lower precision: the trade this whole page is about.
- What this precision is measured over. These figures count a finding against the catalogued-bug set. Vulkro emitted 300 total findings in this run, so the share of ALL findings that map to a catalogued bug is far lower than 0.62. That is expected for a scanner that also reports IaC, secrets and dependency issues the corpus does not catalogue, but it is not the same statistic and we do not present it as one.
- The PHP and Java repos are not hidden. All four tools also scan DVWA (PHP), WebGoat, and java-sec-code (Java); the per-repo rows are published in the scorecard. They are excluded from these totals for two different reasons. PHP is out of Vulkro's language scope. Java is now analysed - see Java and Spring - but that support shipped after this corpus was scored (these numbers come from a Vulkro 0.15.0 run), so the Java repos stay out of the roll-up pending a re-score. Tool A's wins on those repos are called out in Where Vulkro misses.
The corpus, version 2026-07-05
The Tier 1 corpus is 15 public repositories, pinned by SHA: 12 deliberately-vulnerable apps with 90 catalogued bugs, plus 3 clean framework controls (express, fastapi, flask) that must produce zero findings. 76 of the 90 bugs are in the supported-language roll-up above; the other 14 live in the PHP and Java repos.
We grew our own ground truth from 69 to 90 bugs in July 2026 and published the recall drop that came with it. The growth batch added two repos chosen to be hard for us (DVGA, a Python GraphQL app, and govwa, a Go app - both score 0 Vulkro recall today) and 21 previously-missed bugs in existing repos. The corpus is versioned; every change is in the changelog.
The provenance rule. Every catalogued bug must be independently
verifiable and must cite the repo's own vulnerability documentation as
the primary source: its README or solutions doc, an in-source marker
(juice-shop's vuln-code-snippet comments, NodeGoat's tutorial
pages), or a published CVE / GHSA pinned to the commit. Each entry is
then confirmed by reading the sink at the pinned SHA. Selection is
not biased toward what Vulkro detects: catalogued bugs Vulkro misses
stay in the corpus as legitimate false negatives. That honesty is the
point of the benchmark.
[[vuln]]
file = "app/routes/contributions.js"
line = 32
class = "command-injection"
cwe = "CWE-94"
note = "eval(req.body.preTax) - A1 SSJS injection (source comment line 31)"
Methodology
Ground truth and matching
Each catalogued bug is a (file, line, class) tuple. A finding
matches a ground-truth entry iff the files are equal and the finding
line is within 5 lines of the catalogued line (+/-5). Each
ground-truth entry can be claimed by at most one finding (greedy
nearest), so over-firing on the same bug does not inflate the score.
The four buckets
| bucket | meaning |
|---|---|
| TP | Tool fired on a catalogued bug (same file, line within +/-5). |
| FP | Tool fired in a file we catalogued, but at no catalogued bug line. |
| FN | Catalogued bug, no tool finding fired against it. |
| OOS | Tool fired in a file we did not catalogue. Neither rewarded nor penalized: we do not claim to know whether those files are clean or just unannotated. |
The OOS bucket is what separates "noisy" from "broad": a tool that fires thousands of times outside the catalogued files is neither punished for it nor credited with it.
Confidence filtering: Vulkro only, and why
Vulkro is scored at --min-confidence high for the headline (its
recommended CI setting) and at low for the labeled max-recall run.
The other tools' findings are never confidence-filtered, because
their confidence ladders are not directly comparable to ours: dropping
another tool's "low confidence" findings using our idea of what "low"
means would silently rig the comparison. Every other lane runs at its
own defaults instead.
Metrics
- precision = TP / (TP + FP): when the tool fires, how often is it right?
- recall = TP / (TP + FN): of the catalogued bugs, how many did it find?
- F1 = 2 x (precision x recall) / (precision + recall): the standard combined score.
Per-language specialists: the fair-fight rule
Tool F only scans Python, Tool D only Go, Tool C only JavaScript. Rolling a single-language tool up over a mixed corpus would manufacture zeros for it, so the specialists never enter the roll-up. Instead, each specialist is scored only over the repos matching its language, and Vulkro is re-scored over the same repo subset. Every comparison below is a fair fight. Vulkro wins two of the three; the one it loses is printed just as large.
Tool F vs Vulkro, 7 Python repos (42 catalogued bugs)
| Tool F 1.9.4 | Vulkro (same repos) | |
|---|---|---|
| Precision | 0.56 | 0.87 |
| Recall | 0.43 | 0.64 |
| TP / FP / FN | 18 / 14 / 24 | 27 / 4 / 15 |
On its home language, Tool F finds 18 of the 42 catalogued Python bugs with 14 false positives; Vulkro finds 27 with 4.
Tool D vs Vulkro, 1 Go repo (4 catalogued bugs): Tool D wins
| Tool D 2.27.1 | Vulkro (same repo) | |
|---|---|---|
| Precision | 0.20 | 0.00 |
| Recall | 0.75 | 0.00 |
| TP / FP / FN | 3 / 12 / 1 | 0 / 1 / 4 |
On our one Go repo, Tool D finds 3 of 4 bugs; Vulkro finds 0. Go
detection depth is a known gap we publish. The four misses
(a fmt.Sprintf SQL interpolation, an html/template XSS, and two
HttpOnly: false insecure cookies in govwa) stay catalogued as false
negatives rather than being quietly dropped from the corpus. If Go is
your primary language, Tool D is free and it beats us there today.
Tool C vs Vulkro, 4 JS/TS repos (30 catalogued bugs)
| Tool C 0.4.3 | Vulkro (same repos) | |
|---|---|---|
| Precision | 0.50 | 0.77 |
| Recall | 0.33 | 0.57 |
| TP / FP / FN | 10 / 10 / 20 | 17 / 5 / 13 |
One caveat that the per-repo rows keep visible: Tool C targets plain
JavaScript, so on the TypeScript repo (juice-shop) it sees almost
nothing in .ts files and scores 0 of 8 catalogued bugs there.
Max-recall mode
Drop the confidence threshold (--min-confidence low, the default)
and Vulkro trades precision for recall. The published scorecard
labels this run as max-recall:
| metric | Vulkro max-recall | Vulkro headline (high) |
|---|---|---|
| Precision | 0.15 | 0.62 |
| Recall | 0.76 | 0.40 |
| TP / FP / FN | 63 / 347 / 20 | 33 / 20 / 50 |
Max-recall finds 63 of the 83 roll-up bugs at the cost of 347 false
positives. It is the right setting for a one-shot audit where a human
will triage everything once; --min-confidence high is the right
setting for CI. We publish both so the trade-off is yours to make,
not ours to hide.
Where Vulkro misses
The corpus is built to keep our failures on the record:
- GraphQL resolver taint. DVGA (Damn Vulnerable GraphQL Application) has 6 catalogued bugs reached through GraphQL resolvers. Vulkro finds 0 of them. Tracing taint through resolver dispatch is an open detection gap.
- Go. govwa has 4 catalogued bugs; Vulkro finds 0, Tool D finds 3 (see the fair-fight table above).
- PHP and Java. DVWA (7 bugs, PHP) is outside Vulkro's language scope entirely. WebGoat (4) and java-sec-code (3) are Java: this scorecard was generated with Vulkro 0.15.0, before Java/Spring support shipped, so those repos score 0 here and stay excluded from the roll-up until the corpus is re-scored with Java analysis on. Tool A finds 9 of those 14 catalogued bugs. Java coverage today is documented on the Java and Spring page.
By bug class, the 32 headline-run misses across the supported-language repos concentrate in: SQL injection (11), command injection (8), XSS (6), open redirect (4), plus single-digit counts of insecure deserialization, insecure cookies, SSRF, JWT verification, path traversal, and LFI. The full FN-by-class table is regenerated in every scorecard.
Speed
Wall clock for the whole 15-repo corpus, one full end-to-end process invocation per tool per repo. Measured 2026-07-04 on an Apple M5 Pro, 24 GB RAM, macOS 26.5.1.
| tool | whole-corpus scan time |
|---|---|
| Tool D 2.27.1 | 0.6s (1 Go repo only) |
| Tool F 1.9.4 | 3.2s (7 Python repos only) |
| Tool C 0.4.3 | 24.1s (4 JS/TS repos only) |
| Vulkro 0.15.0 (high confidence) | 38.0s |
| Vulkro 0.15.0 (default) | 39.3s |
| Tool E 1.25.0 | 46.1s |
| Tool B 1.136.0 | 48.9s |
| Tool A 2.0.2 | 363.0s |
The honesty notes that come with that table:
- This reference is a single cold run per tool per repo
(
--timing-runs 1), because the 7-tool matrix makes median-of-3 a very long session. For publishable single-tool comparisons the harness supports--timing-runs 3and we recommend it; the aggregation line embedded in the scorecard records exactly what each run did. - Repos are pre-cloned to local disk before timing; clone time is excluded. Every sample includes process start, scan, and report serialization.
- Tool-internal result caches are disabled where the tool supports it (Tool A runs with its cache forced off), so every sample is a cold scan. Tool A's incremental cache makes its repeat scans on unchanged code much faster than shown here; the table just does not mix cold and warm numbers.
- The specialists only scan their matching-language repos, so their totals cover only that subset. A 0.6s single-language total is not comparable to a 38.0s whole-corpus total and the table says so.
- Wall clock is machine-dependent. Compare the ratios between tools on the same machine, not absolute seconds across machines.
How the numbers are produced
You do not have to take any number on this page on faith. Two things make it checkable. First, every catalogued bug cites a public CVE or GHSA advisory in a real project (the provenance rule above), so you can verify any single entry against its source. Second, the exact method is written down: the scoring rules, the five-line match window, the confidence setting each column is scored at, and the pinned tool versions in the table below.
Each configuration writes raw findings, normalized JSON, the markdown
scorecard, runtime.json, and a timing record (hardware, run count,
cache state, and the tool versions actually used), so a result is
always traceable to the run that produced it.
One honest limitation: the corpus and the internal harness that runs it
live in our private repository, so you cannot rerun our exact scoring
yourself. We publish the method, the per-bug advisories, and the
comparisons we lose, not a downloadable corpus. The check that decides
anything is the scanner run on your own code, at the same
--min-confidence high setting the headline row is scored at.
What "reproducible" means for the run itself: same Vulkro binary + same corpus SHAs + same scoring code = same TP/FP/FN counts, every run, on any machine. Known sources of drift:
- The shared community ruleset is not pinned. The ruleset behind the Tool B and Tool E lanes is served live by its own registry and changes over time, so those two counts can move without any change on our side. Our corpus SHAs and scoring code are pinned; someone else's rule feed is not ours to pin.
- Other tools' versions. Newer releases change rules. The scorecard records the versions it ran; compare like with like.
- Hardware moves speed numbers. Compare ratios, not absolute seconds.
Tool versions in the published run
| tool | version | note |
|---|---|---|
| Vulkro | 0.15.0 | scored at --min-confidence high (headline) and low (max-recall) |
| Tool A | 2.0.2 | SAST lane only, result cache disabled |
| Tool B | 1.136.0 | community security ruleset, fetched live, not pinned upstream |
| Tool C | 0.4.3 | JS/TS repos only; bundles a third-party pattern engine (1.86.0) |
| Tool D | 2.27.1 | Go repos only (specialists section) |
| Tool E | 1.25.0 | same live community security ruleset as the Tool B lane |
| Tool F | 1.9.4 | Python repos only (specialists section) |
Tools not included
One authenticated commercial scanner is deliberately left out of the published runs. It cannot run without an authenticated account, and publishing benchmark results may be restricted by its terms of service (a check we have not completed). The scorecards carry a standing "Tools not included" note so this gap is a documented decision, not a silent one.
Two DAST products are also out. They cannot scan a static code directory at all, so a static-corpus comparison would not be meaningful for them.
Caveats: what these numbers do not tell you
- The corpus is deliberately-vulnerable code. Real-world performance varies; deliberately-vulnerable apps are denser in bugs and lighter in framework noise than production code. The 3 clean framework controls (express, fastapi, flask) are the noise-floor check inside the corpus.
- Class matching matters. A finding under the wrong class does not count as a TP even when the line is right. This penalizes every tool roughly equally.
- Ground truth is a floor, not a census. The 90 catalogued bugs are the bugs we could verify by provenance; the OOS bucket exists precisely because we do not pretend the rest of every file is clean.
- A benchmark is one input. Run the tools on your own code. The harness exists so that "run it yourself" is a one-command suggestion rather than a slogan.
Related
- Confidence model - how
HighvsMediumvsLowis computed from evidence weights. - OWASP API Top 10 - per-class detector inventory.
- Reproducing in CI - running the benchmark as part of every PR.