Run a SAST benchmark on your own code: measuring accuracy and noise

Every SAST vendor has a benchmark, and every benchmark shows the vendor doing well. That is not always dishonest. It is usually overfitting: the tool was tuned on the same vulnerable apps it is scored on, so the score says more about the tuning than about your code.

The only benchmark that answers "will this tool be useful for us" is one you run on your own repositories. This post is a method for doing that in a week or two, with numbers you can defend in a procurement meeting.

#Decide what you are measuring

Three numbers matter, and they pull against each other:

• Recall: of the real vulnerabilities in the code, how many did the tool report?
• Precision: of the findings the tool reported, how many were real?
• Noise cost: how much engineer time does it take to get from the raw report to the real issues?

A tool can buy recall by reporting everything, and buy precision by reporting almost nothing. You need ground truth for both, and the two need different sources.

#Build ground truth for recall

You cannot measure recall without knowing where the bugs are. There are two practical ways to get that list on your own code.

#Seeded bugs

Pick a branch of a real service and plant vulnerabilities by hand: a query built from a request parameter, a missing authorization check on one route, a path built from user input, a hard-coded credential. Write each one down with its file, line, class and the severity you expect.

Good seeding rules:

• Write the bugs the way your team writes code, using your framework and helpers, not textbook examples.
• Plant safe lookalikes next to them: the same pattern with a real sanitizer or a real auth check. These measure false positives on code that looks dangerous.
• Have someone other than the evaluator plant them, so the person running tools does not know the answers.

#Known-fixed vulnerabilities

Your issue tracker and git history are a second source. Find security fixes your team already shipped, or CVE fix commits in third-party dependencies you vendor. Check out the parent of each fix commit and scan it. The vulnerability is real, it was found in the wild, and the fix tells you exactly where it lived.

This is the strongest recall test, because nobody designed these bugs to be found.

#Measure precision with a triage sample

Precision needs a different method. On a real codebase you do not know every bug, so you cannot score findings against a complete list. Instead, sample.

1. Run the tool at the settings you would use in CI.
2. Take a random sample of findings. Fifty is enough to see a pattern; stratify by severity so Criticals are not drowned out by Lows.
3. Have an engineer who knows the code mark each one: real and exploitable, real but not reachable, or false positive.
4. Record how long each decision took.

Report precision per severity. A tool whose Criticals are mostly wrong is worse than one whose Lows are noisy, because Criticals are what block releases and page people.

#Count unique root causes, not rows

Tools differ wildly in how they report one bug. One reports a single finding for an injection; another reports it at the source, at every hop and at the sink. If you count rows, the chatty tool looks like it found more.

Before scoring, collapse findings to unique root causes. A practical rule: one finding per vulnerable sink, per data flow. When you match findings to seeded bugs, let each bug be claimed once, and use a fixed line window (a few lines either side) so matching is mechanical rather than a judgement call.

#Separate unseen code from tuned code

If a tool has been tuned on a public vulnerable app, its score on that app is not evidence. Keep two buckets:

Expect the second number to be lower. If it is dramatically lower, that gap is the overfitting, and it is the most useful thing you will learn from the exercise.

#Measure noise per 1,000 lines

Precision is a ratio, so it hides volume. A tool that is 50% precise and reports 400 findings costs far more than one that is 50% precise and reports 40. Normalise:

• Findings per 1,000 lines of code at your CI settings, per severity.
• Median time to triage a finding, from the timings in your sample.
• Triage hours per release: new findings per release multiplied by median triage time.
• Triage minutes per real issue: triage hours per release divided by the real issues among those findings (findings multiplied by precision).

The last two figures are the cost your team will actually pay, and they are the ones to bring to a budget conversation.

#Control the run

Benchmarks drift when the run is not controlled. Pin the tool version, record every flag, disable caches, and run twice to confirm the output is identical. A tool that gives different findings on the same commit makes every other number in your benchmark unreliable.

Also record what the tool did not read: files it skipped, languages it did not parse, routes it could not resolve. A clean report on code the tool never analysed is not a pass.

#How Vulkro publishes its benchmark

We publish our benchmark page with the method behind each number and the exact scan settings, so you can run the same commands on your own code:

export VULKRO_OFFLINE=1 VULKRO_DISABLE_CACHE=1 VULKRO_DETERMINISTIC=1
vulkro scan ./app --format json --all-confidence

The page covers blind holdout applications that nobody working on the engine reads, how findings are matched to labels, how severity is scored, and a known misses section. It also says plainly what the numbers are not: planted-bug apps are denser than production code, so the holdout result is not real-world precision. The holdouts themselves stay private, because publishing them would end their use as a blind test. That is the same reason your own benchmark should keep its answer key away from whoever runs the tools.

Every Vulkro finding is marked Proven, Unproven or not checked, which helps with step three: a triager can start from the attack path behind a Proven finding rather than reconstructing it.

#FAQ

#How do you evaluate a SAST tool for accuracy and noise?

Measure recall against bugs you know are there (seeded bugs and historical security fixes), measure precision on a random, severity-stratified triage sample, and normalise noise as findings per 1,000 lines and triage hours per release. Score unique root causes, not raw rows.

#How many findings should a SAST triage sample include?

Around fifty per tool is enough to see whether precision is high, middling or poor, as long as the sample is random and stratified by severity. Increase it if the decision is close or if Criticals are rare in the sample.

#Are public SAST benchmarks useful?

As a sanity check, yes. As a buying signal, only weakly, because vendors can tune on public corpora. Treat a score on your own unseen code as the real result, and treat a large gap between the two as overfitting.

See the best offline SAST page for how Vulkro fits this checklist. Related reading: how to evaluate a SAST tool.