Skip to main content

How to evaluate a SAST tool: a precision-first checklist

· 9 min read
Vulkro
Vulkro
Security research

Knowing how to evaluate a SAST tool matters more than any feature list, because the demo is the worst evidence you will get. A vendor demo runs on code chosen to show findings, usually a deliberately vulnerable app the tool has seen many times. Your code is different: older, larger, built on frameworks the demo never touched, and full of safe code that looks risky.

The real cost of a static analysis tool is not the licence. It is the hours your engineers spend deciding whether each finding is real. A tool that reports more findings is not better if most of them are wrong. This checklist is built around that: measure precision first, on your code, and ask the questions that a findings count hides.

Why precision matters more than the finding count​

The research on this is old and consistent. Johnson et al. (ICSE 2013) interviewed 20 developers about static analysis. All of them saw value in it, yet false positives and the way warnings were presented stood out as reasons not to use it. At Google, Sadowski et al. (CACM 2018) define an "effective false positive" as an issue developers took no positive action on. Checks shown in code review must stay under 10% of those, and analyzers are switched off when developers mark too many results not useful.

The lesson for a buyer: a tool people stop reading finds nothing. Recall (how many real bugs it catches) still matters, but precision decides whether anyone acts on the results.

How do you evaluate a SAST tool on your own code?​

Pick two or three repositories that represent what you actually ship:

  • One old, large service. This shows how the tool handles legacy code and how much noise a first run produces.
  • One recent service on your current frameworks.
  • One with known history. Find vulnerabilities you already fixed (from a pen test, a bug bounty or an incident) and check out the commit before each fix. A tool that misses a bug you know was there tells you more than any benchmark.

Run each tool with its default settings first, because that is what your developers will see. Then run its strictest and broadest modes to see the range.

Measure the precision of Critical findings​

Do not score every finding. Score the ones that will page someone or block a release.

  1. Take every Critical finding, or a random sample of 30 if there are more.
  2. Have two engineers label each one: real, false, or cannot tell. Disagreements are useful data.
  3. Precision is real divided by (real plus false). Report "cannot tell" separately: a finding nobody can decide is a cost too.
  4. Do the same for High, then check recall against the known historical bugs.

A tool that rates a safe lookup Critical will train your team to ignore Critical. That is the failure to look for.

Ask what the tool did not read​

Zero findings can mean the code is fine, or that the tool never parsed it. Unsupported syntax or a file over a size limit produces silence that looks like a pass. Ask every tool for a coverage report: files present, files analysed, and the specific files it could not parse. We wrote about this failure in a clean scan that did not read your code. Vulkro reports it in scan_coverage in the JSON output and in the local console.

Demand proof per finding​

A finding is a claim. Ask what backs it. For an injection, that means the path from the request to the dangerous call, with the code at each step. For a secret, whether it matches a provider's key format or was confirmed live. For a dependency advisory, whether your code reaches the vulnerable function.

Also ask how the tool labels findings it cannot prove. Some classes, such as whether a record lookup is authorised for this caller, often cannot be decided by static analysis alone. An honest tool says so instead of rating a guess Critical. Vulkro labels each finding Proven, Unproven or Not checked, and keeps Critical and High for findings with evidence; see proven, unproven, not checked.

Is it deterministic?​

Run the same scan twice on the same commit and compare:

vulkro scan . --format json > run1.json
vulkro scan . --format json > run2.json
diff <(jq -S '.findings' run1.json) <(jq -S '.findings' run2.json)

If findings appear and disappear between identical runs, you cannot gate on the tool, compare scans over time, or tell whether a fix worked. That is the core problem with asking a language model to find vulnerabilities, covered in deterministic analysis vs asking a model.

Where does your code go?​

Ask for a written list of every outbound network call, what each sends and how to turn it off. Then test it: run the scan with no network interface and see whether it completes with the same results. For regulated or isolated environments, the licence and the vulnerability data must also work offline; our post on air-gapped SAST covers what that takes. Vulkro publishes its full list on the network egress page.

Does it fit how you ship?​

A tool that cannot gate a pull request on its own changes ends up as a quarterly report. Check for:

  • Diff-scoped gating: failing only on findings a change added, so old debt does not block every build. See security gates in CI.
  • Distinct exit codes for "findings" and "error", so a crash never passes as clean.
  • SARIF output: SARIF 2.1.0 is the OASIS standard most code review platforms read.
  • Findings in the editor, so developers see problems before CI does.

Check coverage of your languages and frameworks, at depth​

A language on a list is not the same as depth. Ask whether the tool follows data across files and function calls in each language you use, and which of your frameworks it models (routing, auth middleware, ORM). Vulkro Core covers JavaScript, TypeScript, Python, Go, Java, PHP, C and C++; the taint analysis page states how deep each one goes, including where analysis stays within a single function.

What about AI-written code?​

In many teams, AI coding agents now write a growing share of the code, and they write it fast. Ask whether the tool can check an agent's edit as it is made, not only in nightly CI, and whether it can scan just the lines a change touched so the feedback is quick. Code from an assistant deserves the same evidence standard as code from a person, no more and no less.

How to read a SAST benchmark​

Benchmarks are useful if you read them carefully. The OWASP Benchmark is a good model: each test case holds one intentional weakness that is either a real vulnerability or a false alarm, and tools are scored on both. Its score is a Youden index, true positive rate minus false positive rate, so a tool that flags everything scores no better than random guessing. Its limits are worth knowing too: the test cases are synthetic, and a tool can be tuned to a public suite.

When a vendor shows you a benchmark, ask: Was the test code ever seen by the people building the tool? Are misses and false alarms published next to the hits? Is it a comparison with other tools, or only with the tool's own earlier releases?

We publish our method on the proof page and in the benchmark docs. The test apps are blind holdouts the engine team never reads, the misses and the safe code flagged by mistake are shown with the hits, and the page prints where results got worse between releases. It also states what the number is not: not a comparison with other tools, and not precision on real-world code. The apps stay private, because publishing them would end their use as a blind test. The test that decides anything is still your own code.

SAST evaluation checklist​

  • Tested on your repositories, including one with known past bugs
  • Precision of Critical and High measured by two reviewers
  • Coverage report: what was not read, and why
  • Evidence per finding, and honest labels for what cannot be proven
  • Same input, same output, run twice
  • Every network call listed, and tested offline
  • Diff-scoped gating, distinct exit codes, SARIF
  • Depth per language and framework, not just a list
  • A way to check AI-written changes as they happen

Frequently asked questions​

What is a good false positive rate for a SAST tool?​

There is no universal number, but Google's code review checks aim for under 10% effective false positives, meaning developers act on at least nine in ten. Measure it on your own Critical and High findings, not on a vendor's demo app.

Should I use the OWASP Benchmark to compare SAST tools?​

It is a useful sanity check, especially for Java, because it scores false alarms as well as hits. Treat it as one input: it is synthetic and public, so it cannot replace a test on your own code.

How long should a SAST proof of concept take?​

Long enough to label the findings, which is usually the slow part. Plan for two or three representative repositories and enough reviewer time to label every Critical and High finding.

Is finding more vulnerabilities always better?​

No. A tool that reports more but is wrong more often costs more triage time and gets ignored. Compare precision and recall together, on the same code.

Where to see it​

Browse the use cases to see where Vulkro fits, read the benchmark method, then run it on your own repositories as part of your evaluation. Start here.