A clean scan that did not read your code is not a clean scan

A security scan finishes, prints zero findings and exits 0. The pipeline goes green and the pull request merges. Two very different situations produce exactly that output. In the first, the scanner read every file and found nothing worth reporting. In the second, it read a fraction of the project, found nothing in that fraction, and said nothing about the rest.

From the outside the two look identical. That is the problem. An empty result is only evidence of safety if you know what the scanner actually looked at, and most scan output does not say.

This is not a bug in any one tool. It is what happens when a scanner meets a language it does not support, a file it cannot parse, or an analysis pass that does not finish, and its output has nowhere to put that fact.

Four ways a scan goes quiet without saying so

A language the scanner does not read

The OWASP page on source code analysis tools puts it first in its selection criteria: "Prerequisite: Support your programming language." That is easy to check for a single-language repository. It is much harder for a real one.

A typical product repository today holds a TypeScript frontend, a Go or Java service, a Python data job, some Terraform and a Dockerfile. A Salesforce team adds Apex, Lightning components and Flow metadata. If a scanner supports three of those five, the other two produce no findings. Not because they are clean, but because nothing looked at them.

A file the scanner cannot parse

Parsers have limits. A generated file several megabytes long, a file in an encoding other than UTF-8, an expression nested deeply enough to hit a recursion limit, or syntax newer than the grammar the tool ships with: each can make a parser give up. The usual response is to skip the file and move on, often with a log line at a verbosity nobody runs in CI.

The same OWASP page lists another weakness that belongs here: many tools "have difficulty analyzing code that can't be compiled." A missing dependency or build step turns into a skipped module, and a skipped module turns into silence.

A pass that did not finish

A modern scanner runs many analyses: pattern checks, data-flow tracing, dependency matching, configuration checks. If one of them times out, runs out of memory or fails on an unexpected input, the others still report. The result looks complete. It is not: one whole class of check never ran.

A path that was not there

The simplest version is a misconfigured CI job that points the scanner at an empty or wrong directory. Zero files, zero findings, exit 0.

We have fixed this class of mistake in our own scanner too, where a check that could not run read as a pass. It is worth naming because it turns "we could not check" into "you are fine", which is the worst direction a scanner can be wrong in.

Why the gap stays invisible

Scan output is built around findings. A list of findings has a natural shape: rule, file, line, severity. "I did not read these 40 files" does not fit that shape, so it tends to end up in a log, a warning on stderr, or nowhere.

The standards are not the obstacle. The SARIF 2.1.0 format, which many code-scanning tools emit, has an executionSuccessful flag and a toolExecutionNotifications list on each run's invocation (sections 3.20.14 and 3.20.21), specifically so a tool can report problems during its own execution. But a dashboard that counts results still counts zero, and a gate that checks for findings still passes.

There is also an uncomfortable incentive. A scanner that says "I could not analyse a third of your project" looks worse in a demo than one that stays quiet. Silence reads as confidence. It should read as a question.

What a coverage report has to answer

To trust an empty result, you need four answers next to it:

1. How many source files were present, and how many were analysed?
2. Which languages were present, and which of them does this scanner read?
3. Which files were skipped, and for what reason?
4. Which analysis passes did not complete?

If all four come back clean, an empty result means something. If any of them does not, you know exactly where the blind spot is, and you can decide whether it matters.

What Vulkro reports

Since 0.29.0, Vulkro's JSON and NDJSON output carry a scan_coverage block that answers those four questions. Here is an illustrative example of its shape (the numbers are made up for the example):

"scan_coverage": {
  "complete": false,
  "files_present": 412,
  "files_analysed": 409,
  "languages": [
    { "language": "TypeScript", "files_present": 301, "files_analysed": 300, "supported": true },
    { "language": "Go", "files_present": 111, "files_analysed": 109, "supported": true }
  ],
  "unanalysed_languages": [
    { "language": "Apex", "files": 38, "extensions": ["cls", "trigger"], "examples": ["force-app/main/default/classes/InvoiceController.cls"] }
  ],
  "skipped_files": [
    { "reason": "file_size_cap", "detail": "...", "partial": true, "files": 3, "examples": ["web/src/generated/schema.ts"] }
  ],
  "degraded_passes": []
}

Each field maps to one of the questions:

• files_present and files_analysed give the headline ratio, and languages breaks it down per language.
• unanalysed_languages lists source the scanner has no extractor for, with the file count, the extensions seen and a few example paths. Apex in the core vulkro scanner is the common case: Salesforce code is analysed by vulkro-sf, and the coverage block says so rather than leaving those files out of the count.
• skipped_files groups skipped files by a stable reason id, such as file_size_cap, not_utf8 or nesting_too_deep, with a plain-language explanation and a partial flag that says whether some text-based checks still read the file.
• degraded_passes names every analysis pass that did not run.
• complete is true only when none of the above reports a gap.

Files we decline to analyse on purpose, because they are recognised as generated, vendored or minified, are listed separately under excluded_files. They are reported so the counts add up, but they do not make the scan incomplete: that is code you did not write.

The local console (vulkro serve) shows the same picture: how many files were analysed against how many are present, and the specific files that could not be parsed. vulkro-sf says in every output what was not analysed and why, and also counts the Apex entry points it analysed and the ones whose analysis was cut short.

A time budget is a good example of the principle. If you set VULKRO_SCAN_BUDGET_SECS to stop a very large repository from hanging a CI job, the scanner stops starting new passes once the budget is spent, names each skipped pass in degraded_passes and adds a partial-results warning. A partial scan is allowed. A partial scan that looks complete is not.

Making it part of your gate

Because the block is plain JSON, a pipeline can act on it. A minimal check that keeps the scanner's own exit code and adds a warning when coverage is incomplete:

vulkro scan . --format json > scan.json; status=$?
jq -e '.scan_coverage.complete == true' scan.json > /dev/null \
  || echo "warning: this scan did not read the whole project, see scan_coverage in scan.json"
exit $status

Some teams will want to fail the build instead of warning. That is a reasonable choice for a repository that should be fully covered, and a bad one for a polyglot monorepo where a language is knowingly out of scope. The point is that it becomes a decision you make, not an accident you never see.

Coverage also feeds the question people ask after a scan: why did it not flag this? When a file was never read, vulkro why-not answers with that before anything else. We cover that command in why did the scanner not flag this.

Silence is a claim

Every scan result is a claim about your code. A finding claims something is wrong. An empty result claims nothing was found in what was examined, and the second half of that sentence is the part that tends to get dropped.

Coverage is one of three things we try to keep honest. The other two are severity that reflects who can reach the code, and findings that say whether they are proven, unproven or not checked.

Where to see it

Run a scan with --format json and look for scan_coverage, or open the console with vulkro serve. The output formats reference and the scan command docs cover the rest, and the local console guide shows where coverage sits in the UI. To get set up, start here.