Skip to main content

Fixes you can trust, because each one is checked

· 8 min read
Vulkro
Vulkro
Security research

Every suggested fix makes two claims. The first is that it removes the problem. The second is that it does not add a new one. Most fixes, whether they come from a colleague, a linter or an AI assistant, arrive with neither claim checked. Someone reads the diff, it looks reasonable, and it is merged.

For ordinary bugs the test suite catches some of the bad ones. Security bugs are different, because the tests rarely exercise the hostile input. A fix that escapes the wrong variable, or moves string concatenation one function further away, passes every test the old code passed.

Since Vulkro 0.30.0, vulkro fix --verify checks both claims before you see the patch. This post is about why that matters and exactly what the check does and does not prove.

Why an unchecked fix is a risk​

Research on AI-written code is a useful warning here, because AI assistants now write a large share of fixes. In a study presented at IEEE S&P 2022, Pearce and colleagues had an AI code assistant complete 89 security-relevant scenarios and found about 40% of the 1,689 programs it produced were vulnerable. A Stanford user study by Perry and colleagues found that participants with an AI assistant wrote less secure code than those without one, and were more likely to believe their code was secure. The same group of researchers behind the first study also tested models on repairing vulnerabilities: the models fixed the synthetic and hand-crafted cases, but the authors' review of real-world examples highlighted the difficulty of producing functionally correct code.

None of that is unique to models. Human fixes fail in the same few ways:

  • The fix lands in the wrong place. One branch is sanitised, the other still reaches the sink.
  • The fix moves the bug. Input is cleaned in the handler, then concatenated into a query again in a helper.
  • The fix adds a different bug. A strict check is loosened to make the patch compile or a test pass.
  • The fix silences the tool, not the flaw. A rename or a refactor that the scanner no longer recognises, with the same data flow underneath.

A reviewer can catch all four, but only by doing the analysis again. That is the step worth automating.

What --verify does​

vulkro fix . --verify      # re-scan a patched copy: fixed / not-fixed / regressed
vulkro fix . --apply       # write only the patches proven fixed
vulkro fix . --bundle fixes.zip

For each finding it has a remedy for, vulkro fix builds a patch. With --verify it applies the patches to a temporary copy of the project, scans that copy again and gives every patch one of three verdicts:

  • fixed: the finding is gone, and no new finding of Medium severity or higher appeared on the changed lines.
  • not-fixed: the finding is still reported after the patch.
  • regressed: the patch introduced a new finding.

Your working tree is never touched by --verify, --bundle or --format diff. --apply (which implies --verify) writes only the patches marked fixed, one file at a time, and skips a file that changed since the scan. A not-fixed or regressed patch is never applied.

--bundle writes a zip file or a directory with the verified patches (per finding and combined), the patched files, every remedy as JSON and a README. It is the format to hand to a reviewer or attach to a ticket: the diff and the evidence that it was checked travel together. --format diff prints the fixed patches as one unified diff for git apply.

The exit codes are built for scripts. --verify and --bundle always exit 0, because they only report. --apply exits 0 when there was nothing to apply, 1 when at least one verified patch was written, and 2 on an IO failure or when patches were generated but none verified as fixed.

SQL fixes use your driver's own parameters​

The most common remedy is for SQL injection, and the most common wrong fix for SQL injection is escaping. Here is the vulnerable shape in Node with pg:

app.get('/orders', requireAuth, async (req, res) => {
  const sql = "SELECT id, total FROM orders WHERE status = '" + req.query.status + "'";
  const { rows } = await pool.query(sql);
  res.json(rows);
});

A hand-written fix often reaches for a replace:

const status = String(req.query.status).replace(/'/g, "''");
const sql = "SELECT id, total FROM orders WHERE status = '" + status + "'";

It may hold for this query. It is also a quoting rule you now own, applied in one place, and the next person to copy the line into a numeric comparison without quotes gets nothing from it. The OWASP SQL Injection Prevention Cheat Sheet lists prepared statements with parameterized queries as the first defence for a reason.

The remedy Vulkro produces is the bound-parameter form of the driver the code already uses:

const { rows } = await pool.query(
  'SELECT id, total FROM orders WHERE status = $1',
  [req.query.status],
);

For pg that is $1. A Python file on a driver that uses %s placeholders gets %s with a parameter tuple, not a placeholder style borrowed from another library. The value never becomes part of the SQL text, so there is nothing left to escape. Then --verify checks that the injection finding is gone and that the rewrite did not introduce anything new.

When the code shape is not one the remedy recognises, Vulkro does not guess a diff. It prints the remedy as steps with a before and after example, and there is nothing to verify or apply.

What a fixed verdict proves, and what it does not​

We want to be exact about this, because a verdict that overstates itself is worse than none.

A fixed verdict means the same deterministic engine that reported the finding no longer reports it on the patched code, and found no new Medium or higher finding on the lines the patch changed. That rules out the four failure modes above as far as the scanner can see them: the wrong branch, the moved concatenation, the new weakness and the cosmetic rename all show up as not-fixed or regressed.

It does not prove the patch keeps your application's behaviour. A parameterized query that filters on the wrong column is safe and wrong. Run your tests, and read the diff. The verdict removes the security re-analysis from the reviewer's job; it does not remove the review.

The deterministic templates and the AI tier​

vulkro fix has two other paths, and both follow the same rule of checking before claiming.

--write applies a closed list of mechanical rewrites in place, such as cookie Secure, HttpOnly and SameSite flags, MD5 or SHA-1 to SHA-256, yaml.load to yaml.safe_load, and turning off a literal debug=True. Each template re-reads the source line and refuses on any ambiguity, with a reason. --dry-run previews them. There is no model in this path.

--ai is off by default. For findings no template or remedy covers, the model you configured with vulkro ai proposes a minimal diff. It is shown as verified only if two gates pass: first a deterministic re-scan (the finding must disappear and no new finding may appear), then a second AI review of correctness. The AI review can never turn a failed re-scan into a pass. Everything from this tier is labelled advisory, and under offline mode only a model on your own machine is allowed.

Letting an agent check its own patch​

AI coding agents now write a lot of security fixes, and an agent is exactly as confident about a wrong patch as about a right one. So the same verdicts are available to agents through the Vulkro MCP server, started with vulkro mcp serve:

  • suggest_fixes returns fix patches for a scan, each re-scanned on a temporary copy and labelled fixed, not-fixed or regressed.
  • verify_fix takes the agent's own diff, applies it to a temporary copy of the project, re-scans and returns the same verdict.

That gives an agent a loop it can run without a human in the middle: write the patch, call verify_fix, and if the answer is not-fixed or regressed, revise before proposing anything. The verdict comes from a deterministic scan, not from a second model's opinion, so the agent cannot talk its way past it. We covered the wider set of guardrails for agents in securing AI coding agents.

Where to see it​

Run vulkro fix . --verify on a project with findings and read the verdicts before you apply anything. The vulkro fix reference lists every flag and exit code, and the MCP server page documents suggest_fixes and verify_fix. To get set up, start here.

Sources​