Security gate in CI: fail on new findings, not old debt
Almost every team that adds a security gate in CI does it the same way. Run the scanner on every pull request, fail the build on any High or Critical finding, and fix what it reports. The first run on a real codebase reports hundreds of findings that were there long before anyone opened this pull request. Every build is red. Within a few weeks someone adds || true and the gate is gone.
The gate was not wrong to exist. It was asking the wrong question. "Does this repository have findings?" is almost always yes. The question a pull request can answer is narrower: "Did this change add one?"
Why does a security gate in CI get switched off?
Two things kill a gate: old debt and noise.
Old debt is the problem above. A developer who renamed a variable cannot fix a five-year-old injection in another module, and should not be blocked by it.
Noise is the other half. Research on developer behaviour is consistent here. In interviews with 20 developers, Johnson et al. (ICSE 2013) found that while everyone saw value in static analysis, false positives and the way warnings are presented were barriers to using it. At Google, Sadowski et al. (CACM 2018) describe checks shown in code review being held to under 10% "effective false positives", and analyzers being switched off when developers mark too many results not useful. A gate that fails on guesses teaches people to ignore it.
So a gate a team keeps on has to do two things: fail only on what the change added, and fail only on findings it can stand behind.
How do you gate on new findings only?
Vulkro offers three ways to scope a gate to the change. They trade cost against what they can see.
| Approach | Command | What it compares | Catches a new caller of an old sink? |
|---|---|---|---|
| Two-tree gate | vulkro gate --base origin/main | Full findings of the working tree vs the base ref | Yes |
| Changed lines | vulkro scan . --gate-vs origin/main | Findings on lines the change touched | No |
| Ratchet | vulkro scan . --ratchet | Findings vs a committed baseline file | Yes, against the baseline |
vulkro gate --base <ref> scans both trees and reports only findings present in the working tree but not in the base. Because it compares whole finding sets, it catches the subtle case: a change that adds a new route calling an existing, previously unreachable sink. The sink's line did not change, but the finding is new.
vulkro scan --gate-vs <ref> restricts the exit code to findings on changed lines. It is cheaper and fine for a fast lane, but it cannot see that new-caller case. Pair it with --diff-only <ref> to scan only the changed files. Vulkro for Salesforce has the same flag: vulkro-sf scan --gate-vs <ref>.
The ratchet uses a baseline file you commit. Run vulkro baseline on a known-good commit, commit .vulkro-baseline.json, and run vulkro scan . --ratchet in CI. The scan still prints everything but exits non-zero only on findings not in the baseline. --ratchet-strict exits 2 if the file is missing, so a deleted baseline cannot silently turn the gate into a full-repository gate. When findings grow on purpose, regenerate the baseline and review the diff in a pull request. The baselines guide covers the workflow.
What should fail the build?
Scoping decides which findings count. Severity and evidence decide whether they fail.
--fail-on sets the severities that fail the run:
vulkro scan . --fail-on critical,high
The default ci preset already fails on Critical and High. For finer thresholds, put a [quality_gate] section in vulkro.toml and pass --gate; when it is set, those thresholds win over --fail-on.
Evidence matters as much as severity. Every Vulkro finding carries a disposition: Proven, Unproven or Not checked. Not checked findings are classes Vulkro flags but cannot decide offline, such as whether a record lookup is authorised for this caller. They are reported, and a warning says how many were withheld, but they never fail the gate. Failing a build would assert a verdict the analysis did not reach. We explain the three labels in proven, unproven, not checked.
Exit codes CI can trust
A gate is only as reliable as its exit code. Every Vulkro command follows one contract:
| Code | Meaning |
|---|---|
0 | No gating findings |
1 | Gating findings (the gate is the report) |
2 | Error: bad ref, git not available, scan failure |
The distinction between 1 and 2 matters. A crash or a bad ref must never look like a pass, and must never look like a finding either. One more rule: a scan that could not read any source file is never 0. "No new findings" from two trees nobody read is empty for the wrong reason. The background is in a clean scan that did not read your code.
A GitHub Actions example
This workflow reports every finding but fails a pull request only on what it adds:
jobs:
scan:
runs-on: ubuntu-latest
env:
VULKRO_ACCOUNT_TOKEN: ${{ secrets.VULKRO_ACCOUNT_TOKEN }} # a CI key from the customer portal
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # the base ref must be in history
- name: Install Vulkro
run: |
curl -fsSL https://dist.vulkro.com/install.sh | VULKRO_BIN_DIR="$HOME/.local/bin" bash
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
- name: Report everything
run: vulkro scan ci . --format sarif > vulkro.sarif || true
- name: Fail on new findings
if: github.event_name == 'pull_request'
run: vulkro gate --base origin/${{ github.base_ref }}
Upload vulkro.sarif to your code scanning view, or use --format github-annotations to put findings on the changed lines. GitLab, Bitbucket, Jenkins and CircleCI follow the same shape; see CI/CD integration.
Pre-commit, pull request or release?
Each stage answers a different question, so use more than one.
Pre-commit: "did I just add something?" vulkro install-hook --pre-commit writes a hook that runs the new-findings gate against HEAD before each commit, with no outbound request. --pre-push adds a fast, high-confidence scan before code leaves the machine. Hooks compose with an existing hook file, and VULKRO_SKIP_HOOKS=1 bypasses them once. Developers who see findings in the VS Code extension first rarely trip the later gates.
Pull request: "did this change add something?" vulkro gate --base or --gate-vs, as above. This is the gate that has to stay on.
Release: "is anything open that we agreed not to ship?" Here a whole-repository view is right. Run a full scan with --fail-on or your [quality_gate]. If your repositories are scanned in Vulkro Cloud, the release job can ask the issue queue directly:
vulkro cloud todo --fail-on high
It exits 1 when open issues at or above that severity exist. Suppressed, accepted-risk and advisory issues do not count, so decisions your team already made are respected.
Checklist for a security gate in CI
- Pull requests gate on new findings only (
gate --base,--gate-vsor the ratchet) - Severity policy set once (
--fail-onor[quality_gate]), in the repository - Not checked findings reported, not blocking
- Exit
2treated as a failed job, never retried into a pass fetch-depth: 0so the base ref exists- Pre-commit hook for fast local feedback
- A release-stage check over everything still open
Frequently asked questions
How do I fail a CI build only on new vulnerabilities?
Compare against the base branch. vulkro gate --base origin/main reports and fails only on findings the change added, and vulkro scan --gate-vs <ref> does the same for changed lines at lower cost.
What is a security ratchet?
A committed list of the findings you already have. vulkro scan --ratchet fails only when a finding appears that is not on the list, so the count can go down but never up without a reviewed change to the baseline.
Should a security gate block on Medium findings?
Usually not at first. Start with Critical and High on new findings, keep the gate on for a few weeks, then widen --fail-on if the team agrees. A gate that stays on beats a strict one that gets removed.
What exit code should a failed scan return?
A non-zero code that differs from "findings found". Vulkro returns 1 for findings and 2 for errors, so a broken scan never passes as clean.
Where to see it
The release gating use case shows the whole flow. The reference pages are vulkro gate, baselines, vulkro install-hook and vulkro cloud. To wire it into your pipeline, start here.