Skip to main content

Evidence and the AI layer

Vulkro's scan engine is fully deterministic and runs entirely on your machine. It calls no model: the same input produces the same findings, severities, and exit code every time. On top sits an optional, advisory AI layer that runs locally through Ollama by default.

This guide covers the boundary between the two: the evidence graph that grounds every AI answer, the advisory tiers (vulkro ai ask, vulkro triage, vulkro fix --ai), and what a verified AI fix means.

The invariant

AI assistance is advisory and local by default. It never changes a deterministic scan result: not a finding, not a severity, not a JSON / SARIF / SBOM document, and not an exit code. The published benchmark is AI-free: every number is produced with the AI layer off.

Evidence graph​

--format evidence-graph emits a stable, versioned JSON document (evidence-graph/1.0) that composes endpoints, taint source-to-sink flows, reachability verdicts, findings, and the dependency SBOM into one graph:

vulkro scan . --format evidence-graph > graph.json

The graph is deterministic ground truth, and it is what grounds AI answers on both sides of the boundary:

  • An external agent (your own tooling, an MCP client) can be handed the graph file and reason about it with the scan result as the source of truth.
  • Vulkro's own AI layer is grounded in the same structure. vulkro ai ask and vulkro ai chat build their context from the scan's evidence graph. vulkro triage assembles each finding's evidence rows, taint trace hops, and reachability verdict into the prompt. vulkro analyze --ai selects its targets from the graph nodes where the engine is structurally weak.

The model never free-associates over your repository: it is asked about what the deterministic scan saw. vulkro discover . --format evidence-graph produces the same shape from the discovery pass when you only need the endpoint and flow structure.

The local AI layer​

The AI layer is opt-in and configured once with vulkro ai. It is CLI-only (no config files); the selection is stored in ~/.vulkro/scans.db and shared with the vulkro-sf binary, so a model chosen once serves both.

vulkro ai setup          # install / point at Ollama, pick a model, self-test
vulkro ai status         # current model, runtime, reachability, license tier
vulkro ai use <model>    # switch the active model
vulkro ai off            # disable the AI layer (deterministic-only)

The default runtime is local Ollama at http://127.0.0.1:11434/v1, and the default model is qwen2.5-coder:7b (Apache-2.0). On a constrained machine the setup wizard recommends the smaller phi4-mini:3.8b (MIT) instead. Any OpenAI-compatible runtime works via --ai-url.

Configuration resolves in a fixed order, highest priority first:

  1. Per-run flags: --ai-model, --ai-url, --no-ai.
  2. Environment variables: VULKRO_AI_MODEL, VULKRO_AI_URL, VULKRO_AI_KEY.
  3. The saved selection from vulkro ai use.
  4. The smart default above.
NameEffect
VULKRO_AI_MODELModel tag override for any AI-enabled command.
VULKRO_AI_URLOpenAI-compatible base url override.
VULKRO_AI_KEYAPI key for a bring-your-own-key cloud endpoint. Read from the environment only, never persisted to disk.

Offline behaviour follows one rule. A loopback (local) endpoint is allowed even under VULKRO_OFFLINE, because nothing leaves the machine. A cloud endpoint is refused offline with an actionable error. Cloud is a deliberate, bring-your-own-key opt-in (vulkro ai setup --byok): pointing the layer at a cloud url means code context leaves the machine, and Vulkro says so before it happens. When the endpoint is the Anthropic API, the legacy VULKRO_TRIAGE_API_KEY / ANTHROPIC_API_KEY variables are still accepted as key fallbacks.

The advisory tiers​

Three tiers cover the ask-understand-fix loop, and every one of them is grounded in the deterministic scan:

Ask: vulkro ai ask and vulkro ai chat​

One-shot or interactive questions answered against the current project's scan:

vulkro ai ask "which endpoints accept user input without auth?"
vulkro ai chat

Both run the normal deterministic scan first, then answer only from that scan's endpoints, findings, and vulnerable dependencies. The answer is advisory text. Exit code: 0 on success, 2 on error; the advisory answer never sets exit 1.

Triage: vulkro triage​

An advisory second opinion on findings the deterministic engine cannot settle on its own: entitlement / paid-tier bypass, PII / object-property over-exposure, operational reachability of one-off scripts, source trust, and any finding whose reachability the scan could not confidently judge.

vulkro triage .

By default this runs against the local model; no API key is needed. The verdicts live in a separate report and are never merged back into the scan result. The JSON output is wrapped in an envelope that states its status explicitly ("advisory": true, "non_deterministic": true, "excluded_from_benchmark": true) alongside the provider, model, and endpoint class that produced it. Exit codes: 0 when no candidates needed triage, 1 when advisory verdicts were reported, 2 on error. See vulkro triage.

Fix: vulkro fix --ai and vulkro-sf fix --ai​

The deterministic vulkro fix templates handle a closed allowlist of safe rewrites, with no model involved. --ai adds an advisory tier for findings no template covers: the local model proposes a minimal diff, and the deterministic engine then judges it (see the verified-fix definition below). --ai alone is read-only. --ai --write applies only verified diffs, prints each one first, and runs only after the deterministic templates. A refused AI fix is a refusal, not an error, and --ai never changes the exit codes of fix.

For Salesforce, vulkro-sf fix --ai covers the mandatory-review Apex classes (CRUD / FLS enforcement, with sharing gaps, SOQL injection) with the same propose-then-verify contract. The model proposes; the deterministic detector judges.

Beyond the three tiers, the same advisory layer powers optional overlays. Each one is additive, labelled advisory, and changes nothing deterministic:

CommandWhat the model adds
vulkro explain <id> --aiA plain-language narrative under the unchanged deterministic explanation.
vulkro diff <base> --aiA security-delta narrative over the deterministic surface diff.
vulkro analyze --aiA graph-guided second pass whose candidates are never merged into the findings.
vulkro huntA model-driven hunt using deterministic checks as tools; a result counts as confirmed only when a deterministic oracle grounds it, otherwise it is a labelled lead.
vulkro slopcheck --aiPlain-language explanations of flagged dependency names.
vulkro-sf scan --ai-explainPer-finding AppExchange Security Review context, printed to stderr so stdout stays byte-identical.

What "verified" means​

A fix is verified when, after applying the candidate diff:

  1. the patched file still parses: a patch that introduces new syntax errors relative to the original is refused before any re-scan, so a broken edit can never clear a finding by making the file unparseable, and
  2. a fresh deterministic re-scan no longer reports the originating finding and reports no new finding.

In the core CLI a second, subordinate AI review then checks the patch looks correct and behaviour-preserving. It can only downgrade a would-be verified fix to refused; it is never consulted when the re-scan failed, so it can never turn a re-scan failure into a pass.

Verified is deliberately narrow. It means the deterministic scanner, re-run from scratch, no longer reports the finding and the file still parses. It does not claim the build compiles or that your tests pass. Review every applied diff the way you would review any patch.

Proof harnesses​

vulkro prove turns eligible findings into runnable test files that a developer can execute to confirm a finding is a true positive:

vulkro prove .

For each injection or broken-authorization finding it emits a harness: a fast-check property test plus a concrete proof-of-concept for JS/TS, a Hypothesis and pytest harness for Python, and a generated @isTest class for Apex (runnable inside a Salesforce org). Vulkro only writes the harness text. It never compiles or runs it. Findings with no runnable proof shape (config, inventory, code quality) or in a language without an emitter are listed with a reason rather than dropped silently. No model is involved.

Cross-repo aggregation​

A single repository rarely tells the whole story. vulkro aggregate links evidence-graph exports from several repositories to surface candidate cross-repo flows:

vulkro scan service-a . --format evidence-graph > a.json
vulkro scan service-b . --format evidence-graph > b.json
vulkro aggregate a.json b.json

The linker is deterministic: an outbound HTTP client call in one repo whose normalised route shape and method match an exposed endpoint in another repo is emitted as a candidate cross-repo link. The links are ground truth for an external agent to investigate, not a verdict.

Feed a scan to an agent live​

vulkro mcp serve runs Vulkro as a Model Context Protocol server that Claude Desktop, Cursor, Windsurf, Continue, and other JSON-RPC MCP clients can call as a tool. This lets an agent request a scan and read the evidence during a coding session rather than after the fact.

vulkro mcp serve

For in-editor diagnostics, the VS Code extension is built and distributed as a .vsix; a Marketplace listing is coming soon.

Last updated on