Skip to main content

Secrets in git history: why deleting the line is not enough

· 9 min read
Vulkro
Vulkro
Security research

Secrets in git history are one of the most common leaks in software, and one of the most misunderstood. Someone commits an API key, notices a few minutes later, deletes the line and commits again. The working tree is clean. The key is still in the repository, in every clone made since, in every fork, and in any CI cache that fetched it.

The scale is not small. Researchers at North Carolina State University scanned public GitHub commits in near real time for almost six months and found secret leakage affecting over 100,000 repositories, with thousands of new, unique secrets leaked every day (Meli, McNiece and Reaves, NDSS 2019). This post covers why deletion does not help, what to do first, when rewriting history is worth the cost, and how to stop the next one.

Why doesn't deleting a secret remove it from git history?​

Git stores every version of every file. A commit that deletes a line adds a new snapshot; it does not change the old one. Anyone with the repository can find the key in seconds:

git log -p -S 'sk_live_' --all       # every commit that added or removed the string
git show 3f2a9c1:config/settings.py  # the file as it was in that commit

Copies spread beyond your control. Clones on laptops and build agents, forks, mirrors and cached pull request views all keep the old commits. Making a public repository private does not recall copies made while it was public.

A real case shows how long this can last. In October 2022 Toyota disclosed that a development contractor had uploaded part of the T-Connect site's source code to a public GitHub repository in December 2017. The code contained an access key to a data server. It stayed public until September 2022, and Toyota said the email addresses and customer management numbers of 296,019 customers may have been exposed. Toyota changed the key on 17 September 2022 and said it could not completely rule out third-party access, as Infosecurity Magazine reported. Nearly five years passed between the commit and the rotation.

What should you do first when a secret is committed?​

Rotate it. GitHub's guide to removing sensitive data from a repository is direct about the order: for a secret, "as a first step you need to revoke and/or rotate that secret." Once revoked, the key can no longer be used, and the guide notes that this may be enough on its own.

In practice:

  1. Revoke the key at the provider and issue a new one.
  2. Deploy the new key from a secret store or environment variable, not from the repository.
  3. Check the provider's access logs for use of the old key during the exposure window, from the first commit that contained it.
  4. Record the incident, including the commit that introduced the key and when it was revoked.

Do not start with a history rewrite. A rewrite takes time and coordination, and until the key is revoked it is still valid in every existing copy.

Should you rewrite git history to remove a secret?​

Sometimes. If the leaked data cannot be rotated (personal data, for example), or a policy requires removal, a rewrite is warranted. GitHub's guide uses git-filter-repo with its sensitive-data mode:

# in a fresh clone; replacements.txt holds lines like: sk_live_abc123==>REMOVED
git filter-repo --sensitive-data-removal --replace-text ../replacements.txt
git push --force --mirror origin

The same guide lists the costs, and they are real:

  • Every commit hash from the leak onward changes, which can break tooling that references commits.
  • Commit and tag signatures are removed.
  • Open pull requests lose their diffs, so merge or close them first.
  • Every collaborator must re-clone or clean their copy. One old clone pushed back undoes the work.
  • Forks keep the old commits, and you cannot erase other people's clones.
  • Cached views and pull request refs need a request to GitHub Support to purge.

So the rewrite reduces exposure; it does not end it. Rotation is what makes the leaked value useless.

How do secrets end up committed?​

Usually by convenience. A key pasted in to test something:

# settings.py: committed with a live key
import stripe
stripe.api_key = "sk_live_51Hx...redacted"

The fix is to read it from the environment, keep local values in a file git ignores, and fail loudly when it is missing:

# settings.py: the key comes from the environment
import os
import stripe

stripe.api_key = os.environ["STRIPE_API_KEY"]  # KeyError if unset, never a fallback key
.env
.env.*
!.env.example

Other common sources: .env files committed before the .gitignore existed, Terraform state, cloud CLI credential files, test fixtures with real tokens, and configuration for AI coding agents and MCP servers with inline keys. We covered that last case in supply-chain attacks are a code problem.

How do you prevent secrets reaching git?​

Catch it before the commit exists, because after that you are in rotation territory.

  • Keep secrets out of files: a secret manager, environment variables, or short-lived credentials issued to CI through workload identity instead of long-lived keys.
  • Ignore the files that hold them from the first commit.
  • Check on every commit. vulkro install-hook --pre-commit installs a hook that scans for new findings against HEAD, secrets included, with no outbound request. A blocked commit costs seconds; a pushed key costs a rotation.
  • Check again in CI, on new findings only, so the gate stays on. See security gates in CI.
  • Audit the history you already have, because prevention does not cover what was committed last year.

How Vulkro finds secrets in code and git history​

Secrets are part of every vulkro scan. Vulkro looks in .env* files, source code in every supported language, config files, and Terraform, Helm and Kubernetes manifests. It also reads git history: up to the last 500 commits within the last two years. A secret removed from the working tree is reported with the commit that added it, and a secret committed and later added to .gitignore is still found.

A generic api_key = "..." match starts at Medium confidence. A value that matches a known provider format, such as an AWS access key, a GitHub token or a Stripe live key, is raised to High.

To know whether a found key still works, opt in to live validation:

vulkro scan . --validate-secrets

Each secret gets a harmless authenticated call to its provider and is tagged [live], [dormant] or [unknown]. A live key is raised to Critical and counts as Proven, because the evidence is the provider's own answer. This is the only part of secret scanning that uses the network; it is off by default and refused under VULKRO_OFFLINE=1. Our post on proven, unproven, not checked explains how that label is used.

In the local console (vulkro serve), the Secrets page shows credentials in the code and in git history together, so a key deleted last month sits next to the one still in a config file. In CI, check out with full history (fetch-depth: 0 on GitHub Actions) or the history audit has nothing to read.

Two limits to know. Commits older than the history window are not checked, so a long-lived repository may need a separate one-off audit. And a credential in a format Vulkro does not recognise may only appear as a lower-confidence generic match, or not at all. The secrets detection page lists what is covered.

Checklist: a secret was committed​

  • Revoke and reissue the key at the provider, today
  • Deploy the new key from a secret store or environment variable
  • Check provider logs from the first commit that contained it
  • Decide on a history rewrite: needed for data that cannot be rotated, optional otherwise
  • If rewriting: merge or close open pull requests, rewrite in a fresh clone, have everyone re-clone, ask GitHub Support to purge cached views
  • Add a pre-commit hook and a CI gate so the next one stops early
  • Run a full history scan for others

Frequently asked questions​

Does deleting a file remove it from git history?​

No. Deleting a file creates a new commit without it, but every earlier commit still contains it. Anyone with a clone can read the old version with git show or git log -p.

Should I rotate a secret or rewrite git history?​

Rotate first, always. Rotation makes the leaked value useless; a rewrite only reduces where it can be found and cannot reach existing clones or forks.

How do I find secrets already in my git history?​

Use a scanner that reads history, not just the working tree. vulkro scan checks the last 500 commits within two years and reports each secret with the commit that added it.

Is a private repository safe for secrets?​

Safer, not safe. Toyota's key sat in a public repository for nearly five years, and private repositories are still cloned to laptops, build agents and backups. Keep secrets out of git entirely.

Where to see it​

The supply chain use case shows secrets alongside dependency checks. The secrets detection docs and the local console guide cover the details. To scan your own history, start here.