Redact sensitive information from repo #24
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
keyop-go/nfq_forwarder#24
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
We currently have commits (especially in the notes/ directory which contains Claude chat exports) which include gitlab access tokens.
We should check the entire repository for any other sensitive information, and force a history rewrite to redact this information from the repo without removing the files entirely.
For now notes/ has been excluded via .gitignore, but this does not remove the history. Once redactions are made (as have been done manually in some of the current files), the .gitignore exclusion can be removed.
It may make sense to include a Makefile target to handle automatically redacting this information, for use prior to committing any further exports.
changed the description
Findings
Scanned the full history (
gitleaks git --log-opts="--all", 101 commits) and the currentworking tree (
gitleaks dir), plus targeted manual checks for anything a regex-based scannermight miss (private key blocks, AWS-style keys, password assignments, accidentally-committed
config.production.yaml, hardcoded secrets in.gitlab-ci.yml/.goreleaser.yaml).Two distinct GitLab Personal Access Tokens are exposed in history, both already manually
redacted (to
glpat-<redacted>) in the current working-tree copies, but still fully presentand retrievable in earlier commits:
notes/issue3_export.txtbcf6133093d9notes/issue5_webUI.txtf0701257c5fanotes/issue14_export.txt30722b03512dnotes/issue19_export.txt9871067144f0notes/issue20_simplify_config.txt(formerlynodes/issue20_simplify_config.txt)4ce5fcfc9754One of the two tokens is (as of this scan) still the active token in use - flagged directly to
Danny separately as needing immediate revocation/rotation, independent of this ticket: rewriting
history doesn't undo prior exposure to GitLab's own server-side copy or anything that already
cloned/fetched the repo.
Nothing else found: no private keys (
BEGIN ... PRIVATE KEY), no AWS-style keys, no otherAPI keys/passwords, and
config.production.yaml(the real, gitignored prod config) was neveraccidentally committed. Every hit was one of the two token values above, in
notes/only - theactual source tree is clean.
Not a secret, but worth a call on: emails appearing in commit authorship/history are all
Danny's own, across several past employers (gmail, keyop.co.uk, cinesite.com, framestore.com)
plus bot addresses (goreleaser/openssh/anthropic) - no third-party PII, but worth deciding
whether that history is fine to have public.
Also worth checking, separate from git history: whether either token was ever echoed into a
GitLab CI job log (this repo's CI runs shell commands against
GITLAB_TOKEN) - job logs are aseparate exposure surface a git history rewrite doesn't touch.
Plan
revoked/rotated in GitLab before or independent of the steps below.
glpat-<redacted>- matching the redaction convention already used in the working tree) forgit filter-repo --replace-text(the modern, git-recommended tool for exactly this - rewritesmatching text within blobs across all history without deleting the files themselves, unlike
BFG's file-deletion-oriented approach).
gitleaksagainst the rewritten mirror (expect zero hits), spot-check a fewrewritten commits by hand, confirm commit messages/authorship/timestamps/graph structure are
otherwise unchanged (only blob text and the resulting SHAs change).
which means every branch and every existing tag (
v1.0.0-v1.0.5) needs re-tagging too.No open MRs currently exist (checked), which simplifies this. Danny's own local clone(s), and
this session's local clone, will need a hard reset or re-clone after the rewrite - a stale
clone can't be reconciled, only discarded. The stale
issue17_unknown_host_keysremotebranch (already merged, not auto-deleted) should be deleted as part of this, since it still
points at pre-rewrite history.
notes/exclusion from.gitignore(per theticket) so future exports track normally again.
maketarget that runs a redaction script (matching theglpat-[A-Za-z0-9_\-\.]+shape generically, not just today's two known values) across
notes/*.txt, so future chatexports get scrubbed automatically before
git addrather than relying on manual edits.No changes have been made yet - this comment is the plan only, per Danny's request, awaiting
sign-off before the history rewrite (step 2 onward) proceeds.
Status update
Token rotated by Danny - confirmed working with a freshly-issued PAT. Proceeding with the plan now:
notes/from.gitignore) already done by Danny - skipping.gitleaksrun, blocking commits that contain secrets rather than just a manual redaction script.Proceeding with steps 1-4 (build replacement list, rewrite history in a fresh mirror clone,
verify, then force-push) now. Will post progress/results here as each stage completes.
mentioned in commit
dec19a6119mentioned in merge request !67
History rewrite complete
the working copy) redacting the exposed token pattern everywhere it appeared.
several previously-affected commits by hand to confirm the redaction landed cleanly and
surrounding content is otherwise untouched; `git fsck --full` clean.
required temporarily relaxing `main`'s "no force push" protection and removing the `v*`
protected-tag rule (both restored by Danny immediately after - please double check they're
back the way you want them).
(#353-356). These are failing - benignly, for an unrelated reason: the project's
`GITLAB_TOKEN` CI/CD variable is currently missing (`missing GITHUB_TOKEN, GITLAB_TOKEN and
GITEA_TOKEN` in the job log), so `goreleaser release` can't even authenticate. Nothing was
re-published or corrupted - it fails before doing anything. Worth restoring that CI/CD
variable (with the newly-rotated token) before the next real release tag is pushed, though,
or that'll fail the same way.
a pre-rewrite clone can't be merged or rebased against it.
Prevention (MR !67)
Item 6 implemented and expanded per your request:
covered by CI's existing lint job automatically.
that blocks committing a likely secret before it's even committed, not just before it's merged.
MR: !67 - https://gitlab.keyop.co.uk/keyop/go/nfq_forwarder/-/merge_requests/26
Once that's reviewed/merged, this ticket should be ready to close.
mentioned in commit
4a3a9b06ffFixed via MR !67 (merge commit
cdeebc12), on top of the history rewrite already pushed directly tomain. Summary of everything done:git filter-repo) to redact the exposed GitLab tokens - verified withgitleaks(zero leaks) andgit fsck. Tokens were also rotated by Danny.mainand all 6 tags; branch/tag protections that had to be temporarily relaxed for that were restored afterward.make check-secrets(a full-historygitleakssweep, now part ofmake lintand therefore CI's existing lint job) andmake install-git-hooks(an opt-in pre-commit hook blocking a secret before it's ever committed).No release tag needed for this one - the history rewrite already happened directly against
main, outside the normal tag/release flow. Closing.