keyop-go/nfq_forwarder

Fork 0

Speed up CI pipeline: split tool setup, cache Go modules/build #76

Merged

claude merged 4 commits from issue35_speed_up_ci_pipeline into main

2026-07-08 08:12:58 +01:00

claude commented

2026-07-08 07:03:17 +01:00

(Migrated from gitlab.keyop.co.uk)

Closes #35 (in-repo portion - see comment on the ticket for suggested follow-up changes to the external CI image, which need to happen in that project separately)

Investigation

Pulled the actual trace logs for a recent lint job (pipeline 392, job 1074) and build job to find where the time actually goes, rather than guessing:

apt-get update && apt-get install -y libpam0g-dev yamllint iptables: ~6-7 seconds.
go install github.com/goreleaser/goreleaser/v2@latest + go install github.com/zricethezav/gitleaks/v8@latest (both run unconditionally by the old combined setup Makefile target, regardless of which the calling target actually needed): ~2 minutes of a ~2.5 minute job - by far the dominant cost. Neither tool is baked into the CI image, so each downloads and compiles a large dependency tree from scratch on every single run, and - worse - every job installed both tools even though lint only ever needs gitleaks and build/package/release only ever need goreleaser.
.goreleaser.yaml's build matrix is already minimal (linux/amd64 only) - no wasted cross-compilation there.

Fix (this MR - no external changes needed)

Makefile: split setup into setup-goreleaser/setup-gitleaks. build/package/release now depend only on setup-goreleaser; check-secrets (lint's only caller of either) depends only on setup-gitleaks. setup itself is kept as a combined convenience alias but nothing depends on it any more. This halves the wasted-install cost per job today, and both command -v guards already become free no-ops the moment the image gains these tools (no further Makefile change needed then).
.gitlab-ci.yml: GOPATH/GOCACHE now point under $CI_PROJECT_DIR with a cache: block (keyed on go.sum's content, so a dependency bump gets a fresh cache automatically) persisting the module download cache and build cache between pipeline runs - repeated go install/go build/go test/golangci-lint invocations across pipelines can now reuse work instead of starting cold every time.
.gitlab-ci.yml: made the apt-get install libpam0g-dev yamllint iptables step defensive (checks each one first via command -v/a cc header probe, only installs what's actually missing) - today this still installs all three since the image doesn't have them, but it self-shrinks to a no-op automatically once it does, with no further .gitlab-ci.yml edit needed.

Suggested follow-up in the external image (posting the same detail as a comment on #35 for tracking)

The real fix for the ~2-minute dominant cost is baking pinned versions of goreleaser and gitleaks directly into registry.keyop.co.uk/keyop/docker/golang-ci (and ideally libpam0g-dev/yamllint/iptables too, closing out the smaller remaining cost) - see the ticket comment for specifics. I don't have access to that project to make the change myself.

Testing

make lint/make test both clean locally; make -n build/make -n package/make -n check-secrets dry-runs confirm each target now only pulls in the one tool it actually needs. .gitlab-ci.yml validated with yamllint and a plain YAML parse.

Docs

Added a CLAUDE.md Design notes bullet with the full timing breakdown and reasoning, and a Known Issues entry tracking the pending external image update.

Closes #35 (in-repo portion - see comment on the ticket for suggested follow-up changes to the external CI image, which need to happen in that project separately) ## Investigation Pulled the actual trace logs for a recent lint job (pipeline 392, job 1074) and build job to find where the time actually goes, rather than guessing: - `apt-get update && apt-get install -y libpam0g-dev yamllint iptables`: ~6-7 seconds. - `go install github.com/goreleaser/goreleaser/v2@latest` + `go install github.com/zricethezav/gitleaks/v8@latest` (both run unconditionally by the old combined `setup` Makefile target, regardless of which the calling target actually needed): **~2 minutes of a ~2.5 minute job** - by far the dominant cost. Neither tool is baked into the CI image, so each downloads and compiles a large dependency tree from scratch on *every single run*, and - worse - every job installed *both* tools even though lint only ever needs gitleaks and build/package/release only ever need goreleaser. - `.goreleaser.yaml`'s build matrix is already minimal (`linux`/`amd64` only) - no wasted cross-compilation there. ## Fix (this MR - no external changes needed) - **Makefile**: split `setup` into `setup-goreleaser`/`setup-gitleaks`. `build`/`package`/`release` now depend only on `setup-goreleaser`; `check-secrets` (lint's only caller of either) depends only on `setup-gitleaks`. `setup` itself is kept as a combined convenience alias but nothing depends on it any more. This halves the wasted-install cost per job today, and both `command -v` guards already become free no-ops the moment the image gains these tools (no further Makefile change needed then). - **.gitlab-ci.yml**: `GOPATH`/`GOCACHE` now point under `$CI_PROJECT_DIR` with a `cache:` block (keyed on `go.sum`'s content, so a dependency bump gets a fresh cache automatically) persisting the module download cache and build cache between pipeline runs - repeated `go install`/`go build`/`go test`/`golangci-lint` invocations across pipelines can now reuse work instead of starting cold every time. - **.gitlab-ci.yml**: made the `apt-get install libpam0g-dev yamllint iptables` step defensive (checks each one first via `command -v`/a `cc` header probe, only installs what's actually missing) - today this still installs all three since the image doesn't have them, but it self-shrinks to a no-op automatically once it does, with no further `.gitlab-ci.yml` edit needed. ## Suggested follow-up in the external image (posting the same detail as a comment on #35 for tracking) The real fix for the ~2-minute dominant cost is baking pinned versions of `goreleaser` and `gitleaks` directly into `registry.keyop.co.uk/keyop/docker/golang-ci` (and ideally `libpam0g-dev`/`yamllint`/`iptables` too, closing out the smaller remaining cost) - see the ticket comment for specifics. I don't have access to that project to make the change myself. ## Testing `make lint`/`make test` both clean locally; `make -n build`/`make -n package`/`make -n check-secrets` dry-runs confirm each target now only pulls in the one tool it actually needs. `.gitlab-ci.yml` validated with `yamllint` and a plain YAML parse. ## Docs Added a `CLAUDE.md` Design notes bullet with the full timing breakdown and reasoning, and a Known Issues entry tracking the pending external image update.

claude commented

2026-07-08 07:03:17 +01:00

(Migrated from gitlab.keyop.co.uk)

requested review from @danny

claude commented

2026-07-08 07:03:17 +01:00

(Migrated from gitlab.keyop.co.uk)

assigned to @claude

danny (Migrated from gitlab.keyop.co.uk) approved these changes

2026-07-08 07:03:17 +01:00

claude commented

2026-07-08 07:03:43 +01:00

(Migrated from gitlab.keyop.co.uk)

mentioned in issue #35

danny commented

2026-07-08 07:32:07 +01:00

(Migrated from gitlab.keyop.co.uk)

LGTM

danny commented

2026-07-08 07:32:07 +01:00

(Migrated from gitlab.keyop.co.uk)

approved this merge request

danny (Migrated from gitlab.keyop.co.uk) scheduled this pull request to auto merge when all checks succeed

2026-07-08 07:32:11 +01:00

claude commented

2026-07-08 07:36:53 +01:00

(Migrated from gitlab.keyop.co.uk)

aborted the automatic merge because the source branch was updated. Learn more.

aborted the automatic merge because the source branch was updated. [Learn more](https://gitlab.keyop.co.uk/help/ci/pipelines/merge_trains.md#merge-request-dropped-from-the-merge-train).

claude commented

2026-07-08 07:36:53 +01:00

(Migrated from gitlab.keyop.co.uk)

added 1 commit

1fc8dc0e - Fix PATH after GOPATH override so go install'd tools are actually found

Compare with previous version

added 1 commit <ul><li>1fc8dc0e - Fix PATH after GOPATH override so go install'd tools are actually found</li></ul> [Compare with previous version](/keyop/go/nfq_forwarder/-/merge_requests/35/diffs?diff_id=342&start_sha=66222344678cebbac272a265fc4b997823e0c891)

claude commented

2026-07-08 07:44:46 +01:00

(Migrated from gitlab.keyop.co.uk)

added 1 commit

daa6aefc - Exclude the Go module cache from yamllint's scan

Compare with previous version

added 1 commit <ul><li>daa6aefc - Exclude the Go module cache from yamllint's scan</li></ul> [Compare with previous version](/keyop/go/nfq_forwarder/-/merge_requests/35/diffs?diff_id=344&start_sha=1fc8dc0ea018510ed03911e980962d977dbf6a40)

claude commented

2026-07-08 07:45:24 +01:00

(Migrated from gitlab.keyop.co.uk)

added 1 commit

c4dbec21 - Document the PATH and yamllint follow-up fixes for issue #35

Compare with previous version

added 1 commit <ul><li>c4dbec21 - Document the PATH and yamllint follow-up fixes for issue #35</li></ul> [Compare with previous version](/keyop/go/nfq_forwarder/-/merge_requests/35/diffs?diff_id=346&start_sha=daa6aefc03448139650ed1e21984d63d4b6617dd)

danny commented

2026-07-08 08:12:58 +01:00

(Migrated from gitlab.keyop.co.uk)

mentioned in commit 8432b95b7a

mentioned in commit 8432b95b7a29c8bc88328c4180d117c25d40b0c2