CI pipeline is slow

danny commented

2026-07-07 15:47:33 +01:00

(Migrated from gitlab.keyop.co.uk)

The CI pipeline takes a long time at the moment, and shouldn't need to.

We can probably pre-install some of the required tools in the image (image is at https://gitlab.keyop.co.uk/keyop/docker/golang-ci.git). Also check if there are any other efficiencies to be gained by restructuring the CI pipeline in any way.

The CI pipeline takes a long time at the moment, and shouldn't need to. We can probably pre-install some of the required tools in the image (image is at https://gitlab.keyop.co.uk/keyop/docker/golang-ci.git). Also check if there are any other efficiencies to be gained by restructuring the CI pipeline in any way.

danny commented

2026-07-07 15:49:01 +01:00

(Migrated from gitlab.keyop.co.uk)

changed the description

claude commented

2026-07-08 07:02:52 +01:00

(Migrated from gitlab.keyop.co.uk)

mentioned in commit 6622234467

mentioned in commit 66222344678cebbac272a265fc4b997823e0c891

claude commented

2026-07-08 07:03:19 +01:00

(Migrated from gitlab.keyop.co.uk)

mentioned in merge request !76

claude commented

2026-07-08 07:03:42 +01:00

(Migrated from gitlab.keyop.co.uk)

In-repo fixes up for review: !76 (branch issue35_speed_up_ci_pipeline) - splits the Makefile's setup target so each job only installs the one tool it actually needs, and adds Go module/build caching to .gitlab-ci.yml. See the MR description for the full trace-log timing breakdown that motivated this - the short version: go install .../goreleaser/v2@latest + go install .../gitleaks/v8@latest together accounted for ~2 of a ~2.5 minute job, by far the dominant cost, since neither is baked into the image.

Suggested changes for registry.keyop.co.uk/keyop/docker/golang-ci (I don't have access to that project to make these myself):

Bake in pinned goreleaser and gitleaks binaries - this is the big one, worth ~2 minutes per job on its own. Something like, in the Dockerfile:
```
RUN go install github.com/goreleaser/goreleaser/v2@v2.x.y && \
    go install github.com/zricethezav/gitleaks/v8@v8.x.y
```
(pin to specific versions rather than @latest - this project's own Makefile currently floats to @latest too, which is a separate, pre-existing minor risk worth fixing at the same time: an image rebuild could otherwise silently pick up a new major version of either tool with no warning). Once these are on $PATH in the image, this project's Makefile needs no changes at all - setup-goreleaser/setup-gitleaks's command -v guards (from !76) already skip the go install entirely when the binary is already present.
Bake in libpam0g-dev, yamllint, and iptables (apt packages) - smaller win (~6-7s/job) but same principle. Once present, .gitlab-ci.yml's defensive package-check (also from !76) automatically shrinks to a no-op with no further change needed on this side either.
Worth checking whether the image already has Go's own build cache warmed with this module's dependencies (or a generic warm cache from building typical Go projects) - if the image is rebuilt periodically, priming $GOPATH/pkg/mod/$GOCACHE with a representative go mod download/go build during the image build could shave additional time off the very first pipeline run against a fresh image, though !76's .gitlab-ci.yml cache should handle every run after that regardless.

Once (1) and (2) land in the image and a new tag is rolled out, the only remaining .gitlab-ci.yml change needed here would be bumping the pinned image tag (currently registry.keyop.co.uk/keyop/docker/golang-ci:1.26.4) - happy to do that once you let me know the new tag is ready.

**In-repo fixes up for review**: !76 (branch `issue35_speed_up_ci_pipeline`) - splits the Makefile's `setup` target so each job only installs the one tool it actually needs, and adds Go module/build caching to `.gitlab-ci.yml`. See the MR description for the full trace-log timing breakdown that motivated this - the short version: `go install .../goreleaser/v2@latest` + `go install .../gitleaks/v8@latest` together accounted for **~2 of a ~2.5 minute job**, by far the dominant cost, since neither is baked into the image. **Suggested changes for `registry.keyop.co.uk/keyop/docker/golang-ci`** (I don't have access to that project to make these myself): 1. **Bake in pinned `goreleaser` and `gitleaks` binaries** - this is the big one, worth ~2 minutes per job on its own. Something like, in the Dockerfile: ```dockerfile RUN go install github.com/goreleaser/goreleaser/v2@v2.x.y && \ go install github.com/zricethezav/gitleaks/v8@v8.x.y ``` (pin to specific versions rather than `@latest` - this project's own Makefile currently floats to `@latest` too, which is a separate, pre-existing minor risk worth fixing at the same time: an image rebuild could otherwise silently pick up a new major version of either tool with no warning). Once these are on `$PATH` in the image, this project's `Makefile` needs **no changes at all** - `setup-goreleaser`/`setup-gitleaks`'s `command -v` guards (from !76) already skip the `go install` entirely when the binary is already present. 2. **Bake in `libpam0g-dev`, `yamllint`, and `iptables`** (apt packages) - smaller win (~6-7s/job) but same principle. Once present, `.gitlab-ci.yml`'s defensive package-check (also from !76) automatically shrinks to a no-op with no further change needed on this side either. 3. Worth checking whether the image already has Go's own build cache warmed with this module's dependencies (or a generic warm cache from building typical Go projects) - if the image is rebuilt periodically, priming `$GOPATH/pkg/mod`/`$GOCACHE` with a representative `go mod download`/`go build` during the image build could shave additional time off the very first pipeline run against a fresh image, though !76's `.gitlab-ci.yml` cache should handle every run after that regardless. Once (1) and (2) land in the image and a new tag is rolled out, the only remaining `.gitlab-ci.yml` change needed here would be bumping the pinned image tag (currently `registry.keyop.co.uk/keyop/docker/golang-ci:1.26.4`) - happy to do that once you let me know the new tag is ready.

claude commented

2026-07-08 07:22:57 +01:00

(Migrated from gitlab.keyop.co.uk)

Submitted the golang-ci image fix: golang-ci MR !46 (assignee/reviewer set the same way as this project's MRs). Turned out to be more than just "add the missing tools" - while testing a real docker build against the current Dockerfile, found it's actually broken today: the existing curl/make version pins are stale (Debian bookworm versions against what's now a trixie-based image) and libpam0g-dev was never actually being installed at all due to a syntax bug (a colon instead of =, placed outside the actual install command) - which explains why this project's own CI has had to apt-get install it directly all along. Fixed both, added yamllint/iptables, and baked in pinned goreleaser/gitleaks. Verified with a real local Docker build (not just reviewed by eye) - every tool confirmed present and functional in the built image, including a real gitleaks scan. See the golang-ci MR for full detail.

Submitted the golang-ci image fix: [golang-ci MR !46](https://gitlab.keyop.co.uk/keyop/docker/golang-ci/-/merge_requests/5) (assignee/reviewer set the same way as this project's MRs). Turned out to be more than just "add the missing tools" - while testing a real `docker build` against the current Dockerfile, found it's actually broken today: the existing `curl`/`make` version pins are stale (Debian bookworm versions against what's now a trixie-based image) and `libpam0g-dev` was never actually being installed at all due to a syntax bug (a colon instead of `=`, placed outside the actual install command) - which explains why this project's own CI has had to apt-get install it directly all along. Fixed both, added yamllint/iptables, and baked in pinned goreleaser/gitleaks. Verified with a real local Docker build (not just reviewed by eye) - every tool confirmed present and functional in the built image, including a real gitleaks scan. See the golang-ci MR for full detail.

claude commented

2026-07-08 07:45:24 +01:00

(Migrated from gitlab.keyop.co.uk)

mentioned in commit c4dbec2115

mentioned in commit c4dbec2115570f6c99ec3aa085c526b9b2570e70

danny commented

2026-07-08 08:12:58 +01:00

(Migrated from gitlab.keyop.co.uk)

mentioned in commit 8432b95b7a

mentioned in commit 8432b95b7a29c8bc88328c4180d117c25d40b0c2

danny (Migrated from gitlab.keyop.co.uk) closed this issue

2026-07-08 08:12:59 +01:00

claude commented

2026-07-08 08:14:49 +01:00

(Migrated from gitlab.keyop.co.uk)

Merged via !76 (merge commit 8432b95). No release for this one - CI-only change, per Danny. Confirmed the speedup materialized: lint/build jobs went from ~150-156s each (pre-fix baseline) to ~16-18s each on pipeline 402 - roughly a 9x improvement, combining the golang-ci image bake-in (golang-ci !46) with the in-repo Go module/build caching.

Merged via !76 (merge commit `8432b95`). No release for this one - CI-only change, per Danny. Confirmed the speedup materialized: lint/build jobs went from ~150-156s each (pre-fix baseline) to ~16-18s each on pipeline 402 - roughly a 9x improvement, combining the golang-ci image bake-in ([golang-ci !46](https://gitlab.keyop.co.uk/keyop/docker/golang-ci/-/merge_requests/5)) with the in-repo Go module/build caching.

claude commented

2026-07-08 10:04:43 +01:00

(Migrated from gitlab.keyop.co.uk)

mentioned in issue #27

Rows
Columns

CI pipeline is slow #35