Beyond Unit Tests: Building a Robust CI Harness for Go OSS Projects

Catching Invisible Degradation in a Go OSS Project: 7 CI Checks Over 11 Months

Kazu, a maintainer of the gomarklint project, developed a comprehensive CI harness to eliminate silent failures. The system evolved after a release where passing tests and local builds still resulted in a broken binary for users.

Why This Matters

Standard test suites often create a false sense of security by verifying logic but ignoring the delivery pipeline. Technical reality shows that module path changes, runner variance in performance benchmarks, and invisible Unicode characters can bypass traditional CI, leading to broken installations or security vulnerabilities like the GlassWorm attack that standard diff reviews cannot detect.

Key Insights

• Coverage Gates (2025-06): Simple test execution is insufficient; enforcing a minimum coverage threshold (e.g., 80%) prevents the silent removal of test helpers or code paths.
• Install Canaries (2025-07): Verifying go install on published modules in clean environments catches failures caused by incorrect version tags or missing main packages that unit tests miss.
• Performance Witnessing (2026-02): Hard-failing CI on benchmarks leads to ‘reviewer fatigue’ due to GitHub Actions runner variance; posting delta comments allows human decision-making based on data.
• Supply Chain Defense (2026-05): The GlassWorm attack demonstrated that malicious code can be hidden via Unicode variation selectors (U+E0100–U+E01EF), necessitating automated regex scanning of source files.

Working Examples

Enforces a minimum test coverage threshold of 80%.

go test ./... -coverprofile=coverage.out
go tool cover -func=coverage.out | \
awk '/total:/ { pct=$3+0; if (pct < 80) { print "Coverage " $3 " is below 80%"; exit 1 } }'

Scans for invisible Unicode characters used in supply-chain attacks.

#!/bin/sh
grep -rPn \
'[\x{200B}\x{200C}\x{200D}\x{FEFF}]|[\x{E0100}-\x{E01EF}]' \
--include='*.go' --include='*.sh' . && {
echo "Invisible Unicode characters detected"
exit 1
}
[ $? -eq 2 ] && { echo "grep error: scan failed"; exit 2; }
exit 0

Pre-push hook to mirror CI checks locally.

#!/bin/sh
set -e
golangci-lint run ./...
go test ./...
go test -tags e2e ./...

Practical Applications

• ). Use Case: gomarklint using reviewdog to surface golangci-lint errors as inline PR comments rather than buried log lines. Pitfall: Treating lint errors as log output leads to them being skimmed and ignored by reviewers.
• ). Use Case: gomarklint using an Action smoke test to run published GitHub Actions against real Markdown fixtures. Pitfall: Testing only the underlying binary while ignoring the Action wrapper layer, leading to broken input names or entrypoint paths.

References:

• https://dev.to/_402ccbd6e5cb02871506/catching-invisible-degradation-in-a-go-oss-project-7-ci-checks-over-11-months-fmb