Skip to content

Repo Cache

Workers don't re-clone a repository for every scan. All workers share one cache directory (a local path, or a mounted ReadWriteMany PVC in Kubernetes) plus Postgres-based locks. This lets a fleet of workers on different pods collaborate on one set of clones and coverage profiles.

On-disk layout

<cache.directory>/
  <repoID>/                    # shallow clone, one per repo (repoID = sha of the URL)
  coverage/
    <repoID>/
      <sha>.out                # go test -coverprofile output, keyed by commit SHA
  • Clone<cache.directory>/<repoID>, one per repo.
  • Coverage<cache.directory>/coverage/<repoID>/<sha>.out. Coverage profiles are stored per commit SHA, so a commit that was already tested is never re-tested. Older profiles are pruned to cache.coverage_keep (default 10) most recent.

Flow per scan

For each job a worker handles:

  1. Acquire the repo lock (Postgres, see Locking).
  2. Prepare the clone — clone if missing; otherwise fetch + hard-reset to the tracked ref (the task ref, the per-repo --ref override, or the clone's default branch).
  3. Coverage — if a profile exists for the commit SHA, reuse it and skip go test entirely; otherwise run go test -coverprofile once and store the result.
  4. Scan — run go-crap scan --coverage-profile <profile> (always), so scores always reflect the tests.
  5. Save — upsert the analysis into Postgres under its deterministic ID.
  6. Release the lock.

If the ref or commit changes, the clone is updated in place; if the update fails, the worker removes the directory and re-clones fresh.

Locking

Locks live in Postgres (repo_locks), not on the filesystem, so any worker on any pod can take them — a ReadWriteMany volume would not give cross-pod filesystem locking.

  • Each lock is a lease (default cache.lock_lease 30s) that the holder refreshes on a background loop (~10s cadence) while it works.
  • If a holder's refresh stops (crash, eviction), the lease expires and the lock becomes stealable.
  • A worker that can't acquire the lock waits up to cache.lock_wait (default 10m) before failing the task.
  • In-flight work aborts immediately if the lock is lost mid-scan.
  • repo unlock <url> --yes and repo delete <url> --yes force-clear a stuck lock (see repo command).

Per-repo overrides

Stored in Postgres (repo_configs) and applied to every scan of that repository. Set them with the CLI or the REST API:

go-crap-tracker repo config set https://github.com/org/repo \
  --ref release \
  --test-args "-race ./pkg/..." \
  --crap-args "--exclude,legacy/.*" \
  --submodules \
  --timeout 30m \
  --git-timeout 30m \
  --go-flags "-p=1 -parallel=2"
Override Effect
ref branch/tag to track instead of the task's ref
test-args extra args after go test
crap-args extra go-crap flags (comma-separated)
submodules clone/update with git submodules
timeout per-repo go test timeout ("0" = global default)
git-timeout per-repo git operation timeout ("0" = global default)
go-flags GOFLAGS for the repo's go test / go-crap subprocesses; empty = Go default parallelism. See Constrained Environments

repo config set is a partial upsert — only the flags you pass change; the others keep their stored value. Remove all overrides with repo config rm <url>.

Tuning knobs

Config Default Meaning
cache.directory (empty → $TMPDIR/go-crap-tracker-cache) shared cache root
cache.lock_lease 5m per-repo lock lease, refreshed while work runs
cache.lock_wait 10m how long a worker waits for a busy repo
cache.test_timeout 10m timeout for go test -coverprofile
cache.git_timeout 5m timeout for git operations (clone/update/reset)
cache.coverage_keep 10 coverage profiles kept per repo (per commit SHA)

Configuration for the full list and env-var equivalents.