Skip to content

Constrained Environments

Single-board-computer clusters and small cloud VPSs share one trait: a few GB of RAM and nothing to spare. A scan is CPU-light but memory-bursty: go test -coverprofile compiles the target repo's test binaries in parallel (Go's default -p = CPU count), and each compile can take 1–2 GB. A large repo on a 3.7 GB node — or a 2 GB VPS — can OOM the pod, and when the worker shares a node with the control plane, the whole cluster goes down with it.

go-crap-tracker is built to run in environments like this. This page explains the knobs and what they do.

What the system does by default

  • Per-pod memory limit — worker pods run with a hard limit (3000Mi in infra/), so one scan cannot eat a node's whole RAM.
  • Workers on dedicated nodes — the worker deployment pins workers to the worker pair (node affinity) and keeps them off the control-plane node and the node running the control app, so a scan's memory burst never competes with NATS, Redis and etcd.
  • Default parallelism — a repo with no override runs go test with Go's default parallelism. That is the fast setting, and it is the right one for small and medium repos on any hardware.

The per-repo knob: go_flags

go_flags is a per-repo GOFLAGS value applied to that repo's go test and go-crap subprocesses:

go-crap-tracker repo config set https://github.com/microsoft/typescript-go \
  --go-flags "-p=1 -parallel=2"
  • -p=1 — one go test package at a time (one compile in flight).
  • -parallel=2 — caps per-test parallelism within a package.

Empty (the default) means "no GOFLAGS, Go's default parallelism" — fastest, and correct for most repos. Only a repo that actually OOMs (or nearly does) on your hardware needs a value; start at -p=1 -parallel=2 on a ~4 GB node and go lower, not higher.

Semantics:

  • The value applies only to that repo's scans; other repos are unaffected.
  • A per-repo go_flags overrides any pod-level GOFLAGS environment variable — the worker drops the inherited GOFLAGS before applying the repo's value, so the per-repo setting always wins.
  • It is a per-repo override like timeout or test-args: repo config set is a partial upsert, repo config rm clears it.

Pod-level GOFLAGS (blanket fallback)

The worker deployment may set a GOFLAGS environment variable, which applies to every scan that has no per-repo value. Use it as a transition fallback or when every repo in your fleet is large; prefer per-repo go_flags once you know which repo needs it — a blanket -p=1 slows small repos down for nothing.

Sizing guide

Hardware What fits
≥ 8 GB node, worker-dedicated Large repos at default parallelism; usually no tuning needed
~4 GB SBC node (e.g. a 4-node cluster of 3.7 GB boards) Large repos need -p=1 -parallel=2; small repos are fine at defaults
1–2 GB VPS Small repos only; even -p=1 may be too much for the largest (10k+ function) repos — exclude them, or scan them from a bigger box

Observed on a 3.7 GB node: a single compile burst peaks around 2.1 GB RSS; with -p=1 -parallel=2 a 14k-function repo (typescript-go) completes its full scan — tests, coverage, analysis — under the 3000Mi pod limit without OOM.

Cold build cache matters for timeouts. go test first compiles every test binary (that is the memory- and time-heavy part), then runs the tests. On this hardware a cold-compile typescript-go run takes over 30 minutes — too long for the default cache.test_timeout — while a warm-cache run of the same repo finishes in about 6. So a large repo's per-repo timeout should cover its cold compile (typescript-go runs with --timeout 60m), not its warm time. A cold-cache first attempt that hits the timeout simply retries and succeeds warm; the timeout headroom makes even that unnecessary.

Symptom → fix

Symptom Fix
Worker pod OOMKilled mid-scan Set --go-flags "-p=1 -parallel=2" on that repo; raise the pod limit only if the node has the RAM
Whole node reboots / control plane goes down Pin workers off the control-plane node (node affinity or taints); set per-repo go_flags
A repo scans fine on big hardware, OOMs on your cluster That is the repo, not the cluster — go_flags is the per-repo valve
  • Repo Cache — per-repo override reference (ref, test-args, go-flags, timeouts).
  • Deployment — worker scheduling, limits, image.
  • repo command — repo config set --go-flags.