Skip to content

Task Lifecycle

A scan task is the unit of work: one scan of one repository at one ref. It moves through a small, enforced state machine.

States

Status Meaning
pending just created, not yet published to the queue
queued published to NATS, waiting for a worker
running a worker claimed it and is scanning
completed finished, an analysis_id is set
failed a worker ran it and it errored
expired was running too long and the reconciler gave up on it

completed, failed, and expired are terminal for the worker — but failed and expired can be retried by the reconciler.

Transitions

                ┌──────────────┐
                │   pending    │
                └──────┬───────┘
                       │ schedule (publish job)
                       ▼
                ┌──────────────┐      claim        ┌──────────────┐
                │    queued    │──────────────────▶│   running    │
                └──────┬───────┘                   └──────┬───────┘
                       │                                  │
              stale queued (no worker)                    │
                       │                         complete │ fail
                       ▼                                  ▼
             re-publish to queue          ┌────────────┐┌────────┐
                                          │ completed  ││ failed │
                                          └────────────┘└───┬────┘
                                                            │
                                   stale running (max_stale_min)
                                                            ▼
                                                       ┌────────┐
                                                       │expired │
                                                       └────────┘

Transitions are validated in the domain (internal/core/domain/scantasks); an illegal move returns ErrInvalidTransition.

  • A task can only be claimed if it is pending/queued and attempt < max_attempts (default max_attempts is 3).
  • Claiming bumps attempt by one and sets running.
  • running → completed (with analysis_id), failed (with an error), or expired (with a reason).

Retries

  • Each task carries attempt / max_attempts (default 3).
  • A worker that fails a scan marks it failed; a task the reconciler expires is expired.
  • The reconciler's retryDead step re-queues any failed/expired task that still has attempts left (attempt < max_attempts), then re-publishes the job to NATS.
  • The NATS consumer is configured with MaxDeliver: 3, so a poison message is also bounded at the transport layer.

The reconciler

Started by serve, the reconciler ticks every 30 seconds and runs three passes:

  1. reconcileStale — find running tasks that have been stale (no progress) longer than max_stale_min and Expire them with reason stale: worker disconnected.
  2. retryDead — re-queue and re-publish failed/expired tasks that still have attempts left.
  3. rescheduleStaleQueued — re-publish queued tasks that have sat untouched longer than queue_stale_min (covers a job lost in the queue).

Each pass broadcasts a status event so SSE subscribers and the UI update live.

Note — while a scan is running, the worker also heartbeats the task's updated_at (well below max_stale_min) so a legitimately long scan is not mistaken for a dead worker.