‹ Back to the moca docs · moca · ADR-0047

Async fan-out

A drain loop inside the ordinary turn worker — how N leaf turns are submitted, queued as bare ids, drained under the same admission budget as interactive turns, granted credentials only at dequeue, and recovered after a crash. Step through the mechanism, then read the decisions behind it. The written version is Dispatch archetypes.

interactive turn async leaf control-plane mint failure path

The decision

Async fan-out is a drain loop inside the ordinary turn worker, taking its slots from the same TurnCounter as interactive turns. The queue carries a turn id and nothing else; the leaf's credentials are minted when a worker claims it.

Shape of the answer

One queue, one trust model

mocactl, an LLM coding agent and a controller all enqueue through the same authenticated, owner-scoped API. The archetypes spec's deterministic non-LLM orchestrator no longer holds (#378), so every property must survive a misbehaving client.

The data-plane route stays removed

/runs {async:true} is unauthenticated when SH_REQUIRE_AUTH is off — the second trust model this design rules out. Two submission paths means every safety property is enforced twice and can diverge once. A controller holds a service-account credential instead (#687).

Shared admission, no headroom

No supervisor change: onChange already sends the load IPC and reconcile already treats the worker as authoritative. Dedicated --role=leaf workers and a second counter both create two capacity budgets that must be reconciled.

No credential in the queue

At claim the worker presents its own identity plus the turn id; the control plane resolves a credential name and mints the two MI1 grants. "Turn start" for turn_exp is dequeue, not enqueue.

turn_exp can only tighten

Clamped at mint: dequeue + min(caps.wallClock ?? 1800s, 1800s). Hour-long turns are an operator decision — raise the ceiling — not a caller's. A caller asking for more is clamped, not refused.

Abandon without acking on drain

The in-flight leaf is left pending for XAUTOCLAIM; the slot is released so awaitIdle exits early. A 1800 s leaf cannot finish in a 20 s grace window, and nobody is waiting on an async leaf — redelivery beats aborting.

History: why the design needed these four things

These were the findings of the design review (2026-11-05). All four are resolved on main; they are kept here because each explains a choice above.

  1. 1

    turn_exp was specified but not implemented. Expiry was always exp = iat + ttlSeconds. It is implemented now: a grant expires at min(now + ttl, turn_exp), and the control plane sets turn_exp when it mints.

  2. 2

    The claim heartbeat, without which every real leaf double-runs. minIdleMs is 90 s against a MOCA_TURN_MAX_S of 1800 s: a leaf running past 90 s goes idle in the PEL while still running, and a peer's XAUTOCLAIM would reclaim it — idempotent in result, double-spent in budget and sandboxes. The drain loop therefore touches its claim while a leaf runs, and the heartbeat dies with the process, so a crashed worker's leaf is reclaimed.

  3. 3

    The synchronous leaf aliases occupied no slot. /runs, /run-leaf and /v1/runs were not counted against the worker's turn budget, which over-admitted. All five paths (/turn, /v1/turn and those three) are in TURN_PATHS now.

  4. 4

    The consumer group needed a lifecycle on P6. The earlier drainer reaped once at process start and GC'd idle consumers so KEDA could scale to zero; neither trigger exists on P6. Reclaim is periodic and armed by every worker, keyed on moca-${HOSTNAME}-w${SH_WORKER_ID} — stable across an in-place restart, unique per live worker.

How it shipped

ADR-0047 is Accepted and all three milestones are implemented. Async fan-out is opt-in: set SH_ASYNC_DRAIN=1 on the workers.

M1

A working drain

Queue (@moca/work-queue), the drain loop in the worker (leaf-drain-loop.ts, async-worker.ts), per-owner quota and the status read (#704, #713, #717).

M2

Multi-tenant

turn_exp, grant-at-dequeue with both audiences, and the single-tenancy gate lifted (#788, #807, #822).

M3

deploy/k8s

Manifests, overlays, setup.sh and a queued-leaf smoke test (#765). Per-turn budgets, running-turn cancel and per-worker fairness followed in #391.