moca · sandbox service

Admission and the microVM pool

A request for a sandbox passes through four admission gates. This page shows each gate, each computation, and the pool that serves the Execs. Step through the sequence with the controls.

‹ moca docs · written page: Limits

standby · paused warming in-flight · resumed destroying parked · workspace only

How the system works

The text below follows the rules of ASD-STE100. Each sentence holds one fact. Each sentence uses one action.

Admission control

  • A sandbox request passes through four gates. The gates are the control plane, the relay, the worker queue, and the pool budget.
  • The control plane checks the enrollment list first. Then the control plane counts the live sandboxes. The count runs in one Lua script. The script is atomic. Two requests cannot take the same free slot.
  • The quotas control disk, not memory. A user holds at most 3 live sandboxes. The service holds at most 40 live sandboxes.
  • The control plane does no worker I/O at create time. The worker builds the workspace at the first Exec.
  • The relay caps the in-flight Execs per sandbox. The default cap is 2. One Exec runs. One Exec waits at the workspace gate.
  • The worker holds a queue of 64 Execs. The queue answers busy: queue full when it is full.
  • The pool checks the memory budget. The check adds the full price of one VM to the committed bytes. The price is guest RAM plus VM overhead.
  • A paused standby costs less than a running VM. The pool charges the measured dirty pages times the headroom factor.
  • A promotion over the budget evicts standbys from colder runs. The eviction picks the run with the oldest last use first.
  • A capacity refusal is retryable. The capacity reasons are memory-budget:, max-runs: and busy: queue full. The caller sees HTTP 429, HTTP 503, or RESOURCE_EXHAUSTED. A host-memory: refusal is not a capacity refusal and is not retried. Each refusal carries a distinct reason code.

Pool management

  • The pool groups VMs by workspace. Each workspace is one run.
  • The pool keeps standby VMs for each run. A standby VM is paused. The target count per run is 2.
  • An Exec acquires one standby. The acquire pops the newest standby first.
  • The VM runs one command. The pool destroys the VM after the command. The pool never reuses a VM.
  • The destroy runs to the end before the workspace gate opens. A benchmark-only option (DeferReapWorkers, default 0) can move the reap to background workers. The deployed worker does not enable it.
  • The pool refills a popped slot after a delay. The delay is 200 ms. The delay keeps a finished run from getting new VMs. A failed refill backs off. The backoff starts at 250 ms, doubles, and stops at 30 s.
  • The sweep runs on every Exec. The sweep also runs on a timer. The timer fires every 22.5 s.
  • A run with no Exec for 90 s drops its standbys. The run keeps the workspace. The run enters the state parked. A parked run holds disk, not memory.
  • The sweep destroys at most 8 VMs per scan. This rate limits the reclaim work.
  • A run with no Exec for a longer time loses the workspace. The default time is 30 minutes. A sandbox-service deployment typically sets it longer than the sandbox lifetime (for example 768 hours with the default 30-day maximum lifetime). A live sandbox keeps its workspace for its whole lifetime.
  • The host janitor runs every 15 minutes. The janitor asks the control plane for the live and terminated sandboxes. The janitor frees the disk of a terminated sandbox after a grace period.

Admission gates and their knobs

GateCheckKnobDefaultRefusal
control planesubject is enrolledSH_ALLOWED_SUBJECTSrequirednot enrolled
control planelive sandboxes per subjectSH_SANDBOX_MAX_PER_SUBJECT3429 quota_exceeded
control planelive sandboxes, totalSH_SANDBOX_MAX_TOTAL40429 quota_exceeded
relayin-flight Execs per sandboxSH_SANDBOX_MAX_INFLIGHT2RESOURCE_EXHAUSTED "sandbox busy"
workerwaiting ExecsQueueCap64, fixed"busy: queue full"
workerExec concurrency slotsWORKER_MAX_CONCURRENT16queueing, back-pressure
workerone Exec per workspaceexecGatefixedwait at the gate
poolmemory budgetSH_MAX_COMMITTED_MB / SH_MEMORY_RESERVE_MBrequired / 0"memory-budget:" → 503
poolstandby chargeSH_STANDBY_HEADROOM4.0 ("full" = full RAM)—
poolunparked runsSH_MAX_RUNS64"max-runs:" → 503
poolhost free memory floorSH_HOST_MEM_FLOOR_MB0, off"host-memory:" → refused, not retryable (no 503)

Pool knobs

KnobDefaultWhat it tunes
SH_STANDBY_DEPTH2standby VMs kept warm per run
SH_GUEST_RAM_MB256guest RAM per VM; the dominant cost term
VMOverheadBytes32 MiBper-VM cost outside guest RAM
StandbyIdle90 sidle time that parks a run (worker flag, not env)
SH_WORKSPACE_IDLE30 min (sandbox service: set longer than the sandbox lifetime)idle time that deletes a workspace
ReplenishDelay200 msgrace before a popped slot is refilled
replenish backoff250 ms → 30 sretry curve for a failed refill
MaxReclaimsPerScan8VMs destroyed per sweep; reclaim rate limit
SH_WORKSPACE_IMAGE_MB2048disk per sandbox workspace
SH_SANDBOX_LIFETIME_SECONDS7 dayssandbox lifetime; expiry is computed, not scheduled
SH_SANDBOX_TOKEN_TTL_SECONDS900sandbox token lifetime
SBX_JANITOR_AGE / cadence≥ token TTL + 5 min / 15 mingrace and run rate of the host janitor