Skip to main content
Version: next

Sandbox service runbook

Preview. The sandbox service is implemented, but not yet declared Stable; interfaces and limits may change.

For the operators of a sandbox-service deployment: the sandbox control plane, the relay and Redis in one namespace of a shared OpenShift cluster, and template workers plus the janitor on a KVM host. The install reference (every input and its rule) is Install the sandbox service; users have the user guide. The design is the sandbox service's design spec in the repository's docs/specs/ (its §8 for operations).

Throughout, <env file> is the operator's settings file outside the repository (README §1), <host> a KVM host, <checkout> this repository on that host, and <id> a worker id. Keep deploy/sbx/.generated/ between runs: setup.sh reads the bundles it last wrote there (it is gitignored).

1. Deploy from scratch​

  1. The namespace. An existing namespace, dedicated to the sandbox service, where you have edit rights.

  2. A GitHub OAuth app of this deployment's own, with device flow enabled: its client id is SBX_GITHUB_CLIENT_ID.

  3. The snapshots. On the KVM host, one golden snapshot per template, built with deploy/microvm/build-snapshot.sh --out /srv/sbx/snapshots/<image> (MicroVM tier (P4) has the full command). Every curated template carries bash, git, tar and ripgrep. Build them with a build-snapshot.sh from P4.1 or later: an older snapshot has no guest egress listener (see Guest egress). The host also needs /usr/local/bin/moca-egress (CGO_ENABLED=0 go build -o /usr/local/bin/moca-egress ./remote-worker/cmd/moca-egress); the worker install refuses without it, unless you pass --no-egress. And the sandbox service's own worker binary, CGO_ENABLED=0 go build -o /usr/local/libexec/sbx/microvm-worker ./remote-worker/cmd/microvm-worker (SBX_WORKER_BIN): the install refuses without it, and never runs /usr/local/bin/microvm-worker, which a P4 worker on the same host may run.

  4. The cluster side. Write <env file> (README §1 lists every input; the required ones are SBX_NAMESPACE, SBX_API_HOST, SBX_ATTACH_HOST, SBX_EXEC_HOST, SBX_IMAGE, SBX_WORKERS, SBX_TEMPLATES_FILE, SBX_GITHUB_CLIENT_ID, SBX_ALLOWED_SUBJECTS and SBX_WORKSPACE_FS_GIB), then:

    deploy/sbx/setup.sh --env-file <env file>

    It generates every token, key and certificate once and keeps them, applies the overlay, waits for the rollouts, and writes one worker bundle per SBX_WORKERS id plus the janitor bundle under deploy/sbx/.generated/.

  5. Each worker, on its KVM host:

    scp -r deploy/sbx/.generated/workers/<id> <host>:sbx-<id>
    ssh <host> sudo <checkout>/deploy/sbx/install-host.sh sbx-<id> --snapshot <image> [--budget-mb N]
    ssh <host> rm -r sbx-<id>

    It waits up to 120 s for the worker's attach to hold (SBX_ATTACH_TIMEOUT). The worker's guest RAM (SH_GUEST_RAM_MB) is taken from the snapshot's manifest.json (guest_ram_mb), never set by hand: a snapshot without it is refused. --budget-mb (default 8192) must hold at least one session, (guest RAM + 32) × 3 + 1024 MiB at the unit's standby depth of 2: 13408 MiB for a 4096 MiB guest. A smaller budget is refused, naming the minimum. The worker runs SBX_WORKER_BIN (default /usr/local/libexec/sbx/microvm-worker, README). When <image> was built by deploy/sbx/template-onboarding/template-build.sh, its template record (/srv/sbx/templates/<image>.json) is wired into the worker env (SH_TEMPLATE_RECORD, SANDBOX_TEMPLATE), so the worker checks the record against its snapshot at every start; a record for another worker is refused before anything is written. On a host run by a host agent, run it as sudo SBX_WORKER_AT_BOOT=0 <checkout>/deploy/sbx/install-host.sh ...: the worker is started once but not enabled for boot, so a reboot leaves starting workers to the agent's wanted state instead of starting all of them.

  6. The janitor, once per KVM host:

    scp -r deploy/sbx/.generated/janitor <host>:sbx-janitor
    ssh <host> sudo <checkout>/deploy/sbx/install-host.sh --janitor sbx-janitor
    ssh <host> rm -r sbx-janitor
  7. Check it with status: every worker attached, the janitor timer active, then log in as a user and run mocactl sbx doctor in a repository.

Which mocactl users install. Since v0.7.0 the default channel, the latest release, carries mocactl sbx, as the user guide says:

curl -fsSL https://raw.githubusercontent.com/rossoctl/moca/main/scripts/install-mocactl.sh | sh
mocactl sbx --help # what an sbx sandbox is, then: usage (the sandbox service):

A mocactl older than v0.7.0 has no sbx: its mocactl sbx --help prints the general usage: list, and a bare mocactl sbx says unknown command "sbx". Such a user re-runs the install line above (or with MOCACTL_VERSION=edge for the newest build of main).

The increments' inputs, for an older deployment: 1a had SBX_CLIENTS (static tokens; now refused) and no sandbox profile; 1b added the control-plane inputs (SBX_API_HOST, SBX_GITHUB_CLIENT_ID, SBX_ALLOWED_SUBJECTS, SBX_TEMPLATES_FILE, the lifetimes and quotas); 1c adds the janitor (SBX_JANITOR_WAIT_MINUTES, SBX_API_CA_FILE) and nothing else that is required.

2. Upgrading​

An image upgrade (a new SBX_IMAGE): set it in <env file> and re-run setup.sh. A changed image rolls the control plane and the relay, and so does a change to what the relay reads (the token issuer, SBX_MAX_INFLIGHT, the signing keyset, the attach or exec certificate). Workers keep running, and reattach when the relay comes back; Execs running through the old relay pod end with it (the worker kills a dropped stream's Execs), so apply those when nothing long is running. Any other setting (templates, subjects, quotas, lifetimes) rolls the control plane alone, and running Execs carry on.

Moving to idle lifetimes (#770). The first run of this setup.sh on an existing deployment changes every worker's idle from 216 h to 768 h at the defaults, and starts idle lifetimes: the control plane counts idle time only when SH_SANDBOX_MAX_LIFETIME_SECONDS is set, and keeps the old absolute lifetime otherwise. Until the hosts can be reinstalled, take this setup.sh with idle lifetimes off: it writes no SH_SANDBOX_MAX_LIFETIME_SECONDS and keeps every worker's idle at 216 h (at the defaults), so no host changes and routine runs (enrolling a user, a quota change) need no ack:

SBX_MAX_LIFETIME_DAYS=off deploy/sbx/setup.sh --env-file <env file>

Idle lifetimes stay off; set a number on the day you reinstall the hosts. On that day, in this order:

SBX_ACK_HOSTS=1 deploy/sbx/setup.sh --env-file <env file>

then re-run install-host.sh with the new bundles on every host the same day, and then run status.sh and check that it prints no worker idle WARNING. See §3.

From 1a: unset SBX_CLIENTS (the run refuses it), move SBX_IMAGE off the team-only 1a image, and run with SBX_ACK_HOSTS=1, because every 1a worker bundle lacks the workspace idle time and must be reinstalled:

SBX_ACK_HOSTS=1 deploy/sbx/setup.sh --env-file <env file>

Then re-run install-host.sh with the new bundle for every worker (step 5 of §1); it refuses a 1a bundle. The run deletes the static client tokens and removes the relay's harness exec token. 1a workspaces are not migrated: tell 1a users to pull their work out first.

The worker binary. install-host.sh runs the sandbox service's own build at SBX_WORKER_BIN (default /usr/local/libexec/sbx/microvm-worker), no longer /usr/local/bin/microvm-worker: build it there (§1 step 3) before re-running install-host.sh on a host installed before, or the install refuses. A new worker binary takes effect when its workers restart.

Redis's image and its data. Since #670 the base runs Valkey 9, which cannot read RDB format 12, the format Redis 7.4 writes. On a namespace whose data a Redis 7.4 wrote, Valkey's AOF load aborts (Can't handle RDB format version 12) and redis-0 crash-loops, taking the control plane down with it. Until a migration exists, keep such a namespace on Redis 7: set SBX_REDIS_IMAGE to the digest it ran, e.g. docker.io/redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499 (the pin before #670). With SBX_REDIS_IMAGE unset, setup.sh now refuses, before it applies anything, when the namespace's redis StatefulSet runs an image other than Valkey, and names that image. Before any Redis image change, take a copy of /data (BGSAVE, then dump.rdb and appendonlydir). The StatefulSet is OrderedReady: a redis-0 that crash-loops on the old revision is never replaced, so setup.sh's rollout wait times out. Then delete the pod (kubectl -n <namespace> delete pod redis-0; its PVC stays) and re-run setup.sh.

Redis's memory limit. setup.sh's redis.conf sets maxmemory 200mb with maxmemory-policy noeviction, under the container's 256 MiB limit: a full Redis refuses writes (the control plane answers 503 redis_unavailable) and evicts nothing, instead of being OOM-killed with every session and sandbox record. A new namespace gets it at Redis's first start. An existing one does not: the Secret's change does not restart redis-0 (the StatefulSet has no settings hash), so apply it live, kubectl -n <namespace> exec redis-0 -c redis -- redis-cli CONFIG SET maxmemory 209715200 and … CONFIG SET maxmemory-policy noeviction (REDISCLI_AUTH is set in the container), or let the next restart read the file. Check with CONFIG GET maxmemory. A namespace that raises the container's memory limit should raise maxmemory with it.

A host bundle change. Re-run install-host.sh with the new worker bundle whenever setup.sh says so, and install-host.sh --janitor with the new janitor bundle after a change to SBX_JANITOR_WAIT_MINUTES, SBX_TOKEN_TTL_MINUTES (its default wait follows the lifetime), SBX_API_HOST or SBX_API_CA_FILE. status.sh --host warns when a host's installed janitor wait no longer exceeds the token lifetime plus 30 s.

A worker installed before the jail path was keyed by a digest of the id has its jail at /srv/sbx/jail/<id>; a reinstall moves it to /srv/sbx/jail/j<12 hex of the id>. Re-running install-host.sh for that worker, or --uninstall <id>, removes the old /srv/sbx/jail/<id> (only that path, once the worker is stopped) and logs removed <path>; its workspace data is not touched.

Rolling a worker back past lease mode (plan 4), then forward again. Every worker since plan 4 writes <workspace root>/.lease-state/.boot at boot, lease=on or lease=off; a lease-on boot after a lease-off one first marks every known lease state root_wrote, because a lease-off worker runs per-Exec commands as root without recording it. A worker binary from before plan 4 writes no marker, so while a host runs one the marker can still say lease=on. A rollback alone is safe: the old binary has no lease mode. Rolling forward again is safe too, as long as it goes through install-host.sh: every worker install deletes .lease-state/.boot before it restarts the worker (a symlinked marker goes as the link; a symlinked .lease-state is refused), so the first new boot finds no marker and marks every known state, conservatively. A rollback done with an older install-host.sh leaves the marker, and the next forward install clears it. Never roll forward by replacing SBX_WORKER_BIN and restarting the unit alone; if that happened, stop the worker, delete /srv/sbx/workspaces/<id>/.lease-state/.boot, and start it again.

3. Changing the sandbox lifetime​

Two settings. SBX_LIFETIME_DAYS (default 7) is the default idle lifetime: a sandbox ends after that many days without use. SBX_MAX_LIFETIME_DAYS (default 30) is the longest an owner may pick, and each worker's idle reclaim, SH_WORKSPACE_IDLE, is that plus two days. The default must not exceed the maximum. SBX_MAX_LIFETIME_DAYS=off keeps idle lifetimes off, with the idle at SBX_LIFETIME_DAYS plus two days (§2).

Changing only SBX_LIFETIME_DAYS touches no host: re-run setup.sh and the control plane rolls.

Changing SBX_MAX_LIFETIME_DAYS means reinstalling every host's workers. The control plane takes the new maximum at once, but a running worker keeps the idle it was installed with until install-host.sh is re-run on its host; after a raise, that idle can be shorter than a live sandbox's lifetime, and the worker would reclaim its workspace.

So setup.sh refuses a maximum that raises the idle in the bundles it last wrote, unless acknowledged:

SBX_ACK_HOSTS=1 deploy/sbx/setup.sh --env-file <env file>

and then, right after, re-run install-host.sh with the new bundle for each worker the run names.

Lowering the maximum needs no ack and touches no host: sandboxes keep the deadlines set under the old maximum (the control plane never shortens a deadline), and a shorter idle would reclaim their workspaces, so the run keeps the bundles' idle and logs kept at <old>h until SBX_LOWER_WORKSPACE_IDLE=1. Once the old maximum has passed since that run:

SBX_LOWER_WORKSPACE_IDLE=1 SBX_ACK_HOSTS=1 deploy/sbx/setup.sh --env-file <env file>

writes the shorter idle into the bundles; the hosts take it when install-host.sh is next run on them. Never set SBX_LOWER_WORKSPACE_IDLE=1 before the old maximum has passed: a routine reinstall would then give the hosts the shorter idle early. Pass it on the command line for that one run; do not keep it in the env file: the env file is sourced with set -a, so kept there it makes a later lowering skip the "kept" step and, with SBX_ACK_HOSTS=1 also in the file, write the shorter idle at once.

status.sh prints worker idle <id>: ... for each host and a WARNING when a worker's idle does not exceed the maximum.

4. Adding or removing a user​

Users are SBX_ALLOWED_SUBJECTS: comma-separated github:<numeric id> or oidc:<provider sub> (a mixed list is accepted), no spaces. Look a GitHub id up with gh api users/<login> --jq .id. A user who is not enrolled is told their own subject by mocactl sbx login. The model behind this section (who is enrolled, who is an admin, the installer token, what a removed user leaves behind) is in Users, enrollment and roles.

Edit the list and re-run setup.sh. The control plane rolls with the new list:

  • adding takes effect as soon as the rollout ends;
  • removing refuses every sandbox route (create, list, mint) at once, and the user's login at its next refresh. A sandbox token already issued keeps working until it expires: within one token lifetime plus 30 s (SBX_TOKEN_TTL_MINUTES, 15 minutes by default), and a command still running then is cancelled. The user's sandboxes are not deleted; delete them as an admin (mocactl sbx ls --all, mocactl sbx rm --force <sandbox id>) if their work must go. The --force is needed: rm first checks a sandbox for work not yet pulled, which takes a sandbox token, and an admin is never minted one for another user's sandbox.

Admins (SBX_ADMIN_SUBJECTS) may list, read and delete every sandbox, but never mint a token for another user's: they cannot run commands in another user's code.

Enrollment and admins through the API (on a deployment with a template store, SBX_TEMPLATE_STORE=1). The settings above stay the bootstrap; an admin named in SBX_ADMIN_SUBJECTS adds to them without a setup.sh run (an API-granted admin can't change access: 403), each change a record on the members provenance stream:

mocactl sbx admin subjects # who is enrolled, and how
mocactl sbx admin subjects enroll github:<id> # or unenroll; oidc:<sub> too
mocactl sbx admin subjects grant github:<id> --until 1h # a temporary admin: lapses by itself (365d at most)
mocactl sbx admin subjects revoke github:<id>

Each change reads the subjects first and prints old -> new with its record; if the subject changed between that read and the write (another admin at once), it says so and exits 1: check and write again. A login issued for another deployment (another control plane or issuer) is refused before the write.

It takes effect at the subject's next request. A subject in SBX_ALLOWED_SUBJECTS or SBX_ADMIN_SUBJECTS can't be unenrolled or revoked this way (409 bootstrap_subject): change the settings for those. Enrolling one of them writes nothing (it is already enrolled). Before removing a subject from the settings, check its storeEnrolled in mocactl sbx admin subjects --json (enrolled (env, api too) in the list): if true, the store still enrolls it, so unenroll it through the API once the settings no longer name it. An admin can't revoke its own role. A subject holding an admin grant can't be unenrolled while it does (409 admin_granted): revoke the role first. If the store's members.json can't be read (damaged by hand, a bad restore), the subjects in the settings keep working, with a warning in the control plane's log (once a minute at most), while API enrollments and grants don't count (those subjects are refused) until the file is restored: the settings stay the way back in.

What this changes for an operator:

  • Who controls access. Without a store, every enrollment or admin change is a setup.sh run. With one, an admin named in SBX_ADMIN_SUBJECTS enrolls subjects and grants admin through the API. The record on the members stream is then the trace, so read it (GET /v1/subjects, the provenance export) as part of any access review.
  • Roles come from the store, not the login. With a store, the control plane replaces a token's roles claim, on every request, with the subject's roles now (the settings plus the store's grants). Two consequences:
    • A login minted as a user acts as an admin if its subject holds an API admin grant. Check GET /v1/subjects before treating a login as user-only.
    • Revocation is immediate. Removing a subject from SBX_ADMIN_SUBJECTS (a setup.sh run), or revoking an API grant, takes effect at the subject's next request, even for an admin token that is still valid. But setup.sh can't remove an API grant (those live in the store, on its volume), so after removing an admin from the settings, check GET /v1/subjects for API admins too. For a temporary admin, prefer an API grant with --until: it lapses by itself.

5. Adding, updating and removing a template​

The template onboarding folder, deploy/sbx/template-onboarding/ (its GUIDE.md and ONBOARDING.md), is the long form, with the tools that do the steps below: template-build.sh (the snapshot, checked in a microVM) and template-register.sh (the two lists).

A template is an id clients choose (mocactl sbx create --template <id>, POST /v1/sandboxes {template}), served by one worker running one snapshot. A sandbox keeps the worker it was created on for its whole life, whatever later happens to its template's entry.

Adding a template:

  1. Build its snapshot on the KVM host (§1 step 3), named <image>. An image built from a container image for a benchmark (one that refuses agent commits, with a read-only rootfs) usually needs build-snapshot.sh --skip-git-identity-check --guest-home /tmp/home, and --vcpus N --guest-ram-mb M for its size (MicroVM tier (P4)).
  2. Pick a worker id (^[A-Za-z_][A-Za-z0-9_]*$, say sbx_node22) and add it to SBX_WORKERS.
  3. Add its entry, { "id", "worker", "description", "tools" }, to SBX_TEMPLATES_FILE. One template per worker; every worker in the list.
  4. Re-run setup.sh: it mints the worker's token and writes its bundle. It rolls the control plane but not the relay (§2), so running Execs on every template carry on.
  5. Install the worker with that bundle and --snapshot <image> (§1 step 5). A new worker's token reaches the relay when the kubelet syncs the Secret, about a minute, so its first attaches may be refused; install-host.sh waits. Pass --budget-mb for a guest larger than 256 MiB (§1 step 5, and §9 for concurrent use).
  6. Check it with deploy/sbx/templates-check.sh (§14).

Updating a template (a new image or snapshot) is versioned: never re-point a worker that serves sandboxes at another snapshot, or their commands change environment mid-work. Instead:

  1. Build the new snapshot under a new <image> name.
  2. Add a new worker id for it (say sbx_node22_v2) to SBX_WORKERS, keeping the old one.
  3. Change the template's worker in SBX_TEMPLATES_FILE to the new id. The old worker serves no template now, but stays in SBX_WORKERS, so it keeps its token and its sandboxes keep working.
  4. Re-run setup.sh, install the new worker (steps 4-5 above), and check it (step 6). New sandboxes of the template go to the new worker; existing ones stay on the old.
  5. Retire the old worker once it serves no live sandbox: GET /v1/janitor/sandboxes (the janitor's token) lists each live sandbox with its wrk, and mocactl sbx ls --all (an admin) shows their templates. Then remove it as below.

Both workers hold their snapshot's memory file in RAM (it is the guest's whole RAM image) until the old one is retired, so the host needs room for both.

Removing a template (or a retired worker) is the reverse of adding one: drop it from both lists, re-run setup.sh (which revokes the worker's token and deletes its bundle; it cannot attach again), then sudo <checkout>/deploy/sbx/install-host.sh --uninstall <id> on its host. Its sandboxes stop working at once (no worker serves them): tell their owners to pull first. With egress credentials on (SBX_EGRESS_CREDENTIALS=1), a worker that serves no template is no egress worker, so its remaining sandboxes get no stored credential (as if their owners had none) and credentialed hosts answer them 401 until it is retired.

One control plane process per template store (a deployment with SBX_TEMPLATE_STORE=1): activation's quota and placement check and its lease write run under a lock inside the control plane process, which is enough because one process owns the store: the Deployment runs one replica, and the template store's volume is ReadWriteOnce with strategy: Recreate, so no second pod writes it, even during a rollout. Keep it that way. A control plane with more than one replica, or a rolling update, on one store would need a store-level lock.

Self-serve templates and the policy (a deployment with a template store and host agents, "The template host agent" below): clients create their own templates with POST /v1/templates, gated by the stored policy (GET /v1/policy), which only an admin changes (PUT /v1/policy, a policy record). What an operator sets there:

  • Profiles, the gate a create names: base (no gate record), benchmark (a gate record an admin trusts in trustedGateRecords), and the self-attested benchmark-self (offline) and general-self (the KVM host's allowlisted egress), which trust a subject's own gate record for its own templates only. A new image's default policy gains new profiles, but a stored policy does not: add them with one PUT /v1/policy, the whole policy as read plus the change (below).
  • Images: allowedRegistries (<default> is the KVM host's own SBX_REGISTRY) and, per registry, allowedRepositories. Docker Hub is docker.io. On a rollback to an image older than the repository allowlist, take docker.io out of allowedRegistries first: an older image ignores allowedRepositories, so it would admit every Docker Hub repository.
  • Quotas: quota, and quotaBySubject for an override.
  • What clients see of it: mocactl sbx policy, mocactl sbx quota and mocactl sbx hosts; they create, follow and activate their templates with mocactl sbx templates (Make your own templates).

Writing the policy. A PUT /v1/policy stores the whole policy, quota overrides included, so send it as a compare-and-set: ?expect= the digest of the GET /v1/policy it was built from. If another admin changed anything since (a quota too), nothing is written and the answer is 409 policy_changed: read the policy again, redo the change on it, and write with the new digest. A write without expect replaces whatever was set meanwhile. Quotas alone are simpler to set with mocactl sbx templates quota, which does the same.

# CP: the control plane's URL; HDR: a 0600 file with one line,
# "Authorization: Bearer <an admin's session token>" (never the token in argv).
curl -fsS -H @"$HDR" "$CP/v1/policy" > policy.json
digest="$(jq -r .digest policy.json)"
jq .policy policy.json > new-policy.json # then make the change in new-policy.json
curl -sS -o put.json -w '%{http_code}\n' -X PUT -H @"$HDR" -H 'Content-Type: application/json' \
--data @new-policy.json "$CP/v1/policy?expect=$digest"
# 200: put.json is the policy as set (its new digest, its record, any `narrowed` builds).
# 409 policy_changed: nothing written; put.json's `digest` is the policy's now. Read again, then write.

Which policy writes need an admin named in the settings. Any admin may narrow the policy and change quotas. A write that widens it is refused 403 forbidden for an admin whose role comes from an API grant (mocactl sbx admin subjects grant); only an admin named in SBX_ADMIN_SUBJECTS (SH_ADMIN_SUBJECTS) may make it. Widening is any of:

  • adding a sha256 to trustedGateRecords, or a registry to allowedRegistries;
  • adding a repository to a registry's allowedRepositories, or dropping an allowed registry's list (which opens the whole registry); registry keys that differ only in case count as one;
  • adding a profile, or loosening one: a rule dropped from requires, requireOfflineGate turned off, selfAttest or selfAttestEgress turned on, an envelope bound raised, or its exact size changed or removed;
  • moving defaultProfile to a profile that accepts more.

The check and the write are one locked step, so a write built from an older policy cannot restore trust a settings admin removed meanwhile.

Trusting a gate record, and self-attested templates. Trust comes from who submitted a gate record or from an admin's list, never from what the record says:

  • Deployment-wide trust is the record's sha256 in trustedGateRecords, which a settings admin adds with PUT /v1/policy (above). The benchmark profile accepts only such a record.
  • Self-attested (the benchmark-self and general-self profiles, selfAttest: true): a record the policy does not trust still passes X1.trusted for the template's owner, the creator (or, when an admin adds a build to another subject's template, that template's owner). A template the deployment declared has no owner, so it takes no self-attested build. A self-attested template stays private: deployment visibility and a transfer are refused 409 self_attested until an admin trusts the record. Removing a record from trustedGateRecords makes those templates private again on the next read; the PUT /v1/policy answer and its policy record list them as narrowed.
  • One subject per self-attested record. An admitted self-attested create claims the record for its owner. The claim holds while any build citing it is queued, building, probing, built, registered, rejected or retired; only a failed build (a cancel included) releases it. A create made while the record was trusted, and a refused create, claim nothing. Another subject's create with a claimed record is refused, rule X1.self.scope. The lookup and the write run as one step per sha256, so two subjects sending the same record at once cannot both claim it.
  • A self-attested record must show commits refused (X12.self: agent_can_commit: "no" and git_identity: "none"), whatever the spec says, and the spec may set egress: true only on a profile with selfAttestEgress (X1.self.egress).
  • skip_git_identity_check needs a gate record under every profile, base included: a skip no gate record backs, or one whose record does not show commits refused, is 422 policy_refused, rule X12. Send such a gate record, or drop the skip.

6. Rotating the signing key​

The control plane signs sandbox tokens with one Ed25519 key, SH_SESSION_TOKEN_PRIVATE_KEY in the moca-mu1 Secret; the relay verifies with its public half, in sbx-token-keys. setup.sh keeps exactly one key: every run rewrites sbx-token-keys with the current key alone, so a second key added by hand is removed by the next run. Rotation is therefore a key replacement:

kubectl patch secret moca-mu1 -n <namespace> --type=merge \
-p '{"data":{"SH_SESSION_TOKEN_PRIVATE_KEY":null}}'
deploy/sbx/setup.sh --env-file <env file>

Remove that one key only: moca-mu1 also holds the janitor's token (SH_JANITOR_TOKEN), which deleting the whole Secret would rotate too. The run generates a new key, publishes its public half as the whole keyset, and rolls the relay and the control plane (both pods' settings hashes cover the keyset). setup.sh does not keep the old key beside the new one, and nothing else does.

What users see: mocactl mints a fresh sandbox token for its calls, and re-mints once when the relay refuses a token before the call started, so sessions carry on with the new key. A token minted with the old key is refused by the new relay (an unknown key id): a direct client must run its SBX_TOKEN_CMD again, and no old-key token outlives one token lifetime in any case. The relay's rollout also ends the Execs running through it (the worker kills a dropped stream's Execs), so rotate when nothing long is running. The gentler form of spec §4.3 (both keys in the keyset while the signer switches) is not what setup.sh does.

The janitor's token rotates the same way, plus a restart, because the control plane reads it only at boot and it is not in the settings hash:

kubectl patch secret moca-mu1 -n <namespace> --type=merge -p '{"data":{"SH_JANITOR_TOKEN":null}}'
deploy/sbx/setup.sh --env-file <env file>
kubectl rollout restart deployment/moca-control-plane -n <namespace>

Then re-run install-host.sh --janitor with the new janitor bundle on every host. Until then, each janitor run there logs skipped (refused) and removes nothing.

Rotating the credential KEK​

Only with egress credentials on (SBX_EGRESS_CREDENTIALS=1). SH_CREDENTIAL_KEK in the sbx-egress Secret is what opens every stored credential: the store — a PVC or Vault — holds ciphertext only, so this one value is the most sensitive in the deployment, and it is the same procedure for either store.

It is a ring, and re-sealing is write-driven. SH_CREDENTIAL_KEK is a comma-separated list, newest first. Writes always seal under the first key; reads try every key. So a credential moves to the new key only when it is next written — a credential nobody touches stays on the old key indefinitely. There is deliberately no bulk re-wrap: no API returns a stored secret (packages/control-plane/src/envelope.ts), so nothing — not even an admin — can export and re-seal.

Dropping the retired key too early is unrecoverable: that credential can never be decrypted, and the only repair is the user entering it again. So the last step waits on evidence, not on a timer.

Set NS first. Each step reads the live Secret and refuses on a value it does not expect, because the failure mode of a half-pasted step is writing an empty SH_CREDENTIAL_KEK: the control plane then refuses to boot (SH_CREDENTIAL_KEK is not set) and the ring is gone from the Secret, so unless you still hold the old value every stored credential becomes unreadable. Step 3 takes the key it keeps from the live ring rather than from a variable, so the steps are safe to run in separate shells and no key is ever echoed to a terminal or a shell history.

NS=<namespace>
# 1. The new key, prepended to the ring. The current value is kept as the SECOND entry: overwriting
# instead of prepending is the mistake that loses every existing credential.
old="$(kubectl get secret sbx-egress -n "$NS" -o jsonpath='{.data.SH_CREDENTIAL_KEK}' | base64 -d)"
[[ -n "$old" ]] || { echo 'refusing: sbx-egress has no SH_CREDENTIAL_KEK to rotate'; return 2>/dev/null || exit 1; }
new="$(openssl rand -base64 32)" # never echoed: step 3 re-reads it from the Secret
kubectl patch secret sbx-egress -n "$NS" --type=merge \
-p "{\"data\":{\"SH_CREDENTIAL_KEK\":\"$(printf '%s,%s' "$new" "$old" | base64 | tr -d '\n')\"}}"
kubectl rollout restart deployment/moca-control-plane -n "$NS"

setup.sh generates the KEK once and then keeps whatever is there, including a ring, so a re-run does not undo this and does not reorder it. A ring entry that is not 32 bytes of base64 makes the next setup.sh run refuse, naming the entry rather than telling you to delete it.

# 2. Watch for reads still landing on the retired key. Both signals are the same event:
kubectl logs deployment/moca-control-plane -n <namespace> | grep 'NON-PRIMARY KEK'
# -> names the subject hash and credential, i.e. WHICH credential is still stale
curl -sS -H @"$HDR" "$CP/metrics" | jq .counters.credential_nonprimary_kek_reads
# -> the same signal as a number, without scraping logs: WHETHER any is

/metrics needs the admin role (a non-admin gets 403, not 404), and $CP/$HDR are the same control-plane URL and header file as §5. The counter is process-local and resets on restart, and with more than one replica a scrape hits one replica — so read it from each, or scale to one while you finish the rotation. Neither signal is deduplicated, on purpose: a once-per-process log would let a long-running pod look clean while credentials were still stale.

# 3. Only once nothing has opened under a non-primary key for long enough that every credential has
# plausibly been rewritten -- a quiet counter across a full cycle of use, not a quiet hour -- drop
# the retired entry, leaving the new key alone. Taken from the LIVE ring's first entry rather than
# from $new, so a fresh shell cannot write an empty key, and refused if the ring is not a ring.
ring="$(kubectl get secret sbx-egress -n "$NS" -o jsonpath='{.data.SH_CREDENTIAL_KEK}' | base64 -d)"
primary="${ring%%,*}"
[[ "$ring" == *,* && -n "$primary" ]] || # names no value: the ring is key material
{ echo 'refusing: SH_CREDENTIAL_KEK is not a ring with a non-empty first entry (nothing to drop)'; return 2>/dev/null || exit 1; }
kubectl patch secret sbx-egress -n "$NS" --type=merge \
-p "{\"data\":{\"SH_CREDENTIAL_KEK\":\"$(printf '%s' "$primary" | base64 | tr -d '\n')\"}}"
kubectl rollout restart deployment/moca-control-plane -n "$NS"

To force a credential forward instead of waiting, have its owner write it again (mocactl sbx credentials add, same name and value): a write re-seals under the primary key. That is the only way to move one on demand, and it needs the user, because the control plane cannot read the secret back out.

If the counter will not go quiet, the stale credentials are named in the log lines from step 2; ask those owners to re-add, or leave the retired key in the ring. Keeping two keys is not an emergency — it costs one failed GCM verification per read of an old credential, and nothing else.

Vault instead of the PVC​

SBX_CREDENTIAL_STORE=vault with SBX_VAULT_ADDR points the control plane at a Vault KV v2 mount instead of the PVC (deploy/k8s/overlays/ocp-sbx/egress-credentials-vault, moca#823). The KEK above is unchanged and still lives in sbx-egress — Vault holds ciphertext, so the key that opens it must not be in Vault.

Minimum policy, which needs no list (one object per subject, by hash, read by exact name) and no delete (deleting a credential rewrites the subject's document, a check-and-set update):

path "secret/data/moca/credentials/*" {
capabilities = ["create", "read", "update"]
}

Two things about that path. It must match SBX_VAULT_KV_MOUNT and SBX_VAULT_PATH — the secret/ and moca/credentials above are the defaults, so a deployment that sets either must adjust the policy to match, and the data/ segment is KV v2's own (v1 will not work). And read is not optional even for writes: a write is read-modify-write under check-and-set, and the version it names comes from reading the entry's metadata.version. A policy of create + update alone would let writes fail in a way that looks like a concurrency problem rather than a permissions one.

Two things this deployment does not do for you. Both must be done before the switch works at all — the component is not startable on its own:

  • The token. VAULT_TOKEN_FILE=/run/vault/token is re-read on every request, so a Vault Agent sidecar can log in (AppRole) and renew it without the control plane ever holding a long-lived token. The sidecar is not in the Component: its auth method is a per-deployment decision.

    Until something writes that file the pod does not start. The store reads the sink once, synchronously, at boot, so a missing or empty file fails startup (with the path in the log) rather than coming up and serving 503s — /run/vault is an empty in-memory volume until an Agent fills it. A 503 is what you get later, if a file that was readable at boot stops being readable. Expect CrashLoopBackOff on the first roll after the switch if the sidecar is not in place; check the control plane's logs for the token path before suspecting Vault itself.

  • Reaching Vault. The control plane's NetworkPolicy allows egress to Redis and to port 443 on public addresses only; it excepts 10/8, 172.16/12, 192.168/16, 100.64/10 and 169.254/16. An internal Vault is typically in one of those ranges and on 8200, so it is unreachable until a narrow egress rule for that address and port is added in the deployment's own overlay. Do not widen the policy to all private addresses to make it work. A private CA on that Vault also needs NODE_EXTRA_CA_CERTS.

Switching an existing deployment does not migrate what the PVC store already holds: the file and Vault stores share the document format and the envelope, so a migration is a re-upload of the existing ciphertext rather than a decrypt/re-encrypt, but nothing in-tree performs it yet (moca#823).

7. The janitor​

A systemd timer on each KVM host, every 15 minutes (sbx-janitor.timer). Each run asks the control plane for the live and terminated sandboxes (GET /v1/janitor/sandboxes, with its own token) and, in each of this host's worker roots, removes:

  • the workspace of a sandbox terminated more than the wait ago (SBX_JANITOR_WAIT_MINUTES, default 60, or the token lifetime plus 5 minutes when that is longer);
  • an unknown workspace (no record at all) left unmodified for longer than that worker's SH_WORKSPACE_IDLE;
  • .reclaiming-* trees older than an hour.

It keeps everything else, and touches nothing when the control plane cannot be asked or answers anything unexpected. The wait must exceed one token lifetime plus 30 s, so a deleted sandbox's last token has expired before its files go; setup.sh refuses a shorter one, and status.sh --host warns when a host's installed bundle predates a lifetime raise.

Each run logs one JSON line, summary or skipped, unless it refuses its own settings (a token file with the wrong mode, a bad URL): then it exits 2 with the reason on stderr and no JSON line. status.sh --host shows the service's result, so such a run shows as FAILED:

ssh <host> sudo journalctl -u sbx-janitor -o cat | grep '"janitor":"summary"' | tail -1
{
"janitor": "summary",
"ts": "…",
"examined": 4,
"removed": 1,
"unknown": 0,
"bytes_freed": 4096,
"errors": 0
}

errors counts what it kept because something did not check out (a missing worker env file, an unreadable time); the lines before the summary say which. A skipped line says why (the control plane unreachable, the token refused, a TLS failure: set SBX_API_CA_FILE if the API Route's certificate is not publicly trusted).

Uninstalling the last worker leaves the janitor. Remove it with sudo <checkout>/deploy/sbx/install-host.sh --uninstall-janitor.

8. Status and the audit lines​

deploy/sbx/status.sh is read-only. It reads the audit lines of the relay's current pod only, so right after the relay rolls (any setup.sh change that alters its settings or keys) the "last hour" figures cover less than an hour. From a machine with kubectl access to the namespace:

deploy/sbx/status.sh --env-file <env file>
token lifetime: 900 s
workers attached: sbx_default sbx_py312
workers not attached: none
sandboxes live: 12
sandboxes terminated (retained, approx.): 31
exec last hour: 214 calls, p50 380 ms, p95 2140 ms, errors 1%
exec refused last hour: 2 refused, 5 busy
exec cut short last hour: 1 timeout, 0 expired, 3 cancelled
  • workers attached reads the relay's presence records in Redis. A worker not attached serves no sandbox: its users see no live worker for sandbox <id>.
  • sandboxes terminated (retained, approx.) is the deleted or expired sandboxes still listed (SBX_RETENTION_DAYS, default 14).
  • exec last hour is from the relay's audit lines. A call is an Exec the relay routed; p50 and p95 are over those calls' durations; errors counts the relay's error outcome only (no live worker, a worker disconnect, a worker-side failure). A command's own non-zero exit is not an error. Refusals (a bad token, a wrong sandbox) and busy (the in-flight cap) are counted apart, as are calls cut short by a timeout, a token expiry or a client's cancel.

On a KVM host, as root:

sudo <checkout>/deploy/sbx/status.sh --host --token-ttl-seconds 900
unit sbx-worker@sbx_default.service: active
unit sbx-janitor.timer: active
unit sbx-sampler.timer: active
workspace filesystem: 45% used, 90G of 200G (110G free) on /srv
janitor last run: {"janitor":"summary",…}
janitor wait: 3600 s (outlasts the token lifetime 900 s + 30 s)
host sample: {"sampler":"sample",…}

host sample is one run of the host sampler, the line its timer writes to the journal, now; a host sample error line before it names each field it could not read. To read it: … | sed -n 's/^host sample: //p' | jq ..

Pass --token-ttl-seconds the cluster's token lifetime, or set SBX_STATUS_HOSTS="<host> …" (ssh targets whose user has passwordless sudo) in <env file>, and the cluster-side run reports each host too, with every line prefixed host <host>. The script is piped over ssh, so the host needs no checkout for this.

Reading the audit lines yourself. The relay writes one JSON line per sandbox Exec to stdout (spec §3.5), mixed with its other output, so select them first:

kubectl logs deployment/sandbox-relay -c sandbox-relay -n <namespace> --since=1h |
grep -F '"audit":"sandbox_exec"' | jq -c 'select(.outcome != "end")'

Each line has ts, wrk (the worker), req_id, ms, outcome (end | timeout | expired | busy | error | cancelled | refused) and out_bytes; sub and sbx (the user and the sandbox) when the token verified; exit_code and truncated on end; and a fixed reason on refused. No command text, output or token is ever logged. A refused Abort is "audit":"sandbox_abort". A "audit":"sandbox_release" line means a worker never ended an expired Exec within 60 s of its abort, and the relay gave its slot back anyway: look at that worker's host (a wedged guest, a stuck vsock). For a debugging session only, SH_SANDBOX_AUDIT_COMMANDS=1 on the relay adds each command's first 200 bytes as cmd.

9. Limits operators must know​

  • The in-flight cap. The relay admits at most SBX_MAX_INFLIGHT (default 2) Execs in flight per sandbox: one running, one waiting at the worker. More are refused as busy (users see "sandbox busy"). It keeps one user from filling a template's worker.
  • The deaf-worker slot hold. When an Exec outlives its token, the relay cancels it and then waits for the worker's acknowledgement before freeing its in-flight slot. A wedged worker that never answers holds that slot until its attach stream drops: the sandbox service has no bound on that drain, so with the default cap of 2, one wedged Exec halves the sandbox's concurrency until the worker disconnects. Restarting the worker (systemctl restart sbx-worker@<id>) drops the stream and frees every slot it held, and also ends every running Exec on that worker, for every user of the template.
  • Delete is not immediate. A deleted sandbox (or a removed user) stays reachable for up to one token lifetime plus 30 s. Raising SBX_TOKEN_TTL_MINUTES allows longer commands at the cost of a slower delete, and needs a janitor reinstall (§2).
  • Disk is not admission-controlled on the worker. The quotas are the disk control: setup.sh refuses SBX_MAX_TOTAL × SBX_WORKSPACE_IMAGE_MB over 80 % of SBX_WORKSPACE_FS_GIB.
  • A worker's budget is sized for concurrent use, not for one session. install-host.sh's smallest --budget-mb holds one session at the unit's standby depth of 2, (guest RAM + 32) × 3 + 1024 MiB (13408 MiB for a 4096 MiB guest). A running VM is charged its full guest size and a paused standby only its dirty pages, so that minimum fits about two running VMs: a third command at once on the template is refused at admission before it starts (memory-budget: …, "template busy" to users). Size it for the commands that may run at once on the template, (guest RAM + 32) × running VMs + 1024 MiB: for example 17536 MiB for four 4096 MiB VMs (two agents on one template, each with a command and a standby being promoted). The static clients (remote-worker/cmd/sbx, docs/api/examples/sbx.py) retry such a refusal and exit 75 if every try is refused. The worker's start-up warning (slots × per-VM size against the budget) is expected at any budget below its slot count.

Reproducing a member's issue​

When a member reports a problem, assemble one bundle for the sandbox: what the member's client sent and got back, what the sandbox reached, and what the control plane, the relay and the KVM host logged for that sandbox in that window. Every query below was checked against the line formats in the code it names; none has been run against a live deployment yet. The jq filters run on your machine (status.sh needs jq too); the host's commands run over ssh as in §8.

Which sandbox, whose, and when. Get the sandbox id from the member (mocactl sbx ls, or .sbx/sandbox.json in their repository), then, as an admin:

SBX=sbx-… # the sandbox id
mocactl sbx ls --all --json | jq --arg id "$SBX" '.[] | select(.id == $id)
| {id, owner, template, state, created: (.createdAt | todate),
ended: (if .terminatedAt then (.terminatedAt | todate) else null end)}'
SUB=github:… # its owner
FROM='2026-11-17 10:00:00' # the window, UTC
TO='2026-11-17 11:00:00'
NS=<namespace>

createdAt and terminatedAt are epoch seconds. The KVM host logs the sandbox under its workspace key, which is the sandbox id itself (wsk is set to the id at create, in packages/control-plane/src/sandbox-handlers.ts), and its worker under the template's worker id, wrk in the relay's lines below.

1. The member's own record. Ask the member to rerun what failed with --record issue.ndjson on the command, and to send the file:

mocactl sbx up --record issue.ndjson # or the command that failed: run, push, pull, …
mocactl run "prompt" --record issue.ndjson # a Moca Pi session's calls

The flag goes after sbx <command> or run (mocactl --record F sbx … is refused, exit 2), and every mocactl sbx command that calls the API takes it except token, login, logout and credentials, which refuse it because it would record a secret (NO_RECORD in packages/mocactl/src/sbx/cli.ts). It appends one JSON line per HTTP attempt, the login's refresh included: the command, an invocation id, seq and try (a retry of the same call), the method and path (never the base URL or host), the status or the transport error's name, both bodies, their times (requestAt, answerAt, UTC) and any retryAfter; a streamed answer gets its line when the headers arrive and a stream_end line with its bytes, sha256 and frame count, never its contents (packages/mocactl/src/sbx/call-record.ts). Token-named fields, gate records and credential values, and any string that looks like a JWT or a refresh token, are kept only as their sha256; exec endpoints and CAs, a proxy's dial_addr, a build log and any absolute URL only as their length; under run, the prompt, the reply, the patch and other free text only as their length too. The file is created 0600 and opened for appending; an existing file readable or writable by anyone but its owner is refused (exit 2). The record has no Exec in it: commands go to the relay over gRPC, which is not recorded, so for sbx run or an agent's commands it shows the token mints and the control-plane calls around them, and the relay's lines (step 3) show the Execs.

2. The egress log. For a template with egress, as an admin (the route answers the owner or an admin, packages/control-plane/src/routes.ts):

mocactl sbx egress "$SBX" --ndjson > egress.ndjson
mocactl sbx egress "$SBX" --verify

They are two runs: egress takes --ndjson or --verify, not both. --ndjson prints the log as the control plane stores it, one event per line (host, method, path, status, bytes up and down, decision, credential name). --verify reads the log again and checks its sha256 and line count against the control plane's summary: verified: N events, sha256:… (with INCOMPLETE: N dropped when the log lost events, and sealed or open (not sealed yet)), or NOT verified: …, exit 1 (cmdEgress and verifyEgress in packages/mocactl/src/sbx/commands.ts). Until the log is sealed, it can still gain events: take the copy after that, as Egress records explains. It is sealed at the host janitor's first sweep at least 600 s after the sandbox ends (EGRESS_GRACE_S), not at the 600 s themselves, so --verify can read open (not sealed yet) well past them; with no host janitor running (the host stopped), it stays open until one runs.

3. The control plane's, the relay's and the host's lines. The control plane writes one "event":"audit" JSON line per decision to stdout (packages/control-plane/src/handler-util.ts, auditBestEffort), with subject, decision and, for a sandbox, sandboxId (sandbox_created, sandbox_deleted, sandbox_mode_changed, sandbox_create_rolled_back, among others). Those lines carry no time of their own, so read them with --timestamps (checked against the code):

kubectl logs deployment/moca-control-plane -c control-plane -n "$NS" \
--since-time="${FROM/ /T}Z" --timestamps |
grep -F '"event":"audit"' |
jq -cR --arg sbx "$SBX" --arg sub "$SUB" --arg to "${TO/ /T}Z" '
capture("^(?<t>[^ ]+) (?<l>.*)$") | .t as $t | select($t <= $to)
| .l | fromjson? | objects | select(.sandboxId == $sbx or .subject == $sub) | {t: $t} + .'

The relay writes the "audit":"sandbox_exec" line of §8 per Exec, and sandbox_abort, sandbox_release, sandbox_lease_control and sandbox_workspace_release (a client's Release of its workspace at its end, #338) lines, each with its own ts, and sub and sbx when the token verified (packages/sandbox-relay/src/sandbox-caller.ts, makeAudit). Checked against the code:

kubectl logs deployment/sandbox-relay -c sandbox-relay -n "$NS" --since-time="${FROM/ /T}Z" |
grep -F '"audit":"sandbox_' |
jq -cR --arg sbx "$SBX" --arg sub "$SUB" --arg to "${TO/ /T}Z" '
fromjson? | objects | select(.ts <= $to and (.sbx == $sbx or .sub == $sub))'

A refusal of a token that did not verify names no sub or sbx: look for it by the worker and the time, select(.wrk == "<wrk>" and .outcome == "refused"). Both Deployments run one replica, so deployment/… reads the one pod. Add --previous for the container before its last restart; anything older than that is gone (see below). On OpenShift, oc logs takes the same flags.

The control plane also writes each decision to the Redis stream sh:cp:audit, with a ts in epoch ms (OwnershipIndex.audit in packages/control-plane/src/ownership.ts). Checked against the code (the stream and its field names; the plain output, one value per line, is redis-cli's own):

ms() { jq -nr --arg t "$1" '$t | strptime("%Y-%m-%d %H:%M:%S") | mktime * 1000'; }
kubectl -n "$NS" exec redis-0 -c redis -- \
redis-cli XRANGE sh:cp:audit "$(ms "$FROM")" "$(ms "$TO")" |
awk -v k=" $SBX " '/^[0-9]+-[0-9]+$/ { if (index(r " ", k)) print r; r = $0; next }
{ r = r " " $0 } END { if (index(r " ", k)) print r }'

Each line printed is one entry: its id, then its fields and values. The filter matches a whole value (spaces on both sides), so github:1 does not also match github:12; use -v k=" $SUB " for the member's other decisions (logins and refreshes among them).

On the KVM host, the worker's lines are plain text, and those about one sandbox name its workspace key, as workspace_key="sbx-…" or for "sbx-…" (remote-worker/internal/vmpool). Lines about the pool itself (its cgroups, an orphan sweep) name no sandbox, so read the worker's whole window too. Checked against the code:

ssh <host> "sudo journalctl -u sbx-worker@<wrk>.service --since '$FROM UTC' --until '$TO UTC' \
-o short-iso-precise --no-pager" | grep -F "$SBX"
ssh <host> "sudo journalctl -u sbx-worker@<wrk>.service --since '$FROM UTC' --until '$TO UTC' \
-o short-iso-precise --no-pager" > worker.log

The janitor's JSON lines (§7) name the workspace as wsk, and moca-egress writes one JSON event per connection with the workspace key as sid (remote-worker/internal/egress/audit.go; the host agent selects the events it ships by via and sid too, egress_select in deploy/sbx/host-agent.sh). This is the host's own copy of what step 2 serves, with vm_id and dial_addr besides. Checked against the code:

ssh <host> "sudo journalctl -u sbx-janitor.service --since '$FROM UTC' --until '$TO UTC' -o cat \
--no-pager" | jq -cR --arg k "$SBX" 'fromjson? | objects | select(.wsk == $k)'
ssh <host> "sudo journalctl -u moca-egress.service --since '$FROM UTC' --until '$TO UTC' -o json \
--all --no-pager" | jq -c --arg k "$SBX" '.MESSAGE | strings | (try fromjson catch null)
| objects | select(.via == "moca-egress" and .sid == $k)'

A janitor run's summary line has its own ts; the lines before it in the same run do not.

4. How loaded the host was. The host sampler writes one JSON line every 15 s; take the window's, from a little before FROM, so the first counter difference covers it:

ssh <host> "sudo journalctl -u sbx-sampler.service --since '$FROM UTC' --until '$TO UTC' -o cat \
--no-pager" > samples.ndjson
jq -c 'select(.sampler == "sample") | {ts, load1: .load.avg1, mem_avail_kb: .mem_kb.available,
psi_mem_full10: .psi.memory.full_avg10, psi_io_some10: .psi.io.some_avg10, errors,
workers: [.workers[] | {wrk, vms, committed: .stats.committedBytes, budget_mb, reserve_mb}]}' samples.ndjson
jq -c 'select(.sampler != "sample")' samples.ndjson # the fields it could not read, and when

The worker's wrk is the relay's (step 3). sudo <checkout>/deploy/sbx/status.sh --host (§8) adds a view of now: the units' states, the janitor's last run and one sample.

What is not stored. No command text and no output is stored anywhere server-side. The relay's audit line has neither (docs/notes/sandbox-security-review.md §1.5 lists it as tested), unless the relay runs with SH_SANDBOX_AUDIT_COMMANDS=1, the debug flag of §8, which adds each command's first 200 bytes as cmd. The egress log is the network's shape only, with paths but no bodies. The member's --record file is the only trace of what was sent, and only of the HTTP calls. What is kept, and for how long, today:

  • Control-plane and relay stdout: the pod's current log, and with --previous the one before its last restart; nothing else keeps them.
  • sh:cp:audit: in Redis, trimmed to about 1,000,000 entries on each write (AUDIT_MAXLEN); refusals of a refresh token nobody issued go to sh:cp:audit:anon instead, about 100,000. Redis has maxmemory 200mb (§2) for everything it holds.
  • Egress logs: at most 10,000 events or 4 MiB per sandbox, 64 MiB for all (packages/control-plane/src/egress-log.ts), kept as long as the sandbox's record (SBX_RETENTION_DAYS, default 14).
  • Host journald: the host's own journald settings; install-host.sh sets none. The host sampler adds about 1 KB per sample with a few workers, about 6-8 MB a day at the default 15 s.

Once a log pipeline is chosen (#857, R1), the control-plane, relay and host lines are queried there instead, by the same fields (sandboxId, subject; sbx, sub; the workspace key), and this section moves its queries to that store.

Guest egress​

install-host.sh installs and starts one moca-egress per host (moca-egress.service) before the worker, and wires every template worker to its control socket (MOCA_EGRESS_CONTROL_SOCK=/run/moca-egress/control.sock in the worker env), with a drop-in sbx-worker@<id>.service.d/egress.conf that orders the worker after moca-egress. A --no-egress worker gets neither, so it never starts a moca-egress.service that the host has. A sandbox can then reach only the hosts on the allowlist; everything else is refused with a named error (destination_denied), never a timeout. No credential is involved. For a plain-HTTP request, the refusal's body tells the user or the agent that the host is not on the allowlist and to ask you, the operator. An HTTPS request is refused at its CONNECT, and most clients (curl, git, pip) show only 403 or CONNECT tunnel failed, with the X-Moca-Egress-Error header under curl -v; the agent's sandbox instructions carry the rule there. The allowlist below is where you add a host.

  • The allowlist is MOCA_EGRESS_ALLOW in /etc/sbx/moca-egress.env, a comma-separated list of hosts. It is seeded on the first install from deploy/sbx/egress-allow.env beside the installer (or from MOCA_EGRESS_ALLOW in the installer's environment, on the first write only) and is never overwritten by a re-run. To change it on one host, edit /etc/sbx/moca-egress.env and run sudo systemctl restart moca-egress; to change the default for new hosts, edit deploy/sbx/egress-allow.env in the repo (see its header for the format and the entry groups). Open tunnels end on the restart, new ones work at once. The same restart is needed after rebuilding the binary. A bare MOCA_EGRESS_ALLOW=* (nothing else on the line) is allow-everything mode: every destination is permitted, private and local addresses included -- only the always-refused tier stays (cloud metadata endpoints, link-local, multicast), so a sandbox still cannot fetch the host's instance credentials. It includes the host's own loopback (127.0.0.0/8, ::1) and interface addresses: a sandbox can reach every service listening on the KVM host. Don't use it on a host that runs anything else, or where loopback services trust the local network. For trusted research rigs only; the startup log says allow-all mode plainly, so a journal reader cannot miss it.
  • Another install's moca-egress.service. A host that also runs deploy/microvm's P6 worker already has one. If /etc/systemd/system/moca-egress.service exists and /etc/sbx/manifest has no egress unit line for it, install-host.sh logs reusing the existing moca-egress.service (not installed by this script), wires the workers to its socket and touches nothing else. Its allowlist is then that install's /etc/serverless-harness/moca-egress.env.
  • --no-egress installs a worker without MOCA_EGRESS_CONTROL_SOCK and installs no egress unit: the worker behaves as before egress existed, and the guest has no network.
  • Snapshot: a golden snapshot built before P4.1 has no guest egress listener. Rebuild it with build-snapshot.sh for egress to work (MicroVM tier (P4)).
  • If it fails to start, the install names journalctl -u moca-egress (often a bad /etc/sbx/moca-egress.env). Fix it and re-run install-host.sh.
  • Ownership is by content. The manifest records a digest of the unit it installed. If another install (setup-microvm.sh) later replaces moca-egress.service, a re-run logs changed by another install; reusing it and leaves it alone, and the last uninstall keeps it.
  • Uninstalling the last worker also stops and removes the moca-egress.service this script installed; an allowlist you edited stays (kept ... (edited)).

Egress records​

The template host agent's wanted-state run (sbx-host-agent-wanted, every 15 s) also ships the proxy's audit events to the control plane, where each lands in its sandbox's egress log: mocactl sbx egress [ID], or GET /v1/sandboxes/{id}/egress. Nothing to install: no new unit, timer or setting.

Paths are logged without query, fragment or userinfo, and an injected credential appears by name only, but a secret a guest puts in a URL path segment is logged as sent.

  • On or off: on when systemctl is-active moca-egress.service says active, off otherwise (a --no-egress host, or a moca-egress not started). Active, not enabled: install-host.sh reuses a moca-egress.service another install owns (a P6 worker's) without enabling it, and that install may start it only for a while. So on such a host, whether the agent reads the proxy's journal follows that unit's state: while it runs for its own install, the agent reads its events every 15 s, ships only this host's own sandboxes' (egressSids), and counts the rest egressNotOurs. Nothing of the other install's leaves the host, but check the unit's state before relying on it. Off, the agent reads no journal, posts nothing and writes no cursor; its summary says "egress":"off".
  • Only this host's sandboxes' events leave it. The wanted state names them (egressSids: the workspace keys of the sandboxes on this host's workers, live or ended within 600 s); an event with any other sid, such as a Moca P6 session's on the same proxy, stays in the journal and is only counted (egressNotOurs). The list is read before the journal, so a batch with a sid not in it (a sandbox created in between, at its first egress) reads the wanted state again, after the journal read, before anything is left out; if that read fails, nothing is sent and the cursor stays. The proxy's audit_dropped reports are sent. Against a control plane whose wanted state has no egressSids (older than openapi 1.14.0), the agent sends nothing and says so in an error line, keeping its cursor.
  • A time budget: a run starts no new batch 120 s after its egress step began, so slow posts never hold the wanted unit (TimeoutStartSec 5 min) and its activations. The cursor stays at the last batch taken; the next run goes on.
  • It only reads. The agent reads journalctl -u moca-egress.service and never starts, stops or configures moca-egress, so a failed post never blocks or changes a guest's egress.
  • Where it is: the journal cursor after the last event the control plane took is /srv/sbx/egress.cursor; a batch being sent is /srv/sbx/egress.pending.json (both 0600). Without a cursor, the first run starts at this boot.
  • If a post fails (the control plane down, a 5xx, or a 404 from one older than this): the summary counts an error (egress post failed: the cursor stays, the next run retries (HTTP <code>)), the cursor stays, and the next run re-sends the same batch first. The control plane takes a repeated batch once (idempotent by host and cursor). Up to 10 batches of 1000 entries per run.
  • A batch refused as malformed (400 invalid_request or invalid_json) would fail every retry, so it is moved to /srv/sbx/egress.rejected/<UTC time>-<pid>.json (0600), the cursor moves past it, and the error names the file (egressRejected in the summary). Its events are not in any sandbox's log: look at the file, and fix the agent or the control plane.
  • A cursor the journal no longer has (rotated away, or from another machine) is an error (the journal cursor is no longer valid; restarting from this boot). journalctl itself does not fail on one: it resumes at the nearest kept entry, or for another machine's cursor by time across boots, with exit 0. So before its first read the agent checks that the cursor still names an entry (journalctl --cursor starts at that entry); journalctl fails only on a cursor it cannot parse, which is the same error. Entries rotated away before they were sent are lost with the journal. The old cursor is kept as /srv/sbx/egress.cursor.stale, and shipping restarts from this boot's first entry. Entries already sent are skipped (egressSkipped), by /srv/sbx/egress.mark (the last sent entry's boot id and monotonic time): the control plane's replay check is by cursor, so it could not tell them apart. Without a mark, the restart re-sends this boot's entries, which the control plane stores twice; the agent says so in an error line (no position mark). Entries of an earlier boot that the old cursor had not reached are not read.
  • What is kept: each event with its fields and values as the proxy wrote them (the agent decodes and re-encodes each one, so the bytes may differ, and the control plane stores its own canonical line), for as long as the sandbox record (SBX_RETENTION_DAYS). A log holds at most 10,000 events (EGRESS_LOG_CAP) and 4 MiB of stored lines (EGRESS_LOG_BYTES), and all logs together at most 64 MiB of stored lines (EGRESS_TOTAL_BYTES), inside the 256 MiB Redis has in all; Redis's own overhead per entry comes on top. The total is a running sum kept by the store's append (packages/control-plane/src/egress-log.ts), so an expired log gives its bytes back at a later append, and a post costs the same however many logs are retained.
  • Completeness: an event over 8 KiB isn't sent (only its sandbox id is); the agent reads the journal with --all, so one up to 8 KiB is sent whole (without it journalctl prints a field over 4096 bytes as null). A journal entry whose message is not text (not UTF-8) cannot be read for its sandbox: the agent says so in an error line. A log's dropped counts those (invalid), events past its cap (cap) or past the total budget (budget), and the proxy's own audit_dropped reports, charged to every sandbox on the host's workers live within 60 s before each. complete is dropped == 0. The sandbox's end record (sandbox, state: terminated) names egress: {events, sha256, dropped, droppedBy, complete}; if events arrive in the 600 s after the end, a later sandbox record with state: egress names the final log. The grader decides whether an incomplete log still counts.
  • Check a saved copy offline: wait until the log is sealed: sealed in mocactl sbx egress ID (summary.sealed: true with --json), at the host janitor's first sweep at least 600 s after the sandbox ends. Until then the control plane still takes the sandbox's last events, and if the log changes, the janitor's seal writes a later state: egress record that a copy saved earlier no longer matches. Then save it with mocactl sbx egress ID --ndjson > log.ndjson, take the provenance export, and run verify-provenance.py EXPORT.json --egress ID=log.ndjson. (mocactl sbx egress ID --verify checks the log served against the control plane's summary of it, not against the end record; on an open log it says when events arrived while it read.)
  • Deploy the control plane first: an older one answers 404 on the egress route, and the agent keeps its batch and retries.
  • To turn it off on a host without disabling egress: there is no switch but the unit's state, by design. Roll back the host agent instead (install-host.sh with the previous tree).

Credential injection (--egress)​

Prerequisite for both procedures: set SBX_EGRESS_CREDENTIALS=1 (and optionally SBX_EGRESS_HOSTS) in setup.sh's env file and run setup.sh. That writes deploy/sbx/.generated/egress/<host id>/ (egress.env, egress.token, optional api-ca.crt); pass that <host id> directory to --egress, not its egress/ parent. --egress generates the host's interception CA once (/etc/sbx/egress/ca.crt, ca.key), writes the token (0600) and the drop-in moca-egress.service.d/sbx-credentials.conf, then restarts moca-egress. It refuses a host whose moca-egress.service it did not install (the P6-owned unit is configured by its own installer) and a host with no worker yet (the unit comes with the first worker). --egress-ca-out prints the CA path (generating it if absent) and does nothing else. Once the drop-in exists, a snapshot whose manifest.json egress_ca_sha256 is missing or differs is refused before systemd is touched.

Fresh host (no worker installed):

  1. sudo deploy/sbx/install-host.sh --egress-ca-out (generates the CA).
  2. deploy/microvm/build-snapshot.sh --egress-ca /etc/sbx/egress/ca.crt --out /srv/sbx/snapshots/<image>, or for a template, template-build.sh SPEC --egress-ca /etc/sbx/egress/ca.crt ... (deploy/sbx/template-onboarding/ONBOARDING.md). A self-serve template is built by the host agent, which passes --egress-ca itself once the drop-in exists (step 4; The template host agent). From creating the CA until the drop-in is in, stop sbx-host-agent (or let no build be queued): a build claimed in between lacks the CA, is refused at worker install, and is rebuilt.
  3. sudo deploy/sbx/install-host.sh WORKER_BUNDLE --snapshot <image>.
  4. sudo deploy/sbx/install-host.sh --egress .generated/egress/<host id>.

Existing host (worker already installed):

  1. sudo deploy/sbx/install-host.sh --egress .generated/egress/<host id>.
  2. deploy/microvm/build-snapshot.sh --egress-ca /etc/sbx/egress/ca.crt --out /srv/sbx/snapshots/<new image>.
  3. Re-install each worker with --snapshot <new image>.

Between --egress and the reinstall, users with a stored credential see guest TLS failures on intercepted hosts, because the old snapshot does not trust the new CA. Do the reinstall promptly. The same applies to a CA rotation.

The default allowlist (MOCA_EGRESS_ALLOW) does not include api.githubcopilot.com. Add it to /etc/sbx/moca-egress.env for the github-mcp preset, which otherwise gets 403 destination_denied.

Turning credential injection off. Run install-host.sh --uninstall-egress on every KVM host first, then run setup.sh with SBX_EGRESS_CREDENTIALS=0. With the cluster off, the lookup route is unmounted (404); a host that still has the drop-in treats the 404 as an error and answers 503 credential_unavailable on every allowlisted request, including a plain git clone by users with no credential. The PVC and the sbx-egress Secret stay, so a later on-run finds the stored credentials again. The reverse rule: never run --egress against a cluster that is still off.

A re-run keeps the CA. To rotate it, delete /etc/sbx/egress/ca.crt and ca.key, re-run --egress, then rebuild every snapshot (step 2). --uninstall-egress removes the drop-in, token and api CA and keeps the CA, because existing snapshots were built with it.

Snapshot read-ahead​

A worker's cold start reads its snapshot's memory file (4 GiB for a 4 GiB guest) from /srv/sbx. On a network block volume whose largest request is 128 KiB (max_hw_sectors_kb), the kernel's default read-ahead of 128 KiB keeps too few reads in flight to reach the volume's throughput. One measurement (one KVM host, a network volume rated 750 MiB/s and 3,000 IOPS, page cache dropped): one reader of the memory files got 227 MiB/s at read_ahead_kb 128, 730 MiB/s at 512 and about 850 MiB/s at 1024, and four readers 435 and about 779 MiB/s (128, 512). Eight cold worker starts posted at once, four at a time, were all active after 100.7 s at 128 (about 420 MiB/s) and after 62.2 s at 512 (736-742 MiB/s, the volume's limit), with the same 4.0 GiB read per start. Every read stayed 128 KiB, at about 5,900-6,900 reads/s at 512 and above (about 5,950 in the cold starts): what bound it was reads in flight, not IOPS.

So install-host.sh sets the read-ahead of the device holding /srv/sbx to SBX_READ_AHEAD_KB (default 512 KiB, at most 16384) and installs sbx-read-ahead.service, a oneshot enabled at boot (a sysfs write does not survive a reboot) that the workers start after. It is enabled whatever SBX_WORKER_AT_BOOT says: on a host whose workers a host agent starts, the boot still sets the read-ahead before the agent starts any. It finds the device by path at every run: a partition's read-ahead is its disk's, an LVM or dm device has its own. The value the device had before the first install is kept in /etc/sbx/read-ahead.orig, and the last worker's uninstall writes it back. A /srv/sbx on no block device (tmpfs, overlay), or a value the device does not take, is a warning in the install output and in journalctl -u sbx-read-ahead, never a failed install.

  • Check it: findmnt -no SOURCE --target /srv/sbx names the device; for a disk or a dm device, cat /sys/block/<dev>/queue/read_ahead_kb (for a partition, its disk's).
  • It is device-wide. Everything else on that volume (another Moca tier, other sandboxes, the host's own files if /srv/sbx is on the root disk) reads ahead as much. Read-ahead changes the size of the reads, not how many bytes a cold start reads; on local NVMe a smaller value may do.
  • Turn it off: re-run a worker's install with SBX_READ_AHEAD_KB=0. That writes the recorded value back and removes the unit. A later install without SBX_READ_AHEAD_KB=0 sets it again.

The host sampler​

Every worker install also installs sbx-sampler.timer (#857, R2), so a KVM host with a worker samples its own load from then on. Every 15 s (SBX_SAMPLER_INTERVAL_S, 5-3600, at the install; in practice every 15-16 s, the timer's 1 s accuracy on top) a oneshot, sbx-sampler.service, runs /usr/local/libexec/sbx-sampler.sh (deploy/sbx/sampler.sh), which writes one JSON line to the journal. It reads /proc and the cgroup files; it starts df twice (3 s each at most; after one that does not answer, as on a hung filesystem, it skips the second and reports workspaces null) and, per worker that runs with SH_DIAG_STATS_ADDR, one curl of that worker's /stats on this host (1 s at most; an address that is neither loopback nor a wildcard is refused, never curled); and it never fails its unit. A sample, shortened:

{
"sampler": "sample",
"ts": "2026-11-17T10:30:00Z",
"load": { "avg1": 3.1, "avg5": 2.8, "avg15": 2.2 },
"mem_kb": {
"total": 65536000,
"available": 40000000,
"swap_total": 0,
"swap_free": 0,
"swap_used": 0
},
"psi": {
"memory": { "some_avg10": 0.2, "some_avg60": 0.1, "full_avg10": 0.1, "full_avg60": 0.05 },
"…": "cpu, io"
},
"workspaces": {
"bytes_total": 214748364800,
"bytes_used": 96636764160,
"…": "bytes_avail, inodes_total, inodes_used, inodes_free"
},
"slice": {
"cpu_usage_usec": 9000000,
"cpu_nr_throttled": 0,
"memory_current": 6442450944,
"memory_peak": 8589934592,
"io_rbytes": 1024,
"…": "…"
},
"workers": [
{
"wrk": "sbx_default",
"vms": 3,
"memory_current": 4294967296,
"budget_mb": 17536,
"reserve_mb": 1024,
"stats": {
"committedBytes": 3221225472,
"standbyCommittedBytes": 268435456,
"refusals": { "memory-budget": 2 }
}
}
],
"errors": []
}
  • load, mem_kb, psi are the host's: /proc/loadavg, /proc/meminfo (kB; swap_used is SwapTotal minus SwapFree) and /proc/pressure/{cpu,memory,io} (the share of time some or all tasks waited, over 10 and 60 s, in percent).
  • workspaces is the filesystem holding /srv/sbx/workspaces, in bytes and inodes.
  • slice is sbx.slice, every worker and its VMs: cpu.stat, memory.current, memory.peak (null before Linux 5.19) and io.stat summed over devices. The CPU, throttling and I/O figures are counters since the slice was made: the use in a window is the difference between two samples (below). memory.current counts page cache too.
  • workers, one per /etc/sbx/workers/<id>.env: vms is the VM cgroups in that worker's slice that hold a process, running and paused standby VMs alike (a stopped worker has 0); memory_current is its slice's (null when it is not running); budget_mb and reserve_mb are its SH_MAX_COMMITTED_MB and SH_MEMORY_RESERVE_MB, from the running worker's environment (the process named microvm-worker in its unit's cgroup; else its env file). A VM is admitted while committedBytes stays within budget_mb minus reserve_mb. stats is from the worker's /stats, named as the worker names them, and null unless the worker runs with SH_DIAG_STATS_ADDR, which is off by default. To turn it on for one worker, a drop-in (an install rewrites the env file, never this), with a port of its own per worker: sudo systemctl edit sbx-worker@<id>, then [Service] and Environment=SH_DIAG_STATS_ADDR=127.0.0.1:9901, and sudo systemctl restart sbx-worker@<id>, which ends that worker's running commands. It serves the pool's counters only, on its own listener.
  • errors names each field group it could not read in this sample; their values are null. The first sample in which a kind fails is preceded by one {"sampler":"error","kind":…,"why":…} line, and the first one in which it reads again by one {"sampler":"recovered","kind":…} line, so a lasting failure (a kernel without PSI, a worker's /stats not answering) is one line in the journal, not one every 15 s.

The unit logs at notice and drops anything below it, so systemd's own lines about each run (started, finished) stay out of the journal. On the host:

sudo journalctl -u sbx-sampler.service -o cat --since -5min | jq -c 'select(.sampler == "sample")'
# How loaded was it at 10:30 UTC: the samples around it.
sudo journalctl -u sbx-sampler.service -o cat --since '2026-11-17 10:29:30 UTC' \
--until '2026-11-17 10:30:30 UTC' | jq -c 'select(.sampler == "sample") | {ts, load, mem_kb, psi}'
# sbx.slice's CPU in cores, between consecutive samples.
sudo journalctl -u sbx-sampler.service -o cat --since -10min | jq -sc '[.[] | select(.sampler == "sample")]
| . as $s | range(1; length) as $i | {ts: $s[$i].ts, slice_cores: (($s[$i].slice.cpu_usage_usec
- $s[$i - 1].slice.cpu_usage_usec) / ((($s[$i].ts | fromdate) - ($s[$i - 1].ts | fromdate)) * 1e6))}'

These run jq on the host; on a host without it, run the journalctl over ssh and jq on your machine, as Reproducing a member's issue does. The journal keeps the samples for as long as the host's journald settings keep anything. Once a log pipeline is chosen (#857, R1), it ships the same lines.

  • Its own cost. The target is under 1% of one core. Measure it on the host: 200 runs as the timer runs them, then the CPU time per run over the interval:

    sudo env SBX_ENV_DIR=/etc/sbx SBX_SRV=/srv/sbx SBX_SAMPLER_STATE_DIR= bash -c \
    'time (for i in $(seq 200); do /usr/local/libexec/sbx-sampler.sh >/dev/null; done)'
    sudo systemd-run --wait -p CPUAccounting=yes /usr/local/libexec/sbx-sampler.sh # one run, as a unit

    (user + sys) / 200 / 15 s is the share of one core; systemd-run --wait prints one run's CPU time consumed.

  • Change the interval, or turn it off: re-run a worker's install with SBX_SAMPLER_INTERVAL_S=60 (or 0, which removes it). The last worker's uninstall removes it; --keep-shared keeps it.

10. A KVM-host outage​

What users see. Every sandbox whose template runs on the host fails: mocactl sbx run and the agents' tools report sandbox failure: … no live worker for sandbox <worker id>, and mocactl sbx doctor fails its exec check. Login, ls and create still work, because the control plane runs in the cluster; a new sandbox fails the same way until the host is back.

What to tell them. The service is best effort, business hours. Their laptops keep the original trees, and anything not yet pulled is on the host. If the host comes back with its disk, every sandbox and its files are as they were (a worker restart keeps workspaces). If the disk is lost, every sandbox on it is gone: they run mocactl sbx rm --force (rm cannot check a lost sandbox for unpulled work, so it refuses without --force), then mocactl sbx up, and redo the work not pulled.

What to check. status.sh (workers not attached), then on the host status.sh --host and journalctl -u sbx-worker@<id>. A worker that crash-looped into failed is brought back by re-running its install-host.sh (it resets the failed state).

A VM left behind by a stopped worker. A worker's Firecracker VMs run in pool-N cgroups under its slice, /sys/fs/cgroup/sbx.slice/sbx-<id>.slice, beside the unit's own cgroup, so stopping the unit does not kill them. The unit's ExecStopPost= does: after every exit (a stop, a crash, a stop timeout) /usr/local/libexec/sbx-reap-vms.sh <id> kills each pool-N cgroup of that slice and waits up to 10 s for it to empty, logging sbx-reap-vms: lines to journalctl -u sbx-worker@<id>. A still populated line there, or a host installed before the step existed, can leave a VM holding its guest memory until that worker next starts. With the worker stopped, find one by its cgroup:

grep -H . /sys/fs/cgroup/sbx.slice/sbx-<id>.slice/pool-*/cgroup.procs # any pid here is a leaked VM
for p in $(pgrep -x firecracker); do echo "$p $(cut -d: -f3 "/proc/$p/cgroup")"; done # every VM's cgroup

and reap it with sudo /usr/local/libexec/sbx-reap-vms.sh <id> (or echo 1 | sudo tee /sys/fs/cgroup/sbx.slice/sbx-<id>.slice/pool-<n>/cgroup.kill). Never while the worker runs: its pool-N cgroups then hold its live sandboxes' VMs.

11. The disk filling up​

  1. status.sh --host: the workspace filesystem line, and the janitor's last run.
  2. If the janitor is skipped on every run, or its timer is not active, nothing is being freed: fix that first (§7).
  3. If it runs but frees nothing, the disk is live sandboxes: status.sh's sandboxes live, and mocactl sbx ls --all as an admin to see whose. Ask users to mocactl sbx rm what they no longer need.
  4. Lower SBX_MAX_TOTAL (and SBX_MAX_PER_SUBJECT) and re-run setup.sh, so fewer sandboxes can exist; existing ones are not affected.

12. Uninstalling​

On each KVM host:

sudo <checkout>/deploy/sbx/install-host.sh --uninstall <id> # each worker
sudo <checkout>/deploy/sbx/install-host.sh --uninstall-janitor

They remove exactly what the installs recorded in /etc/sbx/manifest; the last worker's uninstall also removes the shared sbx-worker@.service, sbx-reap-vms.sh and sbx.slice, the host sampler (The host sampler), and the moca-egress.service it installed (Guest egress). Workspace data under /srv/sbx/workspaces/<id> and its jail /srv/sbx/jail/j<12 hex of the id> (the uninstall names it) is left; remove it by hand.

In the cluster, delete what setup.sh applied, then its Secrets and ConfigMaps:

kubectl delete -k deploy/sbx/.generated
kubectl delete secret -n <namespace> moca-redis moca-relay moca-relay-attach \
moca-relay-sandbox-tokens moca-mu1 sbx-token-keys sbx-egress sbx-template-hosts \
moca-relay-tls sbx-exec-tls --ignore-not-found
kubectl delete configmap -n <namespace> moca-settings sbx-settings --ignore-not-found
rm -rf deploy/sbx/.generated

--ignore-not-found covers the Secrets a stack without egress credentials (sbx-egress) or the template store (sbx-template-hosts) never had. With SBX_PROFILE=combined, the generated folder is deploy/sbx/.generated-<namespace>/, and moca-supervisor-tls is one more Secret to delete.

kubectl delete -k deletes the template store's and the egress credentials' volumes. They are resources of the components setup.sh adds when those features are on: moca-sbx-templates (the catalog, the policy and the provenance records) and the stored egress credentials' claim, moca-sbx-credentials (moca-credentials on the combined profile). Back them up first if you may want them.

The Redis volume (the StatefulSet's data claim, data-redis-0) is kept by the delete. Remove it with kubectl delete pvc data-redis-0 -n <namespace> to delete every sandbox record and every login (the refresh families live there too).

13. Live smoke​

deploy/sbx/smoke.sh runs the spec's live checks (§10.6) against a deployed service. Run it after a deploy or an upgrade, and once for the 1c acceptance. Without SBX_LIVE_SMOKE=1 it prints one SKIP line and does nothing. It reads only the environment. It never logs in, never prints a token, and deletes only the sandboxes it created in that run, even when it fails or is interrupted.

Prepare two logins, once. You need two GitHub accounts, both enrolled (§4), because step 2 checks that one user cannot reach the other's sandbox. Each login lives in its own HOME:

mocactl sbx login --control-plane-url https://<api host> # user A, in your own HOME
mkdir -p ~/sbx-smoke-b
HOME=~/sbx-smoke-b mocactl sbx login --control-plane-url https://<api host> # user B

Log user B in from a browser session signed in to B's GitHub account (a private window works). Both logins refresh themselves, so this lasts as long as a login normally does. Leave XDG_CONFIG_HOME and XDG_STATE_HOME unset: the smoke runs mocactl with them unset.

Run it:

SBX_LIVE_SMOKE=1 SBX_API_URL=https://<api host> \
SBX_SMOKE_HOME_A="$HOME" SBX_SMOKE_HOME_B=~/sbx-smoke-b \
SBX_SMOKE_HOST=<host> deploy/sbx/smoke.sh
ok 1 - round trip: sbx up, run, put, pull --apply on a sample repository
ok 2 - two users: B is refused A's sandbox (404, PERMISSION_DENIED); one req_id from both gets each its own result
ok 3 - a command silent for 5 minutes completes
ok 4 - delete: the mint is refused at once; the kept token is cancelled at exp + leeway, then refused
ok 5 - claude fixes the failing test through mocactl sbx mcp; the tree is unchanged until pull --apply
ok 5 - opencode fixes the failing test through mocactl sbx mcp; the tree is unchanged until pull --apply # SKIP opencode not on PATH
ok 6 - the janitor frees a deleted sandbox's directory
ok 7 - egress credentials: the PAT works from the sandbox without being in it; the placeholder is inert; delete revokes # SKIP SBX_SMOKE_GITHUB_TOKEN_FILE and SBX_SMOKE_GITHUB_REPO not both set
ok 8 - lease mode: one guest across commands; an idle restart is noticed; reset and a downgrade end the lease # SKIP lease not allowed
ok 9 - lease+processes: a tracked process serves later commands; a downgrade to lease stops it # SKIP lease+processes not allowed

It exits 0 when every step is ok or skipped, 1 when any step is not ok (the line gives the reason), and 2 on a usage or settings error, before any sandbox is touched. On a failure it keeps its logs and prints their path. A full run takes about 25 minutes without step 6: step 3 runs a command that is silent for 5 minutes, and step 4 waits for a token to expire (one token lifetime plus the 30 s leeway). With step 6 it is closer to 70–90 minutes, because step 6 waits out the host's own janitor wait (setup.sh keeps it at max(60, SBX_TOKEN_TTL_MINUTES + 5) minutes, so 65 min with 60-minute tokens). Step 8 adds about a minute where lease mode is allowed, and step 9 a few seconds where lease+processes is.

VariableDefaultWhat it is
SBX_API_URL(required)https://<api host>
SBX_SMOKE_HOME_A, SBX_SMOKE_HOME_B(required)the two logged-in HOMEs; they must be two different users
SBX_SMOKE_GITHUB_TOKEN_FILE, SBX_SMOKE_GITHUB_REPO(unset: step 7 is a SKIP)step 7: a file holding a GitHub PAT (a fine-grained one that can push to the repository; one trailing newline is ignored; never an environment variable) and OWNER/REPO
SBX_SMOKE_CRED_TTL60the egress credential cache TTL (SH_EGRESS_CREDENTIAL_TTL_SECONDS); step 7 waits this plus 5 s after deleting the credential
SBX_SMOKE_GITHUB_MCP(unset)1: step 7 also checks the github-mcp preset (api.githubcopilot.com must be in the host's egress allowlist)
SBX_SMOKE_CRED_TWO_USERS(unset)1, with SBX_SMOKE_HOME_B: step 7 also checks that user B's sandbox gets a 401 and a public clone works; otherwise that check is a SKIP
SBX_SMOKE_TEMPLATEdefaultthe template every sandbox is created from
SBX_SMOKE_STEPS1,2,3,4,5,6,7,8,9the steps to run; 1,2,5 leaves out the slow ones
SBX_SMOKE_AGENTSclaude,opencodethe agents step 5 runs, headless; one not on PATH is a SKIP. Each needs its own login and model configured
SBX_SMOKE_HOST(unset: step 6 is a SKIP)an ssh target ([user@]host) for the template's KVM host, with passwordless sudo
SBX_SMOKE_HOST_ENV_DIR, SBX_SMOKE_HOST_SRV, SBX_SMOKE_HOST_LIBEXEC/etc/sbx, /srv/sbx, /usr/local/libexecthat host's SBX_ENV_DIR, SBX_SRV and SBX_LIBEXEC_DIR, if install-host.sh was given others
SBX_SMOKE_JANITOR_AGEthe host's own SBX_JANITOR_AGEstep 6's janitor wait. The one run frees every workspace on the host terminated longer ago than this, other users' too, so it can only be lengthened: a value below the host's wait (which setup.sh keeps above the token lifetime + 30 s) is refused. Step 6 waits until its own deletion is that old: the host's wait, 65 min by default with 60-minute tokens
MOCACTL(unset: installed from the edge channel)a mocactl to test instead, such as a local build (packages/mocactl/dist/mocactl.mjs)
SBX_SMOKE_PYTHONpython3Python 3.10+ with grpcio and grpcio-tools (steps 2 and 4 use docs/api/examples/sandbox_v1_client.py). It also runs the sample repository's tests locally

Step 7 (egress credentials, SBX_EGRESS_CREDENTIALS=1) runs alone with only user A's login:

SBX_LIVE_SMOKE=1 SBX_API_URL=https://<api host> SBX_SMOKE_HOME_A="$HOME" SBX_SMOKE_STEPS=7 \
SBX_SMOKE_GITHUB_TOKEN_FILE=~/.sbx-smoke-pat SBX_SMOKE_GITHUB_REPO=<owner>/<repo> deploy/sbx/smoke.sh

SBX_SMOKE_HOME_B stays required for steps 1 to 6. Step 7 prints the PAT nowhere and passes it to mocactl on stdin only; its trap deletes the credentials (smoke-<ts>-<pid>-gh and -gh-mcp, named per run so two runs as the same user cannot revoke one another), the remote branch sbx-smoke-<ts> and the sandboxes, even on failure.

Step 8 (lease mode, lease-mode spec §13) also runs alone with only user A's login. It needs a deployment that allows lease and a template whose worker's snapshot advertises lease-v1. On the default deployment (SBX_MODES_ALLOWED=per-exec) the create is refused with 403 mode_not_allowed, and the step is a SKIP (# SKIP lease not allowed):

SBX_LIVE_SMOKE=1 SBX_API_URL=https://<api host> SBX_SMOKE_HOME_A="$HOME" SBX_SMOKE_STEPS=8 \
deploy/sbx/smoke.sh

SBX_MODES_ALLOWED is deployment-wide: listing lease for the smoke lets every enrolled user create lease sandboxes. Until docs/notes/sandbox-security-review.md §6 is signed off, run step 8 on a test deployment, or list lease for the run only and set SBX_MODES_ALLOWED back afterwards (§2's re-run of setup.sh; the next token mint then moves any lease sandbox left to per-exec). The create is an upgrade, which mocactl takes only from a terminal, so the step runs it under script (util-linux script -qec on Linux, the BSD script -q on macOS), which must be on PATH.

Step 9 (lease mode, level 2: tracked processes, lease-mode spec §7) also runs alone with only user A's login. It needs a deployment that allows lease+processes, and a template whose worker runs the level-2 code on a snapshot that advertises lease-v1 and lease-processes-v1 and has python3 and curl. A worker without it (no sbx:lease-processes-v1 in its Hello) makes proc start fail ("tracked processes are not available on this sandbox's worker yet"), a not ok 9. Where lease+processes is not allowed (the default) the create is refused with 403 mode_not_allowed, and the step is a SKIP (# SKIP lease+processes not allowed):

SBX_LIVE_SMOKE=1 SBX_API_URL=https://<api host> SBX_SMOKE_HOME_A="$HOME" SBX_SMOKE_STEPS=9 \
deploy/sbx/smoke.sh

Until docs/notes/sandbox-security-review.md §7 (level 2, a draft) is signed off, run step 9 as step 8: on a test deployment, or with lease+processes listed for the run only and SBX_MODES_ALLOWED set back afterwards. Its create runs under script too; the rest of the step needs no terminal.

What each step does:

  1. Creates a sample repository with one failing test, runs sbx up, and checks that sbx run fails the suite. It sbx puts the fix and checks that the suite passes in the sandbox, that the local tree is unchanged and its suite still fails, and that the local suite passes after sbx pull --apply.
  2. Checks that user B's sbx token and sbx env for A's sandbox both say no such sandbox, and that B's token with A's sandbox and workspace key is PERMISSION_DENIED. Then A and B each send the same req_id and the same command at once, each to a sandbox of their own from the same template, and each must read its own sandbox's file.
  3. Runs sbx run --timeout 400 -- sh -c 'sleep 300; echo survived', which must print survived.
  4. Mints a token and keeps it, deletes the sandbox, and checks that the next mint is refused (sandbox was deleted) while the kept token still runs true. 20 s before the token's exp it starts a command that would run past exp. That command must be cancelled (DEADLINE_EXCEEDED) at exp plus 30 s. By this machine's clock, anything from 15 to 60 s after exp passes, to allow for clock skew. After that, the token must be refused (UNAUTHENTICATED).
  5. For each agent, in a fresh sample repository after sbx up: sbx doctor's agent-config check must pass. Then claude -p "…" --mcp-config .mcp.json --strict-mcp-config or opencode run "…" runs, with the smoke's mocactl first on PATH (logged in as user A) for the mocactl sbx mcp it starts. The flight recorder must show the agent's calls, and git status --porcelain must be the same as right after sbx up. The local suite must still fail, then pass after sbx pull --apply.
  6. Deletes a sandbox (step 4's, if that ran), checks that its directory <SBX_SRV>/workspaces/<worker>/<id> exists on the host, and runs the installed janitor once: ssh <host> sudo -n bash -s, which sources janitor.env and then runs /usr/local/libexec/sbx-janitor.sh with the host's own SBX_JANITOR_AGE (or a longer SBX_SMOKE_JANITOR_AGE), after waiting until the deletion is that old. The installed timer and its settings are not changed. After the run, the directory must be gone.
  7. Needs the two GitHub variables, else it is a SKIP. User A stores the PAT (credentials add <run>-gh --preset github, on stdin from the file). In A's sandbox it clones the repository, commits, pushes sbx-smoke-<ts> and deletes it, and checks that gh api user (or curl, without gh) names the PAT's owner. The placeholder sent from this machine must get a 401. User A deletes the credential, the step waits SBX_SMOKE_CRED_TTL + 5 s, and the sandbox must get a 401. Optionally github-mcp, and B's sandbox (401, public clone ok).
  8. Creates a sandbox with sbx create --mode lease --idle-timeout 30 --json under script, and checks that its set_mode was applied and that sbx get --json shows the epoch known and no lease yet. It runs sh -c 'echo warm > /tmp/lease-smoke', and then cat /tmp/lease-smoke must print warm: the same guest. After 45 s idle the same cat must fail (exit 1), with [sandbox] guest restarted (idle, <HH:MM>): ... on stderr. sbx reset must print lease ended (reset), and after one more command sbx mode per-exec must print lease ended (mode_change), both on stdout. Then it deletes the sandbox.
  9. Creates a sandbox with sbx create --mode lease+processes --json under script, and checks that its set_mode was applied. sbx proc start --name web of python3 -m http.server 8080 --bind 127.0.0.1 must print p<id> web pid <n>. A later sbx run of curl -s --max-time 5 http://127.0.0.1:8080/ must get the directory listing: the tracked process outlived the command that started it (10 tries, 1 s apart (each try up to 5 s), while the server binds). sbx proc ls must show web running with that pid, and sbx mode lease must print stopped 1 tracked process: web (pid <n>) on stdout. The same curl must then fail. Then it deletes the sandbox.

What it cannot check for you: the agents' own logs (kept on a failure) show whether an agent fixed the test by the shortest route, or tried its local tools first and was denied. The no network in the guest claim is not one of its steps (docs/notes/sandbox-security-review.md §2).

14. Checking the templates​

deploy/sbx/templates-check.sh checks that a client chooses the template its sandbox runs on: the list, a create of each template, each sandbox's token naming its own template's worker, a command in each answering on its template's image, a token refused at another template's worker, and sbx up refusing a bound sandbox of another template. Run it after adding or updating a template. Like the smoke, it reads only the environment, never prints a token, and deletes only the sandboxes it created; without SBX_LIVE_CHECK=1 it does nothing.

SBX_LIVE_CHECK=1 SBX_API_URL=https://<api host> \
SBX_CHECK_HOME_A=<home A> SBX_CHECK_HOME_B=<home B> SBX_CHECK_TEMPLATES=default,<template> \
SBX_CHECK_PROBE_<TEMPLATE>='<a command that prints yes only on that template>' \
deploy/sbx/templates-check.sh

The two HOMEs are two enrolled users' logins, prepared as for the smoke (§13). Each SBX_CHECK_PROBE_<ID> (the id upper-cased, dashes as underscores) runs in every template's sandboxes: it must print yes in its own template's and anything else in the others'. For a SWE-bench template, for example, check the testbed's baseline commit and the testbed's Python. SBX_CHECK_EXEC=0 checks the control plane alone, before any worker is installed. The direct-client check (a token sent to another template's worker) needs SBX_CHECK_PYTHON, a Python 3.10+ with grpcio and grpcio-tools, and is a SKIP without it. --help lists every input and check. It runs on bash 3.2 too.

ok 1 - list: default, <template> (no worker ids)
ok 2 - create: one sandbox per template, each recording its template; an unknown template is refused (…)
ok 3 - routing: each template's sandboxes go to its own worker: default->… <template>->…; two default sandboxes share it, with their own workspaces
ok 4 - hands: each sandbox answers on its template's image: …
ok 5 - isolation: a default sandbox's token sent to <template>'s worker (…) is PERMISSION_DENIED
ok 6 - isolation: user B gets 'no such sandbox' for user A's default sandbox (env and token)
ok 7 - up: a repository bound to a default sandbox refuses sbx up --template <template>

It exits 0 when every check is ok or skipped, 1 when any is not ok (the line gives the reason), and 2 on a usage or settings error, before any sandbox is touched.

The template host agent​

The self-serve template build of the template lifecycle spec §6.1, and the activation of §6.3. deploy/sbx/host-agent.sh --once is one poll of a KVM host's agent: it reports what the host holds, claims queued template builds (clients create them with POST /v1/templates), runs template-build.sh on each with its gate record, and reports the result; then it starts and stops installed workers by the host's wanted state, and removes the retired builds it is told to (worker and snapshot). The control plane registers or rejects each built build by policy; nobody approves it. It needs:

  • on the control plane, the template store and this host's bearer: run setup.sh with SBX_TEMPLATE_STORE=1 SBX_TEMPLATE_HOSTS='<host id> ...'. It adds the store's PVC (moca-sbx-templates, mounted as SH_PROVENANCE_DIR; SBX_TEMPLATES_FILE is imported into it once), a token per host in the Secret sbx-template-hosts (SH_HOST_TOKEN_<ID>, generated once and kept; a host dropped from the list has its token revoked), and a bundle per host in .generated/host-agent/<id>/ (host-agent.env, host-agent.token, and api-ca.crt with SBX_API_CA_FILE): copy it to the host as /etc/sbx/host-agent.env and /etc/sbx/host-agent.token, add SBX_KERNEL (and SBX_API_CA_FILE, if the bundle has a CA) to the env file, and delete the copy;
  • on the host, /etc/sbx/host-agent.env (SBX_API_URL, SBX_HOST_ID, SBX_KERNEL, optionally SBX_API_CA_FILE, SBX_DISK_FLOOR_GB, SBX_BUILD_CONCURRENCY, SBX_ACTIVE_LIMIT, SBX_IDLE_LIMIT_S, SBX_MEM_FLOOR_MIB, SBX_MEM_PRESSURE_LIMIT (see "Memory-aware placement" below), SBX_START_CONCURRENCY, SBX_START_WINDOW_S, SBX_IDLE_GRACE_S (see "Cold-start pacing" below), SBX_REGISTRY), the token alone in /etc/sbx/host-agent.token (0600, root), the script at /usr/local/libexec/sbx-host-agent.sh, and in /etc/systemd/system sbx-host-agent.service with its timer (every 2 minutes) and sbx-host-agent-wanted.service with its timer (every 15 s, the wanted state alone); then systemctl daemon-reload and systemctl enable --now sbx-host-agent.timer sbx-host-agent-wanted.timer. The unit sets SBX_TEMPLATE_BUILD and SBX_INSTALL_HOST to the checkout at /srv/sbx/src (the agent cannot find them from /usr/local/libexec) and REGISTRY_AUTH_FILE=/run/containers/0/auth.json (it hides root's home): log root's podman in with --authfile /run/containers/0/auth.json. With SBX_REGISTRY set, no build is claimed until that file holds a login for it. The unit also gives Go a home and caches under /srv/sbx (build-snapshot.sh compiles the guest agent): run install -d -m 0700 /srv/sbx/go /srv/sbx/home first. Do not set GOFLAGS=-mod=mod: the repository is a Go workspace, which refuses it. Run a poll through the unit (systemctl start sbx-host-agent.service), not by hand: only the unit loads the env file.

It never claims a build below SBX_DISK_FLOOR_GB of free disk (default 60), claims at most SBX_BUILD_CONCURRENCY builds a poll (default 1, at most 4), and checks a gate record's bytes against their sha256 before building with it. On a host with egress credentials on (the --egress drop-in exists; see Credential injection), a spec with egress: true is built with --egress-ca /etc/sbx/egress/ca.crt, so its snapshot trusts the host's interception CA and the worker install's CA check passes; with credentials on and no CA file, the build is reported failed (step: egress). The CA is part of the build's identity (its manifest hash), so turning credentials on, or rotating the CA, needs a rebuild of the egress templates the host already holds. The token stays in a 0600 header file, never in argv or the log. A bad setting is refused (exit 2) before any call. Each run ends with one summary line, {"agent":"summary","claimed","built","failed","started","stopped","removed","errors"}, in the unit's journal, with "psi":"absent", "sliceUnread":[...] and "memfileMismatch":<n> when there is something to say about memory, "startQueued":<n> when the start limit left wanted workers for a later run, and "graced":<n> when the idle grace keeps any.

Nothing that names where an image lives leaves the host. A failed build's reason and its uploaded log are redacted (SBX_REGISTRY exactly, then URLs, registry hosts, IP addresses and 12-digit account ids); the full log stays on the host in /srv/sbx/agent-logs/ (0700). The reason leads with a code a client can branch on: pull_failed, probe_failed, disk_floor, spec_refused, build_failed, or leak_refused (step report): the template record or manifest, which go byte for byte, names SBX_REGISTRY, so the build is reported failed and neither is sent. Name the image in the spec without its registry; SBX_REGISTRY supplies it at the pull. A build the agent claimed but cannot carry on with is reported failed agent_error, with the agent's own fixed wording and its step (gate: a gate record it could not fetch, or whose bytes do not match; report: a report the control plane did not take, or no template record or manifest after the build), so it does not stay building. Text that reached the store before a redaction rule existed is removed offline with template-onboarding/redact-provenance.py (its --catalog-only mode for a build's stored log).

Activation: the agent runs the workers the control plane wants. A client takes a lease on a registered build (POST /v1/templates/{id}/activate, or mocactl sbx templates activate ID), or a sandbox create on an inactive build takes one (activation on first use: 409 template_not_active with Retry-After, retried by mocactl). A build is wanted running only while a lease stands. The templates declared in SBX_TEMPLATES_FILE (SH_SANDBOX_TEMPLATES) are not affected: their workers are the deployment's to run, they take no lease, and a create on one is never gated. Each host's wanted state (GET /v1/hosts/<id>/wanted) lists the leased builds placed on it, at most SBX_ACTIVE_LIMIT of them, in run, and the builds it holds with no lease and no live sandbox in stop. After each build poll, and every 15 s in sbx-host-agent-wanted.service (host-agent.sh --wanted-only: no claim, no build), the agent stops each stop worker whose unit is active, then starts each run worker whose sbx-worker@<id> unit is installed and not active, and reports each start, refusal and stop (attached, refused, detached) as provenance; readiness is the relay's attached record, never these reports. A worker with a live sandbox is never stopped: it drains first. A lock (/srv/sbx/.activation.lock) keeps the two units from starting or stopping workers at once; the one without it skips, saying so. A host whose last inventory is over 6 minutes old (three build polls) is left out of placement and of GET /v1/templates' hosts until it reports again, so a stopped or removed agent takes no new placement and no slot. Size the limits:

  • SBX_ACTIVE_LIMIT is the most workers the host runs at once: each pins its snapshot's guest memory whole from its start, with its standby VMs. With an agent that reports memory (below), the control plane also keeps new starts within the host's memory, so the limit is an upper bound, not a memory calculation. 0 (unset) places nothing on the host. Every worker that stays running counts against it: the leased builds placed or running on the host (the inventory says which units run; a running build stays on its host), and an unleased one still draining a live sandbox, until it stops.
  • SBX_IDLE_LIMIT_S is the default lease length a client gets when it names none (the largest of the hosts' that hold the build, clamped to 60-86400 s; else 3600 s). A lease is renewed by activating again.
  • The policy's quota.active caps the builds one subject holds a lease on at once (429 quota_exceeded, quota: active); GET /v1/quota shows the use.

A refused activation keeps no lease: 409 template_unavailable (no registered build, no host that reports holds it, or reason: memfile_mismatch) or 409 no_active_slot with Retry-After (every host holding it is at its limit with leased builds, or reason: memory; nothing leased is evicted). The activation check and lease write run one at a time, under a lock in the control plane's process: keep one control-plane process per template store (the Deployment's one replica, the store's ReadWriteOnce volume).

Memory-aware placement (template lifecycle spec §6.2): each inventory also reports the host's memory, and the control plane starts a new worker on a host only if the host has the memory for it, whatever SBX_ACTIVE_LIMIT says. SBX_ACTIVE_LIMIT stays the upper bound. Nothing running is stopped: the rule only holds back new starts.

  • What the agent reports (the inventory's memory object): memTotalMib and memAvailableMib from /proc/meminfo; memPressureFull10, the memory PSI line's full avg10 from /proc/pressure/memory (a percentage: the share of the last 10 s in which every non-idle task was stalled on memory); the workers of held builds whose sbx-worker@<id> runs; per running worker, vmCommittedMib, its slice's memory.stat (sbx.slice/sbx-<id>.slice) anon + shmem + kernel: its VMs' memory the kernel cannot reclaim, never its page cache (a memory file is file-backed and mlocked, so the worker's cost counts it; keep /srv/sbx on a disk volume, never tmpfs: a memory file on tmpfs is shmem, and would be counted twice); and memFloorMib, the effective floor, so the number placement uses is in the inventory and in GET /v1/templates' hosts block (memory.floorMib), with no arithmetic.
  • SBX_MEM_FLOOR_MIB: the memory never placed into, a whole number of MiB. Unset, it is 10% of MemTotal, at least 4096 (a host of 256 GiB: 26214 MiB). Set a fixed amount on a host shared with other work (Moca's own P6 sessions).
  • SBX_MEM_PRESSURE_LIMIT (default 10): while full avg10 is above it, the host takes no new start; what it runs is kept. No /proc/pressure (an older kernel, or psi=0 at boot): the field is left out, which is no brake, never "no starts"; the agent's summary says "psi":"absent" once per run.
  • Both live in host-agent.env (0600), never in argv. A bad value (a floor of 0, a leading zero, a limit that is not a number in (0, 100] with at most two decimals) is refused like the agent's other settings: exit 2, before any call.
  • A worker's cost is its guest's RAM (the registered build's guest size, which registration checked against the manifest's guest_ram_mb) plus 128 MiB; a build with no recorded guest size counts at 4096 + 128. A host's headroom is min(total − floor − used, available − floor − pending), used being its running workers' cost, their VMs' committed memory and the starts placed there but not running yet. The hosts block shows usedMib and headroomMib.
  • A stale host (its last inventory over 60 s old: four missed 15 s reports) takes no new start (brake: stale); one under pressure shows brake: pressure.
  • The memory-file check: for each build it holds, the agent checks that the snapshot's memfile is exactly guest_ram_mb × 1 MiB (the manifest beside it). A worker pins that file whole, so a different size (a future Diff snapshot, a balloon, a trimmed file) would make its cost wrong. A mismatch is an error line naming the build, a "memfileMismatch":<n> count in the summary, and the build in the inventory's memfileMismatch: the control plane never places it on that host (409 template_unavailable, reason: memfile_mismatch, when every holder reports it), and the agent refuses to start it (reported refused). Rebuild the snapshot.
  • A slice it cannot read (memory.stat missing, or with no anon line): that worker's VMs are left out of vmCommittedMib, never guessed, and the summary names it ("sliceUnread":[...]).
  • What a client sees: an activation refused for memory is 409 no_active_slot with reason: memory and the usual Retry-After; at the active limit it answers as before, with no reason. A build waiting for memory reads readiness: unavailable with readinessReason: memory (mocactl sbx templates show ID), and mocactl sbx hosts shows each host's headroom, floor and brake.
  • With the sandbox lease mode, a lease VM's guest memory is in its worker's slice, so vmCommittedMib counts it. Its reclaim acts on VMs and this floor on whole workers: on one host, set the floor at or above lease mode's reclaim target.
  • Version skew, both ways: an agent older than memory-aware placement sends no memory, and the control plane places that host by SBX_ACTIVE_LIMIT alone, as before. A control plane older than it (openapi 1.12.0) ignores the field and places by SBX_ACTIVE_LIMIT alone; the agent still refuses to start a build whose memory file does not match. Upgrade either first; list both in the rollout.

Cold-start pacing (template lifecycle spec §6.5): before it attaches to the relay, a worker reads and locks its whole snapshot memory file, and workers that start together share the host's disk. So the agent starts at most SBX_START_CONCURRENCY workers at once (default 4).

  • What it caps: workers starting (started, not attached yet), not running ones. It is not SBX_ACTIVE_LIMIT, which caps running workers. Each run starts at most SBX_START_CONCURRENCY less those still starting, oldest lease first, and leaves the rest for a later run (every 15 s); the summary counts them ("startQueued":<n>). A run never waits for a start to finish: systemctl start of the worker unit returns at once.
  • The setting: a whole number, at least 1, in host-agent.env (0600), never in argv. 0 is refused (exit 2, before any call), as is a leading zero or anything that is not a number.
  • Where "starting" comes from: the control plane's wanted answer lists, in starting, the builds placed on this host whose worker the relay does not show attached. The limit is the agent's, so it holds with an older control plane too (openapi before 1.13.0, no starting): the agent then counts as starting the workers it started less than SBX_START_WINDOW_S ago (default 300) that are still running and wanted. Its starts are kept in /srv/sbx/start-times (0600).
  • A stuck start: either way, a start counts for SBX_START_WINDOW_S at most. A worker still not attached after it (a crash loop, a hang, a relay it cannot reach) is stuck: each run says so in an error line (worker <id> started <n> s ago and is not attached) and it no longer holds a start slot, so the line moves. Look at that worker's unit (journalctl -u sbx-worker@<id>). Keep SBX_START_WINDOW_S above the host's slowest start in a full wave, or a slow start is taken for a stuck one and one more start shares the disk.
  • What it bounds: each start's wait, not the total. The disk sets the total, at any width. Measured on one KVM host whose snapshots sit on a general-purpose cloud block volume (about 125 MB/s), 4 GiB guests, starts measured after a release (cache not warm): cold starts one at a time took 78 s on average (at most 120 s), a wave of four about 3-4 minutes, and with 28 started at once the last attached after more than twenty minutes. Raise the limit only on a faster disk.
  • What the agent reports: startConcurrency and startSeconds in the inventory's limits. startSeconds is the mean of its last 10 cold starts, start to attach, to within the 15 s poll (kept in /srv/sbx/start-seconds, 0600); it is absent until a start is done, and always with an older control plane. The hosts block of GET /v1/templates shows start: {concurrency, seconds, starting, queued} (mocactl sbx hosts).
  • What a client sees: a build waiting for a start slot is activating with queued: true and its queuePosition (1-based, on its host). 409 template_not_active's Retry-After becomes ceil(ceil(position / concurrency) × startSeconds) once the host has measured a start, else 30 s as before. A queued activation stays activating longer than before, so a client polls readiness with a timeout sized for the host's whole start line; mocactl sbx templates activate --wait sizes its default from it ((ceil(position / concurrency) + 1) × startSeconds × 1.25, at least 1800 s, at most 4 h; --timeout wins).
  • Measure cold starts with the page cache dropped (echo 3 > /proc/sys/vm/drop_caches, or after a reboot): a start right after its build, or after a recent stop, reads its memory file from the cache and looks much faster than a cold one.
  • Placement keeps a build on the host already running its worker (no start at all; sticky), whether or not that host reports its memory; a graced build leased again stays there too.
  • The one change on upgrade: a host whose agent is upgraded, with no new settings and whatever its control plane, now starts at most 4 workers at once instead of every wanted one; a wide activation starts in waves of SBX_START_CONCURRENCY, each about one start time long (a slot frees when a worker attaches, or, with a control plane older than 1.13.0, after SBX_START_WINDOW_S), so 28 builds at the default take about seven waves, many minutes, and a client's first Retry-After is the old 30 s until the host has measured a start. A control plane older than 1.13.0 ignores the new inventory fields; an older agent sends none, and the control plane then shows no queue and answers Retry-After as before.

The idle grace (part of cold-start pacing, openapi 1.13.0): a released worker can stay up for a while, so a template used again soon is warm (no memory-file read at all).

  • The setting: SBX_IDLE_GRACE_S in host-agent.env (0600), never in argv. Default 0: off, and the host behaves exactly as before: a build with no lease and no live sandbox is stopped at the next poll. A whole number of seconds, 0 to 86400, no leading zero; anything else is refused (exit 2, before any call). The inventory reports it (limits.idleGraceSeconds). It is a host setting, not a client's choice.
  • When to use it: long enough to cover the gaps between a workload's rounds on the same templates. Leave it off where each task's cold start is what is measured, or use stopNow (below).
  • What it does: after a build's last lease ends (its release, or the lease running out), the control plane keeps it in the wanted answer's run until that end plus SBX_IDLE_GRACE_S, and lists it in graced. It is graced only if its worker runs on the host (as the agent reports it), has no live sandbox, and the host has room for it: a slot under SBX_ACTIVE_LIMIT beside the workers the host counts (leased and draining ones) and memory-aware placement's headroom. The agent never starts a graced build; it only keeps one running. A use within the grace finds the worker attached and gets a sandbox at once (first use takes a lease; no 409).
  • A wanted build that needs the slot (or the memory) displaces the graced worker that ends soonest: it goes to stop. That is the only stop of a worker nobody asked to stop, and only of one with no lease. At stop time the control plane re-checks that it is still unleased (under the activation lock): a lease taken in between keeps it running. The agent logs such a stop as {"agent":"stopped","worker":"<id>","was":"graced"}.
  • stopNow, to force cold: POST /v1/templates/{id}/deactivate {"stopNow": true} (mocactl sbx templates deactivate ID --stop-now) releases the caller's lease and, if no lease remains, skips the grace: the worker is stopped at the next poll. On a build already in its grace it ends the grace (released: 0). The deactivate record carries stopNow: true. Use it for one-by-one comparisons, so each task still measures its cold start.
  • Counting releases in the records: a host's worker reports share record kinds with clients' calls: attached and refused are written as activate, detached as deactivate, each with subject host:<id> and the report in body.event. So a reader counting a client's releases or activations filters on the subject (or on body.event being absent).
  • What you see: the agent's summary line gains "graced":<n> when it keeps any; the hosts block of GET /v1/templates shows idleGraceSeconds and graced (a count) per host with a grace (mocactl sbx hosts); a graced build reads deactivating with graced: true and gracedUntil (epoch seconds; mocactl sbx templates show ID says "graced until ...").
  • What it costs: a graced worker still pins its whole memory file (it is a running worker). Memory-aware placement counts it in memory.usedMib, and it holds a slot of SBX_ACTIVE_LIMIT until a wanted build needs one. A host over its memory with its graced workers lets the soonest-ending go first.
  • Version skew: an agent without the setting reports no idleGraceSeconds, which the control plane takes as 0 (off). A control plane older than 1.13.0 ignores the field, sends no graced, and stops released workers as before; the agent then just applies stop. Roll out the control plane before setting SBX_IDLE_GRACE_S on a host: the order is the control plane, then the agent, then the setting.

Retired builds are removed by the agent. Once a client retires a template or a build (POST /v1/templates/{id}/retire, DELETE /v1/templates/{id}/builds/{build}, or mocactl sbx templates retire), each host holding it gets it in its wanted state's stop and remove as soon as no live sandbox runs on its worker and no pinned session's hold keeps it; a worker whose sandbox ended less than 600 s ago is in stop only until then, so the host still ships that sandbox's last egress events (retire --wait's default --timeout, 900 s, covers that window). In the same run the agent stops the worker, runs install-host.sh --uninstall <worker> --keep-shared (SBX_INSTALL_HOST), deletes the snapshot directory and its template record under /srv/sbx, and reports removed: the template's provenance gains a retire record by host:<id>, and the build's heldBy (GET /v1/templates/{id}/builds/{build}) no longer lists the host. It deletes only what its own template record names for that build id and worker, never a running worker's; a name no record matches is an error line and deletes nothing. The worker's workspace data stays for the janitor, as with any uninstall. --keep-shared removes the worker's own files only: even the host's last worker leaves the shared units (sbx-worker@.service, sbx.slice), moca-egress and its credential injection in place for the next worker, so a client's retire never undoes an operator's --egress. To remove them from a host with no worker left, run install-host.sh --uninstall <that worker> again without it (§12). Also re-run setup.sh without the worker's id in SBX_WORKERS (§5) to revoke its relay token. mocactl sbx templates retire ID --wait returns once the worker has stopped and every holding host has removed it; a host that is down removes it at its next poll.

A registered build's worker is still installed by an operator. The agent builds, reports, and starts and stops installed workers; it installs none, and a wanted build whose worker is not installed is reported refused (worker not installed). A worker's relay token is minted only by setup.sh, so for a new template: add a worker id to SBX_WORKERS and the template to SBX_TEMPLATES_FILE, re-run setup.sh, and install the worker on the host that built the snapshot, as in §5, with --snapshot <snapshot> (the record's snapshot, $SBX_SRV/templates/<snapshot>.json). install-host.sh then names the record in the worker's env (SH_TEMPLATE_RECORD) and its id as SANDBOX_TEMPLATE, and the worker checks the record against its snapshot and its own id at every start, refusing a mismatch. The worker runs the sandbox service's own binary, SBX_WORKER_BIN (default /usr/local/libexec/sbx/microvm-worker: CGO_ENABLED=0 go build -o /usr/local/libexec/sbx/microvm-worker ./remote-worker/cmd/microvm-worker), never /usr/local/bin/microvm-worker, which a P4 worker on the same host may run; a missing one is refused. Letting the agent obtain a worker token from the control plane is a follow-up.

Known limits: a claimed build is not re-queued if the agent dies mid-build (cancel it, POST /v1/templates/{id}/builds/{build}/cancel, and create it again); one poll builds up to SBX_BUILD_CONCURRENCY builds, and builds are not scheduled across hosts.

To remove the agent: systemctl disable --now sbx-host-agent.timer sbx-host-agent-wanted.timer, delete the four units, /usr/local/libexec/sbx-host-agent.sh, /etc/sbx/host-agent.env and /etc/sbx/host-agent.token, and systemctl daemon-reload; drop the host from SBX_TEMPLATE_HOSTS and re-run setup.sh to revoke its token.

Provenance records: export, verify and redact​

Every template change is a provenance record in the template store (SH_PROVENANCE_DIR; on deploy/sbx, the PVC moca-sbx-templates, mounted at /var/lib/moca/templates in the control-plane container of the moca-control-plane Deployment). Nothing is signed: each record is canonical JSON named by its sha256, and names the digest of the record before it, so an edit to a stored record breaks the chain. Keep a copy of the heads elsewhere: that is what proves an export complete.

The store is append-only:

  • records/<hex>.json: one file per record, its canonical bytes; the file's sha256 is its name.
  • heads/<hex>.json: per stream, {stream, head, count}; <hex> is the sha256 of the stream's name. A stream is policy, members or template/<id>.
  • blobs/<hex>: the documents records cite by sha256, byte for byte (gate records, template records, snapshot manifests).

A record is written before its stream's head moves, so a crash leaves an orphan record, never a head naming a missing one. A record (moca-provenance-record/1) is {schema, stream, seq, kind, prev, at, subject, body}: prev is the previous record's digest (null at seq 1), at is UTC in whole seconds, and subject is who wrote it: a subject, policy (an automatic decision), host:<id> or janitor.

KindWritten when
policyan admin's PUT /v1/policy or PUT /v1/quota/…; the default policy, at its first use
create, gatean admitted create or new build (gate when it sends a gate record)
checkthe policy check of a create, and of a build at registration (a refused create on a new or another subject's id writes none)
builda host's built or failed report; a cancel
activate, deactivatea lease taken or released; a host's attached or refused report (activate) and detached report (deactivate)
tag, visibility, ownera tag moved or removed, a visibility or description change, a transfer
retire, rejecta retire or a reject; a host's removal of a retired build (retire by host:<id>); a rebuild superseding a build (retire by policy)
sandboxa sandbox's create and end (deleted, or expired by the janitor); its egress events arriving after the end
sessiona pinned Moca Pi session's create and delete
member, roleenrollment and admin-role changes, on the members stream

The kind use is defined, but nothing writes it yet.

Export. There is no export route. Export by hand, from a copy of the store:

  1. Copy heads/, then records/ and blobs/, off the volume into an empty directory, say store-copy/ (for example with kubectl cp from the control-plane pod) (unverified on a cluster). Heads first: every head then names a record the copy holds.

  2. If a gate record a gate record cites is missing from blobs/, add it there as blobs/<sha256 hex>: the export leaves out a blob the store does not hold, and the verifier then fails.

  3. From a checkout at the control plane's version, with its dependencies installed, export every stream the copy has a head for, using the control plane's own ProvenanceStore.export:

    cat > /tmp/export-provenance.mts <<'TS'
    import { readdirSync, readFileSync } from 'node:fs';
    import { join } from 'node:path';
    const dir = process.argv[2];
    const { ProvenanceStore } = await import(join(process.cwd(), 'src/provenance.ts'));
    const streams = readdirSync(join(dir, 'heads')).map(
    (f) => JSON.parse(readFileSync(join(dir, 'heads', f), 'utf8')).stream,
    );
    process.stdout.write(JSON.stringify(await new ProvenanceStore({ dir }).export(streams)));
    TS
    cd packages/control-plane
    npx tsx /tmp/export-provenance.mts /abs/path/to/store-copy > EXPORT.json

    The result is one moca-provenance-export/1 document: the streams, the policy stream they cite, and every gate record their gate records cite.

Verify anywhere with Python 3, no moca, cluster or network needed:

python3 deploy/sbx/template-onboarding/verify-provenance.py EXPORT.json \
--head 'template/<id>=sha256:<the head you kept>'

It checks every record's digest, every stream's seq and prev chain up to its head and count, every blob against its sha256, and that each check cites a policy record and each gate a blob of the export. An export can only prove itself consistent; --head STREAM=sha256:… (repeatable) compares a stream's head with one kept elsewhere, which catches an export that dropped its latest records. --egress <sandbox id>=<file> checks a saved egress log against the sandbox's record (Egress records). Exit 0 verified, 1 a check failed (each printed), 2 a usage error or an unreadable file.

Redact only data that must not be kept: it rewrites the chain, and the redaction itself is logged. Stop the control plane first (one process owns the store), and run the script against the store's directory (STORE_DIR, the volume's root) (unverified on a cluster). Without --apply it prints what it would change, counts only, and changes nothing; patterns are never printed.

# One template's stream (template/<id>) and its catalog entry:
python3 deploy/sbx/template-onboarding/redact-provenance.py STORE_DIR TEMPLATE_ID \
--pattern 'REGEX' [--pattern 'REGEX' ...] [--apply]

# The catalog only: every catalog entry (catalog/*.json), no provenance record touched:
python3 deploy/sbx/template-onboarding/redact-provenance.py STORE_DIR --catalog-only \
--pattern 'REGEX' [--pattern 'REGEX' ...] [--apply]
  • The stream form replaces each match in a record's string values (never its keys) with <redacted>, rewrites that record and every later record of the stream (each names the one before), moves the stream's head, and redacts the template's catalog entry the same way.
  • --catalog-only takes no template id. It redacts every catalog entry, and no record: for text that reached only the catalog, such as a build's stored log, which no record copies.
  • With --apply, both append a line to STORE_DIR/redactions.log, never the matched text: the stream form names the stream, its old and new head and how many records it rewrote; --catalog-only how many entries it redacted.
  • An export taken before a redaction no longer verifies against the new head: by design. Keep the new head in place of the old one.