Sandbox service runbook
Preview. The sandbox service is implemented, but not yet declared Stable; interfaces and limits may change.
For the operators of a sandbox-service deployment: the sandbox control plane, the relay and Redis in one
namespace of a shared OpenShift cluster, and template workers plus the janitor on a KVM host. The
install reference (every input and its rule) is Install the sandbox service; users
have the user guide. The design is the sandbox service's design spec
in the repository's docs/specs/ (its §8 for operations).
Throughout, <env file> is the operator's settings file outside the repository (README §1),
<host> a KVM host, <checkout> this repository on that host, and <id> a worker id. Keep
deploy/sbx/.generated/ between runs: setup.sh reads the bundles it last wrote there (it is
gitignored).
1. Deploy from scratch​
-
The namespace. An existing namespace, dedicated to the sandbox service, where you have edit rights.
-
A GitHub OAuth app of this deployment's own, with device flow enabled: its client id is
SBX_GITHUB_CLIENT_ID. -
The snapshots. On the KVM host, one golden snapshot per template, built with
deploy/microvm/build-snapshot.sh --out /srv/sbx/snapshots/<image>(MicroVM tier (P4) has the full command). Every curated template carriesbash,git,tarandripgrep. Build them with abuild-snapshot.shfrom P4.1 or later: an older snapshot has no guest egress listener (see Guest egress). The host also needs/usr/local/bin/moca-egress(CGO_ENABLED=0 go build -o /usr/local/bin/moca-egress ./remote-worker/cmd/moca-egress); the worker install refuses without it, unless you pass--no-egress. And the sandbox service's own worker binary,CGO_ENABLED=0 go build -o /usr/local/libexec/sbx/microvm-worker ./remote-worker/cmd/microvm-worker(SBX_WORKER_BIN): the install refuses without it, and never runs/usr/local/bin/microvm-worker, which a P4 worker on the same host may run. -
The cluster side. Write
<env file>(README §1 lists every input; the required ones areSBX_NAMESPACE,SBX_API_HOST,SBX_ATTACH_HOST,SBX_EXEC_HOST,SBX_IMAGE,SBX_WORKERS,SBX_TEMPLATES_FILE,SBX_GITHUB_CLIENT_ID,SBX_ALLOWED_SUBJECTSandSBX_WORKSPACE_FS_GIB), then:deploy/sbx/setup.sh --env-file <env file>It generates every token, key and certificate once and keeps them, applies the overlay, waits for the rollouts, and writes one worker bundle per
SBX_WORKERSid plus the janitor bundle underdeploy/sbx/.generated/. -
Each worker, on its KVM host:
scp -r deploy/sbx/.generated/workers/<id> <host>:sbx-<id>ssh <host> sudo <checkout>/deploy/sbx/install-host.sh sbx-<id> --snapshot <image> [--budget-mb N]ssh <host> rm -r sbx-<id>It waits up to 120 s for the worker's attach to hold (
SBX_ATTACH_TIMEOUT). The worker's guest RAM (SH_GUEST_RAM_MB) is taken from the snapshot'smanifest.json(guest_ram_mb), never set by hand: a snapshot without it is refused.--budget-mb(default 8192) must hold at least one session, (guest RAM + 32) × 3 + 1024 MiB at the unit's standby depth of 2: 13408 MiB for a 4096 MiB guest. A smaller budget is refused, naming the minimum. The worker runsSBX_WORKER_BIN(default/usr/local/libexec/sbx/microvm-worker, README). When<image>was built bydeploy/sbx/template-onboarding/template-build.sh, its template record (/srv/sbx/templates/<image>.json) is wired into the worker env (SH_TEMPLATE_RECORD,SANDBOX_TEMPLATE), so the worker checks the record against its snapshot at every start; a record for another worker is refused before anything is written. On a host run by a host agent, run it assudo SBX_WORKER_AT_BOOT=0 <checkout>/deploy/sbx/install-host.sh ...: the worker is started once but not enabled for boot, so a reboot leaves starting workers to the agent's wanted state instead of starting all of them. -
The janitor, once per KVM host:
scp -r deploy/sbx/.generated/janitor <host>:sbx-janitorssh <host> sudo <checkout>/deploy/sbx/install-host.sh --janitor sbx-janitorssh <host> rm -r sbx-janitor -
Check it with status: every worker attached, the janitor timer active, then log in as a user and run
mocactl sbx doctorin a repository.
Which mocactl users install. Since v0.7.0 the default channel, the latest release, carries
mocactl sbx, as the user guide says:
curl -fsSL https://raw.githubusercontent.com/rossoctl/moca/main/scripts/install-mocactl.sh | sh
mocactl sbx --help # what an sbx sandbox is, then: usage (the sandbox service):
A mocactl older than v0.7.0 has no sbx: its mocactl sbx --help prints the general
usage: list, and a bare mocactl sbx says unknown command "sbx". Such a user re-runs the
install line above (or with MOCACTL_VERSION=edge for the newest build of main).
The increments' inputs, for an older deployment: 1a had SBX_CLIENTS (static tokens; now refused)
and no sandbox profile; 1b added the control-plane inputs (SBX_API_HOST, SBX_GITHUB_CLIENT_ID,
SBX_ALLOWED_SUBJECTS, SBX_TEMPLATES_FILE, the lifetimes and quotas); 1c adds the janitor
(SBX_JANITOR_WAIT_MINUTES, SBX_API_CA_FILE) and nothing else that is required.
2. Upgrading​
An image upgrade (a new SBX_IMAGE): set it in <env file> and re-run setup.sh. A changed
image rolls the control plane and the relay, and so does a change to what the relay reads (the
token issuer, SBX_MAX_INFLIGHT, the signing keyset, the attach or exec certificate). Workers keep
running, and reattach when the relay comes back; Execs running through the old relay pod end with it
(the worker kills a dropped stream's Execs), so apply those when nothing long is running. Any other
setting (templates, subjects, quotas, lifetimes) rolls the control plane alone, and running Execs
carry on.
Moving to idle lifetimes (#770). The first run of this setup.sh on an existing deployment
changes every worker's idle from 216 h to 768 h at the defaults, and starts idle lifetimes: the control plane counts
idle time only when SH_SANDBOX_MAX_LIFETIME_SECONDS is set, and keeps the old absolute lifetime
otherwise. Until the hosts can be reinstalled, take this setup.sh with idle lifetimes off: it
writes no SH_SANDBOX_MAX_LIFETIME_SECONDS and keeps every worker's idle at 216 h (at the defaults),
so no host changes and routine runs (enrolling a user, a quota change) need no ack:
SBX_MAX_LIFETIME_DAYS=off deploy/sbx/setup.sh --env-file <env file>
Idle lifetimes stay off; set a number on the day you reinstall the hosts. On that day, in this order:
SBX_ACK_HOSTS=1 deploy/sbx/setup.sh --env-file <env file>
then re-run install-host.sh with the new bundles on every host the same day, and then run
status.sh and check that it prints no worker idle WARNING. See
§3.
From 1a: unset SBX_CLIENTS (the run refuses it), move SBX_IMAGE off the team-only 1a image,
and run with SBX_ACK_HOSTS=1, because every 1a worker bundle lacks the workspace idle time and
must be reinstalled:
SBX_ACK_HOSTS=1 deploy/sbx/setup.sh --env-file <env file>
Then re-run install-host.sh with the new bundle for every worker (step 5 of §1); it refuses a 1a
bundle. The run deletes the static client tokens and removes the relay's harness exec token. 1a
workspaces are not migrated: tell 1a users to pull their work out first.
The worker binary. install-host.sh runs the sandbox service's own build at SBX_WORKER_BIN (default
/usr/local/libexec/sbx/microvm-worker), no longer /usr/local/bin/microvm-worker: build it there
(§1 step 3) before re-running install-host.sh on a host installed before, or the install refuses.
A new worker binary takes effect when its workers restart.
Redis's image and its data. Since #670 the base runs Valkey 9, which cannot read RDB format
12, the format Redis 7.4 writes. On a namespace whose data a Redis 7.4 wrote, Valkey's AOF load
aborts (Can't handle RDB format version 12) and redis-0 crash-loops, taking the control plane
down with it. Until a migration exists, keep such a namespace on Redis 7: set SBX_REDIS_IMAGE to
the digest it ran, e.g.
docker.io/redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
(the pin before #670). With SBX_REDIS_IMAGE unset, setup.sh now refuses, before it applies
anything, when the namespace's redis StatefulSet runs an image other than Valkey, and names that
image. Before any Redis image change, take a copy of /data (BGSAVE, then
dump.rdb and appendonlydir). The StatefulSet is OrderedReady: a redis-0 that crash-loops on
the old revision is never replaced, so setup.sh's rollout wait times out. Then delete the pod
(kubectl -n <namespace> delete pod redis-0; its PVC stays) and re-run setup.sh.
Redis's memory limit. setup.sh's redis.conf sets maxmemory 200mb with
maxmemory-policy noeviction, under the container's 256 MiB limit: a full Redis refuses writes (the
control plane answers 503 redis_unavailable) and evicts nothing, instead of being OOM-killed with
every session and sandbox record. A new namespace gets it at Redis's first start. An existing one
does not: the Secret's change does not restart redis-0 (the StatefulSet has no settings hash), so
apply it live, kubectl -n <namespace> exec redis-0 -c redis -- redis-cli CONFIG SET maxmemory 209715200 and … CONFIG SET maxmemory-policy noeviction (REDISCLI_AUTH is set in the container),
or let the next restart read the file. Check with CONFIG GET maxmemory. A namespace that raises
the container's memory limit should raise maxmemory with it.
A host bundle change. Re-run install-host.sh with the new worker bundle whenever setup.sh
says so, and install-host.sh --janitor with the new janitor bundle after a change to
SBX_JANITOR_WAIT_MINUTES, SBX_TOKEN_TTL_MINUTES (its default wait follows the lifetime),
SBX_API_HOST or SBX_API_CA_FILE. status.sh --host warns when a host's installed janitor wait
no longer exceeds the token lifetime plus 30 s.
A worker installed before the jail path was keyed by a digest of the id has its jail at
/srv/sbx/jail/<id>; a reinstall moves it to /srv/sbx/jail/j<12 hex of the id>. Re-running
install-host.sh for that worker, or --uninstall <id>, removes the old /srv/sbx/jail/<id> (only that
path, once the worker is stopped) and logs removed <path>; its workspace data is not touched.
Rolling a worker back past lease mode (plan 4), then forward again. Every worker since plan 4
writes <workspace root>/.lease-state/.boot at boot, lease=on or lease=off; a lease-on boot
after a lease-off one first marks every known lease state root_wrote, because a lease-off worker
runs per-Exec commands as root without recording it. A worker binary from before plan 4 writes no
marker, so while a host runs one the marker can still say lease=on. A rollback alone is safe: the
old binary has no lease mode. Rolling forward again is safe too, as long as it goes through
install-host.sh: every worker install deletes .lease-state/.boot before it restarts the worker
(a symlinked marker goes as the link; a symlinked .lease-state is refused), so the first new boot
finds no marker and marks every known state, conservatively. A rollback done with an older
install-host.sh leaves the marker, and the next forward install clears it. Never roll forward by
replacing SBX_WORKER_BIN and restarting the unit alone; if that happened, stop the worker, delete
/srv/sbx/workspaces/<id>/.lease-state/.boot, and start it again.
3. Changing the sandbox lifetime​
Two settings. SBX_LIFETIME_DAYS (default 7) is the default idle lifetime: a sandbox ends after
that many days without use. SBX_MAX_LIFETIME_DAYS (default 30) is the longest an owner may pick,
and each worker's idle reclaim, SH_WORKSPACE_IDLE, is that plus two days. The default must not
exceed the maximum. SBX_MAX_LIFETIME_DAYS=off keeps idle lifetimes off, with the idle at
SBX_LIFETIME_DAYS plus two days (§2).
Changing only SBX_LIFETIME_DAYS touches no host: re-run setup.sh and the control plane rolls.
Changing SBX_MAX_LIFETIME_DAYS means reinstalling every host's workers. The control plane
takes the new maximum at once, but a running worker keeps the idle it was installed with until
install-host.sh is re-run on its host; after a raise, that idle can be shorter than a live
sandbox's lifetime, and the worker would reclaim its workspace.
So setup.sh refuses a maximum that raises the idle in the bundles it last wrote, unless
acknowledged:
SBX_ACK_HOSTS=1 deploy/sbx/setup.sh --env-file <env file>
and then, right after, re-run install-host.sh with the new bundle for each worker the run names.
Lowering the maximum needs no ack and touches no host: sandboxes keep the deadlines set under
the old maximum (the control plane never shortens a deadline), and a shorter idle would reclaim
their workspaces, so the run keeps the bundles' idle and logs kept at <old>h until SBX_LOWER_WORKSPACE_IDLE=1. Once the old maximum has passed since that run:
SBX_LOWER_WORKSPACE_IDLE=1 SBX_ACK_HOSTS=1 deploy/sbx/setup.sh --env-file <env file>
writes the shorter idle into the bundles; the hosts take it when install-host.sh is next run on
them. Never set SBX_LOWER_WORKSPACE_IDLE=1 before the old maximum has passed: a routine reinstall
would then give the hosts the shorter idle early. Pass it on the command line for that one run; do
not keep it in the env file: the env file is sourced with set -a, so kept there it makes a later
lowering skip the "kept" step and, with SBX_ACK_HOSTS=1 also in the file, write the shorter idle
at once.
status.sh prints worker idle <id>: ... for each host and a WARNING when a worker's idle does
not exceed the maximum.
4. Adding or removing a user​
Users are SBX_ALLOWED_SUBJECTS: comma-separated github:<numeric id> or oidc:<provider sub>
(a mixed list is accepted), no spaces. Look a GitHub id up with gh api users/<login> --jq .id. A
user who is not enrolled is told their own subject by mocactl sbx login. The model behind this
section (who is enrolled, who is an admin, the installer token, what a removed user leaves behind)
is in Users, enrollment and roles.
Edit the list and re-run setup.sh. The control plane rolls with the new list:
- adding takes effect as soon as the rollout ends;
- removing refuses every sandbox route (create, list, mint) at once, and the user's login at
its next refresh. A sandbox token already issued keeps working until it expires: within one
token lifetime plus 30 s (
SBX_TOKEN_TTL_MINUTES, 15 minutes by default), and a command still running then is cancelled. The user's sandboxes are not deleted; delete them as an admin (mocactl sbx ls --all,mocactl sbx rm --force <sandbox id>) if their work must go. The--forceis needed:rmfirst checks a sandbox for work not yet pulled, which takes a sandbox token, and an admin is never minted one for another user's sandbox.
Admins (SBX_ADMIN_SUBJECTS) may list, read and delete every sandbox, but never mint a token for
another user's: they cannot run commands in another user's code.
Enrollment and admins through the API (on a deployment with a template store,
SBX_TEMPLATE_STORE=1). The settings above stay the bootstrap; an admin named in
SBX_ADMIN_SUBJECTS adds to them without a setup.sh run (an API-granted admin can't change
access: 403), each change a record on the members provenance stream:
mocactl sbx admin subjects # who is enrolled, and how
mocactl sbx admin subjects enroll github:<id> # or unenroll; oidc:<sub> too
mocactl sbx admin subjects grant github:<id> --until 1h # a temporary admin: lapses by itself (365d at most)
mocactl sbx admin subjects revoke github:<id>
Each change reads the subjects first and prints old -> new with its record; if the subject changed
between that read and the write (another admin at once), it says so and exits 1: check and write
again. A login issued for another deployment (another control plane or issuer) is refused before the
write.
It takes effect at the subject's next request. A subject in SBX_ALLOWED_SUBJECTS or
SBX_ADMIN_SUBJECTS can't be unenrolled or revoked this way (409 bootstrap_subject): change the
settings for those. Enrolling one of them writes nothing (it is already enrolled). Before removing a
subject from the settings, check its storeEnrolled in mocactl sbx admin subjects --json
(enrolled (env, api too) in the list): if true, the store still enrolls it, so unenroll it through
the API once the settings no longer name it. An admin can't revoke its own role.
A subject holding an admin grant can't be unenrolled while it does (409 admin_granted): revoke
the role first.
If the store's members.json can't be read (damaged by hand, a bad restore), the subjects in the
settings keep working, with a warning in the control plane's log (once a minute at most), while API enrollments and grants
don't count (those subjects are refused) until the file is restored: the settings stay the way back
in.
What this changes for an operator:
- Who controls access. Without a store, every enrollment or admin change is a
setup.shrun. With one, an admin named inSBX_ADMIN_SUBJECTSenrolls subjects and grants admin through the API. The record on themembersstream is then the trace, so read it (GET /v1/subjects, the provenance export) as part of any access review. - Roles come from the store, not the login. With a store, the control plane replaces a token's
rolesclaim, on every request, with the subject's roles now (the settings plus the store's grants). Two consequences:- A login minted as a user acts as an admin if its subject holds an API admin grant. Check
GET /v1/subjectsbefore treating a login as user-only. - Revocation is immediate. Removing a subject from
SBX_ADMIN_SUBJECTS(asetup.shrun), or revoking an API grant, takes effect at the subject's next request, even for an admin token that is still valid. Butsetup.shcan't remove an API grant (those live in the store, on its volume), so after removing an admin from the settings, checkGET /v1/subjectsfor API admins too. For a temporary admin, prefer an API grant with--until: it lapses by itself.
- A login minted as a user acts as an admin if its subject holds an API admin grant. Check
5. Adding, updating and removing a template​
The template onboarding folder, deploy/sbx/template-onboarding/ (its GUIDE.md and
ONBOARDING.md), is the long form, with the tools that do the steps below: template-build.sh (the
snapshot, checked in a microVM) and template-register.sh (the two lists).
A template is an id clients choose (mocactl sbx create --template <id>, POST /v1/sandboxes {template}), served by one worker running one snapshot. A sandbox keeps the worker it was created
on for its whole life, whatever later happens to its template's entry.
Adding a template:
- Build its snapshot on the KVM host (§1 step 3), named
<image>. An image built from a container image for a benchmark (one that refuses agent commits, with a read-only rootfs) usually needsbuild-snapshot.sh --skip-git-identity-check --guest-home /tmp/home, and--vcpus N --guest-ram-mb Mfor its size (MicroVM tier (P4)). - Pick a worker id (
^[A-Za-z_][A-Za-z0-9_]*$, saysbx_node22) and add it toSBX_WORKERS. - Add its entry,
{ "id", "worker", "description", "tools" }, toSBX_TEMPLATES_FILE. One template per worker; every worker in the list. - Re-run
setup.sh: it mints the worker's token and writes its bundle. It rolls the control plane but not the relay (§2), so runningExecs on every template carry on. - Install the worker with that bundle and
--snapshot <image>(§1 step 5). A new worker's token reaches the relay when the kubelet syncs the Secret, about a minute, so its first attaches may be refused;install-host.shwaits. Pass--budget-mbfor a guest larger than 256 MiB (§1 step 5, and §9 for concurrent use). - Check it with
deploy/sbx/templates-check.sh(§14).
Updating a template (a new image or snapshot) is versioned: never re-point a worker that serves sandboxes at another snapshot, or their commands change environment mid-work. Instead:
- Build the new snapshot under a new
<image>name. - Add a new worker id for it (say
sbx_node22_v2) toSBX_WORKERS, keeping the old one. - Change the template's
workerinSBX_TEMPLATES_FILEto the new id. The old worker serves no template now, but stays inSBX_WORKERS, so it keeps its token and its sandboxes keep working. - Re-run
setup.sh, install the new worker (steps 4-5 above), and check it (step 6). New sandboxes of the template go to the new worker; existing ones stay on the old. - Retire the old worker once it serves no live sandbox:
GET /v1/janitor/sandboxes(the janitor's token) lists each live sandbox with itswrk, andmocactl sbx ls --all(an admin) shows their templates. Then remove it as below.
Both workers hold their snapshot's memory file in RAM (it is the guest's whole RAM image) until the old one is retired, so the host needs room for both.
Removing a template (or a retired worker) is the reverse of adding one: drop it from both lists,
re-run setup.sh (which revokes the worker's token and deletes its bundle; it cannot attach again),
then sudo <checkout>/deploy/sbx/install-host.sh --uninstall <id> on its host. Its sandboxes stop
working at once (no worker serves them): tell their owners to pull first. With egress credentials on
(SBX_EGRESS_CREDENTIALS=1), a worker that serves no template is no egress worker, so its
remaining sandboxes get no stored credential (as if their owners had none) and credentialed hosts
answer them 401 until it is retired.
One control plane process per template store (a deployment with SBX_TEMPLATE_STORE=1):
activation's quota and placement check and its lease write run under a lock inside the control
plane process, which is enough because one process owns the store: the Deployment runs one replica,
and the template store's volume is ReadWriteOnce with strategy: Recreate, so no second pod writes
it, even during a rollout. Keep it that way. A control plane with more than one replica, or a
rolling update, on one store would need a store-level lock.
Self-serve templates and the policy (a deployment with a template store and host agents, "The
template host agent" below): clients create their own templates with POST /v1/templates, gated by
the stored policy (GET /v1/policy), which only an admin changes (PUT /v1/policy, a policy
record). What an operator sets there:
- Profiles, the gate a create names:
base(no gate record),benchmark(a gate record an admin trusts intrustedGateRecords), and the self-attestedbenchmark-self(offline) andgeneral-self(the KVM host's allowlisted egress), which trust a subject's own gate record for its own templates only. A new image's default policy gains new profiles, but a stored policy does not: add them with onePUT /v1/policy, the whole policy as read plus the change (below). - Images:
allowedRegistries(<default>is the KVM host's ownSBX_REGISTRY) and, per registry,allowedRepositories. Docker Hub isdocker.io. On a rollback to an image older than the repository allowlist, takedocker.ioout ofallowedRegistriesfirst: an older image ignoresallowedRepositories, so it would admit every Docker Hub repository. - Quotas:
quota, andquotaBySubjectfor an override. - What clients see of it:
mocactl sbx policy,mocactl sbx quotaandmocactl sbx hosts; they create, follow and activate their templates withmocactl sbx templates(Make your own templates).
Writing the policy. A PUT /v1/policy stores the whole policy, quota overrides included, so
send it as a compare-and-set: ?expect= the digest of the GET /v1/policy it was built from.
If another admin changed anything since (a quota too), nothing is written and the answer is
409 policy_changed: read the policy again, redo the change on it, and write with the new digest.
A write without expect replaces whatever was set meanwhile. Quotas alone are simpler to set with
mocactl sbx templates quota, which does the same.
# CP: the control plane's URL; HDR: a 0600 file with one line,
# "Authorization: Bearer <an admin's session token>" (never the token in argv).
curl -fsS -H @"$HDR" "$CP/v1/policy" > policy.json
digest="$(jq -r .digest policy.json)"
jq .policy policy.json > new-policy.json # then make the change in new-policy.json
curl -sS -o put.json -w '%{http_code}\n' -X PUT -H @"$HDR" -H 'Content-Type: application/json' \
--data @new-policy.json "$CP/v1/policy?expect=$digest"
# 200: put.json is the policy as set (its new digest, its record, any `narrowed` builds).
# 409 policy_changed: nothing written; put.json's `digest` is the policy's now. Read again, then write.
Which policy writes need an admin named in the settings. Any admin may narrow the policy and
change quotas. A write that widens it is refused 403 forbidden for an admin whose role comes
from an API grant (mocactl sbx admin subjects grant); only an admin named in
SBX_ADMIN_SUBJECTS (SH_ADMIN_SUBJECTS) may make it. Widening is any of:
- adding a sha256 to
trustedGateRecords, or a registry toallowedRegistries; - adding a repository to a registry's
allowedRepositories, or dropping an allowed registry's list (which opens the whole registry); registry keys that differ only in case count as one; - adding a profile, or loosening one: a rule dropped from
requires,requireOfflineGateturned off,selfAttestorselfAttestEgressturned on, anenvelopebound raised, or itsexactsize changed or removed; - moving
defaultProfileto a profile that accepts more.
The check and the write are one locked step, so a write built from an older policy cannot restore trust a settings admin removed meanwhile.
Trusting a gate record, and self-attested templates. Trust comes from who submitted a gate record or from an admin's list, never from what the record says:
- Deployment-wide trust is the record's sha256 in
trustedGateRecords, which a settings admin adds withPUT /v1/policy(above). Thebenchmarkprofile accepts only such a record. - Self-attested (the
benchmark-selfandgeneral-selfprofiles,selfAttest: true): a record the policy does not trust still passesX1.trustedfor the template's owner, the creator (or, when an admin adds a build to another subject's template, that template's owner). A template the deployment declared has no owner, so it takes no self-attested build. A self-attested template stays private:deploymentvisibility and a transfer are refused409 self_attesteduntil an admin trusts the record. Removing a record fromtrustedGateRecordsmakes those templates private again on the next read; thePUT /v1/policyanswer and itspolicyrecord list them asnarrowed. - One subject per self-attested record. An admitted self-attested create claims the record for
its owner. The claim holds while any build citing it is queued, building, probing, built,
registered, rejected or retired; only a
failedbuild (a cancel included) releases it. A create made while the record was trusted, and a refused create, claim nothing. Another subject's create with a claimed record is refused, ruleX1.self.scope. The lookup and the write run as one step per sha256, so two subjects sending the same record at once cannot both claim it. - A self-attested record must show commits refused (
X12.self:agent_can_commit: "no"andgit_identity: "none"), whatever the spec says, and the spec may setegress: trueonly on a profile withselfAttestEgress(X1.self.egress). skip_git_identity_checkneeds a gate record under every profile,baseincluded: a skip no gate record backs, or one whose record does not show commits refused, is422 policy_refused, ruleX12. Send such a gate record, or drop the skip.
6. Rotating the signing key​
The control plane signs sandbox tokens with one Ed25519 key, SH_SESSION_TOKEN_PRIVATE_KEY in the
moca-mu1 Secret; the relay verifies with its public half, in sbx-token-keys. setup.sh keeps
exactly one key: every run rewrites sbx-token-keys with the current key alone, so a second key
added by hand is removed by the next run. Rotation is therefore a key replacement:
kubectl patch secret moca-mu1 -n <namespace> --type=merge \
-p '{"data":{"SH_SESSION_TOKEN_PRIVATE_KEY":null}}'
deploy/sbx/setup.sh --env-file <env file>
Remove that one key only: moca-mu1 also holds the janitor's token (SH_JANITOR_TOKEN), which
deleting the whole Secret would rotate too. The run generates a new key, publishes its public half
as the whole keyset, and rolls the relay and the control plane (both pods' settings hashes cover
the keyset). setup.sh does not keep the old key beside the new one, and nothing else does.
What users see: mocactl mints a fresh sandbox token for its calls, and re-mints once when the
relay refuses a token before the call started, so sessions carry on with the new key. A token
minted with the old key is refused by the new relay (an unknown key id): a direct client must run
its SBX_TOKEN_CMD again, and no old-key token outlives one token lifetime in any case. The relay's
rollout also ends the Execs running through it (the worker kills a dropped stream's Execs), so
rotate when nothing long is running. The gentler form of spec §4.3 (both keys in the keyset while
the signer switches) is not what setup.sh does.
The janitor's token rotates the same way, plus a restart, because the control plane reads it only at boot and it is not in the settings hash:
kubectl patch secret moca-mu1 -n <namespace> --type=merge -p '{"data":{"SH_JANITOR_TOKEN":null}}'
deploy/sbx/setup.sh --env-file <env file>
kubectl rollout restart deployment/moca-control-plane -n <namespace>
Then re-run install-host.sh --janitor with the new janitor bundle on every host. Until then, each
janitor run there logs skipped (refused) and removes nothing.
Rotating the credential KEK​
Only with egress credentials on (SBX_EGRESS_CREDENTIALS=1). SH_CREDENTIAL_KEK in the sbx-egress
Secret is what opens every stored credential: the store — a PVC or Vault — holds ciphertext only, so
this one value is the most sensitive in the deployment, and it is the same procedure for either store.
It is a ring, and re-sealing is write-driven. SH_CREDENTIAL_KEK is a comma-separated list,
newest first. Writes always seal under the first key; reads try every key. So a credential moves
to the new key only when it is next written — a credential nobody touches stays on the old key
indefinitely. There is deliberately no bulk re-wrap: no API returns a stored secret
(packages/control-plane/src/envelope.ts), so nothing — not even an admin — can export and re-seal.
Dropping the retired key too early is unrecoverable: that credential can never be decrypted, and the only repair is the user entering it again. So the last step waits on evidence, not on a timer.
Set NS first. Each step reads the live Secret and refuses on a value it does not expect, because
the failure mode of a half-pasted step is writing an empty SH_CREDENTIAL_KEK: the control plane
then refuses to boot (SH_CREDENTIAL_KEK is not set) and the ring is gone from the Secret, so
unless you still hold the old value every stored credential becomes unreadable. Step 3 takes the key
it keeps from the live ring rather than from a variable, so the steps are safe to run in separate
shells and no key is ever echoed to a terminal or a shell history.
NS=<namespace>
# 1. The new key, prepended to the ring. The current value is kept as the SECOND entry: overwriting
# instead of prepending is the mistake that loses every existing credential.
old="$(kubectl get secret sbx-egress -n "$NS" -o jsonpath='{.data.SH_CREDENTIAL_KEK}' | base64 -d)"
[[ -n "$old" ]] || { echo 'refusing: sbx-egress has no SH_CREDENTIAL_KEK to rotate'; return 2>/dev/null || exit 1; }
new="$(openssl rand -base64 32)" # never echoed: step 3 re-reads it from the Secret
kubectl patch secret sbx-egress -n "$NS" --type=merge \
-p "{\"data\":{\"SH_CREDENTIAL_KEK\":\"$(printf '%s,%s' "$new" "$old" | base64 | tr -d '\n')\"}}"
kubectl rollout restart deployment/moca-control-plane -n "$NS"
setup.sh generates the KEK once and then keeps whatever is there, including a ring, so a re-run
does not undo this and does not reorder it. A ring entry that is not 32 bytes of base64 makes the next
setup.sh run refuse, naming the entry rather than telling you to delete it.
# 2. Watch for reads still landing on the retired key. Both signals are the same event:
kubectl logs deployment/moca-control-plane -n <namespace> | grep 'NON-PRIMARY KEK'
# -> names the subject hash and credential, i.e. WHICH credential is still stale
curl -sS -H @"$HDR" "$CP/metrics" | jq .counters.credential_nonprimary_kek_reads
# -> the same signal as a number, without scraping logs: WHETHER any is
/metrics needs the admin role (a non-admin gets 403, not 404), and $CP/$HDR are the same
control-plane URL and header file as §5. The counter is process-local and resets on restart, and with
more than one replica a scrape hits one replica — so read it from each, or scale to one while you
finish the rotation. Neither signal is deduplicated, on purpose: a once-per-process log would let a
long-running pod look clean while credentials were still stale.
# 3. Only once nothing has opened under a non-primary key for long enough that every credential has
# plausibly been rewritten -- a quiet counter across a full cycle of use, not a quiet hour -- drop
# the retired entry, leaving the new key alone. Taken from the LIVE ring's first entry rather than
# from $new, so a fresh shell cannot write an empty key, and refused if the ring is not a ring.
ring="$(kubectl get secret sbx-egress -n "$NS" -o jsonpath='{.data.SH_CREDENTIAL_KEK}' | base64 -d)"
primary="${ring%%,*}"
[[ "$ring" == *,* && -n "$primary" ]] || # names no value: the ring is key material
{ echo 'refusing: SH_CREDENTIAL_KEK is not a ring with a non-empty first entry (nothing to drop)'; return 2>/dev/null || exit 1; }
kubectl patch secret sbx-egress -n "$NS" --type=merge \
-p "{\"data\":{\"SH_CREDENTIAL_KEK\":\"$(printf '%s' "$primary" | base64 | tr -d '\n')\"}}"
kubectl rollout restart deployment/moca-control-plane -n "$NS"
To force a credential forward instead of waiting, have its owner write it again
(mocactl sbx credentials add, same name and value): a write re-seals under the primary key. That is
the only way to move one on demand, and it needs the user, because the control plane cannot read the
secret back out.
If the counter will not go quiet, the stale credentials are named in the log lines from step 2; ask those owners to re-add, or leave the retired key in the ring. Keeping two keys is not an emergency — it costs one failed GCM verification per read of an old credential, and nothing else.
Vault instead of the PVC​
SBX_CREDENTIAL_STORE=vault with SBX_VAULT_ADDR points the control plane at a Vault KV v2 mount
instead of the PVC (deploy/k8s/overlays/ocp-sbx/egress-credentials-vault, moca#823). The KEK above is
unchanged and still lives in sbx-egress — Vault holds ciphertext, so the key that opens it must not be
in Vault.
Minimum policy, which needs no list (one object per subject, by hash, read by exact name) and no
delete (deleting a credential rewrites the subject's document, a check-and-set update):
path "secret/data/moca/credentials/*" {
capabilities = ["create", "read", "update"]
}
Two things about that path. It must match SBX_VAULT_KV_MOUNT and SBX_VAULT_PATH — the secret/
and moca/credentials above are the defaults, so a deployment that sets either must adjust the policy
to match, and the data/ segment is KV v2's own (v1 will not work). And read is not optional even
for writes: a write is read-modify-write under check-and-set, and the version it names comes from
reading the entry's metadata.version. A policy of create + update alone would let writes fail in
a way that looks like a concurrency problem rather than a permissions one.
Two things this deployment does not do for you. Both must be done before the switch works at all — the component is not startable on its own:
-
The token.
VAULT_TOKEN_FILE=/run/vault/tokenis re-read on every request, so a Vault Agent sidecar can log in (AppRole) and renew it without the control plane ever holding a long-lived token. The sidecar is not in the Component: its auth method is a per-deployment decision.Until something writes that file the pod does not start. The store reads the sink once, synchronously, at boot, so a missing or empty file fails startup (with the path in the log) rather than coming up and serving 503s —
/run/vaultis an empty in-memory volume until an Agent fills it. A 503 is what you get later, if a file that was readable at boot stops being readable. ExpectCrashLoopBackOffon the first roll after the switch if the sidecar is not in place; check the control plane's logs for the token path before suspecting Vault itself. -
Reaching Vault. The control plane's NetworkPolicy allows egress to Redis and to port 443 on public addresses only; it excepts
10/8,172.16/12,192.168/16,100.64/10and169.254/16. An internal Vault is typically in one of those ranges and on 8200, so it is unreachable until a narrow egress rule for that address and port is added in the deployment's own overlay. Do not widen the policy to all private addresses to make it work. A private CA on that Vault also needsNODE_EXTRA_CA_CERTS.
Switching an existing deployment does not migrate what the PVC store already holds: the file and Vault stores share the document format and the envelope, so a migration is a re-upload of the existing ciphertext rather than a decrypt/re-encrypt, but nothing in-tree performs it yet (moca#823).
7. The janitor​
A systemd timer on each KVM host, every 15 minutes (sbx-janitor.timer). Each run asks the control
plane for the live and terminated sandboxes (GET /v1/janitor/sandboxes, with its own token) and,
in each of this host's worker roots, removes:
- the workspace of a sandbox terminated more than the wait ago (
SBX_JANITOR_WAIT_MINUTES, default 60, or the token lifetime plus 5 minutes when that is longer); - an unknown workspace (no record at all) left unmodified for longer than that worker's
SH_WORKSPACE_IDLE; .reclaiming-*trees older than an hour.
It keeps everything else, and touches nothing when the control plane cannot be asked or answers
anything unexpected. The wait must exceed one token lifetime plus 30 s, so a deleted sandbox's last
token has expired before its files go; setup.sh refuses a shorter one, and status.sh --host
warns when a host's installed bundle predates a lifetime raise.
Each run logs one JSON line, summary or skipped, unless it refuses its own settings (a token
file with the wrong mode, a bad URL): then it exits 2 with the reason on stderr and no JSON line.
status.sh --host shows the service's result, so such a run shows as FAILED:
ssh <host> sudo journalctl -u sbx-janitor -o cat | grep '"janitor":"summary"' | tail -1
{
"janitor": "summary",
"ts": "…",
"examined": 4,
"removed": 1,
"unknown": 0,
"bytes_freed": 4096,
"errors": 0
}
errors counts what it kept because something did not check out (a missing worker env file, an
unreadable time); the lines before the summary say which. A skipped line says why (the control
plane unreachable, the token refused, a TLS failure: set SBX_API_CA_FILE if the API Route's
certificate is not publicly trusted).
Uninstalling the last worker leaves the janitor. Remove it with
sudo <checkout>/deploy/sbx/install-host.sh --uninstall-janitor.
8. Status and the audit lines​
deploy/sbx/status.sh is read-only. It reads the audit lines of the relay's current pod only, so
right after the relay rolls (any setup.sh change that alters its settings or keys) the "last
hour" figures cover less than an hour. From a machine with kubectl access to the namespace:
deploy/sbx/status.sh --env-file <env file>
token lifetime: 900 s
workers attached: sbx_default sbx_py312
workers not attached: none
sandboxes live: 12
sandboxes terminated (retained, approx.): 31
exec last hour: 214 calls, p50 380 ms, p95 2140 ms, errors 1%
exec refused last hour: 2 refused, 5 busy
exec cut short last hour: 1 timeout, 0 expired, 3 cancelled
- workers attached reads the relay's presence records in Redis. A worker
not attachedserves no sandbox: its users seeno live worker for sandbox <id>. - sandboxes terminated (retained, approx.) is the deleted or expired sandboxes still listed
(
SBX_RETENTION_DAYS, default 14). - exec last hour is from the relay's audit lines. A call is an Exec the relay routed; p50 and
p95 are over those calls' durations;
errorscounts the relay'serroroutcome only (no live worker, a worker disconnect, a worker-side failure). A command's own non-zero exit is not an error. Refusals (a bad token, a wrong sandbox) andbusy(the in-flight cap) are counted apart, as are calls cut short by a timeout, a token expiry or a client's cancel.
On a KVM host, as root:
sudo <checkout>/deploy/sbx/status.sh --host --token-ttl-seconds 900
unit sbx-worker@sbx_default.service: active
unit sbx-janitor.timer: active
unit sbx-sampler.timer: active
workspace filesystem: 45% used, 90G of 200G (110G free) on /srv
janitor last run: {"janitor":"summary",…}
janitor wait: 3600 s (outlasts the token lifetime 900 s + 30 s)
host sample: {"sampler":"sample",…}
host sample is one run of the host sampler, the line its timer writes to the
journal, now; a host sample error line before it names each field it could not read. To read it:
… | sed -n 's/^host sample: //p' | jq ..
Pass --token-ttl-seconds the cluster's token lifetime, or set SBX_STATUS_HOSTS="<host> …" (ssh
targets whose user has passwordless sudo) in <env file>, and the cluster-side run reports each
host too, with every line prefixed host <host>. The script is piped over ssh, so the host needs
no checkout for this.
Reading the audit lines yourself. The relay writes one JSON line per sandbox Exec to stdout (spec §3.5), mixed with its other output, so select them first:
kubectl logs deployment/sandbox-relay -c sandbox-relay -n <namespace> --since=1h |
grep -F '"audit":"sandbox_exec"' | jq -c 'select(.outcome != "end")'
Each line has ts, wrk (the worker), req_id, ms, outcome
(end | timeout | expired | busy | error | cancelled | refused) and out_bytes; sub and sbx
(the user and the sandbox) when the token verified; exit_code and truncated on end; and a
fixed reason on refused. No command text, output or token is ever logged. A refused Abort is
"audit":"sandbox_abort". A "audit":"sandbox_release" line means a worker never ended an
expired Exec within 60 s of its abort, and the relay gave its slot back anyway: look at that
worker's host (a wedged guest, a stuck vsock). For a debugging session only, SH_SANDBOX_AUDIT_COMMANDS=1 on the relay
adds each command's first 200 bytes as cmd.
9. Limits operators must know​
- The in-flight cap. The relay admits at most
SBX_MAX_INFLIGHT(default 2) Execs in flight per sandbox: one running, one waiting at the worker. More are refused asbusy(users see "sandbox busy"). It keeps one user from filling a template's worker. - The deaf-worker slot hold. When an Exec outlives its token, the relay cancels it and then
waits for the worker's acknowledgement before freeing its in-flight slot. A wedged worker that
never answers holds that slot until its attach stream drops: the sandbox service has no bound on that drain, so
with the default cap of 2, one wedged Exec halves the sandbox's concurrency until the worker
disconnects. Restarting the worker (
systemctl restart sbx-worker@<id>) drops the stream and frees every slot it held, and also ends every running Exec on that worker, for every user of the template. - Delete is not immediate. A deleted sandbox (or a removed user) stays reachable for up to one
token lifetime plus 30 s. Raising
SBX_TOKEN_TTL_MINUTESallows longer commands at the cost of a slower delete, and needs a janitor reinstall (§2). - Disk is not admission-controlled on the worker. The quotas are the disk control:
setup.shrefusesSBX_MAX_TOTAL×SBX_WORKSPACE_IMAGE_MBover 80 % ofSBX_WORKSPACE_FS_GIB. - A worker's budget is sized for concurrent use, not for one session.
install-host.sh's smallest--budget-mbholds one session at the unit's standby depth of 2, (guest RAM + 32) × 3 + 1024 MiB (13408 MiB for a 4096 MiB guest). A running VM is charged its full guest size and a paused standby only its dirty pages, so that minimum fits about two running VMs: a third command at once on the template is refused at admission before it starts (memory-budget: …, "template busy" to users). Size it for the commands that may run at once on the template, (guest RAM + 32) × running VMs + 1024 MiB: for example 17536 MiB for four 4096 MiB VMs (two agents on one template, each with a command and a standby being promoted). The static clients (remote-worker/cmd/sbx,docs/api/examples/sbx.py) retry such a refusal and exit 75 if every try is refused. The worker's start-up warning (slots × per-VM size against the budget) is expected at any budget below its slot count.
Reproducing a member's issue​
When a member reports a problem, assemble one bundle for the sandbox: what the member's client
sent and got back, what the sandbox reached, and what the control plane, the relay and the KVM host
logged for that sandbox in that window. Every query below was checked against the line formats in
the code it names; none has been run against a live deployment yet. The jq filters run on your
machine (status.sh needs jq too); the host's commands run over ssh as in §8.
Which sandbox, whose, and when. Get the sandbox id from the member (mocactl sbx ls, or
.sbx/sandbox.json in their repository), then, as an admin:
SBX=sbx-… # the sandbox id
mocactl sbx ls --all --json | jq --arg id "$SBX" '.[] | select(.id == $id)
| {id, owner, template, state, created: (.createdAt | todate),
ended: (if .terminatedAt then (.terminatedAt | todate) else null end)}'
SUB=github:… # its owner
FROM='2026-11-17 10:00:00' # the window, UTC
TO='2026-11-17 11:00:00'
NS=<namespace>
createdAt and terminatedAt are epoch seconds. The KVM host logs the sandbox under its workspace
key, which is the sandbox id itself (wsk is set to the id at create, in
packages/control-plane/src/sandbox-handlers.ts), and its worker under the template's worker id,
wrk in the relay's lines below.
1. The member's own record. Ask the member to rerun what failed with --record issue.ndjson
on the command, and to send the file:
mocactl sbx up --record issue.ndjson # or the command that failed: run, push, pull, …
mocactl run "prompt" --record issue.ndjson # a Moca Pi session's calls
The flag goes after sbx <command> or run (mocactl --record F sbx … is refused, exit 2), and
every mocactl sbx command that calls the API takes it except token, login, logout and
credentials, which refuse it because it would record a secret (NO_RECORD in
packages/mocactl/src/sbx/cli.ts). It appends one JSON line per HTTP attempt, the login's refresh
included: the command, an invocation id, seq and try (a retry of the same call), the method and
path (never the base URL or host), the status or the transport error's name, both bodies, their
times (requestAt, answerAt, UTC) and any retryAfter; a streamed answer gets its line when the
headers arrive and a stream_end line with its bytes, sha256 and frame count, never its contents
(packages/mocactl/src/sbx/call-record.ts). Token-named fields, gate records and credential values,
and any string that looks like a JWT or a refresh token, are kept only as their sha256; exec
endpoints and CAs, a proxy's dial_addr, a build log and any absolute URL only as their length;
under run, the prompt, the reply, the patch and other free text only as their length too. The
file is created 0600 and opened for appending; an existing file readable or writable by anyone but
its owner is refused (exit 2). The record has no Exec in it: commands go to the relay over
gRPC, which is not recorded, so for sbx run or an agent's commands it shows the token mints and
the control-plane calls around them, and the relay's lines (step 3) show the Execs.
2. The egress log. For a template with egress, as an admin (the route answers the owner or an
admin, packages/control-plane/src/routes.ts):
mocactl sbx egress "$SBX" --ndjson > egress.ndjson
mocactl sbx egress "$SBX" --verify
They are two runs: egress takes --ndjson or --verify, not both. --ndjson prints the log as
the control plane stores it, one event per line (host, method, path, status, bytes up and down,
decision, credential name). --verify reads the log again and checks its sha256 and line count
against the control plane's summary: verified: N events, sha256:… (with INCOMPLETE: N dropped
when the log lost events, and sealed or open (not sealed yet)), or NOT verified: …, exit 1
(cmdEgress and verifyEgress in packages/mocactl/src/sbx/commands.ts). Until the log is sealed, it can
still gain events: take the copy after that, as Egress records explains. It is
sealed at the host janitor's first sweep at least 600 s after the sandbox ends (EGRESS_GRACE_S),
not at the 600 s themselves, so --verify can read open (not sealed yet) well past them; with no
host janitor running (the host stopped), it stays open until one runs.
3. The control plane's, the relay's and the host's lines. The control plane writes one
"event":"audit" JSON line per decision to stdout (packages/control-plane/src/handler-util.ts,
auditBestEffort), with subject, decision and, for a sandbox, sandboxId (sandbox_created,
sandbox_deleted, sandbox_mode_changed, sandbox_create_rolled_back, among others). Those lines
carry no time of their own, so read them with --timestamps (checked against the code):
kubectl logs deployment/moca-control-plane -c control-plane -n "$NS" \
--since-time="${FROM/ /T}Z" --timestamps |
grep -F '"event":"audit"' |
jq -cR --arg sbx "$SBX" --arg sub "$SUB" --arg to "${TO/ /T}Z" '
capture("^(?<t>[^ ]+) (?<l>.*)$") | .t as $t | select($t <= $to)
| .l | fromjson? | objects | select(.sandboxId == $sbx or .subject == $sub) | {t: $t} + .'
The relay writes the "audit":"sandbox_exec" line of §8 per Exec, and sandbox_abort,
sandbox_release, sandbox_lease_control and sandbox_workspace_release (a client's Release of
its workspace at its end, #338) lines, each with its own ts, and sub and sbx
when the token verified (packages/sandbox-relay/src/sandbox-caller.ts, makeAudit). Checked
against the code:
kubectl logs deployment/sandbox-relay -c sandbox-relay -n "$NS" --since-time="${FROM/ /T}Z" |
grep -F '"audit":"sandbox_' |
jq -cR --arg sbx "$SBX" --arg sub "$SUB" --arg to "${TO/ /T}Z" '
fromjson? | objects | select(.ts <= $to and (.sbx == $sbx or .sub == $sub))'
A refusal of a token that did not verify names no sub or sbx: look for it by the worker and
the time, select(.wrk == "<wrk>" and .outcome == "refused"). Both Deployments run one replica, so
deployment/… reads the one pod. Add --previous for the container before its last restart;
anything older than that is gone (see below). On OpenShift, oc logs takes the same flags.
The control plane also writes each decision to the Redis stream sh:cp:audit, with a ts in epoch
ms (OwnershipIndex.audit in packages/control-plane/src/ownership.ts). Checked against the
code (the stream and its field names; the plain output, one value per line, is redis-cli's own):
ms() { jq -nr --arg t "$1" '$t | strptime("%Y-%m-%d %H:%M:%S") | mktime * 1000'; }
kubectl -n "$NS" exec redis-0 -c redis -- \
redis-cli XRANGE sh:cp:audit "$(ms "$FROM")" "$(ms "$TO")" |
awk -v k=" $SBX " '/^[0-9]+-[0-9]+$/ { if (index(r " ", k)) print r; r = $0; next }
{ r = r " " $0 } END { if (index(r " ", k)) print r }'
Each line printed is one entry: its id, then its fields and values. The filter matches a whole
value (spaces on both sides), so github:1 does not also match github:12; use -v k=" $SUB "
for the member's other decisions (logins and refreshes among them).
On the KVM host, the worker's lines are plain text, and those about one sandbox name its workspace
key, as workspace_key="sbx-…" or for "sbx-…" (remote-worker/internal/vmpool). Lines about
the pool itself (its cgroups, an orphan sweep) name no sandbox, so read the worker's whole window
too. Checked against the code:
ssh <host> "sudo journalctl -u sbx-worker@<wrk>.service --since '$FROM UTC' --until '$TO UTC' \
-o short-iso-precise --no-pager" | grep -F "$SBX"
ssh <host> "sudo journalctl -u sbx-worker@<wrk>.service --since '$FROM UTC' --until '$TO UTC' \
-o short-iso-precise --no-pager" > worker.log
The janitor's JSON lines (§7) name the workspace as wsk, and moca-egress writes one JSON event
per connection with the workspace key as sid (remote-worker/internal/egress/audit.go; the host
agent selects the events it ships by via and sid too, egress_select in
deploy/sbx/host-agent.sh). This is the
host's own copy of what step 2 serves, with vm_id and dial_addr besides. Checked against the
code:
ssh <host> "sudo journalctl -u sbx-janitor.service --since '$FROM UTC' --until '$TO UTC' -o cat \
--no-pager" | jq -cR --arg k "$SBX" 'fromjson? | objects | select(.wsk == $k)'
ssh <host> "sudo journalctl -u moca-egress.service --since '$FROM UTC' --until '$TO UTC' -o json \
--all --no-pager" | jq -c --arg k "$SBX" '.MESSAGE | strings | (try fromjson catch null)
| objects | select(.via == "moca-egress" and .sid == $k)'
A janitor run's summary line has its own ts; the lines before it in the same run do not.
4. How loaded the host was. The host sampler writes one JSON line every
15 s; take the window's, from a little before FROM, so the first counter difference covers it:
ssh <host> "sudo journalctl -u sbx-sampler.service --since '$FROM UTC' --until '$TO UTC' -o cat \
--no-pager" > samples.ndjson
jq -c 'select(.sampler == "sample") | {ts, load1: .load.avg1, mem_avail_kb: .mem_kb.available,
psi_mem_full10: .psi.memory.full_avg10, psi_io_some10: .psi.io.some_avg10, errors,
workers: [.workers[] | {wrk, vms, committed: .stats.committedBytes, budget_mb, reserve_mb}]}' samples.ndjson
jq -c 'select(.sampler != "sample")' samples.ndjson # the fields it could not read, and when
The worker's wrk is the relay's (step 3). sudo <checkout>/deploy/sbx/status.sh --host (§8) adds
a view of now: the units' states, the janitor's last run and one sample.
What is not stored. No command text and no output is stored anywhere server-side. The relay's
audit line has neither (docs/notes/sandbox-security-review.md §1.5 lists it as tested), unless the relay
runs with SH_SANDBOX_AUDIT_COMMANDS=1, the debug flag of §8, which adds each command's first 200
bytes as cmd. The egress log is the network's shape only, with paths but no bodies. The member's
--record file is the only trace of what was sent, and only of the HTTP calls. What is kept, and
for how long, today:
- Control-plane and relay stdout: the pod's current log, and with
--previousthe one before its last restart; nothing else keeps them. sh:cp:audit: in Redis, trimmed to about 1,000,000 entries on each write (AUDIT_MAXLEN); refusals of a refresh token nobody issued go tosh:cp:audit:anoninstead, about 100,000. Redis hasmaxmemory 200mb(§2) for everything it holds.- Egress logs: at most 10,000 events or 4 MiB per sandbox, 64 MiB for all
(
packages/control-plane/src/egress-log.ts), kept as long as the sandbox's record (SBX_RETENTION_DAYS, default 14). - Host journald: the host's own journald settings;
install-host.shsets none. The host sampler adds about 1 KB per sample with a few workers, about 6-8 MB a day at the default 15 s.
Once a log pipeline is chosen (#857, R1), the control-plane, relay and host lines are queried
there instead, by the same fields (sandboxId, subject; sbx, sub; the workspace key), and
this section moves its queries to that store.
Guest egress​
install-host.sh installs and starts one moca-egress per host (moca-egress.service) before the
worker, and wires every template worker to its control socket
(MOCA_EGRESS_CONTROL_SOCK=/run/moca-egress/control.sock in the worker env), with a drop-in
sbx-worker@<id>.service.d/egress.conf that orders the worker after moca-egress. A --no-egress
worker gets neither, so it never starts a moca-egress.service that the host has. A sandbox can
then reach only the hosts on the allowlist; everything else is refused with a named error
(destination_denied), never a timeout. No credential is involved. For a plain-HTTP request, the
refusal's body tells the user or the agent that the host is not on the allowlist and to ask you,
the operator. An HTTPS request is refused at its CONNECT, and most clients (curl, git, pip) show
only 403 or CONNECT tunnel failed, with the X-Moca-Egress-Error header under curl -v; the
agent's sandbox instructions carry the rule there. The allowlist below is where you add a host.
- The allowlist is
MOCA_EGRESS_ALLOWin/etc/sbx/moca-egress.env, a comma-separated list of hosts. It is seeded on the first install fromdeploy/sbx/egress-allow.envbeside the installer (or fromMOCA_EGRESS_ALLOWin the installer's environment, on the first write only) and is never overwritten by a re-run. To change it on one host, edit/etc/sbx/moca-egress.envand runsudo systemctl restart moca-egress; to change the default for new hosts, editdeploy/sbx/egress-allow.envin the repo (see its header for the format and the entry groups). Open tunnels end on the restart, new ones work at once. The same restart is needed after rebuilding the binary. A bareMOCA_EGRESS_ALLOW=*(nothing else on the line) is allow-everything mode: every destination is permitted, private and local addresses included -- only the always-refused tier stays (cloud metadata endpoints, link-local, multicast), so a sandbox still cannot fetch the host's instance credentials. It includes the host's own loopback (127.0.0.0/8,::1) and interface addresses: a sandbox can reach every service listening on the KVM host. Don't use it on a host that runs anything else, or where loopback services trust the local network. For trusted research rigs only; the startup log saysallow-all modeplainly, so a journal reader cannot miss it. - Another install's
moca-egress.service. A host that also runsdeploy/microvm's P6 worker already has one. If/etc/systemd/system/moca-egress.serviceexists and/etc/sbx/manifesthas noegress unitline for it,install-host.shlogsreusing the existing moca-egress.service (not installed by this script), wires the workers to its socket and touches nothing else. Its allowlist is then that install's/etc/serverless-harness/moca-egress.env. --no-egressinstalls a worker withoutMOCA_EGRESS_CONTROL_SOCKand installs no egress unit: the worker behaves as before egress existed, and the guest has no network.- Snapshot: a golden snapshot built before P4.1 has no guest egress listener. Rebuild it with
build-snapshot.shfor egress to work (MicroVM tier (P4)). - If it fails to start, the install names
journalctl -u moca-egress(often a bad/etc/sbx/moca-egress.env). Fix it and re-runinstall-host.sh. - Ownership is by content. The manifest records a digest of the unit it installed. If another
install (
setup-microvm.sh) later replacesmoca-egress.service, a re-run logschanged by another install; reusing itand leaves it alone, and the last uninstall keeps it. - Uninstalling the last worker also stops and removes the
moca-egress.servicethis script installed; an allowlist you edited stays (kept ... (edited)).
Egress records​
The template host agent's wanted-state run (sbx-host-agent-wanted, every 15 s) also ships the proxy's audit
events to the control plane, where each lands in its sandbox's egress log: mocactl sbx egress [ID], or GET /v1/sandboxes/{id}/egress. Nothing to install: no new unit, timer or setting.
Paths are logged without query, fragment or userinfo, and an injected credential appears by name only, but a secret a guest puts in a URL path segment is logged as sent.
- On or off: on when
systemctl is-active moca-egress.servicesays active, off otherwise (a--no-egresshost, or a moca-egress not started). Active, not enabled:install-host.shreuses amoca-egress.serviceanother install owns (a P6 worker's) without enabling it, and that install may start it only for a while. So on such a host, whether the agent reads the proxy's journal follows that unit's state: while it runs for its own install, the agent reads its events every 15 s, ships only this host's own sandboxes' (egressSids), and counts the restegressNotOurs. Nothing of the other install's leaves the host, but check the unit's state before relying on it. Off, the agent reads no journal, posts nothing and writes no cursor; its summary says"egress":"off". - Only this host's sandboxes' events leave it. The wanted state names them (
egressSids: the workspace keys of the sandboxes on this host's workers, live or ended within 600 s); an event with any othersid, such as a Moca P6 session's on the same proxy, stays in the journal and is only counted (egressNotOurs). The list is read before the journal, so a batch with asidnot in it (a sandbox created in between, at its first egress) reads the wanted state again, after the journal read, before anything is left out; if that read fails, nothing is sent and the cursor stays. The proxy'saudit_droppedreports are sent. Against a control plane whose wanted state has noegressSids(older than openapi 1.14.0), the agent sends nothing and says so in an error line, keeping its cursor. - A time budget: a run starts no new batch 120 s after its egress step began, so slow posts never
hold the wanted unit (
TimeoutStartSec5 min) and its activations. The cursor stays at the last batch taken; the next run goes on. - It only reads. The agent reads
journalctl -u moca-egress.serviceand never starts, stops or configuresmoca-egress, so a failed post never blocks or changes a guest's egress. - Where it is: the journal cursor after the last event the control plane took is
/srv/sbx/egress.cursor; a batch being sent is/srv/sbx/egress.pending.json(both 0600). Without a cursor, the first run starts at this boot. - If a post fails (the control plane down, a 5xx, or a 404 from one older than this): the summary
counts an error (
egress post failed: the cursor stays, the next run retries (HTTP <code>)), the cursor stays, and the next run re-sends the same batch first. The control plane takes a repeated batch once (idempotent by host and cursor). Up to 10 batches of 1000 entries per run. - A batch refused as malformed (400
invalid_requestorinvalid_json) would fail every retry, so it is moved to/srv/sbx/egress.rejected/<UTC time>-<pid>.json(0600), the cursor moves past it, and the error names the file (egressRejectedin the summary). Its events are not in any sandbox's log: look at the file, and fix the agent or the control plane. - A cursor the journal no longer has (rotated away, or from another machine) is an error
(
the journal cursor is no longer valid; restarting from this boot). journalctl itself does not fail on one: it resumes at the nearest kept entry, or for another machine's cursor by time across boots, with exit 0. So before its first read the agent checks that the cursor still names an entry (journalctl --cursorstarts at that entry); journalctl fails only on a cursor it cannot parse, which is the same error. Entries rotated away before they were sent are lost with the journal. The old cursor is kept as/srv/sbx/egress.cursor.stale, and shipping restarts from this boot's first entry. Entries already sent are skipped (egressSkipped), by/srv/sbx/egress.mark(the last sent entry's boot id and monotonic time): the control plane's replay check is by cursor, so it could not tell them apart. Without a mark, the restart re-sends this boot's entries, which the control plane stores twice; the agent says so in an error line (no position mark). Entries of an earlier boot that the old cursor had not reached are not read. - What is kept: each event with its fields and values as the proxy wrote them (the agent decodes and
re-encodes each one, so the bytes may differ, and the control plane stores its own canonical line), for as long as the sandbox record
(
SBX_RETENTION_DAYS). A log holds at most 10,000 events (EGRESS_LOG_CAP) and 4 MiB of stored lines (EGRESS_LOG_BYTES), and all logs together at most 64 MiB of stored lines (EGRESS_TOTAL_BYTES), inside the 256 MiB Redis has in all; Redis's own overhead per entry comes on top. The total is a running sum kept by the store's append (packages/control-plane/src/egress-log.ts), so an expired log gives its bytes back at a later append, and a post costs the same however many logs are retained. - Completeness: an event over 8 KiB isn't sent (only its sandbox id is); the agent reads the journal with
--all, so one up to 8 KiB is sent whole (without it journalctl prints a field over 4096 bytes asnull). A journal entry whose message is not text (not UTF-8) cannot be read for its sandbox: the agent says so in an error line. A log'sdroppedcounts those (invalid), events past its cap (cap) or past the total budget (budget), and the proxy's ownaudit_droppedreports, charged to every sandbox on the host's workers live within 60 s before each.completeisdropped == 0. The sandbox's end record (sandbox,state: terminated) namesegress: {events, sha256, dropped, droppedBy, complete}; if events arrive in the 600 s after the end, a latersandboxrecord withstate: egressnames the final log. The grader decides whether an incomplete log still counts. - Check a saved copy offline: wait until the log is sealed:
sealedinmocactl sbx egress ID(summary.sealed: truewith--json), at the host janitor's first sweep at least 600 s after the sandbox ends. Until then the control plane still takes the sandbox's last events, and if the log changes, the janitor's seal writes a laterstate: egressrecord that a copy saved earlier no longer matches. Then save it withmocactl sbx egress ID --ndjson > log.ndjson, take the provenance export, and runverify-provenance.py EXPORT.json --egress ID=log.ndjson. (mocactl sbx egress ID --verifychecks the log served against the control plane's summary of it, not against the end record; on an open log it says when events arrived while it read.) - Deploy the control plane first: an older one answers 404 on the egress route, and the agent keeps its batch and retries.
- To turn it off on a host without disabling egress: there is no switch but the unit's state, by
design. Roll back the host agent instead (
install-host.shwith the previous tree).
Credential injection (--egress)​
Prerequisite for both procedures: set SBX_EGRESS_CREDENTIALS=1 (and optionally SBX_EGRESS_HOSTS)
in setup.sh's env file and run setup.sh. That writes deploy/sbx/.generated/egress/<host id>/
(egress.env, egress.token, optional api-ca.crt); pass that <host id> directory to --egress,
not its egress/ parent. --egress generates the host's interception CA once (/etc/sbx/egress/ca.crt,
ca.key), writes the token (0600) and the drop-in moca-egress.service.d/sbx-credentials.conf, then
restarts moca-egress. It refuses a host whose moca-egress.service it did not install (the P6-owned
unit is configured by its own installer) and a host with no worker yet (the unit comes with the first
worker). --egress-ca-out prints the CA path (generating it if absent) and does nothing else. Once the
drop-in exists, a snapshot whose manifest.json egress_ca_sha256 is missing or differs is refused
before systemd is touched.
Fresh host (no worker installed):
sudo deploy/sbx/install-host.sh --egress-ca-out(generates the CA).deploy/microvm/build-snapshot.sh --egress-ca /etc/sbx/egress/ca.crt --out /srv/sbx/snapshots/<image>, or for a template,template-build.sh SPEC --egress-ca /etc/sbx/egress/ca.crt ...(deploy/sbx/template-onboarding/ONBOARDING.md). A self-serve template is built by the host agent, which passes--egress-caitself once the drop-in exists (step 4; The template host agent). From creating the CA until the drop-in is in, stopsbx-host-agent(or let no build be queued): a build claimed in between lacks the CA, is refused at worker install, and is rebuilt.sudo deploy/sbx/install-host.sh WORKER_BUNDLE --snapshot <image>.sudo deploy/sbx/install-host.sh --egress .generated/egress/<host id>.
Existing host (worker already installed):
sudo deploy/sbx/install-host.sh --egress .generated/egress/<host id>.deploy/microvm/build-snapshot.sh --egress-ca /etc/sbx/egress/ca.crt --out /srv/sbx/snapshots/<new image>.- Re-install each worker with
--snapshot <new image>.
Between --egress and the reinstall, users with a stored credential see guest TLS failures on
intercepted hosts, because the old snapshot does not trust the new CA. Do the reinstall promptly. The
same applies to a CA rotation.
The default allowlist (MOCA_EGRESS_ALLOW) does not include api.githubcopilot.com. Add it to
/etc/sbx/moca-egress.env for the github-mcp preset, which otherwise gets 403 destination_denied.
Turning credential injection off. Run install-host.sh --uninstall-egress on every KVM host
first, then run setup.sh with SBX_EGRESS_CREDENTIALS=0. With the cluster off, the lookup route is
unmounted (404); a host that still has the drop-in treats the 404 as an error and answers
503 credential_unavailable on every allowlisted request, including a plain git clone by users with
no credential. The PVC and the sbx-egress Secret stay, so a later on-run finds the stored credentials
again. The reverse rule: never run --egress against a cluster that is still off.
A re-run keeps the CA. To rotate it, delete /etc/sbx/egress/ca.crt and ca.key, re-run --egress,
then rebuild every snapshot (step 2). --uninstall-egress removes the drop-in, token and api CA and
keeps the CA, because existing snapshots were built with it.
Snapshot read-ahead​
A worker's cold start reads its snapshot's memory file (4 GiB for a 4 GiB guest) from /srv/sbx.
On a network block volume whose largest request is 128 KiB (max_hw_sectors_kb), the kernel's
default read-ahead of 128 KiB keeps too few reads in flight to reach the volume's throughput. One
measurement (one KVM host, a network volume rated 750 MiB/s and 3,000 IOPS, page cache dropped):
one reader of the memory files got 227 MiB/s at read_ahead_kb 128, 730 MiB/s at 512 and about
850 MiB/s at 1024, and four readers 435 and about 779 MiB/s (128, 512). Eight cold worker starts
posted at once, four at a time, were all active after 100.7 s at 128 (about 420 MiB/s) and after
62.2 s at 512 (736-742 MiB/s, the volume's limit), with the same 4.0 GiB read per start. Every
read stayed 128 KiB, at about 5,900-6,900 reads/s at 512 and above (about 5,950 in the cold starts): what bound it was reads in flight, not IOPS.
So install-host.sh sets the read-ahead of the device holding /srv/sbx to SBX_READ_AHEAD_KB
(default 512 KiB, at most 16384) and installs sbx-read-ahead.service, a oneshot enabled at boot
(a sysfs write does not survive a reboot) that the workers start after. It is enabled whatever
SBX_WORKER_AT_BOOT says: on a host whose workers a host agent starts, the boot still sets the
read-ahead before the agent starts any. It finds the device by path
at every run: a partition's read-ahead is its disk's, an LVM or dm device has its own. The value the
device had before the first install is kept in /etc/sbx/read-ahead.orig, and the last worker's
uninstall writes it back. A /srv/sbx on no block device (tmpfs, overlay), or a value the device
does not take, is a warning in the install output and in journalctl -u sbx-read-ahead, never a
failed install.
- Check it:
findmnt -no SOURCE --target /srv/sbxnames the device; for a disk or a dm device,cat /sys/block/<dev>/queue/read_ahead_kb(for a partition, its disk's). - It is device-wide. Everything else on that volume (another Moca tier, other sandboxes, the
host's own files if
/srv/sbxis on the root disk) reads ahead as much. Read-ahead changes the size of the reads, not how many bytes a cold start reads; on local NVMe a smaller value may do. - Turn it off: re-run a worker's install with
SBX_READ_AHEAD_KB=0. That writes the recorded value back and removes the unit. A later install withoutSBX_READ_AHEAD_KB=0sets it again.
The host sampler​
Every worker install also installs sbx-sampler.timer (#857, R2), so a KVM host with a worker
samples its own load from then on. Every 15 s (SBX_SAMPLER_INTERVAL_S, 5-3600, at the install; in practice every
15-16 s, the timer's 1 s accuracy on top) a
oneshot, sbx-sampler.service, runs /usr/local/libexec/sbx-sampler.sh (deploy/sbx/sampler.sh),
which writes one JSON line to the journal. It reads /proc and the cgroup files; it starts df
twice (3 s each at most; after one that does not answer, as on a hung filesystem, it skips the
second and reports workspaces null) and, per worker that runs with SH_DIAG_STATS_ADDR, one curl of that worker's
/stats on this host (1 s at most; an address that is neither loopback nor a wildcard is refused, never
curled); and it never fails its unit. A sample, shortened:
{
"sampler": "sample",
"ts": "2026-11-17T10:30:00Z",
"load": { "avg1": 3.1, "avg5": 2.8, "avg15": 2.2 },
"mem_kb": {
"total": 65536000,
"available": 40000000,
"swap_total": 0,
"swap_free": 0,
"swap_used": 0
},
"psi": {
"memory": { "some_avg10": 0.2, "some_avg60": 0.1, "full_avg10": 0.1, "full_avg60": 0.05 },
"…": "cpu, io"
},
"workspaces": {
"bytes_total": 214748364800,
"bytes_used": 96636764160,
"…": "bytes_avail, inodes_total, inodes_used, inodes_free"
},
"slice": {
"cpu_usage_usec": 9000000,
"cpu_nr_throttled": 0,
"memory_current": 6442450944,
"memory_peak": 8589934592,
"io_rbytes": 1024,
"…": "…"
},
"workers": [
{
"wrk": "sbx_default",
"vms": 3,
"memory_current": 4294967296,
"budget_mb": 17536,
"reserve_mb": 1024,
"stats": {
"committedBytes": 3221225472,
"standbyCommittedBytes": 268435456,
"refusals": { "memory-budget": 2 }
}
}
],
"errors": []
}
load,mem_kb,psiare the host's:/proc/loadavg,/proc/meminfo(kB;swap_usedis SwapTotal minus SwapFree) and/proc/pressure/{cpu,memory,io}(the share of time some or all tasks waited, over 10 and 60 s, in percent).workspacesis the filesystem holding/srv/sbx/workspaces, in bytes and inodes.sliceissbx.slice, every worker and its VMs:cpu.stat,memory.current,memory.peak(null before Linux 5.19) andio.statsummed over devices. The CPU, throttling and I/O figures are counters since the slice was made: the use in a window is the difference between two samples (below).memory.currentcounts page cache too.workers, one per/etc/sbx/workers/<id>.env:vmsis the VM cgroups in that worker's slice that hold a process, running and paused standby VMs alike (a stopped worker has 0);memory_currentis its slice's (null when it is not running);budget_mbandreserve_mbare itsSH_MAX_COMMITTED_MBandSH_MEMORY_RESERVE_MB, from the running worker's environment (the process namedmicrovm-workerin its unit's cgroup; else its env file). A VM is admitted whilecommittedBytesstays withinbudget_mbminusreserve_mb.statsis from the worker's/stats, named as the worker names them, and null unless the worker runs withSH_DIAG_STATS_ADDR, which is off by default. To turn it on for one worker, a drop-in (an install rewrites the env file, never this), with a port of its own per worker:sudo systemctl edit sbx-worker@<id>, then[Service]andEnvironment=SH_DIAG_STATS_ADDR=127.0.0.1:9901, andsudo systemctl restart sbx-worker@<id>, which ends that worker's running commands. It serves the pool's counters only, on its own listener.errorsnames each field group it could not read in this sample; their values are null. The first sample in which a kind fails is preceded by one{"sampler":"error","kind":…,"why":…}line, and the first one in which it reads again by one{"sampler":"recovered","kind":…}line, so a lasting failure (a kernel without PSI, a worker's/statsnot answering) is one line in the journal, not one every 15 s.
The unit logs at notice and drops anything below it, so systemd's own lines about each run
(started, finished) stay out of the journal. On the host:
sudo journalctl -u sbx-sampler.service -o cat --since -5min | jq -c 'select(.sampler == "sample")'
# How loaded was it at 10:30 UTC: the samples around it.
sudo journalctl -u sbx-sampler.service -o cat --since '2026-11-17 10:29:30 UTC' \
--until '2026-11-17 10:30:30 UTC' | jq -c 'select(.sampler == "sample") | {ts, load, mem_kb, psi}'
# sbx.slice's CPU in cores, between consecutive samples.
sudo journalctl -u sbx-sampler.service -o cat --since -10min | jq -sc '[.[] | select(.sampler == "sample")]
| . as $s | range(1; length) as $i | {ts: $s[$i].ts, slice_cores: (($s[$i].slice.cpu_usage_usec
- $s[$i - 1].slice.cpu_usage_usec) / ((($s[$i].ts | fromdate) - ($s[$i - 1].ts | fromdate)) * 1e6))}'
These run jq on the host; on a host without it, run the journalctl over ssh and jq on your
machine, as Reproducing a member's issue does. The journal keeps the samples for as
long as the host's journald settings keep anything. Once a log pipeline is chosen (#857, R1), it
ships the same lines.
-
Its own cost. The target is under 1% of one core. Measure it on the host: 200 runs as the timer runs them, then the CPU time per run over the interval:
sudo env SBX_ENV_DIR=/etc/sbx SBX_SRV=/srv/sbx SBX_SAMPLER_STATE_DIR= bash -c \'time (for i in $(seq 200); do /usr/local/libexec/sbx-sampler.sh >/dev/null; done)'sudo systemd-run --wait -p CPUAccounting=yes /usr/local/libexec/sbx-sampler.sh # one run, as a unit(user + sys) / 200 / 15 s is the share of one core;
systemd-run --waitprints one run'sCPU time consumed. -
Change the interval, or turn it off: re-run a worker's install with
SBX_SAMPLER_INTERVAL_S=60(or0, which removes it). The last worker's uninstall removes it;--keep-sharedkeeps it.
10. A KVM-host outage​
What users see. Every sandbox whose template runs on the host fails: mocactl sbx run and the
agents' tools report sandbox failure: … no live worker for sandbox <worker id>, and
mocactl sbx doctor fails its exec check. Login, ls and create still work, because the
control plane runs in the cluster; a new sandbox fails the same way until the host is back.
What to tell them. The service is best effort, business hours. Their laptops keep the original
trees, and anything not yet pulled is on the host. If the host comes back with its disk, every
sandbox and its files are as they were (a worker restart keeps workspaces). If the disk is lost,
every sandbox on it is gone: they run mocactl sbx rm --force (rm cannot check a lost sandbox
for unpulled work, so it refuses without --force), then mocactl sbx up, and redo the work not
pulled.
What to check. status.sh (workers not attached), then on the host status.sh --host and
journalctl -u sbx-worker@<id>. A worker that crash-looped into failed is brought back by
re-running its install-host.sh (it resets the failed state).
A VM left behind by a stopped worker. A worker's Firecracker VMs run in pool-N cgroups under
its slice, /sys/fs/cgroup/sbx.slice/sbx-<id>.slice, beside the unit's own cgroup, so stopping the
unit does not kill them. The unit's ExecStopPost= does: after every exit (a stop, a crash, a stop
timeout) /usr/local/libexec/sbx-reap-vms.sh <id> kills each pool-N cgroup of that slice and waits
up to 10 s for it to empty, logging sbx-reap-vms: lines to journalctl -u sbx-worker@<id>. A
still populated line there, or a host installed before the step existed, can leave a VM holding its
guest memory until that worker next starts. With the worker stopped, find one by its cgroup:
grep -H . /sys/fs/cgroup/sbx.slice/sbx-<id>.slice/pool-*/cgroup.procs # any pid here is a leaked VM
for p in $(pgrep -x firecracker); do echo "$p $(cut -d: -f3 "/proc/$p/cgroup")"; done # every VM's cgroup
and reap it with sudo /usr/local/libexec/sbx-reap-vms.sh <id> (or
echo 1 | sudo tee /sys/fs/cgroup/sbx.slice/sbx-<id>.slice/pool-<n>/cgroup.kill). Never while the
worker runs: its pool-N cgroups then hold its live sandboxes' VMs.
11. The disk filling up​
status.sh --host: the workspace filesystem line, and the janitor's last run.- If the janitor is
skippedon every run, or its timer is notactive, nothing is being freed: fix that first (§7). - If it runs but frees nothing, the disk is live sandboxes:
status.sh'ssandboxes live, andmocactl sbx ls --allas an admin to see whose. Ask users tomocactl sbx rmwhat they no longer need. - Lower
SBX_MAX_TOTAL(andSBX_MAX_PER_SUBJECT) and re-runsetup.sh, so fewer sandboxes can exist; existing ones are not affected.
12. Uninstalling​
On each KVM host:
sudo <checkout>/deploy/sbx/install-host.sh --uninstall <id> # each worker
sudo <checkout>/deploy/sbx/install-host.sh --uninstall-janitor
They remove exactly what the installs recorded in /etc/sbx/manifest; the last worker's uninstall
also removes the shared sbx-worker@.service, sbx-reap-vms.sh and sbx.slice, the host sampler
(The host sampler), and the moca-egress.service it
installed (Guest egress). Workspace data under /srv/sbx/workspaces/<id> and its jail /srv/sbx/jail/j<12 hex of the id> (the uninstall names it) is left; remove it by hand.
In the cluster, delete what setup.sh applied, then its Secrets and ConfigMaps:
kubectl delete -k deploy/sbx/.generated
kubectl delete secret -n <namespace> moca-redis moca-relay moca-relay-attach \
moca-relay-sandbox-tokens moca-mu1 sbx-token-keys sbx-egress sbx-template-hosts \
moca-relay-tls sbx-exec-tls --ignore-not-found
kubectl delete configmap -n <namespace> moca-settings sbx-settings --ignore-not-found
rm -rf deploy/sbx/.generated
--ignore-not-found covers the Secrets a stack without egress credentials (sbx-egress) or the
template store (sbx-template-hosts) never had. With SBX_PROFILE=combined, the generated folder
is deploy/sbx/.generated-<namespace>/, and moca-supervisor-tls is one more Secret to delete.
kubectl delete -k deletes the template store's and the egress credentials' volumes. They are
resources of the components setup.sh adds when those features are on: moca-sbx-templates (the
catalog, the policy and the provenance records) and the stored egress credentials' claim,
moca-sbx-credentials (moca-credentials on the combined profile). Back them up first if you may want them.
The Redis volume (the StatefulSet's data claim, data-redis-0) is kept by the delete. Remove it
with kubectl delete pvc data-redis-0 -n <namespace> to delete every sandbox record and every
login (the refresh families live there too).
13. Live smoke​
deploy/sbx/smoke.sh runs the spec's live checks (§10.6) against a deployed service. Run it after
a deploy or an upgrade, and once for the 1c acceptance. Without SBX_LIVE_SMOKE=1 it prints one
SKIP line and does nothing. It reads only the environment. It never logs in, never prints a
token, and deletes only the sandboxes it created in that run, even when it fails or is
interrupted.
Prepare two logins, once. You need two GitHub accounts, both enrolled (§4), because step 2
checks that one user cannot reach the other's sandbox. Each login lives in its own HOME:
mocactl sbx login --control-plane-url https://<api host> # user A, in your own HOME
mkdir -p ~/sbx-smoke-b
HOME=~/sbx-smoke-b mocactl sbx login --control-plane-url https://<api host> # user B
Log user B in from a browser session signed in to B's GitHub account (a private window works).
Both logins refresh themselves, so this lasts as long as a login normally does. Leave
XDG_CONFIG_HOME and XDG_STATE_HOME unset: the smoke runs mocactl with them unset.
Run it:
SBX_LIVE_SMOKE=1 SBX_API_URL=https://<api host> \
SBX_SMOKE_HOME_A="$HOME" SBX_SMOKE_HOME_B=~/sbx-smoke-b \
SBX_SMOKE_HOST=<host> deploy/sbx/smoke.sh
ok 1 - round trip: sbx up, run, put, pull --apply on a sample repository
ok 2 - two users: B is refused A's sandbox (404, PERMISSION_DENIED); one req_id from both gets each its own result
ok 3 - a command silent for 5 minutes completes
ok 4 - delete: the mint is refused at once; the kept token is cancelled at exp + leeway, then refused
ok 5 - claude fixes the failing test through mocactl sbx mcp; the tree is unchanged until pull --apply
ok 5 - opencode fixes the failing test through mocactl sbx mcp; the tree is unchanged until pull --apply # SKIP opencode not on PATH
ok 6 - the janitor frees a deleted sandbox's directory
ok 7 - egress credentials: the PAT works from the sandbox without being in it; the placeholder is inert; delete revokes # SKIP SBX_SMOKE_GITHUB_TOKEN_FILE and SBX_SMOKE_GITHUB_REPO not both set
ok 8 - lease mode: one guest across commands; an idle restart is noticed; reset and a downgrade end the lease # SKIP lease not allowed
ok 9 - lease+processes: a tracked process serves later commands; a downgrade to lease stops it # SKIP lease+processes not allowed
It exits 0 when every step is ok or skipped, 1 when any step is not ok (the line gives the
reason), and 2 on a usage or settings error, before any sandbox is touched. On a failure it keeps its logs and prints their path. A full run takes about 25 minutes without
step 6: step 3 runs a command that is silent for 5 minutes, and step 4 waits for a token to expire
(one token lifetime plus the 30 s leeway). With step 6 it is closer to 70–90 minutes, because step 6
waits out the host's own janitor wait (setup.sh keeps it at max(60, SBX_TOKEN_TTL_MINUTES + 5)
minutes, so 65 min with 60-minute tokens). Step 8 adds about a minute where lease mode is allowed,
and step 9 a few seconds where lease+processes is.
| Variable | Default | What it is |
|---|---|---|
SBX_API_URL | (required) | https://<api host> |
SBX_SMOKE_HOME_A, SBX_SMOKE_HOME_B | (required) | the two logged-in HOMEs; they must be two different users |
SBX_SMOKE_GITHUB_TOKEN_FILE, SBX_SMOKE_GITHUB_REPO | (unset: step 7 is a SKIP) | step 7: a file holding a GitHub PAT (a fine-grained one that can push to the repository; one trailing newline is ignored; never an environment variable) and OWNER/REPO |
SBX_SMOKE_CRED_TTL | 60 | the egress credential cache TTL (SH_EGRESS_CREDENTIAL_TTL_SECONDS); step 7 waits this plus 5 s after deleting the credential |
SBX_SMOKE_GITHUB_MCP | (unset) | 1: step 7 also checks the github-mcp preset (api.githubcopilot.com must be in the host's egress allowlist) |
SBX_SMOKE_CRED_TWO_USERS | (unset) | 1, with SBX_SMOKE_HOME_B: step 7 also checks that user B's sandbox gets a 401 and a public clone works; otherwise that check is a SKIP |
SBX_SMOKE_TEMPLATE | default | the template every sandbox is created from |
SBX_SMOKE_STEPS | 1,2,3,4,5,6,7,8,9 | the steps to run; 1,2,5 leaves out the slow ones |
SBX_SMOKE_AGENTS | claude,opencode | the agents step 5 runs, headless; one not on PATH is a SKIP. Each needs its own login and model configured |
SBX_SMOKE_HOST | (unset: step 6 is a SKIP) | an ssh target ([user@]host) for the template's KVM host, with passwordless sudo |
SBX_SMOKE_HOST_ENV_DIR, SBX_SMOKE_HOST_SRV, SBX_SMOKE_HOST_LIBEXEC | /etc/sbx, /srv/sbx, /usr/local/libexec | that host's SBX_ENV_DIR, SBX_SRV and SBX_LIBEXEC_DIR, if install-host.sh was given others |
SBX_SMOKE_JANITOR_AGE | the host's own SBX_JANITOR_AGE | step 6's janitor wait. The one run frees every workspace on the host terminated longer ago than this, other users' too, so it can only be lengthened: a value below the host's wait (which setup.sh keeps above the token lifetime + 30 s) is refused. Step 6 waits until its own deletion is that old: the host's wait, 65 min by default with 60-minute tokens |
MOCACTL | (unset: installed from the edge channel) | a mocactl to test instead, such as a local build (packages/mocactl/dist/mocactl.mjs) |
SBX_SMOKE_PYTHON | python3 | Python 3.10+ with grpcio and grpcio-tools (steps 2 and 4 use docs/api/examples/sandbox_v1_client.py). It also runs the sample repository's tests locally |
Step 7 (egress credentials, SBX_EGRESS_CREDENTIALS=1) runs alone with only user A's login:
SBX_LIVE_SMOKE=1 SBX_API_URL=https://<api host> SBX_SMOKE_HOME_A="$HOME" SBX_SMOKE_STEPS=7 \
SBX_SMOKE_GITHUB_TOKEN_FILE=~/.sbx-smoke-pat SBX_SMOKE_GITHUB_REPO=<owner>/<repo> deploy/sbx/smoke.sh
SBX_SMOKE_HOME_B stays required for steps 1 to 6. Step 7 prints the PAT nowhere and passes it to
mocactl on stdin only; its trap deletes the credentials (smoke-<ts>-<pid>-gh and -gh-mcp, named
per run so two runs as the same user cannot revoke one another), the remote branch
sbx-smoke-<ts> and the sandboxes, even on failure.
Step 8 (lease mode, lease-mode spec §13) also
runs alone with only user A's login. It needs a deployment that allows lease and a template whose
worker's snapshot advertises lease-v1. On the default deployment (SBX_MODES_ALLOWED=per-exec) the
create is refused with 403 mode_not_allowed, and the step is a SKIP (# SKIP lease not allowed):
SBX_LIVE_SMOKE=1 SBX_API_URL=https://<api host> SBX_SMOKE_HOME_A="$HOME" SBX_SMOKE_STEPS=8 \
deploy/sbx/smoke.sh
SBX_MODES_ALLOWED is deployment-wide: listing lease for the smoke lets every enrolled user create
lease sandboxes. Until docs/notes/sandbox-security-review.md §6 is signed off, run step 8 on a test
deployment, or list lease for the run only and set SBX_MODES_ALLOWED back afterwards (§2's
re-run of setup.sh; the next token mint then moves any lease sandbox left to per-exec). The
create is an upgrade, which mocactl takes only from a terminal, so the step runs it under script
(util-linux script -qec on Linux, the BSD script -q on macOS), which must be on PATH.
Step 9 (lease mode, level 2: tracked processes, lease-mode spec
§7) also runs alone with only user A's login. It needs a deployment that allows lease+processes,
and a template whose worker runs the level-2 code on a snapshot that advertises lease-v1 and
lease-processes-v1 and has python3 and curl. A worker without it (no sbx:lease-processes-v1
in its Hello) makes proc start fail ("tracked processes are not available on this sandbox's worker
yet"), a not ok 9. Where lease+processes is not allowed (the
default) the create is refused with 403 mode_not_allowed, and the step is a SKIP
(# SKIP lease+processes not allowed):
SBX_LIVE_SMOKE=1 SBX_API_URL=https://<api host> SBX_SMOKE_HOME_A="$HOME" SBX_SMOKE_STEPS=9 \
deploy/sbx/smoke.sh
Until docs/notes/sandbox-security-review.md §7 (level 2, a draft) is signed off, run step 9 as step 8: on a
test deployment, or with lease+processes listed for the run only and SBX_MODES_ALLOWED set back
afterwards. Its create runs under script too; the rest of the step needs no terminal.
What each step does:
- Creates a sample repository with one failing test, runs
sbx up, and checks thatsbx runfails the suite. Itsbx puts the fix and checks that the suite passes in the sandbox, that the local tree is unchanged and its suite still fails, and that the local suite passes aftersbx pull --apply. - Checks that user B's
sbx tokenandsbx envfor A's sandbox both sayno such sandbox, and that B's token with A's sandbox and workspace key isPERMISSION_DENIED. Then A and B each send the samereq_idand the same command at once, each to a sandbox of their own from the same template, and each must read its own sandbox's file. - Runs
sbx run --timeout 400 -- sh -c 'sleep 300; echo survived', which must printsurvived. - Mints a token and keeps it, deletes the sandbox, and checks that the next mint is refused
(
sandbox was deleted) while the kept token still runstrue. 20 s before the token'sexpit starts a command that would run pastexp. That command must be cancelled (DEADLINE_EXCEEDED) atexpplus 30 s. By this machine's clock, anything from 15 to 60 s afterexppasses, to allow for clock skew. After that, the token must be refused (UNAUTHENTICATED). - For each agent, in a fresh sample repository after
sbx up:sbx doctor's agent-config check must pass. Thenclaude -p "…" --mcp-config .mcp.json --strict-mcp-configoropencode run "…"runs, with the smoke'smocactlfirst onPATH(logged in as user A) for themocactl sbx mcpit starts. The flight recorder must show the agent's calls, andgit status --porcelainmust be the same as right aftersbx up. The local suite must still fail, then pass aftersbx pull --apply. - Deletes a sandbox (step 4's, if that ran), checks that its directory
<SBX_SRV>/workspaces/<worker>/<id>exists on the host, and runs the installed janitor once:ssh <host> sudo -n bash -s, which sourcesjanitor.envand then runs/usr/local/libexec/sbx-janitor.shwith the host's ownSBX_JANITOR_AGE(or a longerSBX_SMOKE_JANITOR_AGE), after waiting until the deletion is that old. The installed timer and its settings are not changed. After the run, the directory must be gone. - Needs the two GitHub variables, else it is a SKIP. User A stores the PAT
(
credentials add <run>-gh --preset github, on stdin from the file). In A's sandbox it clones the repository, commits, pushessbx-smoke-<ts>and deletes it, and checks thatgh api user(orcurl, withoutgh) names the PAT's owner. The placeholder sent from this machine must get a 401. User A deletes the credential, the step waitsSBX_SMOKE_CRED_TTL+ 5 s, and the sandbox must get a 401. Optionallygithub-mcp, and B's sandbox (401, public clone ok). - Creates a sandbox with
sbx create --mode lease --idle-timeout 30 --jsonunderscript, and checks that itsset_modewas applied and thatsbx get --jsonshows the epoch known and no lease yet. It runssh -c 'echo warm > /tmp/lease-smoke', and thencat /tmp/lease-smokemust printwarm: the same guest. After 45 s idle the samecatmust fail (exit 1), with[sandbox] guest restarted (idle, <HH:MM>): ...on stderr.sbx resetmust printlease ended (reset), and after one more commandsbx mode per-execmust printlease ended (mode_change), both on stdout. Then it deletes the sandbox. - Creates a sandbox with
sbx create --mode lease+processes --jsonunderscript, and checks that itsset_modewas applied.sbx proc start --name webofpython3 -m http.server 8080 --bind 127.0.0.1must printp<id> web pid <n>. A latersbx runofcurl -s --max-time 5 http://127.0.0.1:8080/must get the directory listing: the tracked process outlived the command that started it (10 tries, 1 s apart (each try up to 5 s), while the server binds).sbx proc lsmust showwebrunning with that pid, andsbx mode leasemust printstopped 1 tracked process: web (pid <n>)on stdout. The samecurlmust then fail. Then it deletes the sandbox.
What it cannot check for you: the agents' own logs (kept on a failure) show whether an agent
fixed the test by the shortest route, or tried its local tools first and was denied. The
no network in the guest claim is not one of its steps (docs/notes/sandbox-security-review.md §2).
14. Checking the templates​
deploy/sbx/templates-check.sh checks that a client chooses the template its sandbox runs on: the
list, a create of each template, each sandbox's token naming its own template's worker, a command in
each answering on its template's image, a token refused at another template's worker, and sbx up
refusing a bound sandbox of another template. Run it after adding or updating a template. Like the
smoke, it reads only the environment, never prints a token, and deletes only the sandboxes it
created; without SBX_LIVE_CHECK=1 it does nothing.
SBX_LIVE_CHECK=1 SBX_API_URL=https://<api host> \
SBX_CHECK_HOME_A=<home A> SBX_CHECK_HOME_B=<home B> SBX_CHECK_TEMPLATES=default,<template> \
SBX_CHECK_PROBE_<TEMPLATE>='<a command that prints yes only on that template>' \
deploy/sbx/templates-check.sh
The two HOMEs are two enrolled users' logins, prepared as for the smoke (§13). Each
SBX_CHECK_PROBE_<ID> (the id upper-cased, dashes as underscores) runs in every template's
sandboxes: it must print yes in its own template's and anything else in the others'. For a
SWE-bench template, for example, check the testbed's baseline commit and the testbed's Python.
SBX_CHECK_EXEC=0 checks the control plane alone, before any worker is installed. The direct-client
check (a token sent to another template's worker) needs SBX_CHECK_PYTHON, a Python 3.10+ with
grpcio and grpcio-tools, and is a SKIP without it. --help lists every input and check. It runs
on bash 3.2 too.
ok 1 - list: default, <template> (no worker ids)
ok 2 - create: one sandbox per template, each recording its template; an unknown template is refused (…)
ok 3 - routing: each template's sandboxes go to its own worker: default->… <template>->…; two default sandboxes share it, with their own workspaces
ok 4 - hands: each sandbox answers on its template's image: …
ok 5 - isolation: a default sandbox's token sent to <template>'s worker (…) is PERMISSION_DENIED
ok 6 - isolation: user B gets 'no such sandbox' for user A's default sandbox (env and token)
ok 7 - up: a repository bound to a default sandbox refuses sbx up --template <template>
It exits 0 when every check is ok or skipped, 1 when any is not ok (the line gives the
reason), and 2 on a usage or settings error, before any sandbox is touched.
The template host agent​
The self-serve template build of the
template lifecycle spec
§6.1, and the activation of §6.3. deploy/sbx/host-agent.sh --once is one poll of a KVM host's
agent: it reports what the host holds, claims queued template builds (clients create them with
POST /v1/templates), runs template-build.sh on each with its gate record, and reports the
result; then it starts and stops installed workers by the host's wanted state, and removes the
retired builds it is told to (worker and snapshot). The control plane
registers or rejects each built build by policy; nobody approves it. It needs:
- on the control plane, the template store and this host's bearer: run
setup.shwithSBX_TEMPLATE_STORE=1 SBX_TEMPLATE_HOSTS='<host id> ...'. It adds the store's PVC (moca-sbx-templates, mounted asSH_PROVENANCE_DIR;SBX_TEMPLATES_FILEis imported into it once), a token per host in the Secretsbx-template-hosts(SH_HOST_TOKEN_<ID>, generated once and kept; a host dropped from the list has its token revoked), and a bundle per host in.generated/host-agent/<id>/(host-agent.env,host-agent.token, andapi-ca.crtwithSBX_API_CA_FILE): copy it to the host as/etc/sbx/host-agent.envand/etc/sbx/host-agent.token, addSBX_KERNEL(andSBX_API_CA_FILE, if the bundle has a CA) to the env file, and delete the copy; - on the host,
/etc/sbx/host-agent.env(SBX_API_URL,SBX_HOST_ID,SBX_KERNEL, optionallySBX_API_CA_FILE,SBX_DISK_FLOOR_GB,SBX_BUILD_CONCURRENCY,SBX_ACTIVE_LIMIT,SBX_IDLE_LIMIT_S,SBX_MEM_FLOOR_MIB,SBX_MEM_PRESSURE_LIMIT(see "Memory-aware placement" below),SBX_START_CONCURRENCY,SBX_START_WINDOW_S,SBX_IDLE_GRACE_S(see "Cold-start pacing" below),SBX_REGISTRY), the token alone in/etc/sbx/host-agent.token(0600, root), the script at/usr/local/libexec/sbx-host-agent.sh, and in/etc/systemd/systemsbx-host-agent.servicewith its timer (every 2 minutes) andsbx-host-agent-wanted.servicewith its timer (every 15 s, the wanted state alone); thensystemctl daemon-reloadandsystemctl enable --now sbx-host-agent.timer sbx-host-agent-wanted.timer. The unit setsSBX_TEMPLATE_BUILDandSBX_INSTALL_HOSTto the checkout at/srv/sbx/src(the agent cannot find them from/usr/local/libexec) andREGISTRY_AUTH_FILE=/run/containers/0/auth.json(it hides root's home): log root's podman in with--authfile /run/containers/0/auth.json. WithSBX_REGISTRYset, no build is claimed until that file holds a login for it. The unit also gives Go a home and caches under/srv/sbx(build-snapshot.shcompiles the guest agent): runinstall -d -m 0700 /srv/sbx/go /srv/sbx/homefirst. Do not setGOFLAGS=-mod=mod: the repository is a Go workspace, which refuses it. Run a poll through the unit (systemctl start sbx-host-agent.service), not by hand: only the unit loads the env file.
It never claims a build below SBX_DISK_FLOOR_GB of free disk (default 60), claims at most
SBX_BUILD_CONCURRENCY builds a poll (default 1, at most 4), and checks a gate record's bytes
against their sha256 before building with it. On a host with egress credentials on (the --egress drop-in exists; see
Credential injection), a spec with egress: true is built with
--egress-ca /etc/sbx/egress/ca.crt, so its snapshot trusts the host's interception CA and the
worker install's CA check passes; with credentials on and no CA file, the build is reported failed
(step: egress). The CA is part of the build's identity (its manifest hash), so turning credentials
on, or rotating the CA, needs a rebuild of the egress templates the host already holds. The token stays in a 0600 header file, never in argv
or the log. A bad setting is refused (exit 2) before any call. Each run ends with one summary line,
{"agent":"summary","claimed","built","failed","started","stopped","removed","errors"}, in the unit's
journal, with "psi":"absent", "sliceUnread":[...] and "memfileMismatch":<n> when there is
something to say about memory, "startQueued":<n> when the start limit left wanted workers for a
later run, and "graced":<n> when the idle grace keeps any.
Nothing that names where an image lives leaves the host. A failed build's reason and its
uploaded log are redacted (SBX_REGISTRY exactly, then URLs, registry hosts, IP addresses and
12-digit account ids); the full log stays on the host in /srv/sbx/agent-logs/ (0700). The reason
leads with a code a client can branch on: pull_failed, probe_failed, disk_floor,
spec_refused, build_failed, or leak_refused (step report): the template record or manifest,
which go byte for byte, names SBX_REGISTRY, so the build is reported failed and neither is sent.
Name the image in the spec without its registry; SBX_REGISTRY supplies it at the pull. A build
the agent claimed but cannot carry on with is reported failed agent_error, with the agent's own
fixed wording and its step (gate: a gate record it could not fetch, or whose bytes do not match;
report: a report the control plane did not take, or no template record or manifest after the
build), so it does not stay building. Text that
reached the store before a redaction rule existed is removed offline with
template-onboarding/redact-provenance.py (its --catalog-only mode for a build's stored log).
Activation: the agent runs the workers the control plane wants. A client takes a lease on a
registered build (POST /v1/templates/{id}/activate, or mocactl sbx templates activate ID), or a
sandbox create on an inactive build takes one (activation on first use: 409 template_not_active
with Retry-After, retried by mocactl). A build is wanted running only while a lease stands. The
templates declared in SBX_TEMPLATES_FILE (SH_SANDBOX_TEMPLATES) are not affected: their workers
are the deployment's to run, they take no lease, and a create on one is never gated. Each
host's wanted state (GET /v1/hosts/<id>/wanted) lists the leased builds placed on it, at most
SBX_ACTIVE_LIMIT of them, in run, and the builds it holds with no lease and no live sandbox in
stop. After each build poll, and every 15 s in sbx-host-agent-wanted.service (host-agent.sh --wanted-only: no claim, no build), the agent stops each stop worker whose unit is active, then
starts each run worker whose sbx-worker@<id> unit is installed and not active, and reports each start,
refusal and stop (attached, refused, detached) as provenance; readiness is the relay's
attached record, never these reports. A worker with a live sandbox is never stopped: it drains
first. A lock (/srv/sbx/.activation.lock) keeps the two units from starting or stopping workers at
once; the one without it skips, saying so. A host whose last inventory is over 6 minutes old (three
build polls) is left out of placement and of GET /v1/templates' hosts until it reports again, so
a stopped or removed agent takes no new placement and no slot. Size the limits:
SBX_ACTIVE_LIMITis the most workers the host runs at once: each pins its snapshot's guest memory whole from its start, with its standby VMs. With an agent that reports memory (below), the control plane also keeps new starts within the host's memory, so the limit is an upper bound, not a memory calculation. 0 (unset) places nothing on the host. Every worker that stays running counts against it: the leased builds placed or running on the host (the inventory says which units run; a running build stays on its host), and an unleased one still draining a live sandbox, until it stops.SBX_IDLE_LIMIT_Sis the default lease length a client gets when it names none (the largest of the hosts' that hold the build, clamped to 60-86400 s; else 3600 s). A lease is renewed by activating again.- The policy's
quota.activecaps the builds one subject holds a lease on at once (429 quota_exceeded,quota: active);GET /v1/quotashows the use.
A refused activation keeps no lease: 409 template_unavailable (no registered build, no host
that reports holds it, or reason: memfile_mismatch) or 409 no_active_slot with Retry-After
(every host holding it is at its limit with leased builds, or reason: memory; nothing leased is
evicted). The activation check and lease write run one at a time, under a lock in the control
plane's process: keep one control-plane process per template store (the Deployment's one replica,
the store's ReadWriteOnce volume).
Memory-aware placement (template lifecycle spec
§6.2): each inventory also reports the host's memory, and the control plane starts a new worker on a
host only if the host has the memory for it, whatever SBX_ACTIVE_LIMIT says. SBX_ACTIVE_LIMIT
stays the upper bound. Nothing running is stopped: the rule only holds back new starts.
- What the agent reports (the inventory's
memoryobject):memTotalMibandmemAvailableMibfrom/proc/meminfo;memPressureFull10, the memory PSI line'sfull avg10from/proc/pressure/memory(a percentage: the share of the last 10 s in which every non-idle task was stalled on memory); the workers of held builds whosesbx-worker@<id>runs; per running worker,vmCommittedMib, its slice'smemory.stat(sbx.slice/sbx-<id>.slice) anon + shmem + kernel: its VMs' memory the kernel cannot reclaim, never its page cache (a memory file is file-backed and mlocked, so the worker's cost counts it; keep/srv/sbxon a disk volume, never tmpfs: a memory file on tmpfs is shmem, and would be counted twice); andmemFloorMib, the effective floor, so the number placement uses is in the inventory and inGET /v1/templates' hosts block (memory.floorMib), with no arithmetic. SBX_MEM_FLOOR_MIB: the memory never placed into, a whole number of MiB. Unset, it is 10% of MemTotal, at least 4096 (a host of 256 GiB: 26214 MiB). Set a fixed amount on a host shared with other work (Moca's own P6 sessions).SBX_MEM_PRESSURE_LIMIT(default 10): whilefull avg10is above it, the host takes no new start; what it runs is kept. No/proc/pressure(an older kernel, orpsi=0at boot): the field is left out, which is no brake, never "no starts"; the agent's summary says"psi":"absent"once per run.- Both live in
host-agent.env(0600), never in argv. A bad value (a floor of 0, a leading zero, a limit that is not a number in (0, 100] with at most two decimals) is refused like the agent's other settings: exit 2, before any call. - A worker's cost is its guest's RAM (the registered build's guest size, which registration
checked against the manifest's
guest_ram_mb) plus 128 MiB; a build with no recorded guest size counts at 4096 + 128. A host's headroom ismin(total − floor − used, available − floor − pending),usedbeing its running workers' cost, their VMs' committed memory and the starts placed there but not running yet. The hosts block showsusedMibandheadroomMib. - A stale host (its last inventory over 60 s old: four missed 15 s reports) takes no new start
(
brake: stale); one under pressure showsbrake: pressure. - The memory-file check: for each build it holds, the agent checks that the snapshot's
memfileis exactlyguest_ram_mb × 1 MiB(the manifest beside it). A worker pins that file whole, so a different size (a future Diff snapshot, a balloon, a trimmed file) would make its cost wrong. A mismatch is an error line naming the build, a"memfileMismatch":<n>count in the summary, and the build in the inventory'smemfileMismatch: the control plane never places it on that host (409 template_unavailable,reason: memfile_mismatch, when every holder reports it), and the agent refuses to start it (reportedrefused). Rebuild the snapshot. - A slice it cannot read (
memory.statmissing, or with noanonline): that worker's VMs are left out ofvmCommittedMib, never guessed, and the summary names it ("sliceUnread":[...]). - What a client sees: an activation refused for memory is
409 no_active_slotwithreason: memoryand the usualRetry-After; at the active limit it answers as before, with no reason. A build waiting for memory readsreadiness: unavailablewithreadinessReason: memory(mocactl sbx templates show ID), andmocactl sbx hostsshows each host's headroom, floor and brake. - With the sandbox lease mode, a lease VM's guest memory is in its worker's slice, so
vmCommittedMibcounts it. Its reclaim acts on VMs and this floor on whole workers: on one host, set the floor at or above lease mode's reclaim target. - Version skew, both ways: an agent older than memory-aware placement sends no
memory, and the control plane places that host bySBX_ACTIVE_LIMITalone, as before. A control plane older than it (openapi 1.12.0) ignores the field and places bySBX_ACTIVE_LIMITalone; the agent still refuses to start a build whose memory file does not match. Upgrade either first; list both in the rollout.
Cold-start pacing (template lifecycle spec
§6.5): before it attaches to the relay, a worker reads and locks its whole snapshot memory file, and
workers that start together share the host's disk. So the agent starts at most
SBX_START_CONCURRENCY workers at once (default 4).
- What it caps: workers starting (started, not attached yet), not running ones. It is not
SBX_ACTIVE_LIMIT, which caps running workers. Each run starts at mostSBX_START_CONCURRENCYless those still starting, oldest lease first, and leaves the rest for a later run (every 15 s); the summary counts them ("startQueued":<n>). A run never waits for a start to finish:systemctl startof the worker unit returns at once. - The setting: a whole number, at least 1, in
host-agent.env(0600), never in argv. 0 is refused (exit 2, before any call), as is a leading zero or anything that is not a number. - Where "starting" comes from: the control plane's wanted answer lists, in
starting, the builds placed on this host whose worker the relay does not show attached. The limit is the agent's, so it holds with an older control plane too (openapi before 1.13.0, nostarting): the agent then counts as starting the workers it started less thanSBX_START_WINDOW_Sago (default 300) that are still running and wanted. Its starts are kept in/srv/sbx/start-times(0600). - A stuck start: either way, a start counts for
SBX_START_WINDOW_Sat most. A worker still not attached after it (a crash loop, a hang, a relay it cannot reach) is stuck: each run says so in an error line (worker <id> started <n> s ago and is not attached) and it no longer holds a start slot, so the line moves. Look at that worker's unit (journalctl -u sbx-worker@<id>). KeepSBX_START_WINDOW_Sabove the host's slowest start in a full wave, or a slow start is taken for a stuck one and one more start shares the disk. - What it bounds: each start's wait, not the total. The disk sets the total, at any width. Measured on one KVM host whose snapshots sit on a general-purpose cloud block volume (about 125 MB/s), 4 GiB guests, starts measured after a release (cache not warm): cold starts one at a time took 78 s on average (at most 120 s), a wave of four about 3-4 minutes, and with 28 started at once the last attached after more than twenty minutes. Raise the limit only on a faster disk.
- What the agent reports:
startConcurrencyandstartSecondsin the inventory's limits.startSecondsis the mean of its last 10 cold starts, start to attach, to within the 15 s poll (kept in/srv/sbx/start-seconds, 0600); it is absent until a start is done, and always with an older control plane. The hosts block ofGET /v1/templatesshowsstart: {concurrency, seconds, starting, queued}(mocactl sbx hosts). - What a client sees: a build waiting for a start slot is
activatingwithqueued: trueand itsqueuePosition(1-based, on its host).409 template_not_active's Retry-After becomesceil(ceil(position / concurrency) × startSeconds)once the host has measured a start, else 30 s as before. A queued activation staysactivatinglonger than before, so a client polls readiness with a timeout sized for the host's whole start line;mocactl sbx templates activate --waitsizes its default from it ((ceil(position / concurrency) + 1) × startSeconds × 1.25, at least 1800 s, at most 4 h;--timeoutwins). - Measure cold starts with the page cache dropped (
echo 3 > /proc/sys/vm/drop_caches, or after a reboot): a start right after its build, or after a recent stop, reads its memory file from the cache and looks much faster than a cold one. - Placement keeps a build on the host already running its worker (no start at all; sticky), whether or not that host reports its memory; a graced build leased again stays there too.
- The one change on upgrade: a host whose agent is upgraded, with no new settings and whatever
its control plane, now starts at most 4 workers at once instead of every wanted one; a wide
activation starts in waves of
SBX_START_CONCURRENCY, each about one start time long (a slot frees when a worker attaches, or, with a control plane older than 1.13.0, afterSBX_START_WINDOW_S), so 28 builds at the default take about seven waves, many minutes, and a client's first Retry-After is the old 30 s until the host has measured a start. A control plane older than 1.13.0 ignores the new inventory fields; an older agent sends none, and the control plane then shows no queue and answers Retry-After as before.
The idle grace (part of cold-start pacing, openapi 1.13.0): a released worker can stay up for a while, so a template used again soon is warm (no memory-file read at all).
- The setting:
SBX_IDLE_GRACE_Sinhost-agent.env(0600), never in argv. Default 0: off, and the host behaves exactly as before: a build with no lease and no live sandbox is stopped at the next poll. A whole number of seconds, 0 to 86400, no leading zero; anything else is refused (exit 2, before any call). The inventory reports it (limits.idleGraceSeconds). It is a host setting, not a client's choice. - When to use it: long enough to cover the gaps between a workload's rounds on the same
templates. Leave it off where each task's cold start is what is measured, or use
stopNow(below). - What it does: after a build's last lease ends (its release, or the lease running out), the
control plane keeps it in the wanted answer's
rununtil that end plusSBX_IDLE_GRACE_S, and lists it ingraced. It is graced only if its worker runs on the host (as the agent reports it), has no live sandbox, and the host has room for it: a slot underSBX_ACTIVE_LIMITbeside the workers the host counts (leased and draining ones) and memory-aware placement's headroom. The agent never starts a graced build; it only keeps one running. A use within the grace finds the worker attached and gets a sandbox at once (first use takes a lease; no409). - A wanted build that needs the slot (or the memory) displaces the graced worker that ends
soonest: it goes to
stop. That is the only stop of a worker nobody asked to stop, and only of one with no lease. At stop time the control plane re-checks that it is still unleased (under the activation lock): a lease taken in between keeps it running. The agent logs such a stop as{"agent":"stopped","worker":"<id>","was":"graced"}. stopNow, to force cold:POST /v1/templates/{id}/deactivate {"stopNow": true}(mocactl sbx templates deactivate ID --stop-now) releases the caller's lease and, if no lease remains, skips the grace: the worker is stopped at the next poll. On a build already in its grace it ends the grace (released: 0). Thedeactivaterecord carriesstopNow: true. Use it for one-by-one comparisons, so each task still measures its cold start.- Counting releases in the records: a host's worker reports share record kinds with clients'
calls:
attachedandrefusedare written asactivate,detachedasdeactivate, each with subjecthost:<id>and the report inbody.event. So a reader counting a client's releases or activations filters on the subject (or onbody.eventbeing absent). - What you see: the agent's summary line gains
"graced":<n>when it keeps any; the hosts block ofGET /v1/templatesshowsidleGraceSecondsandgraced(a count) per host with a grace (mocactl sbx hosts); a graced build readsdeactivatingwithgraced: trueandgracedUntil(epoch seconds;mocactl sbx templates show IDsays "graced until ..."). - What it costs: a graced worker still pins its whole memory file (it is a running worker).
Memory-aware placement counts it in
memory.usedMib, and it holds a slot ofSBX_ACTIVE_LIMITuntil a wanted build needs one. A host over its memory with its graced workers lets the soonest-ending go first. - Version skew: an agent without the setting reports no
idleGraceSeconds, which the control plane takes as 0 (off). A control plane older than 1.13.0 ignores the field, sends nograced, and stops released workers as before; the agent then just appliesstop. Roll out the control plane before settingSBX_IDLE_GRACE_Son a host: the order is the control plane, then the agent, then the setting.
Retired builds are removed by the agent. Once a client retires a template or a build (POST /v1/templates/{id}/retire, DELETE /v1/templates/{id}/builds/{build}, or mocactl sbx templates retire), each host holding it gets it in its wanted state's stop and remove as soon as no live
sandbox runs on its worker and no pinned session's hold keeps it; a worker whose sandbox ended less
than 600 s ago is in stop only until then, so the host still ships that sandbox's last egress
events (retire --wait's default --timeout, 900 s, covers that window). In the same run the agent stops the worker, runs install-host.sh --uninstall <worker> --keep-shared (SBX_INSTALL_HOST), deletes the snapshot directory and its template record
under /srv/sbx, and reports removed: the template's provenance gains a retire record by
host:<id>, and the build's heldBy (GET /v1/templates/{id}/builds/{build}) no longer lists the
host. It deletes only what its own template record names for that build id and worker, never a
running worker's; a name no record matches is an error line and deletes nothing. The worker's
workspace data stays for the janitor, as with any uninstall. --keep-shared removes the worker's own
files only: even the host's last worker leaves the shared units (sbx-worker@.service, sbx.slice),
moca-egress and its credential injection in place for the next worker, so a client's retire never
undoes an operator's --egress. To remove them from a host with no worker left, run install-host.sh --uninstall <that worker> again without it (§12). Also re-run setup.sh without the
worker's id in SBX_WORKERS (§5) to revoke its relay token. mocactl sbx templates retire ID --wait returns once the worker has stopped and every holding host has removed it; a host that is
down removes it at its next poll.
A registered build's worker is still installed by an operator. The agent builds, reports, and
starts and stops installed workers; it installs none, and a wanted build whose worker is not
installed is reported refused (worker not installed). A worker's relay token is minted only by
setup.sh, so for a new
template: add a worker id to SBX_WORKERS and the template to SBX_TEMPLATES_FILE, re-run
setup.sh, and install the worker on the host that built the snapshot, as in §5, with
--snapshot <snapshot> (the record's snapshot, $SBX_SRV/templates/<snapshot>.json).
install-host.sh then names the record in the worker's env (SH_TEMPLATE_RECORD) and its id as
SANDBOX_TEMPLATE, and the worker checks the record against its snapshot and its own id at every
start, refusing a mismatch. The worker runs the sandbox service's own binary, SBX_WORKER_BIN (default
/usr/local/libexec/sbx/microvm-worker: CGO_ENABLED=0 go build -o /usr/local/libexec/sbx/microvm-worker ./remote-worker/cmd/microvm-worker), never
/usr/local/bin/microvm-worker, which a P4 worker on the same host may run; a missing one is
refused. Letting the agent obtain a worker token from the control plane is a follow-up.
Known limits: a claimed build is not re-queued if the agent dies mid-build (cancel it, POST /v1/templates/{id}/builds/{build}/cancel, and create it again); one poll builds up to
SBX_BUILD_CONCURRENCY builds, and builds are not scheduled across hosts.
To remove the agent: systemctl disable --now sbx-host-agent.timer sbx-host-agent-wanted.timer,
delete the four units,
/usr/local/libexec/sbx-host-agent.sh, /etc/sbx/host-agent.env and /etc/sbx/host-agent.token,
and systemctl daemon-reload; drop the host from SBX_TEMPLATE_HOSTS and re-run setup.sh to
revoke its token.
Provenance records: export, verify and redact​
Every template change is a provenance record in the template store (SH_PROVENANCE_DIR; on
deploy/sbx, the PVC moca-sbx-templates, mounted at /var/lib/moca/templates in the
control-plane container of the moca-control-plane Deployment). Nothing is signed: each record
is canonical JSON named by its sha256, and names the digest of the record before it, so an edit to
a stored record breaks the chain. Keep a copy of the heads elsewhere: that is what proves an export
complete.
The store is append-only:
records/<hex>.json: one file per record, its canonical bytes; the file's sha256 is its name.heads/<hex>.json: per stream,{stream, head, count};<hex>is the sha256 of the stream's name. A stream ispolicy,membersortemplate/<id>.blobs/<hex>: the documents records cite by sha256, byte for byte (gate records, template records, snapshot manifests).
A record is written before its stream's head moves, so a crash leaves an orphan record, never a
head naming a missing one. A record (moca-provenance-record/1) is {schema, stream, seq, kind, prev, at, subject, body}: prev is the previous record's digest (null at seq 1), at is UTC in
whole seconds, and subject is who wrote it: a subject, policy (an automatic decision),
host:<id> or janitor.
| Kind | Written when |
|---|---|
policy | an admin's PUT /v1/policy or PUT /v1/quota/…; the default policy, at its first use |
create, gate | an admitted create or new build (gate when it sends a gate record) |
check | the policy check of a create, and of a build at registration (a refused create on a new or another subject's id writes none) |
build | a host's built or failed report; a cancel |
activate, deactivate | a lease taken or released; a host's attached or refused report (activate) and detached report (deactivate) |
tag, visibility, owner | a tag moved or removed, a visibility or description change, a transfer |
retire, reject | a retire or a reject; a host's removal of a retired build (retire by host:<id>); a rebuild superseding a build (retire by policy) |
sandbox | a sandbox's create and end (deleted, or expired by the janitor); its egress events arriving after the end |
session | a pinned Moca Pi session's create and delete |
member, role | enrollment and admin-role changes, on the members stream |
The kind use is defined, but nothing writes it yet.
Export. There is no export route. Export by hand, from a copy of the store:
-
Copy
heads/, thenrecords/andblobs/, off the volume into an empty directory, saystore-copy/(for example withkubectl cpfrom the control-plane pod) (unverified on a cluster). Heads first: every head then names a record the copy holds. -
If a gate record a
gaterecord cites is missing fromblobs/, add it there asblobs/<sha256 hex>: the export leaves out a blob the store does not hold, and the verifier then fails. -
From a checkout at the control plane's version, with its dependencies installed, export every stream the copy has a head for, using the control plane's own
ProvenanceStore.export:cat > /tmp/export-provenance.mts <<'TS'import { readdirSync, readFileSync } from 'node:fs';import { join } from 'node:path';const dir = process.argv[2];const { ProvenanceStore } = await import(join(process.cwd(), 'src/provenance.ts'));const streams = readdirSync(join(dir, 'heads')).map((f) => JSON.parse(readFileSync(join(dir, 'heads', f), 'utf8')).stream,);process.stdout.write(JSON.stringify(await new ProvenanceStore({ dir }).export(streams)));TScd packages/control-planenpx tsx /tmp/export-provenance.mts /abs/path/to/store-copy > EXPORT.jsonThe result is one
moca-provenance-export/1document: the streams, the policy stream they cite, and every gate record theirgaterecords cite.
Verify anywhere with Python 3, no moca, cluster or network needed:
python3 deploy/sbx/template-onboarding/verify-provenance.py EXPORT.json \
--head 'template/<id>=sha256:<the head you kept>'
It checks every record's digest, every stream's seq and prev chain up to its head and count,
every blob against its sha256, and that each check cites a policy record and each gate a blob of
the export. An export can only prove itself consistent; --head STREAM=sha256:… (repeatable)
compares a stream's head with one kept elsewhere, which catches an export that dropped its latest
records. --egress <sandbox id>=<file> checks a saved egress log against the sandbox's record
(Egress records). Exit 0 verified, 1 a check failed (each printed), 2 a usage
error or an unreadable file.
Redact only data that must not be kept: it rewrites the chain, and the redaction itself is
logged. Stop the control plane first (one process owns the store), and run the script against the
store's directory (STORE_DIR, the volume's root) (unverified on a cluster). Without --apply it
prints what it would change, counts only, and changes nothing; patterns are never printed.
# One template's stream (template/<id>) and its catalog entry:
python3 deploy/sbx/template-onboarding/redact-provenance.py STORE_DIR TEMPLATE_ID \
--pattern 'REGEX' [--pattern 'REGEX' ...] [--apply]
# The catalog only: every catalog entry (catalog/*.json), no provenance record touched:
python3 deploy/sbx/template-onboarding/redact-provenance.py STORE_DIR --catalog-only \
--pattern 'REGEX' [--pattern 'REGEX' ...] [--apply]
- The stream form replaces each match in a record's string values (never its keys) with
<redacted>, rewrites that record and every later record of the stream (each names the one before), moves the stream's head, and redacts the template's catalog entry the same way. --catalog-onlytakes no template id. It redacts every catalog entry, and no record: for text that reached only the catalog, such as a build's stored log, which no record copies.- With
--apply, both append a line toSTORE_DIR/redactions.log, never the matched text: the stream form names the stream, its old and new head and how many records it rewrote;--catalog-onlyhow many entries it redacted. - An export taken before a redaction no longer verifies against the new head: by design. Keep the new head in place of the old one.