Skip to main content
Version: next

Single VM

Preview. The single-VM deployment is implemented but not yet declared Stable; interfaces and limits may change. Known limits: What round one does not claim.

Round one's target for the P6 (process-manager) runtime (spec Β§4.3, Β§4.4, step 4): a single Linux VM running the supervisor, the relay and the MU1 control plane under systemd, with Redis and the sandbox containers as podman containers alongside them. The Knative/KEDA deployment path this once contrasted with has been removed (ADR-0041).

P4 microVM tier on the same host: see MicroVM tier (P4).

Prerequisites​

  • A Linux VM with systemd and podman installed, with a netavark network backend that enforces isolate=strict. setup-vm.sh refuses a podman that rejects the option, but cannot detect one that stores it without enforcing it. Isolation was verified live on netavark 1.17.2.

    • Ubuntu 24.04 packages netavark 1.4.0: podman stores isolate=strict, then the first sandbox fails to start with netavark: unable to parse "isolate": provided string was not true or false, after the env files, units, Redis and the network are already in place. Install a netavark of 1.17.2 or later first. The static release binaries (netavark.gz and aardvark-dns.gz from the netavark and aardvark-dns releases) work when installed as /usr/local/libexec/podman/netavark and /usr/local/libexec/podman/aardvark-dns, which podman searches before /usr/lib/podman; podman info then reports the new version.
    • Some distributions package no podman; Amazon Linux 2023 is one. A static build such as podman-static installs under /usr/local, and sudo's secure_path there drops /usr/local/bin. setup-vm.sh then stops with "missing required commands: podman" and names the fix: run it as sudo env PATH="/usr/local/bin:$PATH" ./deploy/vm/setup-vm.sh. The whole flow, the control plane included, was verified on Amazon Linux 2023 (systemd 252) with podman-static 6.1.2.
  • systemd 247 or newer (LoadCredential=). setup-vm.sh checks and refuses an older one (RHEL 8 ships 239), which would ignore the key and leave the control plane without its secrets.

  • nft (nftables), which loads the sandbox network's firewall

  • Node.js 22+ and pnpm 9+ on the VM (the supervisor and relay run directly via node --import tsx, not containerized)

  • A user able to install systemd units under /etc/systemd/system and run as root (see "Bring it up" below)

  • A system user and group named harness (all three units run as User=harness/Group=harness): e.g. sudo useradd --system --no-create-home --shell /usr/sbin/nologin harness

  • For logins: a GitHub OAuth app with device flow enabled (it is off by default), and outbound HTTPS from the VM to github.com and api.github.com. Its client id goes in control-plane.env (SH_GITHUB_CLIENT_ID), by hand or from setup-vm.sh's environment (see "The GitHub OAuth app" below). Without one, the control plane is installed but not started.

  • The workspace built. ExecStart=node --import tsx src/main.ts needs tsx (a devDependency) and the workspace's link: targets resolved, and those only exist after the checkout is built. Run, in order (spec Β§9), once per checkout:

    git submodule update --init --recursive
    cd pi-fork && npm ci && npm run build && cd ..
    pnpm install

    setup-vm.sh checks for this and refuses to continue with a clear message if it is missing β€” it does not run the build itself, since it can take minutes and does not belong inside a bring-up script.

Upgrading an existing VM​

Re-run setup-vm.sh after pulling; do not just restart the relay. Since MI1 S1 the relay refuses to boot without MOCA_RELAY_EXEC_TOKEN, which git pull && systemctl restart sh-relay.service never creates. sudo ./deploy/vm/setup-vm.sh generates it, migrates relay.env and supervisor.env together to the loopback exec listener, installs the sandbox network and firewall, and restarts what it owns (step 8 below, and the paragraph after it).

Since #366 a re-run also installs the control plane. It writes control-plane.env, installs sh-control-plane.service, generates the MU1 secrets once into /etc/serverless-harness/credentials/, and wires supervisor.env: it adds the public keyset (SH_SESSION_TOKEN_PUBLIC_KEYS) and SH_CONTROL_PLANE_URL, and sets SH_REQUIRE_AUTH=true. From then on, /turn without a session token is refused. A driver that calls it bare, such as an E8 rung, needs SH_REQUIRE_AUTH=false set back in supervisor.env; later re-runs keep that choice.

Async turns (optional). Uncomment SH_ASYNC_DRAIN=1 in supervisor.env and restart sh-supervisor. Every worker then also drains the queue that the control plane's POST /v1/sessions/{id}/turns writes to, so a turn survives the caller disconnecting. No other setting changes: the claim reuses the SH_CONTROL_PLANE_URL and SH_EXCHANGE_TOKEN this VM already wires. The drain shares SH_TURNS_PER_WORKER with sync /turn β€” there is no reserved headroom, so a deep queue can occupy every slot. async_claim_denied_full on the admin /metrics counts the claims that were refused for that reason. It also shares SH_TURNS_PER_SUBJECT with sync /turn, so one owner's running fan-out cannot also fill the worker with interactive turns: an entry claimed past that owner's cap is handed back unrun (counted as async_deferred_subject) and waits out the 90 s reclaim window. That cap does not pace an async-only drain β€” a worker runs one queued leaf at a time, so queued work alone never reaches it, and one owner's deep backlog is still served before another owner's single turn. MOCA_ASYNC_RUNNING_PER_OWNER is the cap that does bound an async-only drain, across every worker, because it counts in Redis rather than in one process; it is unset by default, which is right for a single-user VM, and a deferral costs the entry the same 90 s.

The crash path has its own test, skipped unless you ask for it -- it waits out the production 90 s reclaim window, so it cannot run in CI:

SH_ASYNC_LIVE_CRASH=1 REDIS_URL=redis://127.0.0.1:6379 \
SH_CONTROL_PLANE_URL=http://127.0.0.1:8090 SH_EXCHANGE_TOKEN=... ANTHROPIC_API_KEY=... \
packages/supervisor/node_modules/.bin/vitest run test/real-worker.integration.test.ts

It kills a worker mid-leaf and asserts the turn finishes exactly once -- not merely that it finishes, because a leaf that runs twice also ends with the turn done.

A hand-written MU1 line from the previous mocactl README (for a control plane hosted elsewhere) makes the script stop and name it. SH_EXCHANGE_TOKEN in supervisor.env (or any of the three secrets in either env file) is now a systemd credential: move the value into its file under credentials/ (mode 0600), or delete the line to have a new one generated. A SH_SESSION_TOKEN_PUBLIC_KEYS line with no private key beside it is half a keypair: delete it, and the re-run generates a matching pair. A hand-set SH_CONTROL_PLANE_URL is replaced by this VM's on the run that generates the pair.

An upgraded supervisor.env keeps the old template's "MU1 caller authentication" comment block and its commented #SH_EXCHANGE_TOKEN= line. Both are inert (a comment is never read, and the exchange token now comes from credentials/) and can be deleted.

Bring it up​

cd /opt/serverless-harness # this checkout, on the VM, already built (see Prerequisites)
sudo ./deploy/vm/setup-vm.sh

A first run takes two invocations, by design. The script installs the env files and then refuses to go further until SH_RELAY_TOKEN is set in the relay.env it just wrote (the relay's token validation is fail-closed, so starting containers before that guarantees sandboxes that can never attach β€” see "Sandbox container networking and the relay token" below). So on a fresh VM:

sudo ./deploy/vm/setup-vm.sh # writes the env files, then stops at the token check
sudoedit /etc/serverless-harness/relay.env # SH_RELAY_TOKEN=<a shared secret>
sudoedit /etc/serverless-harness/control-plane.env # SH_GITHUB_CLIENT_ID, SH_PUBLIC_HARNESS_URL
sudoedit /etc/serverless-harness/supervisor.env # SH_TURNS_PER_WORKER
sudo ./deploy/vm/setup-vm.sh # installs units, starts containers, enables the services

The control-plane pair can instead be passed to either run, as sudo env SH_GITHUB_CLIENT_ID=… SH_PUBLIC_HARNESS_URL=… ./deploy/vm/setup-vm.sh. The script writes each one only where control-plane.env has none (see "The GitHub OAuth app" below).

The first invocation exits non-zero with a message naming SH_RELAY_TOKEN and the file. That is the expected first-run path, not a failure to debug. The second invocation keeps the env files you edited (install_env never clobbers an existing one) and continues past the check. Only the relay token blocks the second run. Without the control-plane settings (for SH_PUBLIC_HARNESS_URL behind a tunnel, see "The control plane" below) the control plane is installed but not started; the supervisor is never started by the script, and SH_TURNS_PER_WORKER is what lets you start it (below).

Across those two runs, the script does the following, in this order:

  1. Writes /etc/serverless-harness/supervisor.env, relay.env and control-plane.env from their env/*.example templates β€” only the first time each; an operator-edited env file is never clobbered on a re-run. Then, if SH_GITHUB_CLIENT_ID or SH_PUBLIC_HARNESS_URL is set in the script's environment, writes it into control-plane.env where that file's value is empty (never replacing one; a disagreement is warned about), after checking both values first.
  2. Checks relay.env for a non-empty SH_RELAY_TOKEN, and stops here if there is none. Everything below runs only once that is set β€” which is why a fresh VM needs the second invocation above.
  3. Generates MOCA_RELAY_EXEC_TOKEN β€” the supervisor's credential for the relay's SandboxExec β€” into both env files when absent, one value, never replacing an existing one; and checks that the relay's exec listener (MOCA_RELAY_EXEC_ADDR in relay.env) and the supervisor's dial address (SH_RELAY_ADDR in supervisor.env) agree. On a VM set up before MI1 β€” relay.env with no MOCA_RELAY_EXEC_ADDR, supervisor.env dialing 127.0.0.1:9443 β€” both files are migrated together to the loopback exec listener 127.0.0.1:9444. Any other disagreement (one file migrated and not the other, or two different ports) stops the script with a message naming both values.
  4. Generates the MU1 secrets, once and only those missing (see "The control plane" below). On the run that generates the signing keypair, it points supervisor.env at this VM's control plane (SH_CONTROL_PLANE_URL) and requires session tokens (SH_REQUIRE_AUTH=true).
  5. Installs systemd/sh-supervisor.service, systemd/sh-relay.service and systemd/sh-control-plane.service into /etc/systemd/system, reloads the daemon, and enables podman-restart.service so the containers below come back after a reboot (see "Reboots" below).
  6. Starts a Redis container on podman's default network (published on 127.0.0.1:6379), which isolate=strict below keeps unreachable from sandboxes. Its data is on a named volume with AOF on, and an already-matching container is left running rather than recreated (see "Redis persistence" below).
  7. Creates a dedicated moca-sandbox podman network (fixed subnet 10.89.40.0/24, gateway 10.89.40.1, isolate=strict so it exchanges no traffic with any other podman network) and installs an nftables table that confines it, then pulls SANDBOX_IMAGE and starts SH_SANDBOX_COUNT (default 2) sandbox containers on that network, wired to reach the relay and to authenticate to it. The firewall is in place before the first sandbox starts, and the pull before the first container is created (see "The sandbox image is pulled on every run" below). See "Sandbox container networking and the relay token" below.
  8. Enables and restarts sh-relay.service. Enables sh-control-plane.service, and restarts it once SH_GITHUB_CLIENT_ID and SH_PUBLIC_HARNESS_URL are both set in control-plane.env; until then it is only try-restarted, for the supervisor's reason below. Only enables sh-supervisor.service β€” it is deliberately not started yet (see below).

A re-run restarts what it owns. systemctl enable --now leaves an already-running unit alone, so env and unit changes from a re-run would otherwise wait for the next reboot. setup-vm.sh therefore restarts sh-relay.service on every run, reloads the sandbox firewall table directly (its unit is only started, never restarted: through RequiredBy= a restart would also restart podman-restart.service and every --restart=always container), and try-restarts sh-supervisor.service β€” restarted if it is running, left stopped if it is not, so a supervisor whose SH_TURNS_PER_WORKER is not set yet is never started by the script. sh-control-plane.service is restarted once configured and try-restarted until then. It is enabled either way, though, so an unconfigured one is started on every boot: with no SH_GITHUB_CLIENT_ID it refuses to boot, Restart=always/RestartSec=2 retries it, and systemd's default start limit (5 starts in 10 seconds) leaves it failed about ten seconds after boot, with systemctl is-system-running reporting degraded. That repeats on each boot until you set SH_GITHUB_CLIENT_ID and SH_PUBLIC_HARNESS_URL and re-run setup-vm.sh, which restarts it. (With the client id set but no SH_PUBLIC_HARNESS_URL it does boot, advertising no harness.) To opt out, sudo systemctl disable sh-control-plane.service; a re-run of setup-vm.sh enables it again. A re-run also recreates the sandbox containers (podman run --replace). It does not recreate Redis unless something about it actually differs from what the script would start, and its data is on a named volume either way, so a re-run keeps every session (see "Redis persistence" below).

SH_TURNS_PER_WORKER ships empty on purpose (see below), and readConfig throws on blank, so the supervisor unit is expected to fail if it starts before the operator sets it. With Restart=always/RestartSec=2 and no StartLimitIntervalSec=0, starting it in that state trips systemd's default 5-starts-in-10s limit in about ten seconds, and the unit then refuses even the ordinary recovery command until you systemctl reset-failed it. setup-vm.sh avoids that entirely by enabling the unit (so it starts on future boots) without starting it now. Before starting it for the first time, edit /etc/serverless-harness/supervisor.env and set SH_TURNS_PER_WORKER, then:

sudo systemctl start sh-supervisor.service

Troubleshooting: "start request repeated too quickly"​

If sh-supervisor.service ends up crash-looping anyway (for example, it was started before SH_TURNS_PER_WORKER was set, or some other config problem repeats within the 10-second window), systemd locks it out with Failed to start sh-supervisor.service: Unit sh-supervisor.service is not loaded properly: start request repeated too quickly. Fix the underlying config in supervisor.env, then clear the lockout and start again:

sudo systemctl reset-failed sh-supervisor.service
sudo systemctl start sh-supervisor.service

The control plane​

What runs. sh-control-plane.service runs packages/control-plane on 127.0.0.1:8090 (SH_CONTROL_PLANE_HOST, SH_CONTROL_PLANE_PORT in control-plane.env). Credentials use the file store (SH_CREDENTIAL_STORE=file) in /var/lib/moca-control-plane (SH_CREDENTIAL_DIR, the unit's StateDirectory, mode 0700), envelope-encrypted under the KEK. Sessions live in the same loopback Redis container as the supervisor's, on the named volume described under "Redis persistence" below, so they survive a re-run and a reboot alongside the credentials. Vault (SH_CREDENTIAL_STORE=vault) is the multi-host option and is not wired by setup-vm.sh: a follow-up (#362 item 1).

The GitHub OAuth app​

Login is GitHub's device flow (mocactl login prints a code; the user types it at https://github.com/login/device). The control plane needs one GitHub OAuth app for it, and only its client id: the device flow treats the app as a public client, so there is no client secret to store.

  1. On GitHub: Settings β†’ Developer settings β†’ OAuth Apps β†’ New OAuth App (or the same page under an organization's settings). Any application name. Homepage URL and authorization callback URL are required fields, but the device flow never uses them: use the repository URL for both.

  2. On the created app's page, tick Enable Device Flow and Update application. It is off by default, and forgetting it is the likeliest first-run failure. mocactl login then prints this (one line), and the control plane's log carries the same fix:

    login failed: the control plane cannot log anyone in until its operator fixes it β€” the control plane's GitHub OAuth app has the device flow off (device_flow_disabled): tick Enable Device Flow on the app
  3. Copy the Client ID (Ov23li… for a new OAuth app). Do not generate a client secret. A mistyped id is a client GitHub does not know, and the line ends instead with:

    GitHub knows no OAuth app with client id Ov23li… (20 chars) (Not Found): check SH_GITHUB_CLIENT_ID

    It shows only the start and the length of the id: the reply goes to anyone who can reach the control plane, and a wrong value may be a pasted secret.

  4. Give it to the control plane, in either of two ways:

    • on the setup run, through sudo env (sudo drops the rest of the environment). The same goes for the harness URL, since the control plane starts only once it has both:

      sudo env SH_GITHUB_CLIENT_ID=Ov23li... SH_PUBLIC_HARNESS_URL=http://127.0.0.1:8080 \
      ./deploy/vm/setup-vm.sh

      setup-vm.sh writes each value only where control-plane.env has none. It never replaces one you have already set, and warns if the environment disagrees with the file.

    • or by hand: set both SH_GITHUB_CLIENT_ID= and SH_PUBLIC_HARNESS_URL= in /etc/serverless-harness/control-plane.env, then re-run setup-vm.sh. With only the client id, the script leaves the control plane stopped, and a manual systemctl restart boots it advertising no harness, so mocactl doctor fails at check 5.

The login asks for the read:user scope only. The subject is the numeric GitHub user id (github:<id>), never the login name, and GitHub's access token is used once, to read that id, and then dropped. To make someone an admin (GET /v1/sessions?owner=…), put their subject in SH_ADMIN_SUBJECTS and restart the control plane. Roles are computed at login and carried in the API token, so the user must then log in again (mocactl login). GET /v1/me with the new token shows the role.

Network. The VM must reach https://github.com (the device-code and token endpoints) and https://api.github.com (/user) outbound. Each user's browser must reach https://github.com/login/device. The laptop running mocactl talks only to the control plane and the harness it advertises, never to GitHub.

Who may log in. Anyone with a GitHub account who can reach the control plane. The OAuth app does not restrict users, and the control plane has no allowlist. The SSH tunnel or firewall allowlist below is what limits who can reach it. Two users on this VM each see only their own sessions (404, not 403, for anyone else's), but on the container tier they share sandbox containers. And an SSH account is more than a way to reach the control plane: unless it is restricted to the two forwarded ports ("Reaching it" below), its holder can reach Redis and take over any session. See "What round one does not claim".

Secrets​

They all live in /etc/serverless-harness/credentials/ (directory 0700, files 0600, owner root), and each unit loads its own with LoadCredential= under the setting's own name:

FileSettingLoaded by
session-token-private-keySH_SESSION_TOKEN_PRIVATE_KEYcontrol plane
credential-kekSH_CREDENTIAL_KEKcontrol plane
exchange-tokenSH_EXCHANGE_TOKENcontrol plane and supervisor
operator-inference-tokenSH_OPERATOR_INFERENCE_TOKENcontrol plane, optional (below)

The public half of the signing key is not a secret: it is the SH_SESSION_TOKEN_PUBLIC_KEYS line in supervisor.env. Setting any of them as an env line as well makes the unit refuse to boot (and setup-vm.sh stops before that, naming the line).

setup-vm.sh generates them once, with the checkout's own packages/control-plane/src/genkeys.ts, and a re-run never replaces one. It generates only a file that is missing, so rotation is by hand: delete the file and re-run, which restarts the running units onto the new value. The private key and the SH_SESSION_TOKEN_PUBLIC_KEYS line are one keypair: delete both, or the script stops on half a pair. A new keypair invalidates every live token and re-wires supervisor.env as on first install, SH_REQUIRE_AUTH=true included. A new KEK makes every stored credential undecryptable, so with credential-kek missing and records in SH_CREDENTIAL_DIR, the script stops: restore the KEK from a backup, or empty the store deliberately.

The limit. LoadCredential= keeps the secrets out of env files, out of /proc/<pid>/environ and out of every child process's environment. It does not keep them from a child process: children inherit CREDENTIALS_DIRECTORY and the harness uid, so they can read the files. Nor is it a uid boundary. All three units run as harness, so a compromise of the supervisor's uid can read the control plane's credentials. A dedicated control-plane user is a follow-up.

The operator-key fallback (SH_ALLOW_OPERATOR_FALLBACK=true, off by default) lets a user who has stored no inference credential spend the operator's key. Every such turn is audited as operator_fallback_used. The key is the one secret setup-vm.sh does not create: put it in its file yourself, set the fallback's settings in control-plane.env, and re-run the script. It sees the file and installs a drop-in (sh-control-plane.service.d/50-operator-inference-token.conf) that loads it with LoadCredential=. It never reads the value.

sudo install -m 0600 /dev/null /etc/serverless-harness/credentials/operator-inference-token
sudoedit /etc/serverless-harness/credentials/operator-inference-token # the key; mode is kept
sudoedit /etc/serverless-harness/control-plane.env
# SH_ALLOW_OPERATOR_FALLBACK=true
# A gateway token: SH_DEFAULT_INFERENCE_ENDPOINT=https://<gateway>
# An Anthropic API key: SH_DEFAULT_INFERENCE_ENDPOINT=https://api.anthropic.com
# SH_OPERATOR_INFERENCE_HEADER=x-api-key
cd /opt/serverless-harness && sudo ./deploy/vm/setup-vm.sh

The control plane refuses to boot on a fallback that cannot work, naming the setting to fix:

  • no token;
  • no default endpoint;
  • an Anthropic key on the Bearer header, or aimed at anything but https://api.anthropic.com;
  • an sk-ant-oat… OAuth token: the fallback is shared by every caller, and one Claude Pro/Max subscription must not serve them all. Callers can store their own OAuth token as an inference credential; the shared fallback cannot.

journalctl -u sh-control-plane shows which. setup-vm.sh itself stops on:

  • SH_ALLOW_OPERATOR_FALLBACK=true with no token file;
  • a token file that is empty, a dangling symlink, or readable by anyone but its owner (not 0600 or 0400; for a symlink, its target's mode);
  • the token as a line in control-plane.env, in any spelling systemd accepts (indented, or with spaces around =);
  • another drop-in that also loads SH_OPERATOR_INFERENCE_TOKEN, such as the systemctl edit override.conf this README described before.

For that last one, delete the drop-in's LoadCredential= line and re-run: the script manages the credential itself now.

To turn the fallback off, set SH_ALLOW_OPERATOR_FALLBACK=false. To remove the key as well, delete the file and re-run: the drop-in goes with it.

A session's credential is fixed when the session is created, in both directions:

  • A session started on the fallback keeps spending the operator's key after its user stores a credential of their own. Only a new session picks that credential up. Turning the fallback off ends those sessions' turns with a message saying to start a new one.
  • A session started on the user's own credential never moves onto the operator's key. If the user deletes that credential (because it leaked, say), the session's next turn is refused, naming it.

Reaching it: the supported demo topology​

Both the control plane and the supervisor speak plain HTTP; TLS termination is out of scope for this round. A client needs two ports: the control plane (8090) and the harness it advertises (SH_PUBLIC_HARNESS_URL, the supervisor on 8080). Two topologies are supported:

  • (a) An SSH tunnel, the default. Each user runs ssh -f -N -o ExitOnForwardFailure=yes -L 8090:127.0.0.1:8090 -L 8080:127.0.0.1:8080 <vm> (-f backgrounds it once both forwards are up; ExitOnForwardFailure fails instead of warning when a local port is taken). On the VM, set SH_PUBLIC_HARNESS_URL=http://127.0.0.1:8080; on the laptop, run mocactl --control-plane-url http://127.0.0.1:8090 …. Because the harness URL is advertised, every user must forward the same local port.

    An SSH account reaches every loopback service, including Redis (no password: it holds the session-ownership index) and the supervisor's unauthenticated admin listener on 8081. Anyone who can write that index can make another user's session their own. So give users who are not operators an account that can forward the two ports and nothing else, at the end of /etc/ssh/sshd_config:

    Match User user1,user2
    AllowTcpForwarding local
    PermitOpen 127.0.0.1:8090 127.0.0.1:8080
    AllowStreamLocalForwarding no
    AllowAgentForwarding no
    X11Forwarding no
    PermitTTY no
    ForceCommand /usr/sbin/nologin

    docs/demos/vm-two-user-acceptance.md (0a, 0b) installs it and checks it from a laptop.

  • (b) A cloud firewall allowlist. Set SH_CONTROL_PLANE_HOST=0.0.0.0 and SH_PUBLIC_HARNESS_URL=http://<vm-address>:8080, and allow 8090 and 8080 from the users' addresses only. Tokens and credentials then cross the network in clear.

The supervisor already listens on 0.0.0.0:8080. With SH_REQUIRE_AUTH=true every turn needs a session token, but that is not a substitute for the allowlist.

Session discovery​

/v1/sessions/{id}/resources answers sandbox.phase: "unknown" on a VM: there is no kubectl, and the control plane treats that like a Kubernetes outage. mocactl does not use the route.

Checking it​

On the VM, curl -s 127.0.0.1:8090/readyz. From a laptop, through the tunnel, mocactl --control-plane-url http://127.0.0.1:8090 doctor. Doctor stops at its first failure, and "harness trusts this control plane" (check 7) comes after "logged in" (3) and "inference credential present" (4). mocactl login (GitHub's device flow) gets past check 3. To check the wiring without a GitHub login, mint an API token on the VM the way deploy/compose/smoke.sh does. The key stays in its root-only file, and the output goes to stdout only:

cd /opt/serverless-harness/packages/control-plane
sudo node --import tsx --input-type=module <<'EOF'
import { readFileSync } from 'node:fs';
import { makeSigner } from './src/token.ts';
const s = makeSigner(readFileSync('/etc/serverless-harness/credentials/session-token-private-key', 'utf8'));
const sub = 'demo:1', ttlSeconds = 3600, now = Math.floor(Date.now() / 1000);
const apiToken = s.mint({ sub, tenant: sub, roles: [], scope: ['api'], ttlSeconds, now });
const auth = { apiToken, subject: sub, roles: [], expiresAt: now + ttlSeconds, controlPlaneUrl: 'http://127.0.0.1:8090' };
process.stdout.write(JSON.stringify(auth) + '\n');
EOF

That line is a complete mocactl login cache (expiresAt in epoch seconds; controlPlaneUrl must equal the --control-plane-url you pass). Save it on the laptop as mocactl/auth.json under $XDG_CONFIG_HOME (default ~/.config), mode 0600, creating the directory first:

mkdir -p -m 0700 "${XDG_CONFIG_HOME:-$HOME/.config}/mocactl"
(umask 077 && cat > "${XDG_CONFIG_HOME:-$HOME/.config}/mocactl/auth.json") # paste, then Ctrl-D

A hand-minted cache carries no refresh token, so it works until its own expiresAt and then asks for mocactl login, exactly as before B14. Use the device flow for a login that renews itself.

Then add an inference credential with /credentials in mocactl, and run doctor.

Sandbox container networking and the relay token​

remote-worker/cmd/worker/main.go reads RELAY_ADDR (default localhost:8443), SANDBOX_TOKEN (default dev-token), and SANDBOX_ID (default sbx-laptop-1) from its own environment. None of those defaults work for a podman container started with no -e flags: localhost inside the container resolves to the container itself, not the host running the relay; every container would share one SANDBOX_ID and collide on the same sh:sandbox:records entry in Redis; and dev-token never matches a real, fail-closed relay. setup-vm.sh now sets all three explicitly for each container:

  • SANDBOX_ID=sh-sandbox-<i> β€” unique per container.
  • SANDBOX_TOKEN=<the value of SH_RELAY_TOKEN in the installed relay.env>.
  • RELAY_ADDR=<SH_SANDBOX_RELAY_ADDR, or host.containers.internal:<SH_RELAY_PORT>> β€” see below.

The token preflight. The relay's token validation (makeDefaultValidateToken in packages/sandbox-relay/src/main.ts) is fail-closed: with SH_RELAY_TOKEN unset, every attach is rejected, including a tokenless one. relay.env.example ships it commented out on purpose (it is an operator secret, not a default), so setup-vm.sh checks the installed relay.env for a non-empty SH_RELAY_TOKEN before starting any sandbox container, and refuses to continue with a clear message if it is missing, rather than starting containers that can never attach.

/etc/serverless-harness/relay.env is created by setup-vm.sh itself, so on a fresh VM there is nothing to edit until the script has run once β€” this is the two-invocation first run described under "Bring it up". After that first run:

sudoedit /etc/serverless-harness/relay.env # set SH_RELAY_TOKEN=<a shared secret>
sudo ./deploy/vm/setup-vm.sh # re-run; the edited file is preserved

Appending instead of editing works equally well once the file exists (echo 'SH_RELAY_TOKEN=…' | sudo tee -a /etc/serverless-harness/relay.env) β€” but only then, since the directory and file do not exist before install_env creates them.

Whichever way you set it, the value must match each sandbox worker's SANDBOX_TOKEN; setup-vm.sh reads it back out of relay.env and passes exactly that to every container it starts, so editing this one file is enough.

Reaching the host from a container. host.containers.internal is podman's documented analogue of Docker's host.docker.internal. Podman's own host-gateway special value resolves it to whatever the host actually listens on β€” every port bound to 0.0.0.0, not just the relay's β€” so setup-vm.sh instead runs sandboxes on their own dedicated podman network, moca-sandbox (fixed subnet MOCA_SANDBOX_SUBNET, default 10.89.40.0/24), and pins host.containers.internal to that network's gateway (MOCA_SANDBOX_GATEWAY, default 10.89.40.1) with an explicit --add-host on every podman run, rather than depending on netavark's automatic /etc/hosts population (which differs between rootful and rootless podman and across versions). The default address is therefore host.containers.internal:<port>, where <port> comes from SH_RELAY_PORT in the installed relay.env (falling back to 8443, the code's own default, if that line is missing). Override the whole address with SH_SANDBOX_RELAY_ADDR if this default does not resolve on your VM's actual network setup.

Confining the sandbox network to the relay's attach port (MI1 R8). A dedicated network only changes what address a sandbox dials β€” by itself it does not stop a sandbox from reaching anything else the host listens on, since podman still routes the whole subnet to the host through that gateway. setup-vm.sh also renders an nftables table ($SH_ENV_DIR/moca-sandbox.nft, table inet moca_sandbox) that drops all traffic arriving from the moca-sandbox network to the host except the relay's attach port (SH_RELAY_PORT) and DNS (port 53, answered by podman's own resolver on the gateway), and loads it immediately with nft -f. deploy/vm/systemd/moca-sandbox-firewall.service, a oneshot unit ordered Before= sh-relay.service and podman-restart.service, re-loads that same table on every boot, so the restriction survives a reboot and is in place before the relay β€” and so before any sandbox container β€” starts. The unit is also RequiredBy= both, so the deployment fails closed: if the table does not load at boot, the relay does not start and neither does podman-restart.service, which keeps Redis and every --restart=always container down until the firewall loads. The table filters both the input hook (traffic addressed to the host itself) and the forward hook, but the forward chain drops only link-local destinations β€” 169.254.0.0/16 and the IPv6 metadata address fd00:ec2::254 β€” arriving from the sandbox bridge: link-local cloud instance metadata, which on a cloud VM can expose the instance role's credentials, credentials that belong to the host and not to any session or user (rossoctl/moca#357). All other forwarded traffic (outbound internet access from a sandbox) is untouched in this round β€” that is MI1 S5's moca-egress work, not this one. The rules match the bridge a packet arrives on β€” MOCA_SANDBOX_BRIDGE, default moca-sandbox0, pinned when the network is created and checked on every run β€” not only its source address: netavark leaves IPv6 enabled on the bridge and in every container, so link-local IPv6 reaches the host even though moca-sandbox has only an IPv4 subnet. Three rules accept: replies on connections that are already established (ct state established,related, on that bridge only), and new IPv4 connections from the sandbox subnet to the relay's attach port and to DNS. Everything else arriving on the bridge, IPv6 included, is dropped.

SandboxExec is not served on the moca-sandbox network at all: the relay binds it on loopback (MOCA_RELAY_EXEC_ADDR=127.0.0.1:9444), and the exec token (MOCA_RELAY_EXEC_TOKEN, held only by the relay and the supervisor) is the control.

What has been verified on a real host, and what has not. One live run (2026-09-29) on Amazon Linux 2023 (kernel 6.18), with rootful podman 5.8.7 (a static build β€” Amazon Linux 2023 packages no podman), netavark 1.17.2 using its nftables firewall driver, and nftables 1.0.4, confirmed from inside a sandbox container:

  • the relay's attach port on the gateway connects;
  • every other host port tried (22, 8080, 8081, 9444, 6379) is dropped, over IPv4 and over IPv6 link-local;
  • DNS resolves and outbound internet works;
  • Redis on podman's default network is unreachable (isolate=strict).

On the same host:

  • with the table made unloadable, sh-relay.service and podman-restart.service both refuse to start;
  • a pre-MI1 install migrates to the split exec listener;
  • a fresh install with the default image completes a turn.

A second live run (2026-10-14, same host) verified the forward chain (rossoctl/moca#357): with the original input-only table, http://169.254.169.254/latest/meta-data/ answered from inside a sandbox (HTTP 401 β€” IMDSv2 wanted a token, but the request had reached the service through the forward path); with the forward chain loaded, the same request timed out and the drop rule's counter incremented, while outbound internet, the relay's attach port, and the input chain's other drops (host port 22) were all unchanged. The candidate table also parsed and loaded cleanly on that host's nftables 1.0.4 β€” the fail-closed risk that kept this out of #350. The IPv6 rule (fd00:ec2::254) was not exercised there: that container has no IPv6 route to it. It covers hosts where the IPv6 metadata endpoint is routable.

Not verified: SELinux- or AppArmor-enforcing hosts, rootless podman, a distro-packaged podman, and an actual reboot β€” the fail-closed ordering was exercised with systemctl, not a boot.

If a sandbox container cannot attach on a real VM, SH_SANDBOX_RELAY_ADDR (or, if podman itself cannot resolve host.containers.internal, the VM's actual gateway or bridge IP) is the override to reach for first; sudo nft list table inet moca_sandbox shows the loaded rules.

Link-local instance metadata is blocked in-product (closed rossoctl/moca#357). The moca_sandbox table's forward chain drops link-local destinations β€” 169.254.0.0/16, which carries 169.254.169.254, and the IPv6 metadata address fd00:ec2::254 β€” arriving from the sandbox bridge, so a sandbox cannot reach the link-local metadata endpoints that AWS, GCP, Azure and OCI serve, even though general forwarded traffic stays untouched until S5's moca-egress. Clouds that serve instance data from routable addresses are not covered by a link-local drop β€” Alibaba's 100.100.100.200, Azure's WireServer 168.63.129.16; S5's egress allowlist is what closes those. Platform hardening is still worth it on every cloud β€” defense in depth where the table already covers the endpoint, the only barrier where it does not. On EC2, require IMDSv2 with a hop limit of 1 (aws ec2 modify-instance-metadata-options --instance-id <id> --http-tokens required --http-put-response-hop-limit 1): a forwarded container request then cannot obtain a token even if the table is ever unloaded. Other clouds have the equivalent.

A research turn: curl and git in the sandbox​

Sandbox egress is open in this round. The firewall above filters only traffic to the host. Forwarded traffic isn't filtered, so the agent's bash tool reaches the whole internet from a sandbox container: any host, any port, with no proxy and no allowlist. That includes cloud instance metadata: close it at the platform first (above, #357). Egress control is MI1 S5's moca-egress, not this round. The container image guarantees git, curl and CA certificates (#368), and HOME is a writable /home/sandbox, so git config --global works.

Pi shows the model only the last 2000 lines or 50 KB of a command's output, so the prompt steers the agent to curl -o a file and read it with grep/head, rather than printing it. The demo's prompt, for mocactl run or the interactive UI:

This is a research task. Use your bash tool for every step, and do not answer from memory.
1. Run: git clone --depth 1 https://github.com/rossoctl/moca /workspace/research-demo/moca
2. Run: curl -fsSL -o /workspace/research-demo/node-releases.json https://nodejs.org/dist/index.json
The file is large: do not print it. Read what you need from it with head, grep or python3.
3. Find the commit the clone checked out, and the newest Node.js release in the fetched file (its
first entry) with its release date.
Reply with one short paragraph saying what you found, then end with exactly these three lines:
COMMIT=<the first 12 characters of the commit hash>
NODE_VERSION=<the version, for example v1.2.3>
NODE_DATE=<its date, YYYY-MM-DD>

Neither answer can come from a model's memory: the clone's HEAD changes with every merge, and the newest Node.js release every few weeks. Check them with git ls-remote https://github.com/rossoctl/moca HEAD and https://nodejs.org/dist/index.json.

git clone refuses a destination that already exists, so to run the prompt a second time on the same container, remove /workspace/research-demo first (for example sudo podman exec sh-sandbox-0 rm -rf /workspace/research-demo, on each container), or change the directory name in the prompt.

On the container tier, every session shares one /workspace per container (remote-worker/internal/exec/runner.go ignores the workspace key). So a second user's turn can see the first user's research-demo directory if both land on the same container. It is not a security boundary in this round (epic #370).

The automated check. deploy/vm/research-smoke.sh runs that prompt as a freshly minted user, in a per-run directory. It passes only if:

  • git clone and curl -o both appear in the turn's tool_use frames and succeed;
  • the answer's three values equal what the sandbox's own copy of the fetched files says;
  • the control plane's audit shows the intended credential was spent.

Run it on the VM, as root, on a host with container sandboxes. It refuses to run while a microVM worker is in the pool, because that tier reaches only moca-egress's allowlist, without credentials (MicroVM tier (P4), "Guest egress").

# The user's own credential, from a root-only file holding exactly one line, the key alone (an
# sk-ant-api… key goes as x-api-key to https://api.anthropic.com; anything else as Bearer to
# RESEARCH_ENDPOINT):
sudo install -m 0600 /dev/null /root/inference-key && sudoedit /root/inference-key
cd /opt/serverless-harness
sudo VM_RESEARCH_SMOKE=1 RESEARCH_CREDENTIAL_FILE=/root/inference-key ./deploy/vm/research-smoke.sh
sudo VM_RESEARCH_SMOKE=1 RESEARCH_CREDENTIAL_FILE=/root/gw-token \
RESEARCH_ENDPOINT=https://<gateway> ./deploy/vm/research-smoke.sh
# Or no credential of its own, on the operator-key fallback ("The operator-key fallback", above):
sudo VM_RESEARCH_SMOKE=1 RESEARCH_USE_OPERATOR_FALLBACK=1 ./deploy/vm/research-smoke.sh

It deletes its credential, session and fetched files afterwards (KEEP=1 leaves them), and removes its minted subject's record from the file credential store. That last step is its own: deleting a credential rewrites the subject's document without it and never unlinks, so the empty record would otherwise stay β€” one per run (#418). A record that still holds a credential is kept and reported, not removed. A failed run keeps its SSE transcript in the directory it names.

Reboots​

All three units are WantedBy=multi-user.target, so systemd brings the relay, the control plane and the supervisor back on boot. The podman containers need one extra thing: --restart=always (which setup-vm.sh now passes to Redis and to every sandbox container) covers a container that exits, but podman-run(1) is explicit that it does not cover a host reboot. setup-vm.sh therefore also enables podman-restart.service, podman's own supported mechanism for that. Without it the units would come back while Redis and every sandbox container stayed down β€” sh:sandbox:records empty and every turn failing, on a VM that otherwise looks healthy.

If podman-restart.service is not available on your podman build, setup-vm.sh warns rather than failing (it is one package's unit name, not a hard requirement of the bring-up) and the bring-up still completes. On such a host, re-run setup-vm.sh after a reboot before expecting turns to work.

The sandbox image is pulled on every run​

podman run resolves an image reference only when the host has no copy of it. So a ghcr.io/rossoctl/moca-remote-worker:latest pulled once β€” weeks ago β€” was what every later setup-vm.sh run started the sandboxes on, and nothing in the output said which image that was (#414). The P6.3 rig hit exactly this: its copy predated #372, so HOME inside each sandbox was the shared /workspace rather than /home/sandbox.

setup-vm.sh therefore runs podman pull "$SANDBOX_IMAGE" before it creates the first container, and then logs the image id and creation time the sandboxes are starting from:

==> pulling ghcr.io/rossoctl/moca-remote-worker:latest
==> sandbox image ghcr.io/rossoctl/moca-remote-worker:latest is 102ed4c1cfb9, created 2026-10-01T…

That id is the one podman images prints, so it answers "is this host on the current image?" directly. A failed pull stops the bring-up: the sandbox containers are this deployment's executors, and starting them on an unexamined local copy is the bug being fixed, not a fallback.

SH_SKIP_PULL=1 is the way past it β€” an offline host, or a SANDBOX_IMAGE you built on the VM that no registry serves. It does not attempt the pull at all (so there is no registry timeout to wait out), says so in the output, and still logs the local image the containers will run:

sudo env SH_SKIP_PULL=1 ./deploy/vm/setup-vm.sh

deploy/microvm/build-rootfs.sh pulls the same way, for the same reason: a snapshot is built once and run for a long time, so a rootfs exported from a stale copy outlives it. SH_SKIP_PULL=1 there too (it refuses if the host then has no copy at all), and /etc/moca-rootfs-source inside the tree records the image and digest it came from either way.

Redis persistence​

Redis holds the control plane's session and ownership index, the harness's own session state, the lease store and sh:sandbox:records β€” of which only the last is self-healing (workers re-register when they reconnect to the relay). setup-vm.sh therefore starts it on a named volume (moca-redis, override with MOCA_REDIS_VOLUME) with AOF on (redis-server --appendonly yes), the same shape the Kubernetes path already uses β€” a PVC in deploy/k8s/base/redis.yaml plus appendonly yes in the redis.conf that deploy/k8s/setup.sh writes.

Both halves are needed. The named volume survives podman rm; AOF makes each write durable when it happens, where snapshots alone would drop up to the last save window on a reboot.

A re-run of setup-vm.sh also no longer recreates the container. It inspects the running sh-redis first and leaves it alone when its image, published port, /data volume and --appendonly setting all match what the script would start, logging which field differed when it does replace it. This matters because re-running the installer is how you apply a changed control-plane.env (--restart=always and podman-restart.service bring the container back after a reboot, not the data that was in it).

Hosts installed before this change​

Earlier versions started Redis with no -v. docker.io/redis:7-alpine (what the VM stack ran before the Valkey swap) declares VOLUME /data, so each run's podman run --replace moved the data to a fresh anonymous volume and orphaned the previous one: those sessions were unreachable but never deleted, and a long-lived rig can hold a stack of them (podman volume ls).

On such a host the next run stops and makes you choose. It will not decide the fate of those sessions for you, and it will not carry on and quietly strand them: nothing changes, and sh-redis keeps running on the old volume, until you re-run with one of:

# Migrate the old sessions onto the named volume (what you almost certainly want).
sudo env SH_REDIS_ADOPT_VOLUME=1 ./deploy/vm/setup-vm.sh

# Or: start fresh, leaving them behind β€” not deleted, but no longer reachable.
sudo env SH_REDIS_ADOPT_VOLUME=0 ./deploy/vm/setup-vm.sh

=1 runs redis-cli SAVE on the live container, copies its dump.rdb into moca-redis, converts it to an AOF, and starts Redis on the result. It acts only on a container whose /data is anonymous, it will not touch a named volume, and it refuses outright if moca-redis already holds data rather than overwriting what is in there. If any step fails it puts the host back as it was β€” the old volume is only ever read β€” and says so.

The choice is offered once. After either run, sh-redis is on the named volume and matches what the script starts, so no later run has anything left to adopt. If you have already run with =0 (or with an older version of this script, which did this silently) and want those sessions back, the by-hand recipe below is the route; =1 will no longer do it for you, and that is deliberate β€” moca-redis holds real sessions by then, and adopting over them would trade one loss for another.

Do not just copy a dump.rdb into the volume and start Redis. With --appendonly yes, Valkey loads the AOF and ignores an existing dump.rdb β€” verified against redis:7-alpine (podman 6.1.1) and valkey/valkey:9-alpine β€” so that yields a silently empty Redis. The conversion has to run with appendonly off first, so the RDB is what gets loaded, and only then CONFIG SET appendonly yes. SH_REDIS_ADOPT_VOLUME=1 does this in the right order. The same trap applies to recovering an orphaned volume by hand:

# Inspect candidates first: the newest Mountpoint/dump.rdb mtime is usually the one you want.
podman volume ls
podman run --rm -v <old-volume>:/data docker.io/valkey/valkey:9-alpine ls -l /data
# Load the RDB with appendonly OFF, then convert.
podman run -d --rm --name redis-recover -v <old-volume>:/data \
docker.io/valkey/valkey:9-alpine redis-server --appendonly no
podman exec redis-recover redis-cli DBSIZE # non-zero, or you have the wrong volume
podman exec redis-recover redis-cli CONFIG SET appendonly yes
podman exec redis-recover redis-cli SHUTDOWN NOSAVE # closes the connection; expected

Then point MOCA_REDIS_VOLUME at that volume and re-run setup-vm.sh.

The Valkey swap is one-way: once Valkey rewrites the AOF or RDB, the data needs Valkey to load.

Where the env file lives​

/etc/serverless-harness/supervisor.env (mode 0640, root-owned β€” install_env runs as root and does not chown to harness; that's fine, since systemd reads EnvironmentFile= as PID 1, before dropping privileges to User=harness), installed once from deploy/vm/env/supervisor.env.example. SH_TURNS_PER_WORKER β€” the per-worker cap on in-flight turns (S) β€” has no default anywhere in this deployment: its correct value is an output of experiment E8, not a guess, so shipping one would silently truncate the E8 ladder it exists to measure. Left unset, the supervisor's own startup check (readConfig) refuses to start rather than falling back to a wrong value.

Configuration not in the shipped env files​

Two supervisor-related variables from spec Β§3.8 are deliberately absent from env/supervisor.env.example and from both unit files β€” this is a standing decision, not an oversight, and deploy/vm/tests/setup-vm.test.sh asserts SH_ADMIN_PORT's absence directly (its else-branch depends on it).

  • SH_ADMIN_PORT (default 8081) β€” the loopback-only (127.0.0.1), unauthenticated /metrics listener the supervisor opens whether or not you configure it (Β§5.2). 0 asks the kernel for an ephemeral port instead. readConfig rejects a value equal to PORT (EADDRINUSE at boot otherwise), with 0 exempt since the kernel hands out a distinct ephemeral port each time. The default is correct for this single-VM target, so it is not in supervisor.env.example: an operator who does not need to move the admin port should not have to think about it, or about accidentally setting it equal to PORT.
  • SH_STATS_INTERVAL_MS (default 1000) β€” paces the worker's advisory stats telemetry only; nothing on the routing path depends on it. It is read by the worker process, not the supervisor, so it reaches a worker through the environment the supervisor spawns it with (inherited from the supervisor unit's own environment), not through the supervisor's own readConfig. There is accordingly no supervisor-side reason to set it in supervisor.env, and no unit-file line to set it either.

If an operator needs to change either of these, set them directly in /etc/serverless-harness/supervisor.env (they are ordinary env vars the supervisor process reads at startup) β€” just be aware that adding an uncommented SH_ADMIN_PORT line there will change what deploy/vm/tests/setup-vm.test.sh expects if the test is ever extended to check for it.

SANDBOX_IMAGE (default ghcr.io/rossoctl/moca-remote-worker:latest) is the image setup-vm.sh runs sandbox containers from. It is a variable for the script, not an env-file setting: sudo SANDBOX_IMAGE=<image> ./deploy/vm/setup-vm.sh. The same name means the Kubernetes sandbox pod image to setup-k8s.sh, and compose spells this concept SH_SANDBOX_IMAGE, so do not export one value for all three. The default is a mutable tag, so it is pulled on every run; SH_SKIP_PULL=1 runs the host's copy instead (see "The sandbox image is pulled on every run" above).

Tear it down​

There is no uninstall script. These steps undo what setup-vm.sh installs, in the reverse order. Step 4 deletes every session and every stored credential for good: back up first (Redis persistence) if you may want them.

These steps were run on 2026-11-19 against a host installed by setup-vm.sh at main 175a09a9 (Ubuntu 24.04 on an AWS m7i.metal-24xl, with the supervisor and the control plane running): every command succeeded, and afterwards no moca unit, container, network, volume, nftables table, directory or user was left.

  1. Stop and remove the units (the firewall unit has no stop action, so step 2 removes its table):

    sudo systemctl disable --now sh-supervisor.service sh-control-plane.service sh-relay.service \
    moca-sandbox-firewall.service
    sudo rm -f /etc/systemd/system/sh-supervisor.service /etc/systemd/system/sh-control-plane.service \
    /etc/systemd/system/sh-relay.service /etc/systemd/system/moca-sandbox-firewall.service
    sudo rm -rf /etc/systemd/system/sh-control-plane.service.d # the credential drop-ins, if any
    sudo systemctl daemon-reload
  2. Remove the sandbox firewall table: sudo nft delete table inet moca_sandbox.

  3. Remove the containers and the sandbox network. The sandboxes are sh-sandbox-0 up to sh-sandbox-<SH_SANDBOX_COUNT - 1> (two by default):

    sudo podman rm -f sh-redis sh-sandbox-0 sh-sandbox-1
    sudo podman network rm moca-sandbox

    podman's own firewall tables (ip nat and ip filter, with netavark's NETAVARK_* chains, managed by iptables-nft) stay. They hold no moca rule; leave them to podman.

  4. Delete the data. Redis's named volume holds every session; the control plane's file store holds every stored credential (SH_CREDENTIAL_DIR in control-plane.env, by default /var/lib/moca-control-plane):

    sudo podman volume rm moca-redis
    sudo rm -rf /var/lib/moca-control-plane
  5. Delete the configuration and secrets: sudo rm -rf /etc/serverless-harness (the env files, the relay token, the control plane's keys and moca-sandbox.nft).

  6. Optionally, remove what was there before the install: the harness user (sudo userdel harness), the checkout (/opt/serverless-harness by default), and the images (sudo podman rmi ghcr.io/rossoctl/moca-remote-worker:latest docker.io/valkey/valkey:9-alpine). Every setup-vm.sh run pulls the sandbox image again, so an earlier pull that a newer latest replaced is left untagged, and rmi by tag does not reach it: sudo podman images --filter dangling=true lists such images, and sudo podman rmi <image id> removes one (sudo podman image prune removes every untagged image on the host, moca's or not). setup-vm.sh also enables podman-restart.service, which other containers on the host may rely on: leave it enabled unless nothing else uses it.

What round one does not claim​

The systemd [Service] hardening directives in sh-supervisor.service, sh-relay.service and sh-control-plane.service (ProtectSystem=strict, NoNewPrivileges=true, SystemCallFilter=, and friends) are the VM analogue of a pod's securityContext β€” they narrow the filesystem and syscall surface available to each process. They are present, not equivalent: this round does not claim security-context parity with the Kubernetes deployment, and it does not have any analogue of Kubernetes NetworkPolicy egress control. systemd has no per-unit network-egress primitive comparable to a NetworkPolicy, so a VM deployment is strictly more exposed on that axis until the Z2/Z5 work lands.

Nor does it claim, for the control plane:

  • TLS. The control plane and the supervisor speak plain HTTP. The supported topologies are an SSH tunnel or a firewall allowlist ("The control plane" above), not an encrypted listener.
  • A uid boundary. The control plane runs as harness, the same uid as the supervisor and its workers. LoadCredential= keeps its secrets out of env files and child processes' environments, not out of reach of a compromised harness process.
  • Multi-host credentials. The file store is single-host. Vault, the multi-host store, is not wired by setup-vm.sh.
  • Sandbox egress control. A sandbox container reaches the internet and the cloud's instance metadata ("A research turn" above). Filtering it is MI1 S5.
  • Sandbox isolation between users. Session ownership is enforced: each user lists only their own sessions, and another user's session answers 404. But on the container tier, every user's turns lease the same sandbox containers (harness/src/select-sandbox.ts has no owner filter), and the container worker ignores workspace_key (remote-worker/internal/exec/runner.go). So users share /workspace, the Unix user and the process list. Owner binding is MI1 S5 (docs/specs/2026-09-28-moca-multi-user-isolation-design.md Β§9).
  • Two users after MI1 S2. setup-vm.sh leaves MOCA_TENANCY unset (single). Until MI1 S2, single serves every subject that logs in. From S2 it serves only the first and refuses the rest with 403 single_tenant_deployment (MI1 Β§6.6). setup-vm.sh adds no refusal of its own, since it cannot tell how many people will log in. MOCA_TENANCY=multi serves two users honestly only once S5's owner binding lands β€” that is the real blocker. Until then, run a two-user demo at v0.5.1 (docs/demos/vm-two-user-acceptance.md, #407). On the credential side multi no longer refuses async turns from this release, provided the control plane carries the grant ring and the inference listener; setup-vm.sh configures neither, so enabling multi is three deliberate edits to /etc/serverless-harness/control-plane.env and a systemctl restart sh-control-plane.service:
    1. MOCA_GRANT_SIGNING_KEY β€” an Ed25519 PKCS#8 key, and NOT the one in SH_SESSION_TOKEN_PRIVATE_KEY (a shared key refuses to boot).
    2. MOCA_INFERENCE_ADDR β€” the listener's bind address, e.g. 127.0.0.1:3130 while the workers are on this VM.
    3. MOCA_INFERENCE_URL β€” only when the bind is a wildcard, since 0.0.0.0 is not dialable.
  • Ownership against loopback access. Session ownership is enforced at the control plane's API. Its index is in Redis on 127.0.0.1:6379, unauthenticated, so anyone with a shell or an unrestricted SSH forward on the VM can rewrite it. Restrict non-operator SSH accounts ("Reaching it").