Skip to main content
Version: v0.11.0

Kubernetes and OpenShift

Preview. P6 on Kubernetes is implemented but not yet declared Stable; interfaces and limits may change. Known limits: What this does not claim. It is the first target for Stable: what Stable requires and where this path stands is in Maturity and lifecycle.

The P6 (process-manager) runtime as ordinary Kubernetes workloads: the supervisor and its forked workers, the sandbox relay, Redis, the MU1 control plane and remote-worker container sandboxes. It is milestone P6.1, slice 1 of epic rossoctl/moca#426 (issue rossoctl/moca#423). The design is docs/specs/2026-10-02-p6-on-kubernetes-slice1-design.md.

1. What this isโ€‹

This is the sibling of Single VM (systemd and podman on one VM) and Docker Compose (the same processes under Docker Compose). All three run the same processes, the same environment and the same wire. Only the process manager, the addressing and the way isolation is enforced differ. Here, isolation comes from namespaces, NetworkPolicy, Secrets and pod lifecycle.

This directory uses GrpcRelayTransport and records-based discovery exactly as deploy/vm/ does โ€” the Knative/KEDA deployment model that once shared this name space was removed entirely (ADR-0041). ADR-0037 records the Kubernetes-substrate distinction.

WorkloadNamespacePortsReached by
Deployment moca-supervisormoca8080 data; 8081 admin; 8443 ghostunnel and 8453 its status (OCP)users: port-forward on Kind, the passthrough Route on OCP; kubelet on 8081 (and 8453 on OCP)
Deployment sandbox-relaymoca9443 Attach; 9444 SandboxExecsandboxes on 9443 only; supervisor pods on 9444 only
StatefulSet redismoca6379supervisor, relay, control plane
Deployment moca-control-planemoca8080users (port-forward on Kind, an edge Route on OCP); the supervisor's workers
StatefulSet moca-sandboxmoca-sandboxnonenothing: each sandbox dials out to the relay
per-user credentials (Secrets)moca-credentialsnonethe control plane, through the kube API (K8sSecretStore)

The relay has two Services, sandbox-relay-attach (9443) and sandbox-relay-exec (9444), so the address a sandbox is given has no route to the exec port. The supervisor's Service is moca-supervisor (8080, 8081), and on OCP moca-supervisor-tls (8443).

deploy/k8s/
โ”œโ”€โ”€ base/ namespaces, workloads, Services, NetworkPolicy, RBAC
โ”œโ”€โ”€ overlays/kind/ local images (dev.local/...:local), port-forward access
โ”œโ”€โ”€ overlays/kind-ci/ kind + a loopback mock model in the supervisor pod (CI)
โ”œโ”€โ”€ overlays/ocp/ ghostunnel sidecar, Routes, router and openshift-dns rules
โ”œโ”€โ”€ setup.sh bring it up; idempotent
โ”œโ”€โ”€ smoke.sh the live smoke (K8S_LIVE_SMOKE=1)
โ””โ”€โ”€ tests/setup.test.sh setup.sh with every external command mocked

The manifest tests are in packages/supervisor/test/k8s/ and run with make test.

2. Try it on Kindโ€‹

You need:

  • kind 0.24.0 or newer. It is the first release whose default CNI (kindnet, through kube-network-policies) enforces NetworkPolicy, and this deployment's isolation is NetworkPolicy. setup.sh refuses an older kind and says why.
  • Docker, or podman's docker CLI.
  • kubectl, jq, openssl, and sha256sum or shasum.
  • For --build, a checkout with its submodules (git submodule update --init --recursive). The image builds pi-fork itself.
SH_GITHUB_CLIENT_ID=<client id> deploy/k8s/setup.sh --target kind --build

setup.sh creates the kind cluster moca if it is missing, builds both images and loads them, creates the namespaces, generates the Secrets once, applies the overlay, and waits until every workload is rolled out and every sandbox is attached to the relay. Every kube call is pinned to the context kind-moca, never the shell's current one.

kind create cluster switches your current context. When setup.sh creates the cluster, kind sets the current kube context to kind-moca, as it always does. setup.sh itself never uses the current context, but your next bare kubectl will talk to kind until you switch back (kubectl config use-context <previous>).

Image choice, for --target kind:

  • --build builds both images from this checkout.
  • With neither flag, it pulls ghcr.io/rossoctl/moca:latest and ghcr.io/rossoctl/moca-remote-worker:latest (or the refs given with --image and --sandbox-image), and builds only if the pull fails. Until this slice is published to latest, the published harness image has no /readyz, and the supervisor never becomes ready: use --build.
  • --skip-build assumes both images are already loaded, which makes a re-run take seconds.

Each run stamps the loaded images' IDs on the pod templates (moca.dev/image-id). A rebuilt image keeps its :local tag, so without that the apply would see no change and the pods would keep running the old image. With it, a new image rolls the workloads that run it, and an unchanged image rolls nothing.

Other settings come from the environment: SH_ADMIN_SUBJECTS, SH_ALLOW_OPERATOR_FALLBACK (default false), SH_SANDBOX_COUNT (default 2, at most 4 digits; 0 runs no container sandboxes) and SH_WAIT_SECONDS (default 120, the wait for sandboxes to attach). --help prints the synopsis.

When it finishes, it prints how to reach the stack:

P6 is up on kind (context kind-moca). Reach it with two port-forwards:
kubectl --context kind-moca -n moca port-forward svc/moca-supervisor 8080:8080
kubectl --context kind-moca -n moca port-forward svc/moca-control-plane 8090:8080
then: mocactl --control-plane-url http://127.0.0.1:8090 login

Run the two port-forwards in their own terminals, then log in. Export the URL in the terminal you run mocactl from, so every later command uses the same one (mocactl keys its cached login on the control-plane URL):

export SH_CONTROL_PLANE_URL=http://127.0.0.1:8090
mocactl login

The control plane advertises the harness as http://127.0.0.1:8080 by default, so the supervisor's forward must use local port 8080. If 8080 conflicts with another local service, override it with SH_PUBLIC_HARNESS_URL at setup time (see ยง2's sticky settings). mocactl runs from a checkout (docs/harness/how-to/set-up-mocactl.md). The GitHub OAuth app needs Enable Device Flow ticked (Single VM, "The GitHub OAuth app").

Without SH_GITHUB_CLIENT_ID, the control plane is installed with 0 replicas, because nobody can log in without one, and setup.sh says so. It is rendered that way, not scaled down later, so it never starts into a crash-loop. To fix it, re-run with the client id set:

SH_GITHUB_CLIENT_ID=<client id> deploy/k8s/setup.sh --target kind --skip-build

The settings change rolls the control plane, and the supervisor with it: setup.sh stamps a hash of its settings on both pod templates (moca.dev/settings-hash), so the apply itself restarts them exactly when they change, even if an earlier run failed after writing them. The supervisor is in it because it reads the sandbox tiers from the same ConfigMap (ยง11.1); it drains on the roll, with a 120 s grace.

The first apply of a P6.3 release disrupts every session, so run it at a quiet time. It does more than roll the supervisor and the control plane once:

  • It recreates the relay. base/relay.yaml gains SH_SANDBOX_AFFINITY_TTL_SECONDS, and the relay is strategy: Recreate. While it is down, every sandbox and every P4 host is detached, and in-flight turns fail and are retried.
  • It rolls every moca-sandbox pod, even on an untiered stack: base/sandbox.yaml gains SANDBOX_TIER. A container sandbox's workspace is local to its pod, so every container session's workspace is reset. This happens silently: the pod names do not change, so affinity returns each session to the same sandbox id, and no workspace reset notice is shown.

A re-run with nothing changed converges without restarting any pod or touching any Secret. To remove everything: kind delete cluster --name moca.

Inputs are sticky. A re-run keeps every input it is not given, so you change one input by re-running with just that one:

  • SH_GITHUB_CLIENT_ID, SH_ADMIN_SUBJECTS, SH_ALLOW_OPERATOR_FALLBACK, SH_API_TOKEN_TTL_SECONDS and SH_SESSION_TOKEN_TTL_SECONDS live in the ConfigMap moca-settings. A variable that is unset keeps the stored value. A variable set to empty clears it: SH_ADMIN_SUBJECTS= deploy/k8s/setup.sh ... removes every admin, SH_GITHUB_CLIENT_ID= takes the control plane back to 0 replicas, and an empty token lifetime is the control plane's default (900 seconds for the API token a login gets, 300 for a session token). A lifetime is a whole number of seconds; a change rolls the control plane. The one exception to stickiness is kind-ci's placeholder client id, described under the CI variant below.
  • SH_MODEL_CATALOG_HOSTS lives in moca-settings too: the comma-separated bare hostnames the control plane's model catalog may contact besides api.anthropic.com and the default inference endpoint -- add each gateway your users' inference credentials name (docs/reference/configuration.md; without it, listing and choosing models per session degrades to model_catalog_unavailable). Empty, the default, allows only those two. A scheme, port or path in an entry is refused here, matching the control plane's own boot check. A change rolls the control plane. A pre-0.8 stack may carry the allow-list as a hand-patched SH_MODEL_CATALOG_HOSTS env on the control plane Deployment (the manifests did not wire it then); setup.sh carries that value into moca-settings โ€” a value given for the run or one already stored wins over it โ€” and strips the env before its own apply, without which server-side apply would merge the literal with the ConfigMap ref above and the API server would reject the Deployment. The strip rolls the control plane once on its own, just before the apply rolls it again: a one-time cost of the upgrade.
  • SH_WORKSPACE_IDLE lives in moca-settings too: how long a container sandbox keeps a session's workspace after the session's last command, as hours, minutes and seconds in that order (8h, 90m, 1h30m). Empty, the default, is 8h, the value deploy/microvm writes; the worker's own default is 30 minutes, which resets a human-in-the-loop session that waits longer than that. A value without a unit (30), a sign, a fraction or zero is refused, as the worker would refuse it at boot. The sandboxes run in moca-sandbox and cannot read moca-settings, so setup.sh also writes the value to ConfigMap moca-sandbox-settings there, and stamps it on the sandbox pod template (moca.dev/workspace-idle): a change rolls every moca-sandbox pod, which resets every container session's workspace, so change it at a quiet time. A workload's sandboxes read the same key when their pods next start. Disk is not admission-controlled: /workspace is the pod's writable layer (an emptyDir with no sizeLimit on ocp-single), on the node's disk, and no quota or limit bounds it. What bounds it is the idle sweep, which deletes a workspace at most a minute after it has been idle this long, so a longer value holds more idle sessions' files at once. Under node disk pressure the kubelet evicts a sandbox pod, which loses all its workspaces. The reset when the sweep fires is still silent (#338).
  • SH_PUBLIC_HARNESS_URL (kind and --target ocp-single without Routes) overrides the default http://127.0.0.1:8080 the control plane advertises via GET /v1/discovery; set it when 8080 conflicts with another local service. The port in the URL also becomes the printed kubectl port-forward svc/moca-supervisor local port. Must be http://HOST:PORT (no https, no path). Sticky like the other inputs here: a plain re-run keeps the stored value; SH_PUBLIC_HARNESS_URL= returns to http://127.0.0.1:8080. Refused when Routes are on (SH_ROUTE_DOMAIN non-empty), where the Route host is authoritative.
  • SH_SANDBOX_COUNT, and on OpenShift --image and --sandbox-image, live in the ConfigMap moca-setup (namespace moca, nothing secret in it). A given value replaces the stored one. To go back to a default, pass it explicitly (SH_SANDBOX_COUNT=2, --image ghcr.io/rossoctl/moca:latest), or delete the stored key: kubectl -n moca patch configmap moca-setup --type=json -p '[{"op":"remove","path":"/data/IMAGE"}]'. On Kind the images are never stored: the stack always runs the locally loaded dev.local tags.
  • SH_SANDBOX_EGRESS_EXCEPT (ยง6) lives in moca-setup on every target. On OpenShift, SH_P4_SANDBOX_IDS and SH_SANDBOX_DEFAULT_TIER (ยง11) live there too, and on --target ocp-single so do SH_ROUTE_DOMAIN and --tls-secret (ยง12.5). For the variables, as for the settings, one set to empty clears the stored value.

kubectl -n moca get configmap moca-settings moca-setup -o yaml shows what the next run will reuse. A failed read of either ConfigMap aborts the run rather than resetting the inputs.

The CI variant is --target kind-ci. It adds a scripted mock model to the supervisor pod and a placeholder client id, so the smoke can drive real turns with no model key and no login:

deploy/k8s/setup.sh --target kind-ci --build
K8S_LIVE_SMOKE=1 deploy/k8s/smoke.sh --target kind-ci

That placeholder is the one setting that is not sticky, because both kind targets share one cluster and so one moca-settings: a later --target kind run discards it and goes back to 0 control-plane replicas, rather than coming up with an id nobody can log in with. Give that run a real SH_GITHUB_CLIENT_ID to log in.

3. The front door is L4โ€‹

Nothing that speaks HTTP may sit in front of the supervisor's port 8080. That rules out an Ingress, an OpenShift edge or re-encrypt Route, and a service-mesh sidecar.

The reason is that the supervisor works per connection. It accepts a TCP connection, reads the request head, picks a worker (leastInFlight or stickyBySession), admits or refuses the connection against that worker's capacity, and hands the socket itself to the worker (ADR-0034). An L7 proxy terminates the client's connection and pools its own upstream connections, so many users' turns would arrive over a few long-lived connections. Routing, admission and stickiness would then apply to the proxy's connections, not the users', and they would silently stop meaning anything. Nothing would fail visibly.

On Kind, users reach 8080 by kubectl port-forward, which is a byte stream to the pod.

On OpenShift, TLS is terminated inside the supervisor pod by a ghostunnel sidecar, a byte-for-byte TCP proxy on 8443 that forwards to 127.0.0.1:8080. The Route moca is passthrough, so the OpenShift router routes on SNI and never decrypts. Each client connection becomes exactly one supervisor connection. The control plane is ordinary HTTP, so it gets an edge Route; connection reuse there is harmless.

The image is docker.io/ghostunnel/ghostunnel:v1.11.3, pinned by digest in overlays/ocp/kustomization.yaml. Docker Hub is the project's official channel; it does not publish to GHCR.

Each ghostunnel sidecar's readiness probe is GET /_metrics on its own status port (--status; 8453 beside the supervisor, 8454 and 8455 beside the relay's attach and exec sidecars), never a TCP connect to its TLS port, which logged a handshake error every 5 s. /_metrics dials nothing (/_status would open a connection to the target on every probe), and no NetworkPolicy admits the status ports, so only the kubelet's probes reach them. A cluster that needs the node-CIDR fallback for probes (section 10) must admit the status ports there too.

The manifest tests fail on any Ingress or serving.knative.dev object, and on an OCP supervisor Route that is not passthrough to the sidecar's port. Replacing the sidecar with an HTTP proxy would need an ADR superseding point 3 of ADR-0037.

4. Scalingโ€‹

W, the worker count, is pinned, not inferred. base/supervisor.yaml sets three values together:

SettingValueWhy
SH_WORKERS2W
resources.limits.cpu"2"one CPU per worker
resources.requests.memory768Mi (limit 1Gi)at least W ร— 256Mi (the per-process tsx footprint), plus 256Mi headroom
SH_TURNS_PER_WORKER (trial)4, as in Composea trial value; its real value is an output of E8

Change them together. The manifest tests fail when the memory request is below W ร— 256Mi. Leaving SH_WORKERS unset would follow the CPU limit (#341), but then a CPU change would silently change the memory needed.

The supervisor and its whole pool are one container. The workers are fork()ed and receive sockets over IPC, which cannot cross containers. Never split the workers into their own containers or Deployments; the manifest tests check that only the supervisor container runs the supervisor.

More supervisor pods are allowed. Set replicas in an overlay. With leastInFlight (the default SH_ROUTING_POLICY), the workers already share nothing but Redis, so more pods add no new kind of concurrency. With stickyBySession, affinity holds only within one pod: the Service spreads connections across pods without looking at the session.

Async fan-out is replicas ร— SH_WORKERS wide, and nothing autoscales it. With SH_ASYNC_DRAIN=1 (section 14) every worker also runs a drain loop, and a loop takes one queued leaf at a time: the default W=2 on one pod runs two async leaves at once however deep the queue gets, and the rest wait. SH_TURNS_PER_WORKER does not widen that โ€” it only gates whether a worker claims at all.

No autoscaler ships, by design. KEDA's queue-depth scaling went with the Knative path (ADR-0041) and nothing replaced it. The drain shares one admission budget with interactive turns, so queue depth is the wrong signal to size the front door by: a scale-up on a deep queue adds interactive capacity nobody asked for, and a scale-down evicts pods holding leaves mid-run. Instead, watch async_queue_depth and async_oldest_age_ms on the admin port (section 14 โ€” depth alone cannot tell a deep fast queue from a stalled shallow one), alert on them, and set replicas yourself.

The relay stays at one replica (strategy: Recreate). It is presence-only: each Attach stream is held by one process, so two replicas would split the sandboxes between them and an exec could reach the wrong one.

Sandboxes: SH_SANDBOX_COUNT for setup.sh sets the StatefulSet's replicas. Each pod's SANDBOX_ID is its name (moca-sandbox-0, -1, ...), so replicas never collide on one sh:sandbox:records entry.

5. Secretsโ€‹

setup.sh generates every Secret on its first run and never rotates one. A re-run keeps every existing value, and fills in only a key that is missing. It reads each Secret with --ignore-not-found, so "absent" is the only state that counts as missing: any other API error (a timeout, an RBAC denial, an expired login) aborts the run instead of looking like a missing Secret and replacing it. No value is ever put on a command line. Values travel through pipes and through the environment of the one jq that writes each Secret, and are applied on stdin.

SecretNamespaceKeysConsumed by
moca-relaymocaSH_RELAY_TOKEN, MOCA_RELAY_EXEC_TOKENrelay (both, env); supervisor (MOCA_RELAY_EXEC_TOKEN only, env)
moca-redismocaREDIS_PASSWORD, REDIS_URL, redis.confRedis (redis.conf as a file; REDIS_PASSWORD as REDISCLI_AUTH for probes); supervisor, relay, control plane (REDIS_URL)
moca-redis-provisionermocaREDIS_PASSWORD, REDIS_URL (only with SH_WORKLOAD_SANDBOXES=1; kept when off)provisioner (REDIS_URL: its own Redis user, GET on sh:wl:* only)
moca-mu1mocaSH_SESSION_TOKEN_PRIVATE_KEY, SH_SESSION_TOKEN_PUBLIC_KEYS, SH_CREDENTIAL_KEK, SH_EXCHANGE_TOKENcontrol plane (private key, KEK, exchange token as files); supervisor (exchange token as a file, public keyset as env)
moca-test-usersmocaSH_TEST_USER_SECRET (only while SH_TEST_USERS > 0; deleted when off)control plane (as a file, from an optional projected source; ADR-0045, ยง13)
moca-relay-attachmoca-sandboxSH_RELAY_TOKEN (copied from moca-relay)sandboxes
moca-supervisor-tlsmoca (OCP)tls.crt, tls.keyghostunnel sidecar
  • The tokens and the Redis passwords are openssl rand -hex 32.
  • REDIS_URL and redis.conf are re-derived from the Redis passwords on every run (REDIS_PASSWORD, and with workload sandboxes on, moca-redis-provisioner's too, for its user line), so they never disagree.
  • With workload sandboxes on, moca-provisioner reads Redis as a user of its own, never with REDIS_PASSWORD (#626). Its orphan sweep only reads workload records, so the user may run GET on sh:wl:* and nothing else: no write, no other key, no Pub/Sub. redis.conf defines it beside requirepass, holding the SHA-256 of its password, not the password itself. The password is in moca-redis-provisioner, kept once made, and its REDIS_URL is redis://moca-provisioner:<password>@redis.<ns>.svc:6379. Redis reads redis.conf only at start, so setup.sh puts a hash of the user's line on Redis's pod template: adding the user, changing its password or turning the feature off rolls redis-0.
  • The MU1 keys come from the control plane's own generator, run once in the harness image as the pod moca-genkeys. They are validated with deploy/compose/install.sh's regexes before anything is written, so a garbled generator never leaves half a set. See Troubleshooting for where those values sit while the pod runs.
  • moca-supervisor-tls is replaced only when --tls-cert is given (section 9).

To rotate on purpose, delete the Secret, re-run setup.sh, then restart what consumes it. The re-run needs no other input: settings, the sandbox count and (on OpenShift) the images are sticky (section 2), so it does not reset them. For example, the relay tokens on Kind:

kubectl --context kind-moca -n moca delete secret moca-relay
deploy/k8s/setup.sh --target kind --skip-build
kubectl --context kind-moca -n moca rollout restart deployment/sandbox-relay deployment/moca-supervisor
kubectl --context kind-moca -n moca-sandbox rollout restart statefulset/moca-sandbox

The consumers are in the table. For moca-redis, restart redis as well. For moca-redis-provisioner, the re-run rolls redis itself; restart deployment/moca-provisioner (it reads REDIS_URL at start). For moca-mu1, restart the control plane and the supervisor.

Rotating moca-mu1 logs everyone out: a new signing key invalidates every API and session token. If the KEK changes, every stored credential becomes unreadable: the per-user credentials in moca-credentials are sealed under SH_CREDENTIAL_KEK, and there is no way back without the old KEK. A half keypair (a private key without its public keyset, or the reverse) is refused: delete both keys to regenerate the pair.

6. Isolationโ€‹

Each namespace has a default-deny NetworkPolicy, both directions. Each workload's own policy then allows exactly its edges. This replaces the VM's podman isolate=strict and nftables.

PodIngress allowedEgress allowed
supervisor8080: nothing from the pod network (port-forward and the OCP sidecar's loopback are not filtered). 8443 (OCP): the router namespace. 8081: pods in moca; kubelet probes are admitted by the CNI (section 10).DNS; redis 6379; sandbox-relay 9444; moca-control-plane 8080; 0.0.0.0/0 TCP 443 (model API)
relay9443 from pods in moca-sandbox; 9444 from moca-supervisor pods onlyDNS; redis 6379
Redis6379 from supervisor, relay, control planenone
control plane8080 from moca-supervisor pods; the router namespace (OCP edge Route)DNS; redis 6379; 0.0.0.0/0 TCP 443 and 6443 (kube API, GitHub)
sandboxnoneDNS; sandbox-relay 9443; 0.0.0.0/0 except 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 100.64.0.0/10, 169.254.0.0/16 (all ports)

DNS is kube-system port 53; the OCP overlay adds openshift-dns port 5353.

The sandbox rule lets research turns reach the internet while blocking every cluster-internal address: Pod and Service CIDRs sit inside the private ranges on Kind and OCP. It also blocks cloud instance metadata, which the VM path leaves open (#357).

A publicly routable node or infrastructure network is not covered. The rule assumes everything that is not the internet uses one of those ranges. Some clusters put their nodes on a publicly routable range, and there the rule admits them: a sandbox can reach its node's kubelet (10250) and SSH (22), and other hosts on that network (ยง12.3, #446). Add those ranges with SH_SANDBOX_EGRESS_EXCEPT, a comma-separated list of IPv4 CIDRs appended to the except list on every target:

SH_SANDBOX_EGRESS_EXCEPT=203.0.113.0/24,198.51.100.0/24 deploy/k8s/setup.sh --target ocp ...

It is empty by default, so the rule is the one in the table. Each entry must be a canonical CIDR (network address, prefix 1-32) not already in the built-in list; setup.sh refuses anything else before it touches the cluster. It is sticky (section 2): a re-run keeps the list, and SH_SANDBOX_EGRESS_EXCEPT= clears it. Find the node network with kubectl get nodes -o wide (the INTERNAL-IP column) when you have node read access, or ask the cluster administrator when you do not (--target ocp-single).

The only Secret any moca object references in the sandbox namespace is the attach token (OpenShift adds its own per-ServiceAccount pull Secrets there): the exec token, the Redis password and the MU1 keys never exist there. The manifest tests check that nothing in it references any other Secret.

What the smoke's claim 7 proves. From inside moca-sandbox-0, TCP connects to redis.moca.svc:6379, sandbox-relay-exec.moca.svc:9444 and 169.254.169.254:80 are blocked, and sandbox-relay-attach.moca.svc:9443 is open. On --target ocp it also requires kubernetes.default.svc:443 to be blocked. On Kind it only prints that probe's result as a note: single-node kindnet does not filter traffic to its own node (section 10).

A CNI that enforces NetworkPolicy is a hard requirement. On a CNI that ignores NetworkPolicy, every row above is open, and nothing else stands in. kind โ‰ฅ 0.24.0 and OpenShift's OVN-Kubernetes enforce it.

7. What this does not claimโ€‹

These are the container tier's non-claims, the same as on the VM (docs/demos/vm-multi-user-demo.md, "Notes and limits").

  • No sandbox isolation between users. Every user's turns lease the same sandbox pods, which share /workspace, the Unix user and the process list (remote-worker/internal/exec/runner.go ignores workspace_key, #408). The fix is MI1 S5, owner binding.
  • Direct credential mode. With no injector, a user's real inference secret reaches the shared harness worker for the length of a turn, which puts it in the model request's headers. The sandbox never receives it: an Exec carries only the command, stdin, timeout and workspace key. So the agent's bash cannot read a user's key, but one harness process holds every user's key during their turns. MI1 S2's grants replace this.
  • Open egress from sandboxes, to the whole internet. Unlike the VM, not to instance metadata and not to cluster ranges (section 6). Egress control is MI1 S5's moca-egress.

8. CIโ€‹

.github/workflows/k8s-e2e.yml, job k8s-kind-e2e, runs on pushes to main and on pull requests touching deploy/k8s/**, the mock model, packages/**, harness/**, remote-worker/**, pi-fork, the Dockerfile or the lockfile. It installs kind v0.33.0 (checked by sha256), runs SH_ASYNC_DRAIN=1 setup.sh --target kind-ci --build โ€” the drain is opt-in (section 14) and the smoke's async claims skip themselves without it โ€” then K8S_LIVE_SMOKE=1 smoke.sh --target kind-ci under pipefail, so a failed claim fails the job. On failure it uploads the artifact k8s-kind-e2e-logs: pods, describe, every container's log, events, the smoke's output and its kept log directory, with every token header file deleted first.

It is not a required check until it has passed ten consecutive runs on main. Until then, a red run is investigated, not waved through, but it does not block a merge.

9. Demo on OpenShiftโ€‹

This is the slice's acceptance run: the container-tier acts of docs/demos/vm-multi-user-demo.md, against --target ocp. It is performed by a person and recorded on rossoctl/moca#423. Every outward step (pushing an image, creating objects on a shared cluster, posting on the issue) needs that person's go-ahead.

You need:

  • an OpenShift 4.x cluster where you can grant SCCs (cluster-admin or equivalent), and oc login done;
  • a harness image built from this branch, pushed to a registry the cluster can pull from (the published latest predates /readyz);
  • a GitHub OAuth app with Enable Device Flow ticked;
  • two GitHub accounts, two laptops (or one laptop with two XDG_CONFIG_HOME directories and a private browser window, as the VM demo's 1a trap describes), and two inference credentials;
  • optionally, a certificate and key valid for the supervisor Route's host, moca-moca.<apps domain>.

The cluster pulls ghostunnel and valkey/valkey:9-alpine from Docker Hub, so it needs access to docker.io.

9.1 Bring it upโ€‹

Without a certificate of your own (setup.sh generates a self-signed one, see below):

export LOG_DIR=/tmp/kagenti/tdd/moca; mkdir -p "$LOG_DIR"
SH_GITHUB_CLIENT_ID=<client id> SH_ADMIN_SUBJECTS= \
deploy/k8s/setup.sh --target ocp --image <registry>/moca:<branch-tag> \
>"$LOG_DIR/t19-setup.log" 2>&1; echo "EXIT:$?"
tail -6 "$LOG_DIR/t19-setup.log"

With a certificate and key valid for moca-moca.<apps domain>:

export LOG_DIR=/tmp/kagenti/tdd/moca; mkdir -p "$LOG_DIR"
SH_GITHUB_CLIENT_ID=<client id> SH_ADMIN_SUBJECTS= \
deploy/k8s/setup.sh --target ocp --image <registry>/moca:<branch-tag> \
--tls-cert <fullchain.pem> --tls-key <key.pem> >"$LOG_DIR/t19-setup.log" 2>&1; echo "EXIT:$?"
tail -6 "$LOG_DIR/t19-setup.log"

setup.sh --target ocp checks oc whoami and prints the target context. It reads the apps domain from ingresses.config/cluster, grants nonroot-v2 to each ServiceAccount before the apply, and sets both Route hosts: moca-moca.<apps domain> (supervisor, passthrough) and moca-control-plane-moca.<apps domain> (control plane, edge). --sandbox-image overrides the sandbox image the same way --image overrides the harness.

Without --tls-cert, it generates a self-signed certificate for the supervisor host (825 days) and prints a warning. Every run then writes the file mocactl users trust, and prints the line they need:

export NODE_EXTRA_CA_CERTS=<checkout>/deploy/k8s/.generated/ocp/moca-ca.crt

moca-ca.crt holds the supervisor's certificate when it is self-issued, and the cluster's default ingress CA (ca-bundle.crt of openshift-config-managed/default-ingress-cert), which signs the control plane's edge Route and is usually not publicly trusted either. Both are read back from the cluster on every run, so a re-run, or a run from a second checkout, prints the line and rebuilds the file too. If the ingress CA cannot be read, the run warns and the file holds the supervisor's certificate alone. With an issued --tls-cert, the file holds only the ingress CA.

Copy that file to both laptops and export it there before running mocactl.

Look for: EXIT:0, N sandbox(es) attached to the relay, and the two Route URLs.

If redis-0 is never created, check for the SCC:

kubectl -n moca get events | grep -i 'security context constraint'

unable to validate against any security context constraint means the explicit Redis UID 999 (or its fsGroup 1000) is refused, even with nonroot-v2. The fallback (spec ยง10) is a patch in overlays/ocp that drops runAsUser, runAsGroup and fsGroup from the Redis pod's securityContext, so restricted-v2 assigns both the UID and the fsGroup from the namespace's range:

# overlays/ocp/kustomization.yaml, under patches:
- target: { kind: StatefulSet, name: redis }
patch: |-
- { op: remove, path: /spec/template/spec/securityContext/runAsUser }
- { op: remove, path: /spec/template/spec/securityContext/runAsGroup }
- { op: remove, path: /spec/template/spec/securityContext/fsGroup }

Keeping fsGroup: 1000 does not work: restricted-v2 refuses it (its fsGroup must lie in the namespace's range), so nonroot-v2 admits the pod with no UID at all, and the image runs as root. The symptom is redis-0 in CreateContainerConfigError, with container has runAsNonRoot and image will run as root in kubectl -n moca describe pod redis-0.

Commit it, re-run setup.sh, and record that it was needed.

9.2 Live smoke on OpenShiftโ€‹

With a real model. Put the token in the environment from a password manager, never typed on a command line (the smoke reads it from the environment only):

export SMOKE_MODEL_TOKEN="$(<your password manager's CLI>)"
K8S_LIVE_SMOKE=1 SMOKE_MODEL_URL=https://api.anthropic.com SMOKE_MODEL_KIND=api-key \
deploy/k8s/smoke.sh --target ocp 2>&1 | tee "$LOG_DIR/t19-smoke.log"
unset SMOKE_MODEL_TOKEN

SMOKE_MODEL_KIND is api-key for a raw Anthropic key and bearer (the default) for a gateway token. The smoke reaches everything by port-forward, as on Kind; the Route path is the demo's.

Look for: PASS=12 FAIL=0. On OCP, claim 7 must also report kube API ... BLOCKED. Run the smoke before the acts: claim 9 deletes the supervisor pod and claim 10 deletes redis-0.

Re-check two things that Kind could not prove, because OVN-Kubernetes behaves differently from kindnet:

  • Probes. On Kind, kubelet probes are admitted because kindnet accepts all traffic the node's root originates. If the supervisor or control plane is not Ready under OVN-Kubernetes, and kubectl -n moca describe pod shows probe timeouts, apply the node-CIDR fallback (section 10).

  • The kube API and the kubelet from a sandbox. Claim 7 probes kubernetes.default.svc:443. Also probe the API server's endpoint address and a node's kubelet port directly, which is how the Kind spike found the gap:

    kubectl get endpoints kubernetes -n default -o jsonpath='{.subsets[0].addresses[0].ip}'; echo
    kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}'; echo
    for t in <apiserver ip>:6443 <node ip>:10250; do
    kubectl -n moca-sandbox exec moca-sandbox-0 -c sandbox -- bash -c \
    'timeout 3 bash -c "</dev/tcp/${0%:*}/${0#*:}" 2>/dev/null && echo "$0 OPEN" || echo "$0 BLOCKED"' "$t"
    done

    Both must be BLOCKED.

9.3 The actsโ€‹

One user per laptop. On each, mocactl runs from a checkout (docs/harness/how-to/set-up-mocactl.md):

alias mocactl="node $PWD/packages/mocactl/bin/mocactl.mjs"
export NODE_EXTRA_CA_CERTS=<the CA file>
export SH_CONTROL_PLANE_URL=https://moca-control-plane-moca.<apps domain>

mocactl renews its API token by itself, for 30 days after its last use (90 at most), so one login before the acts is enough. The token lasts 15 minutes unless setup.sh was run with SH_API_TOKEN_TTL_SECONDS (ยง2's sticky settings). A longer value buys nothing with a renewing login and costs revocation latency: mocactl logout stops new tokens at once, but a token already minted stays valid for its whole lifetime. MOCA_TENANCY is unset, as on the VM demo; see its note on tenancy.

L4 baseline (operator), before act 1. Record the supervisor's counters:

kubectl -n moca exec deploy/moca-supervisor -c supervisor -- wget -qO- 127.0.0.1:8081/metrics |
jq '.counters'

Act 1. Log in, and doctor. Both users:

mocactl login
mocactl doctor

Look for: logged in as <GitHub name> (the device flow approves for whichever account the browser is signed into), and every doctor check green. --control-plane-url URL on each command does the same as SH_CONTROL_PLANE_URL.

Act 2. Each user's own credential. Each user stores theirs, under a name of their own (the secret is read from stdin), then runs a turn:

mocactl credentials add anthropic-user1 --host api.anthropic.com --kind api-key \
--endpoint https://api.anthropic.com </path/to/a/0600/file/holding/the/key
mocactl run "Run uname -a in your sandbox and reply with its output only."

A gateway token is --kind bearer with its own host and endpoint. Then the operator shows whose credential each turn spent:

kubectl -n moca exec redis-0 -- sh -c 'redis-cli XREVRANGE sh:cp:audit + - COUNT 6'

Look for: for each user's session, session_created and credential_issued entries with that user's subject and their own credential name. The audit records names, never values.

Act 3. Disjoint sessions, and another user's session does not exist. Each user:

mocactl sessions

Each list holds only that user's sessions. User 1 sends a session id to user 2, who tries it, then an id that never existed:

mocactl run "what did you find?" --session <user 1's session id>; echo "exit $?"
mocactl run "what did you find?" --session sess-does-not-exist; echo "exit $?"

Look for the same answer both times, that session no longer exists, or is not yours and exit 1. User 1 then resumes it, to show it was real.

Act 4. Research from the sandbox. Each user runs the VM demo's 2a prompt, with D=research-user1 or D=research-user2, and then its 2b check on any machine (git ls-remote https://github.com/rossoctl/moca HEAD | cut -c1-12 and the nodejs.org index). Look for the agent's COMMIT=, NODE_VERSION= and NODE_DATE= lines matching the world. Then the honest beat, where the two research directories landed:

for i in 0 1; do echo "moca-sandbox-$i:"; kubectl -n moca-sandbox exec moca-sandbox-$i -c sandbox -- ls /workspace; done

Both users' turns lease the same pods (section 7).

L4 check (operator), after the acts. This is spec ยง10's third risk: the passthrough path must keep connections 1:1. Read the counters again (as in the baseline), and while two users' turns are running at once, count the established connections inside the supervisor pod, on 8443 (from the router) and on 8080 (from the sidecar). Read both socket tables: ghostunnel is a Go program, and its 0.0.0.0:8443 listener is a dual-stack IPv6 socket, so the router's connections appear in /proc/net/tcp6 (as ::ffff: addresses), not in /proc/net/tcp:

kubectl -n moca exec deploy/moca-supervisor -c supervisor -- awk \
'$4 == "01" { split($2, a, ":"); if (a[2] == "20FB") s++; if (a[2] == "1F90") d++ }
END { print "8443:", s + 0, "8080:", d + 0 }' /proc/net/tcp /proc/net/tcp6

Look for: the two numbers equal, and as many as the clients connected (two concurrent turns from two laptops give at least 2). A router that pooled would show fewer 8443 connections than clients. Take two readings a few seconds apart: the sidecar's tcpSocket readiness probe and keep-alive connections come and go. Stop any port-forward to the supervisor first, since its connections count on 8080 too.

/metrics has no per-connection hand-off counter, so record handoff_retries, handoff_failures and over_admission before and after as context only. over_admission is not a pooling signal on its own: in the Kind spike it rose once per turn, with no proxy in front at all. If the connections are not 1:1, the fallback is an L4 LoadBalancer Service for the sidecar's port.

9.4 Recordโ€‹

Post on rossoctl/moca#423, with the go-ahead: the commit and image, the setup tail, the smoke summary, the probe and kube-API re-checks, each act's result, the connection counts and counters, and any fallback applied (the Redis SCC patch, the node-CIDR policy). Then tick slice 1 in epic #426.

9.5 Cleanupโ€‹

The smoke's residue. smoke.sh stores a credential smoke-inference and creates sessions, all under the subject smoke:1, and deletes none of them. Deleting the namespaces (below) removes them. To keep the stack, delete them through the control plane with a token minted the way the smoke mints it, inside the control-plane pod:

kubectl -n moca port-forward svc/moca-control-plane 18090:8080 >/dev/null 2>&1 &
PF=$!; sleep 3
HDR="$(mktemp)"
kubectl -n moca exec deploy/moca-control-plane -c control-plane -- \
node --import tsx --input-type=module -e \
"import { readFileSync } from 'node:fs'; import { makeSigner } from './src/token.ts';
const s = makeSigner(readFileSync('/run/credentials/SH_SESSION_TOKEN_PRIVATE_KEY', 'utf8'));
process.stdout.write(s.mint({ sub: 'smoke:1', tenant: 'smoke:1', roles: [], scope: ['api'], ttlSeconds: 900 }));" |
sed 's/^/Authorization: Bearer /' >"$HDR"
curl -s -o /dev/null -w '%{http_code}\n' -X DELETE -H @"$HDR" http://127.0.0.1:18090/v1/credentials/smoke-inference
for id in $(curl -s -H @"$HDR" http://127.0.0.1:18090/v1/sessions | jq -r '.sessions[].sessionId'); do
curl -s -o /dev/null -w "$id %{http_code}\n" -X DELETE -H @"$HDR" "http://127.0.0.1:18090/v1/sessions/$id"
done
rm -f "$HDR"; kill "$PF"

Each delete answers 204.

On each laptop, delete the run's sessions (mocactl sessions delete ID) and credentials (mocactl credentials delete NAME), and run mocactl logout (it revokes the login on the control plane and deletes auth.json). On the cluster, delete the three namespaces (moca, moca-sandbox, moca-credentials); deleting moca deletes the Redis PVC with it. The nonroot-v2 grants are cluster objects: oc adm policy remove-scc-from-user nonroot-v2 -z <sa> -n <ns> for each ServiceAccount.

10. Troubleshootingโ€‹

The commands here are written for Kind (--context kind-moca, as setup.sh and smoke.sh always pin it). On OpenShift, drop --context kind-moca.

Kind on macOS with Podman. Both workarounds were needed on the Task 16 run (macOS arm64, Podman 5.7.1 behind the docker CLI). Neither is expected on Linux with Docker.

  • A proxy in your shell. kind create copies HTTP_PROXY/HTTPS_PROXY/NO_PROXY into the node's containerd. A proxy on 127.0.0.1 then points at the node itself, and in-cluster pulls (valkey/valkey:9-alpine) fail with proxyconnect tcp: dial tcp 127.0.0.1:.... Create the cluster without them, then run setup.sh as usual (it reuses an existing moca cluster):

    env -u HTTPS_PROXY -u https_proxy -u HTTP_PROXY -u http_proxy -u NO_PROXY -u no_proxy \
    kind create cluster --name moca

    A cluster already created with the proxy has to be deleted and recreated.

  • Flaky DNS from pods. The node's /etc/resolv.conf from the Podman network lists an IPv6 nameserver first that is unreachable, and the IPv4 one timed out under load. Research turns then fail with Could not resolve host. Point CoreDNS at public resolvers:

    kubectl --context kind-moca -n kube-system get configmap coredns -o yaml |
    sed 's#forward . /etc/resolv.conf#forward . 1.1.1.1 8.8.8.8#' |
    kubectl --context kind-moca apply -f -
    kubectl --context kind-moca -n kube-system rollout restart deployment/coredns

    Before suspecting the egress policy, look at kubectl --context kind-moca -n kube-system logs -l k8s-app=kube-dns for upstream errors.

Where the smoke keeps its logs. On any failed claim, smoke.sh keeps its working directory and prints logs kept in <dir> (a mktemp -d directory, mode 0700). It holds the port-forward logs and each turn's SSE stream; the files holding tokens are deleted first. A run that passes removes it. A smoke that stops before the claims (port-forward to the supervisor never came up) prints the port-forward log itself and keeps nothing. In CI, the directory is in the k8s-kind-e2e-logs artifact.

The generated MU1 keys sit in the moca-genkeys pod's log until --rm deletes it. The generator writes the four values to its stdout, which setup.sh reads over the attach, so for the pod's few seconds of life they are also in its container log on the node, readable by anyone with pods/log in moca. --rm deletes the pod and its log when the run completes. If setup.sh is killed in that window, delete it yourself: kubectl --context kind-moca -n moca delete pod moca-genkeys --ignore-not-found (setup.sh also deletes a leftover one before it starts).

On single-node Kind, a sandbox can reach the kube API and its own node's kubelet. kubernetes.default.svc:443 and the node's port 10250 are open from a sandbox, although the egress rule excepts 10.0.0.0/8. kindnet enforces NetworkPolicy only in the forwarding path, and pod-to-own-node traffic never passes it. The exposure is anonymous only: sandboxes mount no ServiceAccount token, so /version answers and secrets is forbidden. No manifest change can fix it on kindnet, so the smoke reports it as a note on Kind and enforces it on --target ocp (section 9.2). Kind CI therefore does not prove the kube-API carve-out; the OpenShift run does. kindnet also accepts all UDP/53, so the DNS policies are not exercised on Kind either.

Kubelet probes and the node-CIDR fallback. If NetworkPolicy filtered the kubelet's probes, the supervisor (8081) and control plane (8080) would never become Ready, with probe timeouts in kubectl describe pod. The fallback (spec ยง10) is a per-overlay policy allowing ingress from the node CIDR to those two ports:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: probes-from-nodes, namespace: moca }
spec:
podSelector:
{
matchExpressions: [{ key: app, operator: In, values: [moca-supervisor, moca-control-plane] }],
}
policyTypes: [Ingress]
ingress:
- from: [{ ipBlock: { cidr: <node CIDR> } }]
ports: [{ protocol: TCP, port: 8081 }, { protocol: TCP, port: 8080 }]

On OpenShift, the ghostunnel sidecars are probed too, on their status ports: 8453 on the supervisor pods, 8454 and 8455 on the relay pods (section 3). A cluster that needs this fallback adds sandbox-relay to the selector's values and { protocol: TCP, port: 8453 }, { protocol: TCP, port: 8454 } and { protocol: TCP, port: 8455 } to its ports, or the sidecars stay NotReady.

It was not needed on Kind: kindnet accepts traffic originated by the node's root user before any policy, which covers the kubelet. It is still to be checked on OVN-Kubernetes (section 9.2).

The control plane stays 0/1. It reconnects to Redis forever, with a bounded backoff, and its /readyz fails fast while Redis is down, so it recovers by itself once Redis is up (the common case on a first apply, since everything is applied at once). If it stays unready, its log shows [control-plane] redis client error: ...; check redis-0. Two smokes run back to back can catch claim 5 inside that 5โ€“10 s reconnect window after the previous run's Redis restart: wait for /readyz, or re-run setup.sh, in between.

only 0 of N sandboxes attached to the relay. The relay retries each sandbox's presence write until it lands or the sandbox detaches, so a late Redis delays the records but no longer loses them. If the count stays short, the message names the two logs to read: kubectl --context kind-moca -n moca logs deployment/sandbox-relay (presence put failed ...) and kubectl --context kind-moca -n moca-sandbox logs statefulset/moca-sandbox.

redis-0 stays Pending. Its PVC binds only through a default StorageClass. setup.sh warns when the cluster has none: mark one default (storageclass.kubernetes.io/is-default-class=true).

A container restarted. Smoke claim 11 fails on any restart in either namespace. The likeliest is the supervisor OOM-killed at its 1Gi limit (kubectl --context kind-moca -n moca describe pod shows OOMKilled): raise the memory request and limit together (section 4). A worker that dies inside the pod is not a container restart: look for worker_exit in the supervisor's log.

Claim 12 notes could not tell affinity from luck. Its four first turns run at once so the least-loaded choice spreads them over the sandboxes; when they all land on one, every second turn matches by luck too. The claim still passes; re-run the smoke for a run that proves affinity. A claim 12 FAIL names the session that moved and both pod names.

The supervisor never becomes Ready on a pulled image. The published latest may predate /readyz, so the startup probe fails. Use --build on Kind, or a branch image with --image on OCP.

port-forward is filtered. Not seen on Kind (port 8080 has no ingress rule at all, yet answers through port-forward). On a CNI that does filter it, smoke claims 1, 3 and 5 fail; the fallback is an allow from the node CIDR on that overlay, as for the probes.

11. P4 on Kubernetes: microVM hosts outside the clusterโ€‹

P6 runs in the cluster; only the P4 microVM tier needs a KVM host (bare metal or a nested-virt VM). Each such host runs microvm-worker and attaches to the in-cluster relay over TLS, through an L4 sidecar (ghostunnel) behind the passthrough Route moca-relay. The cluster issues each host's token; the relay reads it from a mounted Secret on every attach, so adding a host restarts nothing. Design: docs/specs/2026-10-04-p6-on-kubernetes-slice2-design.md.

OpenShift only: setup.sh refuses SH_P4_SANDBOX_IDS on kind and on ocp-single. A stack may run P4 hosts beside its container sandboxes, and several P4 hosts: since P6.3 (#425) a session returns to the sandbox that holds its workspace (docs/specs/2026-10-04-p6-on-kubernetes-slice3-design.md).

11.1 Container sandboxes and P4 together, or P4 onlyโ€‹

# A mixed stack: two container sandboxes and one P4 host.
SH_SANDBOX_COUNT=2 SH_P4_SANDBOX_IDS=moca_microvm_0 deploy/k8s/setup.sh --target ocp
# P4 only.
SH_SANDBOX_COUNT=0 SH_P4_SANDBOX_IDS=moca_microvm_0 deploy/k8s/setup.sh --target ocp

A mixed stack is tiered (SH_SANDBOX_COUNT above 0 and SH_P4_SANDBOX_IDS not empty): setup.sh writes SH_SANDBOX_TIERS=container,microvm to moca-settings, which the supervisor and the control plane both read, and logs two sandbox tiers. A session is created in a tier and every turn of it runs there. A user picks the tier at creation, in mocactl's New Session overlay or with mocactl run --new --option sandboxTier=microvm; a session that names none gets the default tier.

  • SH_SANDBOX_DEFAULT_TIER (container or microvm, default container) sets that default. It is sticky, stored in moca-setup; SH_SANDBOX_DEFAULT_TIER= clears it back to container. Any other value is refused before anything is stored. Kind is never tiered and stores none, but it refuses a bad value too.
  • A single-tier stack stays untiered (containers only, or P4 only): both tier settings are empty, and a stored SH_SANDBOX_DEFAULT_TIER is kept but unused. Untiered on purpose: a worker older than P6.3 advertises no tier, and a tiered supervisor would exclude it.
  • On a mixed stack every worker must be P6.3 or later (the container sandboxes' image and each P4 host's microvm-worker): an older one advertises no tier and receives no sessions.
  • Any settings change now rolls the supervisor too (section 2): a change of client id or admins restarts both Deployments, and the first run of a P6.3 setup.sh on an existing stack rolls both once, because the settings it hashes gained the two tier keys.
  • That first run also recreates the relay and rolls the container sandboxes (section 2), on every stack, tiered or not. Every sandbox and P4 host detaches while the relay is down, and every container session's workspace is reset with no workspace reset notice, since each session returns to a sandbox of the same name. Run it at a quiet time.

The rest applies to every stack with P4 hosts:

  • IDs match ^[A-Za-z_][A-Za-z0-9_]*$ (no dashes: the relay looks each up by name), comma-separated, each listed once.
  • DIR is reserved and refused: SH_RELAY_TOKEN_DIR is the relay's token-directory setting, so the relay refuses that ID before any lookup.
  • The list is sticky like every other input; SH_P4_SANDBOX_IDS= (empty) clears it (see 11.4 for the objects that clearing leaves behind).
  • --relay-tls-cert FILE --relay-tls-key FILE installs an operator certificate for moca-relay-moca.<apps domain>. Without them a self-signed one is generated once and kept.
  • Every bundle carries the relay certificate as relay-ca.crt whenever it is self-issued (setup.sh's own, or a self-signed one you supplied). A certificate issued by another CA is not shipped: that CA must be one the hosts already trust system-wide.

setup.sh ends by printing, per ID, the bundle path and the commands to install it:

P4 hosts attach to https://moca-relay-moca.<apps domain> (the relay, TLS passthrough). One bundle each -- it holds
that host's relay token, so copy it straight to the host, install it, then delete the copy:
moca_microvm_0: <checkout>/deploy/k8s/.generated/ocp/p4/moca_microvm_0
ssh <kvm-host> rm -r moca-p4-moca_microvm_0 (an earlier copy, if any: scp -r would nest into it)
scp -r <checkout>/deploy/k8s/.generated/ocp/p4/moca_microvm_0 <kvm-host>:moca-p4-moca_microvm_0
on <kvm-host>, from a moca checkout: sudo deploy/microvm/setup-microvm.sh --remote ~/moca-p4-moca_microvm_0

The bundle (worker.env and relay-ca.crt, mode 0600, in a 0700 directory) holds that host's relay token. Copy it straight to its host and delete the copy once installed.

Re-copying a bundle (after a new certificate, or an ID removed and added again): first remove ~/moca-p4-<id> on the host, as the printed ssh <kvm-host> rm -r moca-p4-<id> does. scp -r <dir> <kvm-host>:moca-p4-<id> into an existing directory nests the copy as moca-p4-<id>/<id>, and --remote would then install the stale bundle left at the top. On a first copy the rm only reports that nothing is there.

11.2 Install a hostโ€‹

The host needs what MicroVM tier (P4) lists for P4 (/dev/kvm, Firecracker and jailer, a golden snapshot, Go or MICROVM_BIN) and outbound TCP 443 to the relay Route host. Check that first with curl -v https://moca-relay-moca.<apps domain>: a completed TLS handshake proves it is reachable, even when curl then stops at a certificate verification error (as it does with the default self-signed certificate); a timeout or Connection refused does not. It does not need P6 installed.

setup-microvm.sh --remote <bundle> installs the worker with the bundle's address, TLS, ID and token (the cluster's token replaces any local one), installs relay-ca.crt as /etc/serverless-harness/microvm-relay-ca.crt, and swaps the worker's drop-in to 50-moca-remote.conf. Of a local P6 it touches only the tier files: on a host an earlier mixed local run tiered, it removes microvm-tiers.env and both drop-ins (sh-supervisor.service.d/50-microvm-tiers.conf, sh-control-plane.service.d/50-microvm-tiers.conf) and try-restarts the installed sh-supervisor and sh-control-plane, because the host's own P6 is container-only from then on (P6.3 spec ยง7).

It finishes only once the worker's current process has logged attached, serving execs and the attach has then held for MICROVM_ATTACH_SETTLE seconds (default 5, at least 1) with no reconnect, all within MICROVM_ATTACH_TIMEOUT (default 60; a hold that starts before then may finish). The hold matters because the worker logs attached when its stream opens, before the relay checks the token; a refused token ends the stream milliseconds later. Otherwise it dies naming that process's journal (journalctl -u microvm-worker _SYSTEMD_INVOCATION_ID=โ€ฆ), which carries the TLS or token error.

Running setup-microvm.sh again without --remote switches the host back to its local P6 relay, restoring its local token.

11.3 Verifyโ€‹

K8S_LIVE_SMOKE=1 SMOKE_MODEL_URL=โ€ฆ SMOKE_MODEL_TOKEN=โ€ฆ deploy/k8s/smoke.sh --target ocp --tier p4

Claims 1, 5 and 6 are the container tier's. P2: every P4 ID is attached. On a P4-only (untiered) stack no container sandbox may be; on a tiered stack container sandboxes may be attached beside them, and each P4 ID's record must advertise the microvm tier (moca.dev/tier). P3: a turn runs on a kernel that is no node's and writes a file. P4: a second turn reads it back. P6: through the relay Route, a wrong attach token is refused at the relay, and SandboxExec is not served even with the exec token. P7 (only with SMOKE_P4_ADD_ID=<scratch id>): setup.sh adds that ID and removes it again, both runs keeping the sticky SH_SANDBOX_COUNT (so a mixed stack keeps its container sandboxes), and the relay pod is the same one, with the same start time and no restart. Claims 7โ€“12 do not run on this tier. On a tiered stack every smoke session asks for its tier (microvm here, container without --tier p4) and the smoke prints a note naming the stack's tiers, so both smokes test the right tier whatever the default; on an untiered stack sessions name none. A moca-settings the smoke cannot read is a FAIL, not "untiered".

11.4 Add, revoke, and their timingโ€‹

  • Add an ID to the list and re-run setup.sh: the new token reaches the relay's directory within the kubelet's Secret sync (about a minute; P7 prints the measured time), and the host's next attach succeeds. Attached hosts stay attached.

  • Revoke by removing the ID: its token and bundle are deleted, and once the removal reaches the relay's directory (the same kubelet Secret sync, about a minute) its next attach is refused. A host already attached stays attached until its stream drops; to cut it off at once, restart the relay (kubectl -n moca rollout restart deployment/sandbox-relay), which drops every attached sandbox.

  • Clear every ID (SH_P4_SANDBOX_IDS=; on a P4-only stack, with SH_SANDBOX_COUNT back above 0 to return to container sandboxes; a mixed stack becomes untiered): the relay pod loses its TLS sidecar, so nothing listens on 8444 any more. setup.sh applies with a non-pruning kubectl apply -k, though, so four objects of the p4-relay component stay behind. Delete them by hand:

    kubectl -n moca delete route/moca-relay service/sandbox-relay-tls \
    networkpolicy/sandbox-relay-from-router secret/moca-relay-tls

    Secret moca-relay-sandbox-tokens stays, emptied, on purpose: an emptied Secret revokes every host deterministically. Deleting moca-relay-tls means a later switch back to P4 generates a new self-signed certificate (unless --relay-tls-cert is given), so every bundle is then re-copied.

11.5 Troubleshootingโ€‹

The relay pod has two containers here, and the tls sidecar is listed first. The pod's kubectl.kubernetes.io/default-container: sandbox-relay annotation sends plain kubectl logs and kubectl exec to the relay; name the sidecar to read it: kubectl -n moca logs deployment/sandbox-relay -c tls.

setup-microvm.sh --remote dies with did not stay attached. The message names the worker journal to read. A wrong or revoked token shows there as attached, serving execs followed at once by stream ended (โ€ฆ); reconnecting: the bundle is stale, or its ID is no longer listed. Re-run setup.sh, copy the fresh bundle and install it again.

The worker journal says missing selected ALPN property. The relay sidecar is not negotiating h2 (--alpn h2 in overlays/ocp/p4-relay); grpc-go 1.67+ requires it.

x509: certificate signed by unknown authority. The bundle's relay-ca.crt was not installed (re-run --remote with the full bundle), or an operator certificate does not chain to a CA the host trusts.

The worker attaches, then drops every few minutes. The router's idle timeout is shorter than the stream's quiet periods. The Route carries haproxy.router.openshift.io/timeout: 5m, above the worker's 30 s keepalive; check the annotation survived.

P2 fails with a P4 ID missing. The host's worker is not attached; read its journal on the host (journalctl -u microvm-worker).

12. Single-namespace dev/test target (--target ocp-single)โ€‹

For a developer or CI tenant on a shared OpenShift cluster where you have edit rights in one namespace and nothing else: no cluster-scope RBAC, no SCC grants, no permission to read cluster-scoped objects. The whole stack -- supervisor, relay, control plane, Redis, the container sandbox pool, every Secret, the credential Role -- runs in that one namespace, reached by port-forward as on Kind -- or, when your tenant has a custom domain, by opt-in Routes (ยง12.5).

export LOG_DIR=/tmp/kagenti/tdd/moca; mkdir -p "$LOG_DIR"
SH_GITHUB_CLIENT_ID=<client id> SH_SINGLE_NAMESPACE=<your-namespace> \
deploy/k8s/setup.sh --target ocp-single >"$LOG_DIR/t-single-setup.log" 2>&1; echo "EXIT:$?"
tail -4 "$LOG_DIR/t-single-setup.log"

The namespace must already exist (creating one needs cluster scope, which this target assumes you lack); SH_SINGLE_NAMESPACE defaults to moca-single. A ResourceQuota of 8 CPU / 32 Gi covers the default stack several times over (ยง4). The one requirement the cluster must meet anyway: a default StorageClass, for the Redis PVC.

12.1 How it stays within the namespaceโ€‹

Everything the ocp target does outside the namespace is skipped or replaced:

The ocp targetocp-single
creates the moca, moca-sandbox, moca-credentials namespacesuses your existing namespace (existence checked, never created)
grants nonroot-v2 to each ServiceAccount (oc adm)no SCC: every explicit UID/GID is stripped, restricted-v2 assigns from the namespace's range
Routes, ingresses.config/cluster read, TLS secretsno Routes by default (kubectl port-forward, like Kind); opt-in Routes with SH_ROUTE_DOMAIN (ยง12.5), which needs no cluster-scope read because you give the domain
requires ockubectl only

The isolation the base draws from the namespace boundary moves to pod-label NetworkPolicies: the default-deny, the sandbox's egress (relay attach port and the public internet only), the relay's exec-port rule (supervisor only) are all enforced per pod label. The sandbox's egress policy is the one that carries the isolation: a tenant's own namespace policies can widen ingress (ยง12.3). The sandboxes mount no ServiceAccount token, so the kube API refuses them even where the cluster leaves it reachable (ยง12.3).

Three consequences of the collapse, accepted for a dev/test tenant:

  • The secrets the base keeps out of moca-sandbox (the exec token, the Redis password, the MU1 keys) share the namespace with the sandbox pods -- anyone who can edit the namespace can read them.
  • The control plane's Secret Role (moca-control-plane-credentials: get, create, update, patch, delete; no list) is scoped to moca-credentials in the base, so it reaches user credentials only. In one namespace it reaches every Secret: the relay attach and exec tokens, the Redis password, the MU1 keys. A compromised control plane can read and overwrite platform secrets by name, not just user credentials.
  • The sandbox workspace and home are emptyDirs, so they are lost when a sandbox pod restarts (they are container-local in the base too).

12.2 Smokeโ€‹

K8S_LIVE_SMOKE=1 SMOKE_MODEL_URL=<endpoint> SMOKE_MODEL_TOKEN=<token> \
deploy/k8s/smoke.sh --target ocp-single >"$LOG_DIR/t-single-smoke.log" 2>&1; echo "EXIT:$?"

There is no in-pod mock model on this target (the kind-ci one rides the dev.local image), so the smoke needs a real model credential, like --target ocp. The claims are the container tier's: pods up, sandboxes attached, authenticated turns in a sandbox, isolation (redis, relay exec, metadata BLOCKED; relay attach OPEN), drain, Redis-restart persistence, no restarts, and a session staying on its sandbox (claim 12). The kube API probe is reported as a note, not a claim (ยง12.3). For a custom namespace, export the same SH_SINGLE_NAMESPACE you installed with; without it, smoke reads the moca-setup ConfigMap in the kubeconfig's current namespace, so it finds the stack only when the context is already set to it.

12.3 Verifiedโ€‹

Brought up and smoked (11/11) on a shared OpenShift 4.x tenant with edit-only namespace rights, an 8 CPU / 32 Gi quota, RHACS image admission active, and no SCC grants: restricted-v2 assigned UID/GID/fsGroup 1002610000 from the namespace's range, the emptyDirs over /workspace and /home/sandbox were owned by the assigned fsGroup, and the cluster's CNI (Calico, on IBM Cloud ROKS) enforced the pod-label policies, with the exceptions below.

The kube-API Service VIP was reachable from the sandbox. The VIP (172.21.0.1:443) is DNAT'd to ROKS's node-local apiserver proxy, 172.20.0.1:2040, and the sandbox reaches that address directly too, although it is inside the excepted 172.16.0.0/12; other ports on it, other private IPs, and the DNS VIP were blocked. The namespace's own policies do not explain it, so a cluster-level allowance does (not readable with namespace rights). The sandboxes mount no ServiceAccount token, so the API refuses them; treat it as defence-in-depth lost, not as an escape.

The node network is outside the egress except list. That tenant's nodes sit on a publicly routable (non-RFC 1918) range, so the sandbox's internet rule admits it: the sandbox reached its node's kubelet (10250) and SSH (22). The except list covers RFC 1918, CGNAT and link-local only; on a cluster whose node or infrastructure network is publicly routable, add those CIDRs with SH_SANDBOX_EGRESS_EXCEPT (ยง6, #446).

A tenant allow-same-namespace policy (ingress from any pod in the namespace, to every pod) was preinstalled. Policies are additive, so the overlay's ingress rules (the relay's supervisor-only exec port, Redis) admit any pod in the namespace there. The sandboxes stay isolated because their own egress policy is enforced: Redis and the relay exec port were BLOCKED from a sandbox.

12.4 What does not work hereโ€‹

P4 microVM hosts cannot attach: their path is the relay's OpenShift Route (ยง11), which this target does not render. Container sandboxes only, so this target is never tiered; a given SH_SANDBOX_DEFAULT_TIER is stored in moca-setup but unused.

Renaming on re-run: SH_SINGLE_NAMESPACE is not sticky -- it is validated and applied fresh every run. Pointing it at a different namespace installs a second stack there; the first one stays until its namespace is deleted.

12.5 Routes, opt-in (SH_ROUTE_DOMAIN)โ€‹

The port-forwards are the default because the target cannot learn a Route hostname on its own: reading the cluster's apps domain (ingresses.config/cluster) is a cluster-scoped read, which this target assumes you lack. But a tenant whose operator set up a custom domain -- a wildcard DNS record and certificate for *.<namespace>.<domain>, plus an ingress controller -- can be reached properly. Opt in with the domain:

SH_GITHUB_CLIENT_ID=<client id> SH_SINGLE_NAMESPACE=<ns> SH_ROUTE_DOMAIN=<domain> \
deploy/k8s/setup.sh --target ocp-single [--tls-secret NAME]

Two Routes render, in the namespace: moca.<domain> (the supervisor, TLS passthrough to a ghostunnel L4 sidecar, exactly ยง11's shape -- an edge Route there would silently break the supervisor's per-connection routing) and moca-control-plane.<domain> (the control plane, ordinary edge). moca-settings's SH_PUBLIC_HARNESS_URL becomes the supervisor's Route, so /v1/discovery advertises it and mocactl needs no port-forward:

mocactl --control-plane-url https://moca-control-plane.<domain> login

The supervisor's certificate comes from one of three sources, in this order:

SourceWhat setup.sh does
--tls-secret NAMEreferences a preinstalled kubernetes.io/tls Secret by name; never copies it
--tls-cert FILE --tls-key FILEinstalls the pair as the moca-supervisor-tls Secret
neithergenerates a self-signed one, and says so (NODE_EXTRA_CA_CERTS, as on --target ocp)

The edge Route embeds no certificate: the ingress controller serves its own (on a custom-domain tenant, that is the operator's wildcard; on a standard cluster, the router's default for the apps domain).

SH_ROUTE_DOMAIN and --tls-secret are sticky like every input: a re-run without them keeps the Routes (and the certificate) it finds in moca-setup; SH_ROUTE_DOMAIN= set-but-empty turns the Routes off and returns to port-forward access. The three certificate sources are mutually exclusive per run (--tls-secret with --tls-cert is refused), and --tls-cert/--tls-key on a re-run replaces a saved --tls-secret rather than being silently beaten by it -- the saved name is dropped from moca-setup, so a later re-run with no certificate input cannot resurrect it over what the pair installed. The smoke still reaches the stack by port-forward, Routes or not.

The ยง12.3 tenant verified all of it: the operator's Let's Encrypt wildcard for *.moca-ns.vpc-int.res.ibm.com (served by --tls-secret, and also covering the edge Route as the controller's default certificate), a namespace-scoped ingress controller admitted by the policy-group.network.openshift.io/ingress selector, and mocactl logins and turns over the Routes.

13. Test users, for unattended multi-user testing (SH_TEST_USERS)โ€‹

Preview. Test-user logins are implemented but not yet declared Stable; interfaces and limits may change. Known limits: dev and test only, container tier only, and not available on the sandbox profile (see below and ADR-0045).

Every login is GitHub's device flow, which needs a person and a browser. SH_TEST_USERS=N adds N synthetic subjects test:1 โ€ฆ test:N that log in with a shared secret instead, so a script can exercise the multi-user claims โ€” each subject spending only its own credential, and one user being refused another's session. A test user is configured exactly like a GitHub user: the control plane issues the same refresh family and the same token lifetimes, so rotation, roles (SH_ADMIN_SUBJECTS), the ownership index and the audit trail behave identically.

It is off unless asked for, and gated by a secret, because the endpoint issues a real login for an arbitrary subject: without one, anyone who can reach the control plane is every test user. POST /v1/auth/test-token answers 404 while SH_TEST_USERS is unset, the route is not mounted on the sandbox profile, and the control plane refuses to boot if the two are combined.

13.1 Bring it upโ€‹

SH_TEST_USERS is sticky like every other setting (section 2); SH_TEST_USERS= turns it off and deletes the secret with it. On --target ocp -- the shared-cluster target -- it additionally needs SH_TEST_USERS_ACK=i-know-this-is-not-for-production, so a stale export cannot quietly leave a password-gated login path on a cluster other people use.

export LOG_DIR=/tmp/kagenti/tdd/moca; mkdir -p "$LOG_DIR"
SH_TEST_USERS=10 SH_GITHUB_CLIENT_ID=<client id> SH_SINGLE_NAMESPACE=<ns> \
deploy/k8s/setup.sh --target ocp-single --skip-build \
--image <registry>/moca:<tag> --sandbox-image <registry>/moca-remote-worker:<tag> \
>"$LOG_DIR/t-users-setup.log" 2>&1; echo "EXIT:$?"

Look for: WARNING: SH_TEST_USERS=10: test:1..test:10 can log in with the shared secret, and either generated SH_TEST_USER_SECRET or nothing (a re-run keeps the stored one and never rotates it). The secret lives in a Secret of its own, moca-test-users, and reaches the control plane as a credential file under CREDENTIALS_DIRECTORY, never an environment line. Its own Secret because the credential loader treats an empty file as an error and only an absent one as "unset": setup.sh therefore writes it only while test users are on and deletes it when they go off, and control-plane.yaml projects it from an optional source so a stack without it still boots. Give your own instead of a generated one with SH_TEST_USER_SECRET=<at least 16 characters>.

kubectl -n <ns> get secret moca-test-users -o jsonpath='{.data.SH_TEST_USER_SECRET}' | base64 -d

13.2 Check the gate firstโ€‹

Before trusting the feature, prove it is invisible without it. On a stack where SH_TEST_USERS is unset this must answer 404:

curl -s -o /dev/null -w '%{http_code}\n' -X POST -H 'Content-Type: application/json' \
-d '{"index":1}' http://127.0.0.1:8090/v1/auth/test-token

13.3 One user, by handโ€‹

The secret travels in MOCA_TEST_USER_SECRET, never a flag โ€” a flag would land in the shell history and in every ps listing on the machine.

export MOCA_TEST_USER_SECRET=<the secret>
mocactl --control-plane-url http://127.0.0.1:8090 login --test-user 3
# logged in as Test User 3 (test:3)

13.4 N users, and the cross-user caseโ€‹

deploy/k8s/test-users.py drives N users (default 10, --users N) through real mocactl commands, then asserts the cross-user claims. --api runs the same claims straight against the HTTP API, which separates a client bug from a server one.

export MOCA_TEST_USER_SECRET=<the secret>
./deploy/k8s/test-users.py --users 10 --control-plane-url http://127.0.0.1:8090
./deploy/k8s/test-users.py --users 10 --api --json

Look for: users 10/10 fully ok; cross-user 4/4 and RESULT: PASS.

Each user gets its own XDG_CONFIG_HOME, because mocactl keeps one login in one auth.json (packages/mocactl/src/config.ts): a second login in the same directory overwrites the first, so one process in one directory cannot hold ten logins. That is also what makes --jobs safe.

The operator's check, that each turn spent its own user's credential โ€” the audit records credential names, never values:

kubectl -n <ns> exec redis-0 -- sh -c \
'redis-cli --no-auth-warning -a "$REDISCLI_AUTH" XREVRANGE sh:cp:audit + - COUNT 300' |
grep -A1 '^subject$' | grep '^test:' | sort -u # ten distinct subjects

Stop any port-forward on the harness port (8080) and on 18090 before running smoke.sh: it binds 18090 itself and reaches the harness at the URL the control plane advertises, so a leftover forward makes every /turn answer 404. Section 9.3 gives the same warning for its connection count.

Tested on OpenShift 4.x (IBM Cloud ROKS, --target ocp-single, edit-only namespace rights), 2026-10-26: N=10 passed in both CLI and --api modes, with ten distinct test:N subjects and ten credential names in the audit stream.

14. Async fan-out, opt-in (SH_ASYNC_DRAIN)โ€‹

An orchestrator that wants to dispatch many leaves and collect them later submits them to the control plane and polls; it does not hold a connection per leaf. This is archetype A (dispatch archetypes), the KEDA ScaledObject's replacement, decided in ADR-0047.

Status: single-tenant in v0.10.0. Async fan-out under MOCA_TENANCY=multi (grants minted at dequeue, #388) is being worked on, on main, and is targeted for v0.12.0. Until it ships, run the drain only under MOCA_TENANCY=single; the multi notes in this section describe work in progress.

The consumer is not a new workload. No Deployment, no --role=leaf pod, no autoscaler: with the knob on, each of the supervisor's ordinary --role=turn workers also runs a drain loop that claims one queued turn at a time and runs it through the same runLeaf a synchronous /v1/runs uses. Both things it needs are already on the pod โ€” SH_CONTROL_PLANE_URL and the SH_EXCHANGE_TOKEN file (section 5) โ€” so turning it on takes no other wiring.

14.1 Turn it onโ€‹

SH_ASYNC_DRAIN is a sticky setup.sh input, like the token lifetimes (section 2). Set it on a re-run; moca-settings keeps it afterwards, and the settings hash on the pod template rolls the supervisor as part of the same apply, so nothing needs restarting by hand:

SH_ASYNC_DRAIN=1 deploy/k8s/setup.sh --target kind --skip-build

Only 1 and empty are accepted โ€” SH_ASYNC_DRAIN=true is refused rather than read as off, because a typo that silently costs you the whole feature is worse than one re-run. To turn it off again, pass it set but empty (SH_ASYNC_DRAIN= deploy/k8s/setup.sh โ€ฆ); leaving it unset keeps whatever the ConfigMap holds.

Check that it armed, on the admin port:

kubectl --context kind-moca -n moca port-forward svc/moca-supervisor 18081:8081 &
curl -s http://127.0.0.1:18081/metrics | jq '{async_queue_depth, async_oldest_age_ms}'

A number (even 0) means worker slot 0's drain loop is running and reporting. "NaN" means it never reported โ€” the knob is off, or a prerequisite was missing and the worker said so and stayed off rather than failing: kubectl -n moca logs deploy/moca-supervisor | grep '\[async\]'. The drain arms under either tenancy: under multi the dequeue hands it grants instead of a key.

14.2 Submit and collectโ€‹

All three routes are on the control plane and take an api token โ€” the one a mocactl login returns โ€” not a session token:

# 202 {"sessionId":"โ€ฆ","turnId":"t-โ€ฆ"} -- the queue carries the turn id and nothing else.
curl -s -X POST -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
-d '{"async":true,"kind":"prompt","prompt":"summarise README.md","idempotencyKey":"job-1"}' \
"$CP/v1/sessions/$SID/turns"

# {"turnId","state","attempts"} always; "result" only once state is terminal, because a result
# written by a worker that then died is withheld until the reaper has ruled on it.
curl -s -H "Authorization: Bearer $TOKEN" "$CP/v1/sessions/$SID/turns/$TURN_ID"

# Cancel one that is still queued (409 turn_not_cancellable once a worker has it).
curl -s -X DELETE -H "Authorization: Bearer $TOKEN" "$CP/v1/sessions/$SID/turns/$TURN_ID"

idempotencyKey defaults to the session id, so N async turns on one session collapse into the first one's turn. Fanning out N leaves means N sessions, or N distinct keys.

GET /runs/status on the harness never reports an async turn, and does not fail either: it reads the session-wide result key, which an async turn never writes, so it answers queued forever โ€” or, if that session also ran a synchronous /v1/runs leaf, returns that leaf's result, which looks like an answer. Poll the turn route above.

14.3 Watchโ€‹

On the admin port's /metrics, under counters (counts, so missing and zero mean the same thing) and as two top-level gauges:

ReadingMeans
async_claimsleaves claimed by this pod's workers since they started
async_reclaimsof those, the ones redelivered: a worker died holding one, OR it was deferred
async_dead_lettersof those, the ones that exhausted their attempts
async_claim_denied_fullclaims this pod's workers refused because their admission cap was full
async_deferred_subjectentries handed back unrun because their owner was at SH_TURNS_PER_SUBJECT
async_queue_depthqueued turns, from slot 0 only
async_oldest_age_msage of the oldest queued turn, from slot 0 only

async_deferred_subject rising while slots sit idle is the per-owner cap pacing one submitter -- the fairness working, not a fault. Rising with no second owner submitting means SH_TURNS_PER_SUBJECT is too low for this deployment: a deferred entry waits out the 90 s reclaim window before any worker may take it again, so each deferral costs that turn up to 90 s. That window is also why it inflates async_reclaims: the entry comes back by the same XAUTOCLAIM a dead worker's entry does, so subtract async_deferred_subject before reading a reclaim rate as workers dying mid-leaf. A flat zero is not proof of fairness: the cap counts sync and async together and a worker runs one queued leaf at a time, so an async-only backlog never reaches it and is drained FIFO -- one owner's deep queue ahead of another's single turn. At replicas > 1 that is what MOCA_ASYNC_RUNNING_PER_OWNER is for (#391): it caps an owner's RUNNING leaves across every pod, counted in Redis, and is the only one of the two caps a pod-local counter cannot provide. Unset by default; add it to moca-settings to turn it on, and give every pod the same value. Then roll the supervisor yourself -- kubectl -n moca rollout restart deployment/moca-supervisor. The pods read the key as environment at start, and their moca.dev/settings-hash annotation covers only the keys setup.sh writes, so a key added by hand changes no annotation and the running pods keep their old environment indefinitely. Its defers land in this same counter, so a rise after the restart is the new cap working.

The five counters are one pod's, not the deployment's: /metrics answers from whichever pod you reached, each worker counts in its own memory, and a worker that respawns starts its own count at zero. At replicas > 1 (section 4) add the pods up yourself. The two gauges describe the one shared queue, so every pod's slot 0 reports the same numbers.

Read the last two as a pair. Depth alone cannot tell a deep queue that is draining fast from a shallow one that is stalled; a rising async_oldest_age_ms at a flat depth is the stall. async_queue_depth is also the signal to size replicas by โ€” section 4 says why nothing does that for you.

14.4 Limitsโ€‹

  • Under multi, inference is isolated and egress is not yet. multi works: the dequeue returns MI1 ยง6.1 grants and the worker spends the inference one at the control plane's inference listener. It costs three control-plane settings โ€” MOCA_GRANT_SIGNING_KEY, MOCA_INFERENCE_ADDR, and MOCA_INFERENCE_URL when the bind is a wildcard โ€” and the boot refuses without them. These manifests wire none of them: no ConfigMap key and no Service port ship for the listener, so turning multi on here is a manifest change first. The residual: the SANDBOX grant has no redeemer, so an async turn gets inference isolation and not egress isolation.
  • Two leaves per pod at the default W. The loop is sequential; see section 4 for the width rule and why no autoscaler ships.
  • Caps: both of them, enforced by the worker. turn_exp โ€” anchored at the dequeue and clamped to min(caps.wallClock, MOCA_TURN_MAX_S) โ€” is the deadline the worker stops the leaf at, and under multi it also bounds how long the turn's grants are usable. caps.tokens is enforced inside the run. Either one spent ends the turn failed with result.reason: "budget" (#391). caps.costUsd is refused at submission.
  • The scheduled on-ramp has no equivalent. Archetype C went with the KEDA path and has no P6 design (#536).

14.5 Smokeโ€‹

smoke.sh claims 13 and 14 cover this path, and skip with a note on a stack whose moca-settings has the knob off. Claim 13 fans five leaves out over a stack that can run two at once and requires all five to reach a result; claim 14 deletes the supervisor pod under a running leaf and requires that leaf to finish on a second delivery (attempts โ‰ฅ 2), which is the redelivery acceptance of #389 on this path. CI brings its Kind stack up with SH_ASYNC_DRAIN=1, so both run on every k8s-kind-e2e (section 8).

Claim 14 assumes one supervisor pod, as claim 9 does: it picks the pod before submitting the leaf, and cannot learn which pod's drain then claimed it. Above replicas: 1 claims 9 and 14 skip with a note rather than delete a pod that may be holding nothing. If you raised replicas (section 4) and want the redelivery claim back, scale the supervisor to one for the run.