Troubleshooting
Preview. For P6 on Kubernetes. Commands are the ones
deploy/k8suses; they target Kind (--context kind-moca). On OpenShift omit the context. Rows marked (inferred) come from reading the code, not from a run.
By symptom​
| Symptom | Likely cause | Check | Fix |
|---|---|---|---|
429 {"error":"overloaded"} | Every healthy worker is at its turn cap, or none is healthy (a restart window) | GET :8081/metrics: workers[].inFlight, healthy, counters.restarts | Retry after the delay; add supervisor pods or raise W and S (Sizing). It is not a readiness failure |
503 sandbox_unavailable, with Retry-After | Pool saturated or empty, or the session's own sandbox is briefly absent | kubectl -n moca exec redis-0 -- sh -c 'redis-cli HLEN sh:sandbox:records'; kubectl -n moca-sandbox get pods | Raise SH_SANDBOX_COUNT or KAGENTI_SANDBOX_CAP; check the sandboxes attached (below) |
503 redis_unavailable | Redis down or wedged | kubectl -n moca get pod redis-0; control plane log redis client error | Fix Redis; the control plane recovers on its own |
503 credential_unavailable | Control plane unreachable or answering badly; keyset unusable; credential store not answering | kubectl -n moca logs deployment/moca-control-plane; check SH_CONTROL_PLANE_URL and the NetworkPolicy | Fix the control plane; for the store, check the Role on moca-credentials |
401 token_invalid / token_expired / token_required | Expired token, or the signing key changed after moca-mu1 was replaced | mocactl login; compare SH_SESSION_TOKEN_PUBLIC_KEYS with the control plane's key (same moca-mu1) | Log in again; after rotating moca-mu1 restart the control plane and supervisor |
400 invalid_grant on refresh | Refresh token unknown, revoked, expired, or reused outside the 30 s grace | Audit decisions refresh_refused, refresh_reuse_detected | Log in again |
404 session_not_found | Someone else's session (the control plane answers 404, never 403), a deleted one, or Redis was lost | redis-cli EXISTS session:<sid>; HGETALL sh:cp:session:<sid> | Create a new session |
410 config_bundle_not_found (a turn) or 404 (the control plane, for example when creating a session on an unknown digest) | The session's bundle expired (30 days), was deleted, or Redis was wiped | n/a | Re-promote with mocactl promote DIR |
429 bundle_quota_exceeded | The bundle byte budget is full | n/a | mocactl bundles delete DIGEST (an admin can delete anyone's) |
| Supervisor never becomes Ready | The published latest image predates /readyz; startup compile exceeds 30 x 5 s; probes filtered | kubectl describe pod; curl :8081/readyz | Kind: rebuild with --build. OpenShift: use --image with a current image. For filtered probes see the README |
Control plane 0/1, or 0/0 | 0/1: Redis is not up yet. 0/0: no GitHub client id was given, so setup.sh renders zero replicas | kubectl -n moca get deployment moca-control-plane; log line redis client error | Wait for Redis, or SH_GITHUB_CLIENT_ID=<id> deploy/k8s/setup.sh --target kind --skip-build |
only 0 of N sandboxes attached to the relay | Presence not written, or sandboxes cannot reach the relay | kubectl -n moca logs deployment/sandbox-relay (look for presence put failed); kubectl -n moca-sandbox logs statefulset/moca-sandbox | Fix Redis; check the attach token; after a relay token change restart the sandboxes |
| Relay rejects an attach | SH_RELAY_TOKEN differs from the sandbox's SANDBOX_TOKEN; a per-sandbox token is wrong; the relay refuses a token equal to the exec token | Relay log lines relay token for <id> unreadable / equals the exec token; refused | Re-run setup.sh; to rotate, delete moca-relay, re-run, then restart relay, supervisor and sandboxes |
redis-0 Pending | No default StorageClass | setup.sh prints a warning | Mark one StorageClass default |
| Supervisor container OOMKilled | Memory limit (1Gi) too low for W workers | kubectl -n moca describe pod shows OOMKilled; a worker dying alone shows worker_exit in the log | Raise the memory request and limit together with W. Redis at 256Mi is a separate OOM risk (inferred) |
| Sandboxes can reach things they should not | The CNI does not enforce NetworkPolicy; every rule is then open | The smoke test's isolation claim; the README's manual probes | Use kind 0.24.0 or later, or OVN-Kubernetes. On single-node Kind a sandbox can still reach the kube API and its node's kubelet |
| A P4 worker drops every few minutes (OpenShift) | The Route's idle timeout is shorter than the stream's quiet period | Route annotation haproxy.router.openshift.io/timeout | Set it to 5m and confirm it persisted |
| A workspace silently disappeared | Idle reclaim (SH_WORKSPACE_IDLE) or a sandbox pod restart; the turn is not told (#338) | Sandbox pod age; SH_WORKSPACE_IDLE | Raise SH_WORKSPACE_IDLE; re-create the files |
| Smoke test fails | A transient reconnect window, or a genuine regression | The smoke keeps its working directory and prints logs kept in <dir>; CI uploads k8s-kind-e2e-logs | Wait for /readyz and re-run; back-to-back smokes can fail one claim inside Redis's 5 to 10 s reconnect window |
Leftovers​
- A killed
setup.shcan leave a podmoca-genkeyswhose log holds the generated keys, readable by anyone withpods/loginmoca. Delete it:kubectl -n moca delete pod moca-genkeys --ignore-not-found. - On macOS with Podman, a proxy set in the shell is copied into the Kind node's containerd; create the cluster with the proxy variables unset (the deploy README has the exact command and the CoreDNS forwarder fix).
- To remove a Kind install entirely:
kind delete cluster --name moca.
When none of this helps​
Run mocactl doctor; it runs seven ordered checks (control plane, login, credentials, harness) and stops at the first
failure. Its last check is not read-only: it creates and then deletes a scratch session (reference). The full operator troubleshooting section
and the OpenShift Route and TLS notes are in deploy/k8s/README.md.