Skip to main content
Version: v0.11.0

Troubleshooting

Preview. For P6 on Kubernetes. Commands are the ones deploy/k8s uses; they target Kind (--context kind-moca). On OpenShift omit the context. Rows marked (inferred) come from reading the code, not from a run.

By symptom​

SymptomLikely causeCheckFix
429 {"error":"overloaded"}Every healthy worker is at its turn cap, or none is healthy (a restart window)GET :8081/metrics: workers[].inFlight, healthy, counters.restartsRetry after the delay; add supervisor pods or raise W and S (Sizing). It is not a readiness failure
503 sandbox_unavailable, with Retry-AfterPool saturated or empty, or the session's own sandbox is briefly absentkubectl -n moca exec redis-0 -- sh -c 'redis-cli HLEN sh:sandbox:records'; kubectl -n moca-sandbox get podsRaise SH_SANDBOX_COUNT or KAGENTI_SANDBOX_CAP; check the sandboxes attached (below)
503 redis_unavailableRedis down or wedgedkubectl -n moca get pod redis-0; control plane log redis client errorFix Redis; the control plane recovers on its own
503 credential_unavailableControl plane unreachable or answering badly; keyset unusable; credential store not answeringkubectl -n moca logs deployment/moca-control-plane; check SH_CONTROL_PLANE_URL and the NetworkPolicyFix the control plane; for the store, check the Role on moca-credentials
401 token_invalid / token_expired / token_requiredExpired token, or the signing key changed after moca-mu1 was replacedmocactl login; compare SH_SESSION_TOKEN_PUBLIC_KEYS with the control plane's key (same moca-mu1)Log in again; after rotating moca-mu1 restart the control plane and supervisor
400 invalid_grant on refreshRefresh token unknown, revoked, expired, or reused outside the 30 s graceAudit decisions refresh_refused, refresh_reuse_detectedLog in again
404 session_not_foundSomeone else's session (the control plane answers 404, never 403), a deleted one, or Redis was lostredis-cli EXISTS session:<sid>; HGETALL sh:cp:session:<sid>Create a new session
410 config_bundle_not_found (a turn) or 404 (the control plane, for example when creating a session on an unknown digest)The session's bundle expired (30 days), was deleted, or Redis was wipedn/aRe-promote with mocactl promote DIR
429 bundle_quota_exceededThe bundle byte budget is fulln/amocactl bundles delete DIGEST (an admin can delete anyone's)
Supervisor never becomes ReadyThe published latest image predates /readyz; startup compile exceeds 30 x 5 s; probes filteredkubectl describe pod; curl :8081/readyzKind: rebuild with --build. OpenShift: use --image with a current image. For filtered probes see the README
Control plane 0/1, or 0/00/1: Redis is not up yet. 0/0: no GitHub client id was given, so setup.sh renders zero replicaskubectl -n moca get deployment moca-control-plane; log line redis client errorWait for Redis, or SH_GITHUB_CLIENT_ID=<id> deploy/k8s/setup.sh --target kind --skip-build
only 0 of N sandboxes attached to the relayPresence not written, or sandboxes cannot reach the relaykubectl -n moca logs deployment/sandbox-relay (look for presence put failed); kubectl -n moca-sandbox logs statefulset/moca-sandboxFix Redis; check the attach token; after a relay token change restart the sandboxes
Relay rejects an attachSH_RELAY_TOKEN differs from the sandbox's SANDBOX_TOKEN; a per-sandbox token is wrong; the relay refuses a token equal to the exec tokenRelay log lines relay token for <id> unreadable / equals the exec token; refusedRe-run setup.sh; to rotate, delete moca-relay, re-run, then restart relay, supervisor and sandboxes
redis-0 PendingNo default StorageClasssetup.sh prints a warningMark one StorageClass default
Supervisor container OOMKilledMemory limit (1Gi) too low for W workerskubectl -n moca describe pod shows OOMKilled; a worker dying alone shows worker_exit in the logRaise the memory request and limit together with W. Redis at 256Mi is a separate OOM risk (inferred)
Sandboxes can reach things they should notThe CNI does not enforce NetworkPolicy; every rule is then openThe smoke test's isolation claim; the README's manual probesUse kind 0.24.0 or later, or OVN-Kubernetes. On single-node Kind a sandbox can still reach the kube API and its node's kubelet
A P4 worker drops every few minutes (OpenShift)The Route's idle timeout is shorter than the stream's quiet periodRoute annotation haproxy.router.openshift.io/timeoutSet it to 5m and confirm it persisted
A workspace silently disappearedIdle reclaim (SH_WORKSPACE_IDLE) or a sandbox pod restart; the turn is not told (#338)Sandbox pod age; SH_WORKSPACE_IDLERaise SH_WORKSPACE_IDLE; re-create the files
Smoke test failsA transient reconnect window, or a genuine regressionThe smoke keeps its working directory and prints logs kept in <dir>; CI uploads k8s-kind-e2e-logsWait for /readyz and re-run; back-to-back smokes can fail one claim inside Redis's 5 to 10 s reconnect window

Leftovers​

  • A killed setup.sh can leave a pod moca-genkeys whose log holds the generated keys, readable by anyone with pods/log in moca. Delete it: kubectl -n moca delete pod moca-genkeys --ignore-not-found.
  • On macOS with Podman, a proxy set in the shell is copied into the Kind node's containerd; create the cluster with the proxy variables unset (the deploy README has the exact command and the CoreDNS forwarder fix).
  • To remove a Kind install entirely: kind delete cluster --name moca.

When none of this helps​

Run mocactl doctor; it runs seven ordered checks (control plane, login, credentials, harness) and stops at the first failure. Its last check is not read-only: it creates and then deletes a scratch session (reference). The full operator troubleshooting section and the OpenShift Route and TLS notes are in deploy/k8s/README.md.