Cluster state is the source of truth. vk-cocoon keeps no persistent
pods.json — it rebuilds its in-memory tables from the Kubernetes API on
every restart, and it reacts to cocoon’s live event stream in between.
On every restart vk-cocoon:
fieldSelector=spec.nodeName=<VK_NODE_NAME>. os=macos pods are
split out here and adopted via cocoon-macos vm inspect plus a PID
probe (they never appear in Runtime.List); a dead or missing record
is left for the CreatePod replay to restart in place.Runtime.List.creating through cocoon’s lock-checked
vm reconcile-stale-create verb (needs cocoon > v0.5.7 on the node)
before any adoption index is built — adopting a creating placeholder
deadlocks its pod. collected/not-found free the name for a clean
recreate; busy (an in-flight clone still owns the record) and
transient verb or inspect errors hand the record to a bounded
background watcher. Each watcher tick re-invokes the verb:
collected/not-found free the name for a clean recreate; otherwise
an inspect indexes a committed running VM or applies the orphan
policy to a terminal record. An unresolved creating/created state
or transient error remains under bounded retry. A record that already
left creating is adopted only when running. Every verb attempt is
counted on
cocoon_vk_stale_create_reconcile_total.vm.cocoonstack.io/id annotation by matching
the VMID against the runtime list.VK_ORPHAN_POLICY:
destroy (default): remove the VM so pod-less VMs don’t accumulate
after restart or pod chaos.alert: log + bump cocoon_vk_orphan_vm_total, leave the VM alone.keep: no log, no metric.dispatchOwedWork),
derived per tracked pod from persisted facts only: a hibernate
annotation with a VM re-enters hibernate() (booting a crashed VM
first); lifecycle=creating splits on post-clone-state (running
re-runs the fixup, done re-checks SAC then holds Ready until the
lease lands, absent derives the plan — for CH+Windows drop-NIC specs
hibernate evidence decides restore vs fresh, retried under the
recheck budget); failed stays parked. Every resumed op claims the
pod for its duration; UpdatePod and DeletePod back off with an
error while the claim is held. Dispatches are counted on
cocoon_vk_startup_resume_total and emit ResumedAfterRestart.A pod whose annotated VMID does not appear in the local runtime list
logs a warning and is left to CreatePod to recreate on the next
reconcile.
Right after the listing and before the stale-create sweep, every non-terminal pod on the node is checked against the node’s snapshot CPU class (see Snapshot CPU compatibility); a mismatch fails startup reconcile, and vk-cocoon refuses to register the node rather than resume snapshots on an incompatible host.
In addition to the periodic probe, vk-cocoon subscribes to cocoon’s
real-time VM event stream via cocoon vm status --event --format json.
This provides sub-second detection of VM state changes (DELETED, stopped,
error) without waiting for the next probe tick.
The watcher goroutine (vmWatchLoop) runs for the lifetime of the
process with automatic restart on subprocess failure (exponential
backoff from 1 s to 60 s, reset on successful connect). Normal stream
closes (cocoon restart) use a fixed 2 s reconnect delay. When an event
arrives:
| Event | Inspect result | Action |
|---|---|---|
DELETED |
VM not found | evictPod: delete pod (phase=Failed, reason=VMGone) → operator recreates |
MODIFIED (state ≠ running) |
state = stopped/error | cocoon vm start (in-place restart, preserves disk/network) |
MODIFIED (state ≠ running) |
state = running | False alarm — ignore |
A 30-second restart cooldown (restartCooldown) prevents tight
restart loops when a VM keeps crashing. If the cooldown has not elapsed
since the last restart, the pod is evicted (phase=Failed,
reason=RestartCooldown) so the operator can do a clean recreation.
Stale cooldown entries are garbage-collected on each event stream
reconnect.
| Mechanism | Worst-case latency |
|---|---|
| Probe only (old: 15 s × 5 failures) | ~75 s |
| Probe only (current: 5 s × 3 failures) | ~24 s |
| VM event watcher | < 1 s |