Cluster state is the source of truth. vk-cocoon keeps no persistent
pods.json — it rebuilds its in-memory tables from the Kubernetes API on
every restart, and it reacts to cocoon’s live event stream in between.
On every restart vk-cocoon:
fieldSelector=spec.nodeName=<VK_NODE_NAME>. Unmanaged pods are adopted
from their operator-supplied VMID/IP annotations without runtime
lifecycle operations. Managed os=macos pods are split out and adopted
via cocoon-macos vm inspect plus a PID
probe (they never appear in Runtime.List); a dead or missing record
is left for the CreatePod replay to restart in place.Runtime.List.creating through cocoon’s lock-checked
vm reconcile-stale-create verb (needs cocoon > v0.5.7 on the node)
before any adoption index is built — adopting a creating placeholder
deadlocks its pod. collected/not-found free the name for a clean
recreate; busy (an in-flight clone still owns the record) and
transient verb or inspect errors hand the record to a bounded
background watcher. Each watcher tick re-invokes the verb:
collected/not-found free the name for a clean recreate; otherwise
an inspect settles the record. A committed running VM is indexed for
adoption while a pod on the node that is not being deleted still names
it (a failed pod lookup indexes it too); one whose pod was deleted
during the restart gets the orphan policy, as does a terminal record.
An unresolved creating/created state
or transient error remains under bounded retry. A record that already
left creating is adopted only when running. While a watcher holds a
name, CreatePod for that name waits for it to settle instead of
failing on the name clash; a pod whose create fails 20 times is dropped
by virtual-kubelet until the pod changes. Every verb attempt is
counted on
cocoon_vk_stale_create_reconcile_total.vm.cocoonstack.io/id annotation by matching
the VMID against the runtime list. A pod with no VMID annotation is
instead adopted by matching its spec.VMName against the runtime list.VK_ORPHAN_POLICY:
destroy (default): remove the VM and release its DHCP leases so pod-less VMs don’t accumulate
after restart or pod chaos.alert: log + bump cocoon_vk_orphan_vm_total, and index the VM by name.keep: index the VM by name, no log, no metric.Both non-destroy policies publish the VM to vmsByName, so the next
CreatePod for its pod takes the adopt branch instead of creating a second VM.
dispatchOwedWork),
derived per tracked pod from persisted facts only: a hibernate
annotation with a VM re-enters hibernate() (booting a crashed VM
first); lifecycle=creating (or unset) splits on post-clone-state (running
re-runs the fixup, done re-checks SAC then holds Ready until the
lease lands, absent derives the plan — for CH+Windows drop-NIC specs
hibernate evidence decides restore vs fresh, retried under the
recheck budget); failed stays parked. Every resumed op claims the
pod for its duration; UpdatePod and DeletePod back off with an
error while the claim is held. Dispatches are counted on
cocoon_vk_startup_resume_total and emit ResumedAfterRestart.A managed non-macOS pod whose annotated VMID does not appear in the local
runtime list is, if hibernated, handled by clearing its VMID/IP annotations and
tracking it without a VM so a wake retry starts clean; otherwise it logs
a warning and is left to CreatePod to recreate on the next reconcile.
Right after the listing and before the stale-create sweep, every non-terminal pod on the node is checked against the node’s snapshot CPU class (see Snapshot CPU compatibility); a mismatch fails startup reconcile, and vk-cocoon refuses to register the node rather than resume snapshots on an incompatible host.
Every 30 seconds, the status reconciler re-resolves each tracked VM’s DHCP lease and repairs drifted pod status. The same pass idempotently reconciles the published IP and macOS VNC-port annotations, removing an endpoint when the runtime no longer serves it; failed patches are retried on the next pass.
At startup and every 10 minutes, the snapshot reclaimer walks the local snapshot
store. It skips any snapshot that a pod on this node runs, wakes or clones from.
A clone source vk-cocoon pulled from the registry is imported with the
description vk-cocoon clone source; once no pod on this node clones from it,
it is removed whatever its name, since the registry still holds it. The
description travels with a snapshot export, so a hand re-import of such a copy
stays reclaimable. A vk- copy whose name carries a colon (a non-default tag or
a digest, pulled before the description was set) is removed the same way. Any
other vk- snapshot is removed with its fork snapshot once the registry’s
:hibernate tag is gone or names another snapshot. A copy the tag still names
stays as the warm-wake cache, and a registry error keeps the copy until the next
pass with a warning. A leftover -hibernate-import snapshot no pod on this node
wakes from is removed. A snapshot outside the vk- names without the
description, such as one an operator imported by hand, is never touched.
In addition to the periodic probe, vk-cocoon subscribes to cocoon’s
real-time VM event stream via cocoon vm status --event --format json.
This provides sub-second detection of VM state changes (DELETED, stopped,
error) without waiting for the next probe tick.
The watcher goroutine (vmWatchLoop) runs for the lifetime of the
process with automatic restart on subprocess failure (exponential
backoff from 1 s to 60 s, reset on successful connect). Normal stream
closes (cocoon restart) use a fixed 2 s reconnect delay. When an event
arrives:
Inspect result (after a DELETED event, or an ADDED / MODIFIED event for a non-running VM) |
Action |
|---|---|
| VM not found | evictPod: delete pod (phase=Failed, reason=VMGone) → operator recreates |
| Inspect inconclusive (transient error) | Deferred recheck loop; on budget exhaustion (30 min) removes the VM and evicts the pod (phase=Failed, reason=VMInspectTimeout) |
| state = stopped/error | cocoon vm start (in-place restart, preserves disk/network); on failure removes the VM and evicts the pod (phase=Failed, reason=RestartFailed) |
| state = running | False alarm — ignore |
While a hibernate of the pod runs, a gone event is not acted on: the hibernate’s own remove emits one. Once that hibernate ends, the VM is inspected again and handled by the table above.
Every stream connect opens with an ADDED event per VM, so a VM that stopped
while vk-cocoon was down is restarted or evicted the same way. When the pod
delete of an eviction fails, the deferred recheck keeps retrying it under its
backoff until the apiserver accepts it.
A 30-second restart cooldown (restartCooldown) prevents tight
restart loops when a VM keeps crashing. If the cooldown has not elapsed
since the last restart, vk-cocoon removes the VM, evicts the pod
(phase=Failed, reason=RestartCooldown), and releases its DHCP leases
so the operator can do a clean recreation.
Stale cooldown entries are garbage-collected on each event stream
reconnect.
| Mechanism | Worst-case latency |
|---|---|
| Probe only (old: 15 s × 5 failures) | ~75 s |
| Probe only (current: 5 s × 3 failures) | ~24 s |
| VM event watcher | < 1 s |