DeletionTimestamp is set, walk owned pods, delete them, then GC registry tags on every owned VM: probe and delete :hibernate when present (hibernate pushes ignore snapshotPolicy, so any main-only gate would orphan tags pushed by sub-agents), and probe/delete :latest only where shouldKeepLatestTag says vk-cocoon never pushed one for that VM (always keeps every :latest, main-only keeps slot 0’s, never drops stale tags). Probing avoids issuing DELETE for a nonexistent manifest, which can materialize an empty repository on some registries. A failed :hibernate probe or delete keeps the finalizer and stashed VM names for retry; :latest cleanup remains best-effort. Then drop the finalizer. VM names to GC are stashed onto the cocoonset.cocoonstack.io/delete-vm-names annotation before pod deletion, and again whenever a scale-down or toolbox removal deletes a pod, so the cleanup survives a CocoonSet deleted before Status.Agents was ever patched and still covers slots that left the spec earlier. Every stash also takes the VMs of the reclaim record of step 6, so a pod deleted while the set was suspended, which Status.Agents no longer lists, still has its :hibernate tag dropped.cocoonset.cocoonstack.io/finalizer is in place.cocoonset.cocoonstack.io/name=<cs.Name>, drop any with stale labels that aren’t actually controller-owned, and classify the rest by role label.cs.Generation onto each owned pod’s cocoonset.cocoonstack.io/generation annotation so vk-cocoon can echo it back as lifecycle-observed-generation, giving clients a counter-based completion signal immune to wallclock skew.spec.suspend == true, record every managed VM of the owned pods (main, sub-agents, toolboxes) in the cocoonset.cocoonstack.io/hibernate-reclaim annotation, merged with the VMs it already records, and list the same VMs as suspended until step 10 observes the unsuspend; the record is written before any hibernate annotation, so an operator restart mid-suspend keeps it. Then write meta.HibernateState(true) onto every owned pod and poll for completion on every managed VM: vk-cocoon must report lifecycle-state=hibernated with lifecycle-observed-generation >= cs.Generation AND the :hibernate manifest must be in the registry. vk writes state and observed-generation atomically, so the gate rejects a stale tag from a prior suspend cycle (step 10 reclaims it only after that cycle’s wake) and a lagging informer snapshot of a prior round alike. Pods in Pod.Phase=Failed are skipped: they have no live VM to snapshot. A pod that only carries vm.cocoonstack.io/lifecycle-state=failed is not skipped — the suspend keeps waiting for its VM to hibernate, because that VM may still be live. Stay in Phase=Suspending (requeueing every 5 s) until then, and transition to Phase=Suspended. This runs before the failed-state check of step 7, so a main that fails while the CocoonSet is suspended never reaches the failed path until unsuspend. With spec.hibernatePolicy: release (admitted only on a spec with agent.replicas: 0 and no toolboxes) the same gate is followed by a slot release: stash the VM names onto cocoonset.cocoonstack.io/delete-vm-names and the main’s node onto cocoonset.cocoonstack.io/hibernated-on-node, flag each pod with vm.cocoonstack.io/keep-snapshot-on-delete so vk-cocoon keeps the node-local snapshot as the warm-wake cache, then delete the pods so their scheduling seats free. The flag is best-effort — a failed patch costs the wake a registry pull, whereas blocking the delete would forfeit the seat the policy exists to free. Wake recreates the main with a weight-100 preferred affinity on the stashed node, so it lands back on its snapshot whenever that node still has room and cold-pulls elsewhere when it does not. The wait is bounded: after suspendTimeout (3 min) in Suspending the set reports Phase=Failed with a SuspendTimedOut Warning, and the next pass re-enters Suspending with a fresh deadline, so a hibernate that vk-cocoon refuses (a macOS agent, which HibernateUnsupported fails on every attempt) surfaces every three minutes instead of parking silently. The deadline lives in the cocoonset.cocoonstack.io/suspending-since annotation, restarted on every entry into Suspending and cleared on Suspended; a registry probe that keeps failing counts against it too, so an unreachable registry surfaces the same way instead of starving the phase behind the requeue backoff.Pod.Phase=Failed, or it carries vm.cocoonstack.io/lifecycle-state=failed from vk-cocoon while still Running) and still matches its spec, patch Phase=Failed and emit MainAgentFailed / PodLifecycleFailed; a terminal main that has also drifted from spec takes the rebuild budget of step 11 instead. Before either, the un-suspend of step 10 clears the hibernate annotation a prior suspend left on the owned pods, so a main whose hibernate vk-cocoon refused is not refused again on its next update: a macOS agent, which fails every hibernate with HibernateUnsupported, surfaces that while suspended (step 6) and returns to Ready on unsuspend without a manual delete. While a migration is pending (spec.nodeName is set and the main runs on another node) it skips that clear: the annotation is then the quiesce of step 8, and clearing it would let vk-cocoon lift the failure and the migration start over on every pass, so a main that fails its quiesce parks Failed until the pin changes. The Failed phase is recoverable: when the main pod becomes Ready again the operator emits RecoveredFromFailure and resumes normal reconciliation.spec.nodeName pins the main agent (slot 0) to a node it is not currently on, reconcileMigration moves it — quiesce the old pod, snapshot it, delete it (flagged vm.cocoonstack.io/keep-snapshot-on-delete, best-effort as in slot release, so the target node can fetch the snapshot from the source node instead of the registry), recreate on the target node with restore-from-hibernate, and drop the snapshot only once the new VM runs. When the snapshot image rule below drops a snapshot pushed from another image, the recreate is left to step 11, which boots the main fresh on the target. Phase=Migrating persists throughout; while the restored main is Unschedulable the set emits MigrateNoCapacity with the scheduler’s message and keeps polling. A main still unscheduled under the required hostname affinity of a node spec.nodeName no longer names (a re-pin away from a full node, or a removed pin) is deleted and recreated under the current pin, from the snapshot when one exists; it has no VM yet, so nothing is lost, and the check reads the pod only, so a settled pass makes no extra call. It runs before un-suspend so the migration’s own hibernate annotation is not cleared; a main quiesced on its target wakes in place here only while the set is Migrating (a re-target back), and a plain unsuspend of a pinned set is left to step 10. It short-circuits when the target pod is already owned by a desire=Hibernate CocoonHibernation CR, or is tracked by a CocoonHibernation that has not settled Active (a wake in flight, or a failed wake that retries and still needs the snapshot), to avoid racing that reconciler. A :hibernate tag found while the main runs off target without this controller having quiesced it is a leftover and is dropped before the migration starts, but only once that main runs a live VM: until then a wake or restore may still read the tag, so the main stays with the normal flow and the migration starts on a later pass. Live state is never lost to the migration: the old pod dies only after the snapshot exists and this controller quiesced it; the snapshot drops only after the new VM runs, or when an image edit leaves the snapshot nothing to restore into.reconcileWake engages while the CocoonSet is Waking or Suspended, while it is Suspending and a node hint is stashed, or whenever a restore-marked main that is not hibernating still carries a hint. Once the released main is gone it takes over the reconcile if a snapshot exists to restore, recreates the main from that snapshot, keeps the reconcile while the recreated main comes up, and clears the hint on completion. When no snapshot is left to restore (a set suspended before its first boot has none, and the snapshot image rule below drops one pushed from another image), it clears the hint and the normal flow boots the main fresh; while the original main is still being deleted it waits, and only a main that is live and undeleted falls through to the normal flow. It runs before the main-agent step, whose CR-only restore intent would otherwise fresh-boot over that snapshot.spec.suspend == false, clear the hibernate annotation left on an owned pod via podpatch.HibernateState(false) so vk-cocoon wakes the VM. Pods that are the active target of a desire=Hibernate CocoonHibernation CR are skipped to avoid racing the hibernation reconciler. The first pass that finds the suspended list of step 6 does the unsuspend bookkeeping once, whether or not a pod still carries the annotation: it sets the record’s generation to the set’s, records every listed VM whose pod is missing (deleted while suspended, or still terminating) as owed a restore, clears the annotations, and clears the list last, each write durable before the next. A VM an earlier cycle left recorded is not listed, so its returning slot is not owed a restore. A pod that carries the annotation without a suspended list (a set suspended by an earlier operator version, or a quiesce the migration of step 8 abandoned) is recorded and cleared the same way. Steps 11 to 13 create a pod whose VM is owed a restore marked for restore when its :hibernate tag exists (a probe the registry refuses fails the pass), so a main, sub-agent or toolbox deleted while suspended keeps its suspended state instead of booting fresh or, for a sub-agent, being refused as a fork-from pod. The migration of step 8 and the wake of step 9 run before this step, so an unsuspend that also edits spec.nodeName migrates the main on the suspend’s snapshot, and a set whose pods are all gone wakes its main from the snapshot; the bookkeeping then runs on the first pass they hand back, before any sub-agent or toolbox is created. The bookkeeping makes no registry call, and a pass with neither an annotation to clear nor a suspended list skips it without an API call. Every pass that reaches this step drops the :hibernate tag of a recorded VM once its pod no longer carries the hibernate annotation and runs a live VM that vk-cocoon reports lifecycle-state=ready at that generation or later, as a CocoonHibernation wake does, so a later recreate of the pod fresh-boots instead of restoring the snapshot; the VM then leaves the record, and the annotation goes with the last one. A VM whose pod is missing, rebuilding or not yet woken stays recorded, except where vk-cocoon would refuse the new pod while the tag exists; there the tag is dropped before the pod is created, so the new pod boots fresh, and the VM leaves the record. That is a pod the snapshot image rule below creates fresh, and a sub-agent slot about to be created (step 12) that is neither owed a restore nor restored by a CocoonHibernation, such as a slot rebuilt after a failed wake, since vk-cocoon refuses a fork-from pod while the tag exists. A tag delete the registry refuses keeps its VM recorded and fails the pass, so the controller retries it with backoff; until the delete succeeds, the rest of that pass, scaling and the status patch included, does not run. A VM whose slot left the spec stays recorded until the set is deleted or the slot returns: a returning slot owed a restore, or a returning toolbox, leaves the record once its pod wakes, any other returning sub-agent slot before its pod is created.Ready, requeue in 5 s and report Phase=Pending.[1..Replicas] (creates are fanned out via an errgroup capped at 8 concurrent pod creates so a large scale-up does not burst the apiserver, each under the snapshot image rule below and marked for restore when step 10 owes it a restore); delete extras above the requested count. A sub-agent is pinned to the main’s node at creation with a hard pod.Spec.NodeName bind; a cross-node migration (step 8) moves slot 0 only, and existing sub-agents keep their original bind until they are next rebuilt./status from the classification taken in step 4 (with structural diff so unchanged status patches are no-ops).Snapshot image rule: vk-cocoon records the pod’s image on every :hibernate manifest it pushes, and refuses to create a pod, restore-marked or not, over a tag recorded from another image. Before the operator creates any agent or toolbox pod (the main in steps 6 and 11, the seat-release wake in step 9, the migration recreate in step 8, a sub-agent in step 12, a toolbox in step 13), it reads that VM’s :hibernate manifest; when the recorded image is non-empty and differs from the new pod’s image, it deletes the tag and creates the pod fresh, and the VM leaves the reclaim record of step 10. A manifest read or tag delete the registry refuses fails the pass, which retries with backoff, and no pod is created over the tag meanwhile. The rule costs one manifest read per attempt to create a managed pod, and nothing on a pass that attempts none.
Pods are constructed via meta.FromAgentSpec / meta.FromToolboxSpec factory helpers so the operator never touches the annotation map directly. These factories propagate the full VMOptions surface (OS, Backend, ConnType, Network, ForcePull, NoDirectIO, ProbePort, Storage) into the pod annotations that vk-cocoon consumes; Resources is not part of the annotation contract and lands on the pod’s container instead. The For watch admits a spec change (a new generation) or a status change, not an annotation-only patch: each status patch the operator makes brings one more reconcile, which reads a cache that holds that patch, so a status computed from a stale view of the pods is corrected, and an unchanged status is not written again, so it never loops. The Owns side filters pod events to creation, deletion, and meaningful transitions (phase change, readiness flip, label/annotation mutation) via a podRelevantChange predicate so pure VK status churn does not trigger reconcile storms.
spec.nodePool and spec.snapshotCompatibilityClass are independent hard placement dimensions. Every managed pod selects cocoonstack.io/pool=<nodePool>; a non-empty snapshot class also selects cocoonstack.io/snapshot-cpu-class=<class>. The class identifies a certified guest-visible CPU ABI, not merely amd64, and remains in force across release-policy wake, rebuild, and migration; it is immutable once set. Hostname affinity from spec.nodeName (main pin or migration target) is a required nodeAffinity — the pod stays Pending rather than landing off that host — and neither it nor the release-policy wake’s weight-100 preferred hostname affinity relax either selector. Pre-feature pods whose immutable nodeSelector lacks the class are adopted rather than drift-deleted; their next normal recreate or released-seat wake converges them onto the hard selector.
See Observability for the Event reasons and metrics this loop emits, and Configuration for the operator’s environment variables.