Reconciler failures surface as K8s Events on the CR plus Prometheus metrics (counters, and a duration histogram each for hibernate and wake) on the controller-runtime /metrics endpoint:
| Event reason (CocoonHibernation) | Type |
|---|---|
HibernateTimedOut, WakeTimedOut |
Warning |
Hibernated, WokenActive, RetryRequested |
Normal |
| Event reason (CocoonSet) | Type |
|---|---|
PodLifecycleFailed, MainAgentFailed, SubAgentDeadLetter, WakeNoCapacity, MigrateNoCapacity, SuspendTimedOut |
Warning |
SubAgentRebuilding, RecoveredFromFailure |
Normal |
WakeNoCapacity fires while a hibernatePolicy: release wake sits
Unschedulable: the seat the policy freed is not available again yet. The
reconciler keeps waiting instead of failing the wake, so the event is the
out-of-capacity signal, not a terminal error. MigrateNoCapacity is the same
signal for a cross-node migration whose restored main cannot be scheduled; the
set stays Migrating and the event repeats on every poll until a seat opens
on the node the pending pod is pinned to. A main still pending under a node
that spec.nodeName no longer names is recreated under the current pin (step
8 of CocoonSet). The scheduler message usually aggregates
reasons without naming nodes, so read the node being waited on from the
pending pod’s required node affinity (kubectl get pod <main> -o
jsonpath='{.spec.affinity.nodeAffinity}'). SuspendTimedOut fires
when a spec.suspend: true set has sat in Suspending for suspendTimeout
(3 min) without every managed VM hibernated and its snapshot in the registry;
the set reports Failed, then re-enters Suspending with a fresh deadline on
the next pass, mirroring HibernateTimedOut on the CR path.
Metrics:
cocoon_operator_subagent_rebuild_total{namespace, cocoonset}
cocoon_operator_subagent_dead_letter_total{namespace, cocoonset}
cocoon_operator_hibernate_phase_duration_seconds{result} # result=ok|timeout
cocoon_operator_wake_phase_duration_seconds{result}
cocoon_operator_lifecycle_state_failed_observed_total{phase}
cocoon_operator_slot_release_pods_deleted_total{namespace, cocoonset}
cocoon_operator_slot_release_wake_total{namespace, cocoonset, placement} # placement=hint-node|pool
cocoon_operator_slot_release_wake_unschedulable_total{namespace, cocoonset}
cocoon_operator_migrate_unschedulable_total{namespace, cocoonset}
The three slot_release_* families cover hibernatePolicy: release
(see CocoonSet reconcile loop): how many seats a suspend
freed, and where each wake landed — hint-node restored from the
node-local snapshot on the node it hibernated on, pool was rescheduled
elsewhere and cold-pulled from the registry. A rising
slot_release_wake_unschedulable_total means wakes are queuing behind
cluster capacity, which is the cost side of the policy.
CocoonSet consumes the vm.cocoonstack.io/lifecycle-state=failed annotation that vk-cocoon writes on terminal failures; the rebuild and triage path treats it as terminal on every owned pod role (main, sub-agent, toolbox) so reconciliation reacts immediately instead of waiting for Pod.Status.Phase to follow; the suspend gate and the slot-release drain wait for Pod.Status.Phase alone, bounded by suspendTimeout (see CocoonSet reconcile loop step 6). A drifted pod (main, sub-agent or toolbox) or a terminal sub-agent or toolbox is deleted for recreate up to four times with 0/1/5/30 s backoff between attempts, then marked cocoonset.cocoonstack.io/dead-letter=<CocoonSet generation> and left in place, so a slot that cannot converge (a terminal image, or a pod an external defaulter such as a LimitRange keeps rewriting) stops churning VMs. While any owned pod is parked at the current generation the CocoonSet’s Progressing condition reads False with reason DeadLettered and names the pods, so the set never looks converged with a pod still on the old spec. Dead-lettering also blocks recreation of a missing slot at that generation: a missing main reports Phase=Failed, and a missing sub-agent or toolbox is simply skipped, until a spec edit lifts the budget. A terminal main that still matches its spec is not rebuilt: the CocoonSet reports Failed until the main is edited or recovers. A spec edit lifts the dead-letter: at a newer generation the pod is rebuilt with a fresh budget. Rebuild counts persist in the cocoonset.cocoonstack.io/rebuild-history annotation on the CocoonSet, keyed by pod name, so they survive the pod delete; entries for pods the spec no longer names are garbage-collected on every write.