vk-sandbox is a single flag-driven daemon: main.go wires a virtual-kubelet
node, and two packages carry the work.
| Package | Responsibility |
|---|---|
provider/ |
The virtual-kubelet PodLifecycleHandler: claim on CreatePod, the delete-authorization contract on DeletePod, cache-fed status reads, the audit-only orphan scan, and the persisted claims table |
inventory/ |
The node’s L3 contribution: LiveSource (a scale.NodeLiveSource over sandboxd’s index plus the claims table), NodeInfoSource (advertise address + warm-pool capacity), and the Publisher that server-side-applies one NodeInventory per node |
Nothing on the sandboxd wire is reimplemented here – Claim, Release, the
live index (Sandboxes) and warm-pool capacity (Info) all come from
github.com/cocoonstack/sandbox-operator/pkg/sandboxd, so the wire
contract has exactly one home. The same repo supplies pkg/scale’s selector
keys, the entry mapping (scale.EntryFromSummary) and the InventoryApplier
used for the NodeInventory apply.
The load-bearing rules, carried over from the production vk-cocoon provider and pinned by intent tests:
DeletePod releases only when the owning Sandbox is confirmed gone (a
structured NotFound naming it in Details.Name), replaced by another UID,
in teardown (deletionTimestamp set), or expired (Ready reason
SandboxExpired). Otherwise the claim is preserved, and a
same-name replacement pod of the same owner adopts it in place — no
second claim, same VM.
A preserved claim’s owner is re-checked with backoff, so a namespace
deletion that removes Pod and owner independently still releases the VM. See
delete authorization.Sandbox→sandboxes); an endpoint-level 404 without
Details.Name is treated as “GVR guess wrong”, never as “owner deleted”.
(A naive +"s" once destroyed a live-owner VM.)Failed
on confirmed absence; listing errors defer the decision. virtual-kubelet
never polls an asynchronous provider. See leases.Pod bound to the virtual node
|
v
CreatePod
|
+-- runtime annotation != "sandboxd" ------------> error (wrong node)
|
+-- a claim already exists for <namespace>/<name>
| -> adopt in place: rebind the pod UID, no new VM
| (only the same Pod or a Pod of the claim's owner adopts;
| a claim held for another owner is queued for release instead)
|
+-- otherwise: POST /v1/claim {template, net, size, ttl_seconds, claim_ref}
|
+-- error (incl. sandboxd 429 / redirect = no warm capacity)
| -> CreatePod fails, the Pod stays Pending;
| virtual-kubelet retries the create with backoff
| (restartPolicy Never: the Pod goes Failed instead)
|
+-- ClaimResult {id, token, owner_addr}
|
v
claims[key] = {ID, Token, Address, PodUID, Authority, ClaimedAt, Deadline}
persist the table, then notify the kubelet:
phase Running, PodIP/HostIP = host of owner_addr,
one synthetic ready container per spec container
(ImageID "sandboxd://<claim id>"),
annotation sandbox.cocoonstack.io/claim-id = <claim id>
UpdatePod retries creation for a tentative claim whose persistence and
compensating release failed. Otherwise it records the newest Pod object –
running sandbox Pods are immutable at the runtime level. GetPod, GetPods, and
GetPodStatus are served from the in-memory table, so no control-loop read
ever hits the apiserver or sandboxd.
The kubelet interactive surfaces – GetContainerLogs, RunInContainer,
AttachToContainer, PortForward – return ErrNotImplemented by design:
access to a sandbox goes through the sandbox SDK and preview URLs, not the
kubelet exec path. GetStatsSummary returns an empty summary and
GetMetricsResource no metric families.
A Node-NotReady taint eviction deletes every Pod on a node while the microVMs
keep serving users. For a controller-owned Pod, DeletePod reads its owner
through the dynamic client before deciding; a bare Pod is its own authority:
| Owner state | Verdict |
|---|---|
| No controller ownerReference (bare pod) | release – the Pod is its own teardown intent |
Structured NotFound whose Details.Name matches the owner |
release – owner confirmed gone |
Owner exists with a non-zero deletionTimestamp |
release – owner in teardown |
| Owner exists with a different UID than the ownerReference | release – the referenced generation is gone |
Owner Sandbox alive but expired (Ready reason SandboxExpired) |
release – the operator tore its workload down and no replacement Pod comes |
| Owner alive | preserve |
404 without Details.Name (endpoint shape unverified) |
preserve |
| Any query error, or no dynamic client configured | preserve |
Preserve drops only the Pod entry; the claim – and with it the release token – stays in the table so a same-name replacement Pod of the same owner adopts the same VM. The stale-UID guard runs first: a request carrying a previous Pod generation’s UID is ignored outright.
Preserve also records the controller owner with the claim. Deleting a namespace
deletes its Pods and their owning Sandbox objects independently, so about half
the Pods reach DeletePod while their owner is still alive; nothing would look
at those claims again until the lease ends. A re-check loop therefore asks about
the recorded owner of every pod-less claim, first after ownerRecheckInterval
(10s) and then with doubling backoff up to ten minutes, applying the verdict
table above: once the owner is confirmed gone, in teardown, expired or replaced,
the sandbox is released and the claim dropped; while it lives or cannot be
verified, the claim stays. A replacement Pod adopting the claim ends the re-check.
A claim the re-check releases is first withdrawn from its key into a persisted
release queue, so a replacement Pod arriving meanwhile claims fresh instead of
adopting a sandbox on its way out; a release that fails stays queued with its
credential and is retried every tick, across restarts, until sandboxd confirms
it gone. The recorded owner survives a restart with the claim. A same-name Pod
whose controller owner is not the recorded one, such as the Pod of a Sandbox
deleted and recreated under the same name, never adopts the claim: it claims
fresh, and the old claim goes to the release queue.
A Pod that leaves the apiserver while the provider is down, such as one
force-deleted during an outage, never reaches DeletePod, so no owner is
recorded for its claim. Every claim therefore also records its authority
when it is taken or adopted: the Pod’s controller owner UID, or the Pod’s own
UID for a bare Pod. A same-name Pod, after a restart or otherwise, adopts only
if it is the Pod that holds the claim or has the same authority; the
preserve-time owner, when recorded, takes the authority’s place. Any other Pod
claims fresh and the old claim goes to the release queue. A claim from a table
written before the field existed has no authority and is adopted as before.
The owner GVR is derived from apiVersion + kind with the English plural
rules (Sandbox -> sandboxes, policy -> policies), never a naive
+ "s". Even so, a wrong guess is safe: an endpoint-level 404 carries no
Details.Name and is read as “unverifiable”, not “owner deleted”.
On an authorized release, the sandboxd client treats a 404 as already gone;
any other release error keeps the claim and the Pod entry so the kubelet
retries with the credential intact.
The claims table ({id, token, address, podUID, authority, claimedAt, deadline, owner} per
pod key, plus the releasing list of withdrawn claims awaiting release) is written to --state-path as JSON with a tmp-file + rename, mode
0600, directory mode 0700. It is reloaded at startup, so a provider
restart keeps the authority to tear down exactly what it delivered. The
binary requires the flag; only the in-process constructor accepts an empty
path, for tests.
A claim is durable before it counts: until its own write lands it is invisible to Running status and to adoption. A create whose write fails tries to return the sandbox; failed compensation retains the credential for update retries or deletion (see Pod contract). At startup the reloaded table is checked against the node’s live sandboxes and rows the node no longer holds are dropped. A failed listing is not an empty list, so an unreadable sandboxd drops nothing — the rows are quarantined instead: still releasable, but not adoptable and not Running until a later listing vouches for them. A verification loop, independent of the audit scan so that switching the audit off cannot strand them, lifts the quarantine as soon as a listing succeeds and then stops.
Ordinary claims request a 24-hour lease by default, with ttl-seconds to
override it, and have no automatic renewal. The provider persists the returned
deadline and watches for expiry. It pushes terminal status through NotifyPods;
virtual-kubelet installs no status poller for an asynchronous provider.
The cached deadline is not authoritative: the archive lifecycle rewrites a
claim’s lease on the node. Failed is terminal, so before publishing it — and
before replacing a past-deadline row — the node’s listing confirms the sandbox
is really gone; a still-listed claim just has its lease refreshed from the
listing, and an unlistable node defers rather than declares. The publication
itself re-checks identity: if the key changed hands behind the listing (old
Pod deleted, successor claimed fresh), the expiry belonged to the old claim
and nothing is published against the new one.
--orphan-scan-interval runs a background comparison of sandboxd’s live index
against the claims table. It reports only – background reconciliation
cannot prove user intent, so it never releases:
claim_ref are logged
as possible orphans and retained; a non-empty claim_ref marks an
apiserver-direct claim and is recorded as external, not an orphan;A failed list is not an empty list: if GET /v1/sandboxes errors, the whole
cycle is skipped rather than treating “query failed” as “zero sandboxes”.
With --publish-inventory, a goroutine server-side-applies one NodeInventory
object named after the node, every --publish-interval, with field owner
vk-sandbox. That is the node’s entire L3 write path – one O(nodes) apply, no
per-sandbox etcd object, the same summarize-on-the-node pattern as
metrics.k8s.io.
Each apply names this node’s Node as the object’s owner, so deleting the
Node garbage-collects its NodeInventory: the node leaves the claim path and
its sandboxes leave the read view, even while sandboxd keeps serving them and
their leases expire there. On a node’s first start the virtual node registers
after the first publish, so that one goes out unowned and the next sets the
owner. Once this process has seen its Node, a Node that is gone means it
was deleted: the publish stops instead of bringing the collected inventory back
unowned, and the node returns when vk-sandbox restarts and registers again. A
publish that cannot read the Node for any other reason is skipped until the
next tick.
Every apply stamps publishedAt with the publish time. The aggregated
apiserver drops a node whose publishedAt trails the newest publish in the
fleet by more than its --inventory-stale-after (90 s by default), so a node
that dies, or whose vk-sandbox stops, leaves the claim path and the read view
within that window, and comes back on its next publish. A vk-sandbox restart
shorter than the window is invisible, and a control-plane outage, which stops
every publish at once, drops no node.
sandboxd GET /v1/sandboxes ---+
(authoritative live set) |
+--> LiveSource --> entries[]
provider claims table --------+ (scale.EntryFromSummary, the mapping the
(address per claim id) apiserver's live reads use:
name = claim_ref, else sandboxd id;
ID = sandboxd claim id;
phase = Running | Hibernated;
deadline = the row's lease end;
claimedAt = the row's first grant;
address from this node's own claim)
sandboxd GET /v1/info --------> NodeInfoSource --> {Address, Pools[]}
(per-pool warm/target) advertise address is static config
entries[] + Address + Pools[]
|
v
NodeInventory (server-side apply, one per node, owned by its Node)
|
v
sandbox-operator aggregated apiserver serves
`kubectl get sandboxes` from the summaries
Entries are read from sandboxd’s index rather than the claims table, so claims the aggregated apiserver made directly against sandboxd are published too, and a claim whose VM is gone is simply not listed. Each entry carries sandboxd’s own claim id, which is the handle the node’s release verb needs – so a delete through the aggregated path releases exactly that microVM and never resolves by Kubernetes name. Publish failures are logged, not fatal: the next tick rebuilds from live state.
kubernetes-sigs/agent-sandbox
owns the record plane – the agents.x-k8s.io CRDs, warm pools, claims, and
the controller that turns a Sandbox into a Pod. A SandboxTemplate routes that
Pod here by carrying the Pod contract itself.
sandbox-operator owns
the L3 aggregated apiserver and the NodeInventory CRD
(sandbox.cocoonstack.io) this provider publishes into. vk-sandbox owns the
node transaction plane and imports the operator’s pkg/sandboxd for the
client, pkg/scale for selector keys and the inventory applier, and
pkg/logbridge to route controller-runtime’s and client-go’s log output through core/log. The
dependency points one way: the operator does not import this repo.
sandbox ships sandboxd, the node-local daemon holding the warm microVM pools this provider claims from.
vk-cocoon is the sibling
virtual-kubelet for full cocoon MicroVM Pods. The two providers co-exist on
one physical node as two distinct virtual nodes: different node names,
different kubelet listen ports (vk-cocoon :10250, vk-sandbox :10260), and
different routing labels (cocoonstack.io/pool=<pool> for vk-cocoon,
sandbox.cocoonstack.io/runtime=sandboxd here). Both carry the
virtual-kubelet.io/provider taint, which one Exists toleration in the Pod
template covers. vk-sandbox can reuse the co-located vk-cocoon kubelet
certificate when its files exist, and self-signs otherwise, so its API surface
is uniform either way. The delete-authorization and audit-only-GC contracts
implemented here are carried over from vk-cocoon.