sandbox-operator

A Kubernetes operator and aggregated apiserver for fast, warm-poolable agent sandboxes backed by real microVMs. It implements the kubernetes-sigs/agent-sandbox API in full — Sandbox, SandboxTemplate, SandboxWarmPool, SandboxClaim, v1alpha1 and v1beta1 with conversion webhooks — and is driven entirely through the standard Kubernetes API, with no proprietary SDK. A pre-warmed sandbox is acquired in ~33 ms at p50, and each sandbox is a genuine Cloud-Hypervisor/KVM microVM rather than a shared-kernel container.

any Kubernetes client (kubectl / client-go / controller-runtime)
        |                                        e2b SDK (E2B_API_URL)
        v                                                |
kube-apiserver + agent-sandbox CRDs                       v
        |                                     sandbox-apiserver (aggregated)
        v                                                |
sandbox-operator                                          |
Sandbox / SandboxTemplate / SandboxWarmPool / SandboxClaim
        |
        v
warm pool: N pre-booted microVMs
        |
        +-- claim: adopt one, control-plane only (~33 ms) --> delivered sandbox
        |
        +-- runtime: standard   -> ordinary Pod on a standard kubelet node
        +-- runtime: vk-cocoon  -> Cocoon microVM on a virtual-kubelet node
        +-- runtime: sandboxd   -> node-local sandboxd hot pool (0.2-0.7 ms)

The claim model

A cold Sandbox creates a Pod on demand; with the microVM backend that Pod boots a VM, which costs tens of seconds. The warm path keeps that off the request path: a SandboxWarmPool pre-provisions N Ready microVMs, and a SandboxClaim adopts one — the VM is already booted, so the claim is a single optimistic PATCH guarded by resourceVersion, with no scheduler, no kubelet bind, no image pull and no etcd write on the claim path. The pool replenishes in the background.

That is the same ownership-transfer shape Kubernetes already ships for PersistentVolumeClaim → PersistentVolume binding, which is why the whole model stays expressible in ordinary CRDs: kubectl get sandboxes keeps working, RBAC and audit keep applying, and any Kubernetes client is a valid client.

Runtime backends

Runtime selection is per-Sandbox, via the Pod-template annotation sandbox.cocoonstack.io/runtime, falling back to the operator’s --default-runtime:

standard (the default) — an ordinary Pod on a standard kubelet node. Portable to any conformant cluster; no special substrate required.
vk-cocoon — a hardware-isolated Cloud-Hypervisor/KVM guest, materialized on a vk-cocoon virtual-kubelet node by Cocoon.
sandboxd — routes the Pod to a vk-sandbox virtual node, which serves the claim from the node-local sandboxd hot pool of cocoonstack/sandbox. The ownership transfer itself is 0.2–0.7 ms.

The adapter only fills in the scheduling fields a backend needs, and rejects — never overwrites — a conflicting explicit value.

Scaling design: L0 through L3

Reaching a million sandboxes means removing the centralized transaction path, not the API semantics. The thesis is to keep Kubernetes as the record-of-intent and policy plane and push the transaction plane down to the node, behind CRDs, RBAC and watch. Four layers:

layer	what it does	status
L0 — API hygiene	cache-fed reads, diff-before-write, no control-loop `LIST` against etcd; the qualifier that stops APF seat exhaustion at scale	shipped
L1 — ownership transfer	claim = one select + one `PATCH`; `O(nodes)` pool status; per-pool sharded operator with a `coordination.k8s.io` Lease	implemented here
L2 — node-local claim gateway	a DaemonSet fronts `sandboxd`, delivers a running microVM in 0.2–0.7 ms, records `Bound` asynchronously; authorization stays central	designed, `ClaimGateway` skeleton
L3 — aggregated apiserver	`sandboxes` served by scatter-gathering per-node `NodeInventory`; etcd stores intent only, so object count drops from `O(sandboxes)` to `O(pools + nodes)`	designed, `SandboxStore` skeleton

The measured consequence: one kubectl patch taking a SandboxWarmPool from 0 to 50 000 microVMs on 20 bare-metal nodes reaches full supply in 10–15 s at 99 MB net RAM per microVM, while etcd sees ~2 writes/s across the whole run — independent of sandbox count. Full methodology, per-round sampling and the memory ledger are in PERFORMANCE.md.

The same design has one visible cost: the read view is synthesized from NodeInventory published on a ~30 s cadence, so list/get are eventually consistent and a just-created sandbox is briefly invisible. Callers poll.

Guides

API reference — the generated reference for every type in agents.x-k8s.io (v1alpha1, v1beta1) and extensions.agents.x-k8s.io (v1alpha1, v1beta1)
Operator configuration — every flag: runtime and API surface, controller concurrency, webhook and leader election, observability, and a Helm/Deployment patch example
Runtime backends — why standard kubelet is the rollout default, the deterministic selection rules and their conflict cases, and the full annotation contract each backend supplies
e2b-compatible API — serving the e2b REST surface from the aggregated apiserver so an unmodified e2b SDK claims from these warm pools: flags, endpoint mapping, and the limits worth knowing
Lifecycle verbs — pause, resume, fork and snapshot as subresources, plus a runnable walk-through over both API surfaces.
Scaling design — how claims stay off etcd and what the per-node control plane owns.
Snapshot placement — where a checkpoint lives, how a branch reaches it from another node (local hit, gossip + redirect, peer heal), why shared filesystems are ruled out, and the durability this does not give

Repository

Source and issue tracker: github.com/cocoonstack/sandbox-operator. Part of the cocoonstack MicroVM platform.

This site is open source. Improve this page.