sandbox-operator

Configuration

This repository ships three binaries and one Helm chart. Flag text below is the binaries’ own --help output; sandbox-e2b, the mesh-mode e2b front, is configured in e2b-compat.

Installation

Pod path

Upstream’s release installs the CRDs and the controller that serves Pods:

kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/v1.0.5/sandbox-with-extensions.yaml

That is the whole install for the Pod path; nothing from this repository is needed. Routing a Sandbox Pod to a microVM node is the pod-template contract. Two of the controller’s flags drop one apiserver write per claim each, with no effect on latency (performance):

kubectl -n agent-sandbox-system patch deployment agent-sandbox-controller --type=json -p '[
  {"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--disable-claim-events"},
  {"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--disable-claim-observability-annotations"}]'

How long a claim waits for a warm Sandbox’s Pod IP before it creates its own is a flag on upstream main, --sandbox-claim-warm-candidate-grace-period (default 2 s, in no release up to v1.0.5). On sandboxd nodes the default is the right value (performance).

L3 path

Prerequisites: cert-manager, unless you set certManager.enabled=false and supply the serving certificate yourself; and, for the e2b data plane, wildcard DNS for *.{domain} resolving to the sandbox-envd-proxy Service.

The warm-pool driver reads two of upstream’s CRDs. Apply those, then the chart:

VERSION=v1.0.5
for crd in extensions.agents.x-k8s.io_sandboxtemplates extensions.agents.x-k8s.io_sandboxwarmpools; do
  kubectl apply -f "https://raw.githubusercontent.com/kubernetes-sigs/agent-sandbox/${VERSION}/k8s/crds/${crd}.yaml"
done

helm upgrade --install sandbox-operator ./helm \
  --namespace sandbox-system --create-namespace \
  --set apiserver.image.tag="$VERSION_OF_THIS_REPO" \
  --set envdProxy.image.tag="$VERSION_OF_THIS_REPO"

Image tags default to latest, so the chart renders with no --set at all; pin both to a release tag in production. The e2b surface is off by default and needs five values together; the envd proxy it deploys exits at startup without the fleet sandboxd token:

helm upgrade --install sandbox-operator ./helm \
  --namespace sandbox-system --create-namespace \
  --set apiserver.e2b.enabled=true \
  --set apiserver.e2b.domain=sandbox.example.com \
  --set apiserver.e2b.apiKeySecret.name=e2b-api-keys \
  --set apiserver.e2b.envdSecret.secretName=e2b-envd-secret \
  --set apiserver.sandboxdToken.secretName=sandboxd-token

Enabling it without the key secret fails at render (apiserver.e2b.apiKeySecret.name is required when apiserver.e2b.enabled is true) — the surface refuses to serve unauthenticated — and so does enabling it without apiserver.e2b.envdSecret.value or .secretName.

The chart’s default render is 16 objects: the nodeinventories.sandbox.cocoonstack.io CRD, the apiserver’s Deployment, Service, ServiceAccount and PodDisruptionBudget, the APIService, the RBAC (a ClusterRole granting get/list/watch on nodeinventories plus the pool rules, the system:auth-delegator binding, the leader-election Role, and the sandbox-apiserver-auth-reader RoleBinding that must live in kube-system to read extension-apiserver-authentication), and the cert-manager chain — a self-signed Issuer, a CA Certificate, a CA Issuer and the serving Certificate. Enabling the e2b surface adds its own Service, sandbox-apiserver-e2b, and the proxy’s Deployment, Service, ServiceAccount and nodeinventories ClusterRole and binding (22 objects, 23 when the chart renders the envd secret from apiserver.e2b.envdSecret.value): the proxy is that surface’s data plane and renders only with it. The e2b port has a Service of its own so that the build affinity (apiserver.e2b.builds) never pins the aggregated API. Every other object goes to .Release.Namespace; object names are fixed (sandbox-apiserver, sandbox-apiserver-e2b, sandbox-envd-proxy), not release-prefixed, so the release name is free.

It ships no upstream CRD.

Two rules follow from the design, and breaking either one breaks the cluster:

Images are published to GHCR on a release tag: ghcr.io/cocoonstack/sandbox-apiserver, ghcr.io/cocoonstack/sandbox-envd-proxy and ghcr.io/cocoonstack/sandbox-e2b, multi-arch (amd64/arm64).

sandbox-apiserver

The L3 aggregated apiserver. It reads NodeInventory through an informer and needs get/list/watch on nodeinventories.sandbox.cocoonstack.io; the chart’s RBAC grants that. It takes its kube client from the in-cluster service account, or from KUBECONFIG when run outside a cluster.

Node-local claim path

Flag Default Help
--sandboxd-token — sandboxd api_token presented to every node; a binary that reads claim tokens needs the root one. Prefer --sandboxd-token-file for a Secret mount.
--sandboxd-token-file — Path to a file (Secret mount) holding the sandboxd api_token; overrides --sandboxd-token when set.

With both empty, node-local calls carry no token. A sandboxd with an api_token refuses them all, and even an open one refuses Delete and the lifecycle verbs, which act on a sandbox only under its own token or the node’s api_token: reads of published inventory still work, and a Create against an open node holds its microVM until the lease expires.

Warm-pool driver

Flag Default Help
--enable-warm-pool-driver true Run the in-process SandboxWarmPool → sandboxd pool reconcile loop (control-plane warm-capacity surface; pool-level, never per-sandbox).
--warm-pool-sync-interval 0 (means 5s) Resync cadence for the SandboxWarmPool driver, and with it the sampling period of the warm count in pool status.

The driver watches SandboxWarmPool and NodeInventory, resolves each pool’s SandboxTemplate into a (template, net, size) key, spreads spec.replicas evenly across the nodes that advertise a sandboxd address, and PUTs each node its full pool set. It writes status.replicas and status.readyReplicas back from the warm counts those calls report. It runs under leader election (cocoon-warmpool-driver), so one replica drives the pools. A driver that stops, most often because its lease could not be renewed during a control-plane outage, is rebuilt after 5 s and waits in leader election again, so the replica keeps serving the aggregated API and the e2b API; only a driver that cannot be built at startup stops the process. An empty sandboxd token leaves it fail-closed: it logs and sets no pools.

Node inventory freshness

Flag Default Help
--inventory-stale-after 90s Drop a node from this process’s inventory reads once its NodeInventory publishedAt trails the newest publish in the fleet by more than this; set the same value on every binary that reads inventory. An inventory without publishedAt is stale.

A node that stops publishing, because it died or its vk-sandbox stopped, leaves every read and claim path once its publishedAt trails the newest publish by more than this. At vk-sandbox’s default 30 s publish cadence that is three missed publishes, and since the newest publish only advances when a live node publishes, a dead node leaves between this age and one publish interval later. Keep it above vk-sandbox’s --publish-interval.

The age is measured against the newest publish the process has seen, capped by its own clock. A control-plane write outage freezes every stamp together, so it never empties the fleet, and a node’s clock matters only relative to the other nodes: one running ahead cannot evict the rest. A fleet whose nodes all stop publishing at once, or a one-node fleet, keeps its last inventory until the nodes publish again.

The window works on a node once the nodeinventories CRD from helm/crds is applied and that node runs a vk-sandbox that stamps publishedAt. An older CRD prunes the field and an older vk-sandbox never sets it; an inventory without publishedAt is stale once any node stamps one, and a fleet where none does keeps every node. Apply the CRD and upgrade vk-sandbox before relying on the window.

e2b-compatible surface

Flag Default Help
--enable-e2b-api false Serve the e2b-compatible REST surface, so an unmodified e2b SDK can claim from the same warm pools (point E2B_API_URL at it).
--e2b-bind-address :8080 Address the e2b-compatible surface listens on.
--e2b-namespace default Namespace a key that names none claims in, and where anonymous claims land; e2b has no namespace concept.
--e2b-domain — Base domain the SDK derives the in-sandbox envd host from, as {port}-{sandboxID}.{domain}. Required whenever the surface is served: without it a created sandbox has no reachable data plane.
--e2b-envd-version 0.8.0 when empty envd version reported to the SDK. It must name the envd actually installed in the pool’s image; the SDK version-compares it and kills the sandbox when it cannot parse one.
--e2b-default-timeout 300 when 0 Lease in seconds granted to a create that names no timeout, and the lease an SDK refresh renews for.
--e2b-api-key-file — Path to a file (Secret mount) of accepted e2b API keys, one per line as key or key namespace, presented by the SDK as X-API-KEY; a key sees only the sandboxes and snapshots of its namespace, --e2b-namespace when none is given.
--e2b-allow-anonymous false Serve the e2b surface with NO API key. Development only: it leaves the claim endpoint open to anyone who can reach the port.
--e2b-template-alias-file — Path to a file of e2b template aliases, one per line as alias pool-image [size], so a create naming the alias (the SDK’s default is base) claims from that image’s pool at size, small when none is given. The chart writes apiserver.e2b.templateAliases to it.
--e2b-envd-secret-file — Path to a file (Secret mount) holding the key every sandbox’s envd access token derives from; the e2b surface and the envd-proxy must read the same one. Required with --enable-e2b-api.
--e2b-builds false Serve the e2b template build API (Template.build). Builds run in the process that took the request and live in its memory, so every status poll must reach the same replica.
--e2b-build-parallel 2 Builds that run at once; a start beyond them answers 429 and the build stays waiting.
--e2b-build-timeout 30m Bound on one build, claim through publish; it is also the lease of the build’s sandbox.
--e2b-build-log-lines 10000 Log lines kept per build; later lines are dropped.
--e2b-build-dir — Directory that keeps the archives COPY steps upload, taken through this surface’s own signed PUT. Every replica must see the same directory, or use --e2b-build-store-s3-bucket.
--e2b-build-upload-max 1073741824 Largest archive in bytes a signed PUT to --e2b-build-dir takes.
--e2b-build-store-s3-bucket — S3 bucket that keeps the archives COPY steps upload, through presigned PUTs; credentials come from the default AWS chain.
--e2b-build-store-s3-prefix — Key prefix for the archives in --e2b-build-store-s3-bucket; they live under <prefix>/e2b-files/.
--e2b-build-store-s3-endpoint — Endpoint of an S3-compatible store instead of AWS.
--e2b-build-store-s3-region — Region of --e2b-build-store-s3-bucket.
--e2b-build-store-s3-force-path-style false Address the bucket in the path, as most S3-compatible stores need.

Startup fails when --enable-e2b-api is set with neither a key file nor --e2b-allow-anonymous, and when --e2b-domain is empty. Details of the surface are in e2b compatibility.

Serving and delegated auth

The binary also takes the standard aggregated-apiserver option sets from k8s.io/apiserver — secure serving, delegated authentication and authorization, and profiling. --help prints all of them. The ones a deployment sets are --secure-port (default 6443), --tls-cert-file / --tls-private-key-file (or --cert-dir, which self-signs when the pair is absent), and --authentication-kubeconfig / --authorization-kubeconfig when the server runs outside the cluster. There is deliberately no etcd option: this server stores nothing.

sandbox-envd-proxy

The e2b data plane. It resolves a sandbox from the same NodeInventory informer and relays the request into the owning node’s guest-port endpoint.

Flag Default Help
--bind-address :8443 Address the proxy listens on.
--domain — Base domain sandbox hosts are derived from, as {port}-{sandboxID}.{domain}. Must match the apiserver’s --e2b-domain.
--namespace default Namespace inventory lookups are filtered to; empty matches every namespace. Not an access boundary: a caller holding a sandbox’s token reaches it in any namespace.
--tls-cert-file — Wildcard certificate for *.{domain}. Omit to serve cleartext h2c behind an edge that terminates TLS.
--tls-private-key-file — Private key for --tls-cert-file.
--guest-http2 false Forward to the guest over cleartext HTTP/2. Off by default: envd 0.8.0 installs no h2c handler and refuses it. Clients still reach this proxy over HTTP/2.
--e2b-envd-secret-file — The e2b surface’s envd secret file; access tokens are verified against it. Required.
--sandboxd-token / --sandboxd-token-file — sandboxd api_token presented to every node; the proxy needs the root one, since it reads claim tokens by id and relays signed file URLs with it. The file overrides the literal. Required.
--inventory-stale-after 90s Drop a node from this process’s inventory reads once its NodeInventory publishedAt trails the newest publish in the fleet by more than this; set the same value on every binary that reads inventory. An inventory without publishedAt is stale.

The two TLS flags must be set together. Routing, authorization and failure mapping are in envd-proxy.

Chart values

Value Default Effect
apiserver.image.repository ghcr.io/cocoonstack/sandbox-apiserver Aggregated apiserver image
apiserver.image.tag latest Pin a release tag in production
apiserver.image.pullPolicy IfNotPresent  
apiserver.replicaCount 2 Apiserver replicas
apiserver.securePort 6443 --secure-port
apiserver.warmPoolDriver true --enable-warm-pool-driver
apiserver.sandboxdToken.secretName "" Secret holding the fleet sandboxd api_token, mounted for --sandboxd-token-file. Empty sends node-local calls without a token
apiserver.sandboxdToken.key token Key within that Secret
apiserver.e2b.enabled false --enable-e2b-api
apiserver.e2b.domain "" --e2b-domain; required once enabled
apiserver.e2b.namespace default --e2b-namespace
apiserver.e2b.envdVersion "" --e2b-envd-version; empty takes the binary’s 0.8.0
apiserver.e2b.defaultTimeoutSeconds 0 --e2b-default-timeout; 0 takes the binary’s 300
apiserver.e2b.port 8080 --e2b-bind-address
apiserver.e2b.apiKeySecret.name "" Secret of accepted X-API-KEY values, mounted for --e2b-api-key-file; required once e2b is enabled
apiserver.e2b.apiKeySecret.key keys Key within that Secret
apiserver.e2b.envdSecret.value "" Key the chart renders into a Secret for --e2b-envd-secret-file; set this or secretName once e2b is enabled
apiserver.e2b.envdSecret.secretName "" Existing Secret for --e2b-envd-secret-file when value is empty; mounted into the apiserver and the proxy
apiserver.e2b.envdSecret.key secret Key within that Secret
apiserver.resources 100m / 128Mi requests, 512Mi limit Apiserver resources
envdProxy.image.repository ghcr.io/cocoonstack/sandbox-envd-proxy Proxy image
envdProxy.image.tag latest Pin a release tag in production
envdProxy.image.pullPolicy IfNotPresent  
envdProxy.replicaCount 2 Proxy replicas
envdProxy.domain "" --domain; falls back to apiserver.e2b.domain
envdProxy.namespace "" (every namespace) --namespace; the proxy is not an access boundary, and a key’s sandboxes live in the key’s namespace
envdProxy.port 8443 --bind-address
envdProxy.tlsSecretName "" Wildcard certificate for *.{domain}; empty serves cleartext h2c
envdProxy.resources 100m / 128Mi requests, 512Mi limit Proxy resources
certManager.enabled true Render the Issuer/Certificate chain for the apiserver’s serving cert
certManager.servingCertSecret sandbox-apiserver-serving-certs TLS Secret the apiserver serves from
certManager.caBundle "" Base64 PEM; required when certManager.enabled=false

Four behaviours are worth knowing before you set these:

helm show values ./helm prints the chart’s full value set.

Observability

sandbox-apiserver serves the generic apiserver’s own /metrics, /healthz, /readyz and /livez on its secure port, and --profiling (on by default) exposes /debug/pprof. The warm-pool driver’s controller-runtime metrics and health servers are disabled: the aggregated apiserver owns the serving port. sandbox-envd-proxy serves an unauthenticated GET /healthz for probes and no metrics endpoint.

sandbox-apiserver and sandbox-envd-proxy log to stderr through projecteru2/core/log; their controller-runtime and klog output (client-go, and in sandbox-apiserver the generic apiserver library) is routed into the same stream.

Variable Default Effect
OPERATOR_LOG_LEVEL info Log level of both binaries: debug, info, warn or error. An unknown level fails startup.
This site is open source. Improve this page.