cocoon sandbox

Deploying sandboxd

sandboxd is a single static binary per node. It drives VM lifecycle through the cocoon CLI and needs a template image with silkd baked in.

Prerequisites

Prebuilt static linux/amd64 and linux/arm64 binaries (sandboxd, sandboxd.dbg, sandbox-mcp, silkd, with checksums.txt) ship with every GitHub release; the boot artifact and the base/rt/python images are multi-arch manifests (browser and android remain amd64-only). Build from source with make sandboxd (produces dist/sandboxd); either way sandboxd -version reports what you are running.

Upgrading

This release does not convert VM or snapshot state from older releases. Drain old claims and use fresh data_dir and checkpoint_dir locations when upgrading; older checkpoints and promoted templates must not be reused.

The scalar egress-attachment keys are retired: rename "bridge": "br0" to "bridges": ["br0"] and "network": "cni" to "networks": ["cni"] before starting the new binary — config loading rejects the old spellings loudly rather than silently dropping the egress lane.

Dataset volumes require a lockstep rollout. Upgrade every sandboxd node and cocoon to the required version before enabling the catalog or shipping an SDK that requests volumes. Mixed-version serving is unsupported. Once a volume claim has finalized, do not roll a node back to an older sandboxd until all volume claims are gone; the older daemon cannot preserve the volume claims’ admission holds and marker bookkeeping recorded in claims.json.

Configuration

sandboxd reads one JSON file (-config, default /etc/sandboxd/config.json):

{
  "listen": ":7777",
  "data_dir": "/var/lib/sandboxd",
  "cocoon_bin": "cocoon",
  "restore_mode": "mmap",
  "no_direct_io": true,
  "advertise_addr": "10.0.0.5:7777",
  "bridges": ["br0"],
  "volumes": [
    {"name": "imagenet", "path": "/srv/datasets/imagenet.img"},
    {"name": "weights-llama", "path": "/srv/datasets/llama.img", "directio": "on"},
    {"name": "acme-corpus", "path": "/srv/datasets/acme.img", "tenants": ["acme"]}
  ],
  "api_token": "…",
  "tenants": [{"name": "acme", "token": "…"}],
  "mesh": {
    "node_id": "node-a",
    "bind": "10.0.0.5:7946",
    "join": ["10.0.0.6:7946"],
    "cluster_key": "base64…"
  },
  "pools": [
    {"template": "base:24.04", "net": "none",   "size": "small", "warm": 4},
    {"template": "base:24.04", "net": "egress", "size": "small", "warm": 2}
  ]
}
field default meaning
listen :7777 control- and data-plane HTTP listener
data_dir /var/lib/sandboxd golden snapshot exports, the claims journal, the usage/audit journals (usage.jsonl, audit.jsonl + .1 backups), and checkpoints/ by default
cocoon_bin cocoon cocoon CLI binary
restore_mode unset clone and wake-restore memory mode: copy, ondemand, or mmap; use mmap for dense pools
no_direct_io false use buffered writable disks for Cloud Hypervisor cold boots and clones; recommended for dense ephemeral pools to avoid direct-I/O CoW journal contention
advertise_addr = listen the host:port clients reach this node at; returned as a claim’s owner address and gossiped to peers. Must be routable when listen is a wildcard
bridges / networks unset egress-lane attachment: a list of host bridge devices, or a list of CNI conflist names. Mutually exclusive; with neither set the node serves only the no-network lane. A Linux bridge holds at most 1024 ports (kernel BR_MAX_PORTS), so an N-entry list raises the node’s egress ceiling to N×1024 — VMs spread over the list by a stable hash of the VM name, so size it with headroom (the spread is statistical, not exact). bridges keeps the raw TAP-on-bridge attachment (taps in the root netns, no per-VM network namespace or CNI plugin execution); networks runs the CNI chain per VM. Guarded egress needs bridges and rejects a CNI network at load
volumes unset node-local catalog of operator-managed dataset images: [ {"name":"imagenet","path":"/srv/datasets/imagenet.img","directio":"off","tenants":["acme"]}, {"name":"scratch-db","path":"/srv/datasets/scratch.img","writable":true} ]. Names match ^[a-z][a-z0-9_-]{0,19}$ and cannot start with cocoon-; paths are absolute; directio is on, off, or auto and defaults to off for both read-only and writable entries. tenants is an optional access list: empty means every authenticated scope, while every listed name must exist in the node’s tenants config; root always has access. writable (default false) lets a claim request mode: "rw" on that entry — see Dataset volumes. The catalog is intentionally not part of the cluster digest
secrets unset node-side credentials the egress proxy injects by name: [{"name": "gh", "header": "Authorization", "value_env": "GH_TOKEN"}]. A pool or tenant rule references the name; the value comes from the environment, never this file. See egress
egress_internal_allow unset CIDR prefixes re-admitted through the egress proxy’s SSRF guard, node-wide (every pool and tenant). Prefixes, not a permit-private switch — the guest bridges are themselves ULA/RFC1918. See egress
egress_ca unset HTTPS-interception PKI: root_cert (the cluster root baked into intercepted guests; may bundle old+new roots during rotation) plus this node’s intermediate_cert/intermediate_key from sandboxd ca issue-intermediate. Required when any pool rule sets intercept
api_token unset the operator (root) credential: when set, guards the node-level endpoints (Bearer) with full access, including release-by-id cleanup. Per-sandbox tokens guard ordinary sandbox-scoped calls
tenants unset multi-tenant tokens next to api_token: [{"name": "acme", "token": "…", "max_claims": 50}]. A tenant token reaches the resource-creating verbs (claim, fork, promote, checkpoint, preview), catalog discovery, and its own sandbox/checkpoint listings; everything it creates is stamped with the tenant name. Root-only surfaces (per-id sandbox reads, GET /v1/info, PUT /v1/pools, POST/DELETE /v1/drain, /metrics) answer it 403. max_claims (0 = unlimited) caps that tenant’s live claims next to the node-wide cap. Requires api_token set. Names and tokens must be unique, tokens distinct from api_token. On a cluster all nodes must carry the same tenants set (the SDK replays whichever token authorized a redirect), and per-node caps mean a tenant’s effective cluster limit is max_claims × nodes. Empty = exactly the single-token behavior
max_fork_count 16 children a single fork may create; each is a full-RAM VM, so this bounds one request’s memory blast radius to the node’s capacity
refill_concurrency 0 (auto) concurrent VM provisioning budget, shared by warm-pool refills, fork clones, and the reap/hibernate/reconcile engine batches. 0 sizes it from the node: NumCPU*2/3 clamped to [4, 256] — a 384-core node gets 256; small nodes keep a floor of 4
preview_listen (off) address for a preview HTTP server that serves guest ports under signed URLs; needs preview_secret
preview_secret — cluster-shared HMAC secret signing preview tokens (all nodes share one)
preview_advertise = preview_listen the browser-facing preview base URL; nodes behind one TLS proxy may share it, while signed tokens route internally through each owner’s advertise_addr
checkpoint_dir <data_dir>/checkpoints where checkpoints and promoted templates live. Point it at a shared FUSE mount (JuiceFS over object storage, NFS) and every node sharing the mount can branch every checkpoint — records are generation-addressed with meta.json as the atomic commit pointer, so no cross-node locking is required of the filesystem. A re-publish retains its superseded export generation for at least ~1h and until a following hourly sweep so an in-flight clone that resolved the old metadata can finish; budget the current generation plus every generation retained across that grace-and-sweep window. An explicit delete can make a concurrent clone fail visibly. One contract on any shared root (mount or bucket): a template key has a single writer — promotes go to the sandbox’s owner node, and operators must not race promotes of one name from different nodes (checkpoint ids are node-generated and never collide)
checkpoint_store dir checkpoint AND promoted-template backend (both live in one store root, id-namespaced ck_/tp_): {"kind": "s3", "s3": {"bucket": "…", "prefix": "ck/", "endpoint": "…", "region": "…", "force_path_style": true}} stores checkpoints in object storage (any node claims any checkpoint, no shared mount needed). Credentials come from the standard AWS chain (env/IAM role), never this file. Re-publish retains prior export generations until Delete so an in-flight fetch that selected old metadata can finish; budget storage for those generations. An explicit S3 Delete can still make a concurrent fetch that has not finished materializing fail visibly. A crash between upload and the meta.json commit marker leaves orphan objects invisible to listings — add an S3 lifecycle rule to reclaim them. Absent = the dir backend at checkpoint_dir
checkpoint_ttl_hours 0 (keep forever) ages out checkpoints older than this; the sweep runs hourly and at startup. Explicit deletes never wait for it. Must be nonzero and match fleet-wide when checkpoint_peer_heal is on — it is the expiry eligibility point for a healed replica a delete broadcast missed, after which its next successful hourly sweep removes it; persistent sweep failure extends retention until one succeeds, so it is not a hard ceiling
checkpoint_peer_heal false on a cluster, lets a node pull a checkpoint it lacks from a peer — found via a live probe, not gossip — rather than failing the branch; see placement lifecycle. Three requirements, all enforced at config load: a nonempty api_token (the blob transfer between peers authenticates with it; without one the raw record stream would be open), mesh.cluster_key set (the pull presents the fleet api_token to an address learned from the peer probe, so the gossip layer carrying that address must itself be authenticated), and checkpoint_ttl_hours nonzero (a replica a delete broadcast missed becomes eligible for expiry after it, and its next successful hourly sweep removes it — so it is the finite eligibility point, not an exact ceiling). A shared checkpoint store (checkpoint_store kind s3) ignores this setting — every node already resolves every checkpoint directly, so there is nothing to heal
warm_max (pool entry) 0 (static) turns on the demand-adaptive watermark for that pool: the warm target rises from warm toward warm_max while claims arrive faster than the measured provision lead covers, and decays back over ~a minute of silence
max_claims 0 (unlimited) node-wide cap on live claims; claim/fork/branch requests beyond it answer 429 with the pool state unharmed (on a cluster, normal warm-candidate placement applies, with volume claims limited to candidates holding every requested volume)
audit_log false append every relayed request frame’s op + addressing fields (never payloads) to <data_dir>/audit.jsonl, size-rotated with one .1 backup. Records are {t, id, op} plus whichever addressing fields the op carries (argv, path, dest, from, to, url, session, port), plus decision and secret (the ref name, never its value) on egress records; preview accesses record as op preview, one per request. A request frame whose first line exceeds 4 KiB is skipped, never truncated
idle_hibernate_seconds 0 (off) node-wide idle policy for unpooled claims (template/checkpoint claims): a none-lane claim with no data-plane connection for this long is hibernated; the next call wakes it transparently. Per-pool idle_hibernate_seconds does the same for that pool’s claims; pooled keys ignore the node-wide value, and egress pools reject it because they cannot resume safely. Opt in deliberately: a wake costs latency and the snapshot, so callers with their own idle logic must not pay twice
archive_after_seconds 0 (off) tier below hibernation: a hibernated claim idle this long is checkpointed to the store and its local VM dropped, freeing the node entirely; the next call restores it transparently (a checkpoint restore’s latency). Requires idle_hibernate_seconds > 0 and must exceed it. Node-wide for unpooled keys; per-pool overrides for that pool
archive_delete_after_seconds 0 (keep) purge an archived claim’s store checkpoint this long after it was archived, reclaiming storage; the claim is then gone for good. Same node-wide/per-pool split
mesh unset join a cluster (Clusters); unset = single node
pools[] — warm pools, keyed by (template, net, size). warm defaults to 4; net is none or egress; size is a tier, below. Retune online without a restart via PUT /v1/pools — omitted pools drain. This is the first-boot seed: once a node takes a PUT /v1/pools, the applied set persists to <data_dir>/pools.json and overrides this section on every later boot (a startup log notes it); delete pools.json to return to config-owned pools. Egress stays config-owned either way. See state ownership

Size tiers (free-form CPU/memory is deliberately not accepted — it would fragment the warm pools):

size CPU memory
small 1 512M
medium 2 1G
large 4 4G
xlarge 4 8G

Dataset volumes

Each catalog path must name a disk image containing a mountable whole-device filesystem. A read-only entry’s image must stay immutable: do not replace, truncate, or delete it while attached; publish a new catalog name or path instead. A missing path produces a startup warning and fails only claims that request it, allowing images to be distributed after sandboxd starts.

For example, build a whole-device ext4 image directly from a prepared tree, then make the published file host-read-only:

truncate -s 200G /srv/datasets/imagenet.img
mkfs.ext4 -F -d /srv/datasets/imagenet-root /srv/datasets/imagenet.img
chmod 0444 /srv/datasets/imagenet.img

Use the same dataset identity and access list for a volume name on every node, then distribute its immutable image to each node that should advertise it. sandboxd gossips only catalog names; it never copies content or gossips paths or access lists.

A claim requests up to eight unique names and may set an absolute, clean custom mount for each; the default is /volumes/<name>. Mounts must stay outside the guest OS directories and cannot duplicate or nest within one claim. Volume mounts may shadow an existing populated guest directory for that claim’s life.

A volume claim may consume an ordinary warm Cloud Hypervisor VM. sandboxd attaches after the warm pop or provision, waits for the device settle — a serial match under /sys/block, then the /dev/<name> node itself, since the kernel publishes sysfs before devtmpfs creates it — up to 2 seconds, then mounts the device before finalizing the claim: mode: "ro" (the default) attaches and mounts read-only; mode: "rw" requires the catalog entry’s writable: true and attaches and mounts read-write. Setup failure destroys the VM without quiescing — the claim was never handed out, so no workload write happened — and a popped warm VM is refilled normally.

A multi-volume claim brings every volume up concurrently: each volume’s own marker→attach→mount order is preserved, but cocoon serializes the hypervisor attach per VM, so what actually overlaps across volumes is the CLI spawns, device settles, and guest mounts. A partial failure still fails the whole claim and destroys the VM without quiescing, so any rw volume already attached keeps its dirty marker — including one the caller never got a handle to; recovery is the normal rw cycle (see Writable dataset volumes).

Warm candidates retain their normal ranking, but a volume claim’s candidate set, promoted-template intersection, and redirect follow the fleet-wide rule in cluster.

An empty catalog tenants list allows every authenticated scope; a nonempty list limits the image to those tenants, while root always bypasses it. Removing a tenant therefore requires removing every catalog reference in the same edit. GET /v1/volumes reports the caller-visible fleet union and holder count, plus the answering node’s current local availability and whether the entry is writable, without exposing host paths or node addresses. Applied names, effective mounts, and (for rw entries) mode are persisted with the claim. Such a claim cannot hibernate, fork, checkpoint, or promote; the idle hibernate sweep leaves it running. Release removes the VM but never deletes the operator-owned backing image — an rw release quiesces (unmounts) first, below. With directio=off, readers share the host page cache; use directio=on when cache interference matters. A dataset mounted into an egress-lane sandbox can be uploaded wherever that tenant’s egress policy permits, so treat the ACL and egress policy as one access decision. Image replication, detach, and refcounting remain out of scope.

Writable dataset volumes

A catalog entry with writable: true may be claimed with mode: "rw" (omitted, or "ro", always mounts read-only, even against a writable entry). directio still defaults to off for a writable entry — the durability nuance is that the host page cache sits between the guest’s flush and the device, so directio=on is the knob when that gap matters, not just cache interference.

Single holder, by operator contract. A writable name must be attached from exactly one node — the same weight as “replacing an attached image is operator error.” Nodes advertise only the catalog paths they actually hold, so every claim for that name, ro or rw, already funnels to that one node; running the same writable name from two nodes at once is dataset divergence that nothing in the protocol detects for you.

The <path>.dirty marker. Before the first rw attach of an image, sandboxd durably creates a <path>.dirty sidecar beside it; a clean rw release removes it. It is a write-ahead record, not a lock: it survives a killed sandboxd or a dead VMM, so anything short of a clean unmount leaves it in place. It is operator-visible — ls next to the image shows whether it’s mid-write — and travels with the image on shared storage. A leftover marker is not corruption (a journaling filesystem already makes a hard VM kill crash-consistent); it means the image needs one rw claim, which replays the journal as a side effect of mounting, followed by a clean release, before it can serve ro again. A ro claim against a dirty image is refused rather than auto-healed: replaying the journal takes the write lock, which conflicts with concurrent readers.

Concurrency, for one image name:

Attach-only claims opt out of the marker, not out of admission. A claim sent with volumes_attach_only gets the device attached and nothing else, so sandboxd neither writes nor clears <path>.dirty for it — it cannot verify a mount it did not perform. Everything in this section still applies to the default, mounting claims, and the exclusion rules above apply to attach-only claims exactly the same way, which is what keeps other tenants safe. The operator-visible difference: an image whose attach-only writer released without unmounting cleanly carries no marker, so the next ro claim is admitted and fails at mount time (500) instead of being refused early (409). Whoever hands out attach-only rw access owns that trade; see sandboxd-api.

A fuller config

The block above is the minimum. A production node with tenants, guarded egress + HTTPS interception, previews, an object-store checkpoint backend, the idle→hibernate→archive tiers, and a mesh looks like this — every field here validates on load:

{
  "listen": ":7777",
  "data_dir": "/var/lib/sandboxd",
  "advertise_addr": "10.0.0.5:7777",
  "bridges": ["br0"],
  "restore_mode": "mmap",
  "no_direct_io": true,

  "api_token": "op-root-token",
  "tenants": [
    {"name": "acme", "token": "acme-token", "max_claims": 50}
  ],

  "secrets": [
    {"name": "gh", "header": "Authorization", "value_env": "GH_TOKEN"}
  ],
  "egress_ca": {
    "root_cert": "/etc/sandboxd/egress-ca/root.crt",
    "intermediate_cert": "/etc/sandboxd/egress-ca/node-a.crt",
    "intermediate_key": "/etc/sandboxd/egress-ca/node-a.key"
  },

  "preview_listen": ":8443",
  "preview_secret": "cluster-shared-hmac-secret",
  "preview_advertise": "https://preview.example.com",

  "checkpoint_store": {
    "kind": "s3",
    "s3": {"bucket": "sandbox-ckpt", "prefix": "ck/", "region": "us-east-1"}
  },
  "checkpoint_ttl_hours": 168,

  "max_claims": 200,
  "audit_log": true,
  "idle_hibernate_seconds": 300,
  "archive_after_seconds": 3600,
  "archive_delete_after_seconds": 604800,

  "mesh": {
    "node_id": "node-a",
    "bind": "10.0.0.5:7946",
    "join": ["10.0.0.6:7946"],
    "cluster_key": "MDEyMzQ1Njc4OWFiY2RlZg=="
  },

  "pools": [
    {"template": "rt:24.04", "net": "none", "size": "small", "warm": 4, "warm_max": 12},
    {"template": "rt:24.04", "net": "egress", "size": "medium", "warm": 2,
     "egress": {"allow": [
       {"host": "api.github.com", "methods": ["GET", "POST"], "secret": "gh", "intercept": true},
       {"host": "*.googleapis.com"}
     ]}}
  ]
}

Secret values come from the environment named by value_env (here GH_TOKEN), never the file. The egress_ca files are provisioned with sandboxd ca — see Guarded egress. On a cluster, api_token, tenants, preview_secret, and cluster_key must match on every node.

High-density no-network pools

Both lanes use Cloud Hypervisor; net: "none" launches it with no NIC. For dense ephemeral pools, set restore_mode to mmap, enable no_direct_io, place Cocoon’s run_dir on a capacity-sized tmpfs, and set Cocoon’s cni_conf_dir empty on none-only nodes to avoid per-VM network namespaces. The last setting also removes the VMM’s network-namespace quarantine. On a 384-core host, start with refill_concurrency: 64; higher values need measurement. Pause cocoon daemon or lengthen its reconcile interval during a mass fill.

Host tuning for dense egress-lane nodes

The no-network lane is CPU-bound and needs none of this. The egress lane serializes on the kernel’s global rtnl lock — tap create, bridge attach, and the VMM’s own tap open all take it — so at hundreds of VMs host configuration dominates fill throughput:

Reserving CPU for the control plane

A saturated node can leave clone/wake execution and sandboxd itself nothing to run on — under full-node load CH clone p95 has measured in the tens of seconds. cocoon newer than v0.5.8 runs every VMM in its own cgroup v2 CPU scope (Guaranteed-at-N: a VM’s CPU count is also its hard host-CPU cap) and adds cgroup_cpus, a one-time cpuset fence keeping the whole VM population — including the virtio and io-wq worker threads vCPU affinity cannot reach — off reserved host cores. On dense nodes set the fence in cocoon’s config (e.g. cgroup_cpus: "0-379" on a 384-core host) so the reserved cores stay free for sandboxd, its cocoon invocations, and the OS; cocoon’s CPU-isolation docs cover validation semantics and the per-VM knobs.

Auth model

Three token kinds. The root api_token has full access — operators and single-tenant deployments need nothing else. Tenant tokens (the tenants list) create and manage their own resources: claims, forks, checkpoints, promoted templates, and preview URLs are stamped with the tenant name; sandbox and checkpoint listings filter to the caller’s tenant, and a tenant can delete only its own checkpoints and templates (root sees everything). Operator surfaces stay root-only — a tenant token there is authenticated but not authorized, so it answers 403 (a wrong token stays 401). Per-sandbox tokens are unchanged: whoever holds a sandbox’s token drives that sandbox. Fork children inherit the parent’s tenant and count against its max_claims; a tenant at its cap gets 429 exactly like a node at max_claims, and the usage journal’s claim events carry the tenant for per-tenant billing.

Running

sandboxd -config /etc/sandboxd/config.json

On start the node reconciles: persisted claims whose VMs still run are re-adopted, everything else sbx--prefixed is removed. A record still in cocoon’s creating state is reclaimed through vm reconcile-stale-create (cocoon ≥ v0.5.8), which refuses while a clone is in flight instead of deleting the VM out from under it; older cocoons fall back to forced removal. Then the refill loop builds one golden snapshot per pool (a one-time cold boot + snapshot export, tens of seconds) and keeps each pool topped up with claim-ready clones. GET /v1/info shows "golden": true and warm at target when the node is ready to serve warm claims.

A minimal systemd unit (shipped as packaging/sandboxd.service):

[Unit]
Description=sandboxd
Wants=network-online.target
After=network-online.target

[Service]
ExecStart=/usr/local/bin/sandboxd -config /etc/sandboxd/config.json
Restart=on-failure
Environment=SANDBOXD_LOG_LEVEL=info

[Install]
WantedBy=multi-user.target

Stopping sandboxd leaves VMs alive; the next start reconciles them. Claimed sandboxes are reaped when their TTL expires (default 5m, capped at 24h).

To empty a node for maintenance, cordon it first: POST /v1/drain (root) stops new claims and drains the warm pools; poll GET /v1/info until claimed reaches zero (or let the leases expire), then stop sandboxd. DELETE /v1/drain uncordons. The drain is not persisted — a restarted node serves again.

Verifying a node

curl -s -H "Authorization: Bearer $TOKEN" http://127.0.0.1:7777/v1/info | jq .

The repository’s scripts/sandboxd-e2e.sh runs the full loop on a real node (golden build → warm pool → claim tiers → the complete verb smoke → reap → restart reconcile); set BRIDGE=<dev> to include the egress lane.

To include the read-only volume proof, put a nonempty volume-e2e.txt in the filesystem image and run VOLUME_IMAGE=/srv/datasets/imagenet.img scripts/sandboxd-e2e.sh. The script verifies two concurrent mounts, read-only enforcement, warm-pool consumption, and an unchanged source checksum. On a node using prebuilt binaries, also set VOLUME_SMOKE_BIN beside SANDBOXD_BIN, DEMO_BIN, and SMOKE_BIN.

Add VOLUME_RW_IMAGE=/srv/datasets/scratch.img (a second, writable image) to also run the writable leg: a durable write across release, second-writer exclusion, and a clean read-only claim afterward.

Preview URLs

preview_listen starts a second HTTP server that serves a sandbox’s guest HTTP port under a signed, expiring shareable URL. The whole mechanism is in sandboxd:

This site is open source. Improve this page.