cocoon sandbox

sandboxd HTTP API

All bodies are JSON. Three token kinds:

Endpoints below that say “node API token” accept root or tenant unless marked root-only. Errors are {"error": "message"} with the status codes listed per endpoint.

POST /v1/claim

Auth: Authorization: Bearer <api_token> (when configured).

{"template": "base:24.04", "net": "none", "size": "small",
 "ttl_seconds": 300,
 "volumes": [{"name": "imagenet"}, {"name": "weights", "mount": "/models"},
             {"name": "scratch-db", "mode": "rw"}],
 "claim_ref": "namespace/workload", "no_redirect": false,
 "require_promoted": false}

Success:

{"id": "sb_…", "token": "…", "deadline": "2026-07-06T00:05:00Z",
 "owner_addr": "10.0.0.5:7777", "template_digest": "sha256:…",
 "volumes": [{"name": "imagenet", "mount": "/volumes/imagenet"},
             {"name": "weights", "mount": "/models"},
             {"name": "scratch-db", "mount": "/volumes/scratch-db", "mode": "rw"}]}

A claim cloned from a promoted template carries template_digest, the exact export generation fetched for that clone. It is absent for configured pools, cold image boots, forks, checkpoints, and templates published by an older sandboxd until they are re-promoted.

A claim branched from a checkpoint (fork children included) additionally carries "from_checkpoint": "ck_…" — the lineage edge for reconstructing the checkpoint tree.

volumes reports the names, effective mounts, and (rw only) mode applied and persisted at finalization — mode is omitted from the echo for ro entries, matching the request shape. sandboxd attaches each disk, waits for the device settle — a serial match under /sys/block, then the /dev/<name> node itself, since the kernel publishes sysfs before devtmpfs creates it — up to 2 seconds total, and mounts the filesystem — read-only, unless the entry requested and was granted rw — before returning. A custom mount may shadow an existing populated guest directory for the claim’s life.

Attach-only volumes

"volumes_attach_only": true stops after the attach. The device appears in the guest, nothing is mounted, and the echoed entries carry no mount key:

{"id": "sb_…", "token": "…",
 "volumes": [{"name": "imagenet"}, {"name": "scratch-db", "mode": "rw"}]}

Find each device by its virtio serial, which is the catalog name — poll this, not a one-shot lookup: the claim can return before the guest enumerates the device (typically within ~100ms, never guaranteed), and the /dev/<blk> node can lag the serial match the same way it does for an eager mount’s device settle. Confirm both before mounting:

device=
tries=0
while [ "$tries" -lt 200 ]; do
  for serial in /sys/block/*/serial /sys/block/*/device/serial; do
    [ -r "$serial" ] || continue
    [ "$(cat "$serial")" = scratch-db ] || continue
    block=${serial#/sys/block/}
    candidate="/dev/${block%%/*}"
    [ -b "$candidate" ] && { device="$candidate"; break 2; }
  done
  tries=$((tries + 1))
  sleep 0.01
done
[ -n "$device" ] || { echo "scratch-db device not ready" >&2; exit 1; }
printf '%s\n' "$device"

Then mount it however the workload needs. A ro entry is attached --readonly and stays read-only at the guest block layer no matter who mounts it, so the guarantee does not depend on your mount flags.

The whole consistency contract moves to the caller with the mount:

What still protects other claims is unchanged: admission excludes an attach-only rw claim against every other claim of the name (and readers against a writer), an attach-only ro claim of a marker-bearing image is refused with 409, and a claim with volumes still refuses checkpoint, fork and hibernate. An attach-only claim costs one disk attach per volume — no device settle, no mount round-trip — and release costs nothing at all: removing the VM closes the devices.

Redirects (mutually exclusive with the fields above) name peers to retry at — sent on a warm miss with warm peers, when the node lacks a golden for the key but gossip names a template owner, and when the node is at max_claims but a peer reports warm capacity:

{"redirect": ["10.0.0.6:7777", "10.0.0.7:7777"],
 "require_promoted": true}

Retry the same body (+no_redirect: true) at each candidate until one answers. Preserve require_promoted: true when the redirect carries it; ordinary redirects omit the field.

A volume claim may consume an ordinary warm VM; candidate ranking, the promoted-template intersection, and the no_redirect/require_promoted retry follow the fleet-wide rule in cluster. Redirect responses never carry volumes.

A tenant token claims the same way; the sandbox is stamped with the tenant name (attributed in the usage journal and counted against the tenant’s max_claims). A catalog access list may restrict an entry to named tenants; an unknown and a forbidden volume return the same error text.

Errors: 400 unknown template axis, invalid/duplicate volumes, volumes_attach_only with no volumes or with an entry carrying a mount, mode: "rw" against a non-writable entry, or a volume that is unknown or forbidden (the latter two are deliberately indistinguishable), or bad body; 401 bad api token; 409 egress requested on a node without an egress attachment, a writable name already claimed in a conflicting mode (volume busy — a live writer excludes every other claim for that name, live readers exclude a writer), or a ro claim against a writable image left dirty by an unclean rw release (needs recovery — one rw claim must replay and cleanly release before ro claims resume; see deploy); 429 node at max_claims, the calling tenant at its own max_claims, or the node draining; 500 provisioning failed.

GET /v1/volumes

Auth: node API token (root or tenant). Lists the fleet catalog entries the caller may use, without host paths or holder addresses:

{"volumes": [{"name": "imagenet", "default_mount": "/volumes/imagenet",
              "size_bytes": 214748364800, "available": true, "nodes": 3}]}

Root sees the gossiped union, including peer-only entries (available: false); a tenant sees only entries this node declares locally and whose access list permits it. nodes counts members advertising the name; size_bytes and available are a best-effort stat of the answering node’s image. Membership is eventually consistent by one gossip tick. writable is the entry’s catalog configuration, fleet-uniform like the access list; the field is emitted (as true) only for a writable entry and omitted otherwise, so a read-only entry’s response is byte-identical to v1.

POST /v1/sandboxes/{id}/release

Auth: the sandbox’s own token, or the root token for operator cleanup by id. Destroys the VM. 204 on success, 404 for an unknown id or wrong token. Releasing an already-gone sandbox is 404 — the SDK treats it as success.

POST /v1/sandboxes/{id}/hibernate

Auth: the sandbox’s own token, or the root token by id (operator, like release). Atomically snapshots the VM and stops it, freeing its memory; the next agent access restores it transparently (sessions, processes, and memory state intact — cocoon’s hibernate keeps the snapshot point and the stop coincident). Idempotent on an already-hibernated sandbox. The TTL keeps running: a hibernated sandbox is still reaped (VM and snapshot) at its deadline. When to hibernate is the caller’s policy — the node only provides the transition. 204 on success, 404 unknown id or wrong token, 409 on the egress lane or when volumes are attached (neither kind of sandbox hibernates; see egress).

POST /v1/sandboxes/{id}/wake

Auth: the sandbox’s own token, or the root token by id (operator). Restores a hibernated (or archived) sandbox and leaves it running — waking is otherwise only a side effect of the next agent access, so this is the explicit form for warming a sandbox ahead of use. Idempotent on one already running. 204 on success, 404 unknown id or wrong token.

POST /v1/sandboxes/{id}/fork

Auth: the node api_token (Bearer) — forking creates node resources, like a claim. The sandbox’s own token rides in the body as the ownership proof:

{"token": "…", "count": 2, "ttl_seconds": 300}

Clones the sandbox into count fresh claims (1 up to the node’s max_fork_count, default 16). Memory, disk, and guest state (sessions, processes, tmpfs) duplicate at the fork point; cocoon’s clone reseed gives every child a distinct machine identity. Children get their own lease — ttl_seconds (0 = server default), never the parent’s remainder. A running parent is snapshotted in a brief pause window; a hibernated parent forks from its existing memory image without waking. All-or-nothing: on error no child survived. 200 with one claim per child:

{"children": [{"id": "sb_…", "token": "…", "deadline": "…", "owner_addr": "…"}]}

Children inherit the parent’s tenant and count against its max_claims, whoever calls. 400 invalid count or body, 401 bad api token, 404 unknown id or wrong sandbox token, 409 egress-lane or volume parent (neither forks, checkpoints, or promotes; see egress), 429 node or the parent’s tenant at max_claims, or the node draining.

POST /v1/sandboxes/{id}/promote

Auth: like fork — node api_token in the header, the sandbox’s own token in the body:

{"token": "…", "template": "myproj:v1"}

Publishes the sandbox’s current state as a node-local template under (template, this sandbox’s net, its size): later claims for that key clone from it, provision-on-demand — no warm pool unless the node config adds one. Re-promoting to the same name replaces the template. A hibernated sandbox is promoted from its memory image without waking. 200 returns the template’s full key and immutable content identity. On the default local-disk backend a template is node-local, so a cluster client claims from and deletes on this node (name-based calls route via gossip); a shared checkpoint store makes every node resolve it. Under exactly this key:

{"key": {"template": "myproj:v1", "net": "none", "size": "small"},
 "content_digest": "sha256:…"}

content_digest is SHA-256 over a versioned canonical stream of the published export’s regular files: slash-relative path, byte length, and bytes, ordered lexically. Directory entries, modes, mtimes, and the template’s ownership/ creation metadata do not affect it. The digest is computed once while promoting, stored in meta.json, and therefore has identical semantics on the directory and S3 backends. Re-promoting unchanged export bytes keeps the digest; changing any exported path or bytes changes it.

400 invalid name, 401 bad api token, 409 when the name collides with a configured pool, the template is owned by another tenant, or the sandbox is on the egress lane or has volumes attached (see egress), 404 unknown id or wrong sandbox token.

DELETE /v1/templates?template=…&net=…&size=…

Auth: node API token. Removes a promoted template (the query parameters default like a claim’s: net=none, size=small). A tenant may delete only templates it promoted — anything else is 404, root deletes anything. 204 on success, 404 unknown template, 409 when the key belongs to a configured pool (those goldens are owned by the node config). On a cluster, a node that does not hold the template but sees an owner in gossip answers 200 {"redirect": [addrs]} — the claim redirect shape — and the SDK retries the delete at the owner. The retry carries no_redirect=1, mirroring the claim protocol: a node answering a no_redirect delete speaks only for itself.

PUT /v1/pools

Auth: root only (tenant tokens get 403). Replaces the node’s desired warm targets online — no restart, live claims untouched:

{"pools": [{"template": "base:24.04", "net": "none", "size": "small",
            "warm": 4, "warm_max": 16, "idle_hibernate_seconds": 0}]}

Pools omitted from the list are drained: their unclaimed warm VMs are destroyed and the pool entry retires. net/size default like a claim’s. Answers the fresh GET /v1/info payload. 400 bad key, negative warm/idle, warm_max below warm, or duplicate pool; 401 bad api token; 409 egress pool on a node without an egress attachment.

POST /v1/drain

Auth: root only (tenant tokens get 403). Cordons the node for maintenance: claim/fork/branch answer 429 node draining (on a cluster a non-volume claim tries a warm-peer redirect first, and gossip stops naming this node within a tick as its warm counts hit zero), unclaimed warm VMs are destroyed, and live claims keep serving until release or TTL. Pool ownership is untouched — no pools.json write, no config change. Answers the fresh GET /v1/info payload; poll claimed to zero to know the node is empty. Deliberately not persisted: a restarted node serves again.

DELETE /v1/drain

Auth: root only (tenant tokens get 403). Lifts the drain and kicks an immediate refill. Answers the fresh info payload.

POST /v1/sandboxes/{id}/preview

Auth: like fork — node api_token in the header, the sandbox’s own token in the body. Mints a signed URL serving a guest HTTP port from a browser: body {"token": "...", "port": 8080, "ttl_seconds": 0} → {"url": "http://<preview_advertise>/p/<token>/"}. The URL’s life is clamped to the claim’s remaining lease. 501 when the node has no preview_listen. The signed token embeds the sandbox id, port, and owner advertise_addr, so any node’s preview listener can serve it (forwarding to the owner’s main listener) and a released sandbox’s URL simply stops resolving — no revocation list. See deploy.

POST /v1/sandboxes/{id}/checkpoint

Auth: node API token; body {"token": "<sandbox token>", "name": "..."} (name optional). Captures the sandbox’s full state without stopping it and answers 200 {"checkpoint": {id, name, sandbox_id, key, tenant?, created_at}} — tenant records the calling tenant, absent for root. 400 bad body or name, 401 bad api token, 404 unknown id or wrong sandbox token, 409 egress-lane sandbox or one with volumes attached (see egress).

POST /v1/checkpoints/{id}/claim

Auth: node API token; body {"ttl_seconds": 0, "no_redirect": false}. Claims a fresh sandbox branched from the checkpoint (a normal claim response, attributed to the caller); the checkpoint’s recorded key applies — the unguessable id is the capability to branch.

Checkpoints are node-local (unless the store is shared — see Configuration), so a miss here runs a tier order:

  1. This node checks its own store first.
  2. On a miss, it probes up to 3 peers directly — a parallel HEAD to each (authenticated on an encrypted mesh, see below) — and answers exactly like a warm-miss POST /v1/claim: 200 {"redirect": ["10.0.0.6:7777", "10.0.0.7:7777"]}, retry the same body (+no_redirect: true) at each candidate until one answers. The probe and the follow-up claim are not atomic: a peer can answer the probe, then lose the record — a delete’s broadcast lands, or its own TTL sweep runs (below) — before the retry reaches it, so a redirect can go stale between the two calls.
  3. If nothing answers the probe (or no_redirect is set), and the node has checkpoint_peer_heal enabled, it pulls the record from a probed peer itself, validates it, publishes it locally, and serves the claim from there — paid once per node. See the full placement lifecycle for how the three tiers fit together.

404 for an unknown checkpoint (locally, and after redirect and heal both miss), 409 for an egress-lane checkpoint (see egress), 429 node or calling tenant at max_claims or the node draining, 503 when the node’s concurrent-heal cap is already full — retryable, and the response carries a Retry-After hint.

GET/HEAD /v1/checkpoints/{id}/blob

The peer-transfer route behind the probe and heal above — internal, not part of the public API; an SDK caller has no reason to call it directly.

GET /v1/checkpoints

Auth: node API token. Lists this node’s checkpoints, newest first. A tenant sees only its own records; root sees everything.

DELETE /v1/checkpoints/{id}

Auth: node API token. A tenant may delete only its own records — anything else is 404, never a hint the id exists; root deletes anything. 204 on success, 404 unknown.

Delete removes the local record, then best-effort broadcasts to peers so a healed replica does not outlive it — eventual cleanup, not a fleet-wide revocation. A peer offline during the broadcast keeps its copy until the checkpoint TTL ages it out, so an id-holder can still branch it for that window; placement lifecycle has the bound and why heal requires a nonzero, fleet-matching TTL. A shared checkpoint store skips the broadcast: every node already resolves every record directly, so there is no replica to chase. ?no_forward=1 marks a delete already arriving from another node’s own broadcast, so it is not itself re-broadcast (loop prevention); it is an internal parameter, not one an SDK caller should set.

GET /v1/sandboxes

Auth: node API token. Root sees every live claim; a tenant sees only its own. The index is {"sandboxes": [{id, key, deadline, hibernated, archived?, from_checkpoint?, claim_ref?, volumes?: [{name, mount, mode?}]}]} — mode is omitted for ro, matching the claim echo; never sandbox tokens, volume host paths, or catalog access lists.

GET /v1/sandboxes/{id}

Auth: root only. One live claim in the index-row shape above, so a reconcile loop can read a single sandbox without scanning the whole node listing. 404 unknown id.

GET /v1/sandboxes/{id}/stats

Auth: root only. One sandbox’s resource usage — the per-sandbox counterpart to the node-scoped /metrics:

{"id": "sb_…", "cpu_count": 2, "mem_total_bytes": 1073741824,
 "mem_used_bytes": 187654144, "mem_used_measured": true,
 "hibernated": false, "measured_at": "…"}

cpu_count/mem_total_bytes come from the size tier. mem_used_bytes is the host VMM process’s resident set — the only usage signal available without a guest agent; mem_used_measured is false when there is no VMM process to read (hibernated, or the PID is not yet known), so a zero is never mistaken for idle. 404 unknown id.

GET /metrics

Auth: root only (tenant tokens get 403). Prometheus text format, hand-rendered: pool warm/target gauges, claimed/hibernated/archived/draining gauges, a per-tenant live-claim gauge (sandboxd_tenant_claims{tenant="…"}, configured tenants only), sandboxd_config_digest_mismatch on a mesh, claims by tier (warm/clone/cold), wake/hibernate/fork/checkpoint/promote/release/reap counters plus archive/unarchive/archive-delete counters, and claim/wake *_seconds_total for average latency. /metrics is a derived ops view; the billing source of truth is the usage journal below.

Usage journal (usage.jsonl)

Always on: every lifecycle transition appends one JSONL event to <data_dir>/usage.jsonl — {"t": <RFC3339>, "ev": "claim|hibernate|wake|fork|checkpoint|promote|release|reap|archive|unarchive|archive_delete|egress", "id": "sb_…", "vm": "sbx-…"} plus key and tenant (the pool key’s stable hash and the owning tenant, claim events), children (fork) and ref (the promoted template / checkpoint id, or the egress host). A volume claim also carries volumes, the applied catalog names, and — omitted when empty — volumes_rw, the subset of those names claimed rw, so billing can discriminate write access (mounts and host paths are not billing dimensions). The file rotates at 64 MiB keeping one .1 backup, so a tailing collector never loses a window silently. Folding rules: billable compute seconds per sandbox = Σ(claim→release/reap) − Σ(hibernate→wake); hibernated storage seconds = Σ(hibernate→wake) minus the archived span; archived storage seconds = Σ(archive→unarchive/archive_delete), a cheaper store tier than a hibernated VM’s RAM. An interval left open by a crash clamps to the claim’s deadline, and the next reconcile emits reap for claims it drops. The vm name joins cocoon’s machine-level metering ledger for audit cross-checks.

GET /v1/sandboxes/{id}/agent

Auth: the sandbox’s own token. Requires Upgrade: silkd + Connection: Upgrade; answers 101 Switching Protocols and from then on the connection is a byte-for-byte relay to the guest’s silkd (one silkd RPC per connection — see silkd). 426 without the upgrade header, 404 unknown sandbox, 502 guest unreachable.

GET /v1/sandboxes/{id}/owner

Auth: the sandbox’s own token. Answers {"owner_addr": "host:port"} when this node owns the sandbox, 404 otherwise. Used by the SDK’s Lookup scatter.

GET /v1/info

Auth: root only (tenant tokens get 403). Node pools, claim count, and mesh peers:

{"pools": [{"key": {"template": "base:24.04", "net": "none", "size": "small"},
            "warm": 4, "refilling": 0, "target": 4, "golden": true}],
 "claimed": 2,
 "hibernated": 1,
 "archived": 0,
 "peers": ["10.0.0.6:7777"]}

hibernated counts claims whose VM is currently hibernated, archived those checkpointed to the store with the local VM dropped (see archive tiers); both are included in claimed. A node cordoned via POST /v1/drain additionally reports "draining": true.

golden reports whether the pool’s snapshot exists (refill can clone); warm at target with golden: true means warm claims are served in sub-millisecond time.

GET /v1/peers

Auth: node API token (root or tenant). Answers {"peers": [addr, …]} — the cluster’s other node addresses, for the SDK’s redirect follow and Lookup scatter. Cluster topology, not operator state, so a tenant token may read it (unlike GET /v1/info).

GET /healthz

Unauthenticated liveness probe; answers ok.

Operational notes

This site is open source. Improve this page.