cocoon sandbox

sandboxd HTTP API

All bodies are JSON. Four token kinds:

Endpoints below that say “node API token” accept root or tenant unless marked root-only; the operator token only where the endpoint names it. Errors are {"error": "message"} with the status codes listed per endpoint. Request bodies are capped at 1 MiB; a larger one is rejected with 400 and an error beginning invalid request body, and so is a body with a field its endpoint does not declare. PUT /v1/sandboxes/{id}/instance-metadata is the exception: it takes one raw JSON object of at most 4 KiB.

Guest errors

Wake, the agent relay, the port relay and exec restore a sleeping sandbox before they reach its guest, so they share these answers:

Admission refusals

A claim, fork or checkpoint branch answers 429 when the node refuses it for capacity, with the pool state unharmed. The causes:

On a cluster, a claim without volumes, no_redirect or require_promoted that meets one of these redirects to a peer with warm capacity or, failing that, one that serves the pool; see POST /v1/claim. Wake and the guest-reaching verbs have their own 429 causes, listed with each.

POST /v1/claim

Auth: Authorization: Bearer <api_token> (when configured), a tenant token, or the operator token, which claims as root but never with volumes (403).

{"template": "base:24.04", "net": "none", "size": "small",
 "ttl_seconds": 300,
 "volumes": [{"name": "imagenet"}, {"name": "weights", "mount": "/models"},
             {"name": "scratch-db", "mode": "rw"}],
 "claim_ref": "namespace/workload", "metadata": {"team": "a"},
 "on_expire": "archive", "no_redirect": false, "require_promoted": false,
 "egress": true,
 "env": {"MODE": {"value": "prod"}, "GW_KEY": {"value": "Bearer …", "guest": false}}}

Success:

{"id": "sb_…", "token": "…", "deadline": "2026-07-06T00:05:00Z",
 "owner_addr": "10.0.0.5:7777", "template_digest": "sha256:…", "net_route": "relay",
 "volumes": [{"name": "imagenet", "mount": "/volumes/imagenet"},
             {"name": "weights", "mount": "/models"},
             {"name": "scratch-db", "mount": "/volumes/scratch-db", "mode": "rw"}]}

A claim cloned from a promoted template carries template_digest, the exact export generation fetched for that clone. It is absent for configured pools, cold image boots, forks and checkpoints.

net_route says how the guest reaches the network: relay through the host proxy behind the guest’s bound doors, direct over an egress-lane NIC the node does not lock, or none (a claim with no effective egress policy, a claim with "egress": false, or a locked egress-lane NIC with no policy). silkd hands its children the http_proxy, https_proxy and no_proxy of its unit unless the lane is direct; a guest agent of another kind reads net_route and hands its processes http://127.0.0.1:3128 when it is relay. "egress": false closes the doors, not an unlocked NIC, so such a claim on a direct lane still reaches out directly.

A claim branched directly from a checkpoint additionally carries "from_checkpoint": "ck_…" — the lineage edge for reconstructing the checkpoint tree. Fork children do not repeat that edge: they branch from the parent sandbox, not directly from its checkpoint.

volumes reports the names, effective mounts, and (rw only) mode applied and persisted at finalization — mode is omitted from the echo for ro entries, matching the request shape. sandboxd attaches each disk, waits for the device settle — a serial match under /sys/block, then the /dev/<name> node itself, since the kernel publishes sysfs before devtmpfs creates it — up to 2 seconds total, and mounts the filesystem — as ext4 first, then with mount’s own probe if that is refused; read-only, unless the entry requested and was granted rw — before returning. A mount that reports not a valid block device is retried within the 10-second setup budget, since the new node can exist before the kernel opens the disk. A custom mount may shadow an existing populated guest directory for the claim’s life.

Attach-only volumes

"attach_only": true stops after the attach for that entry. The device appears in the guest without a mount. Responses preserve attach_only: true and omit mount:

{"id": "sb_…", "token": "…",
 "volumes": [{"name": "imagenet", "attach_only": true},
             {"name": "scratch-db", "mode": "rw", "attach_only": true}]}

Find each device by its virtio serial, which is the catalog name — poll this, not a one-shot lookup: the claim can return before the guest enumerates the device (typically within ~100ms, never guaranteed), and the /dev/<blk> node can lag the serial match the same way it does for an eager mount’s device settle. Confirm both before mounting:

device=
tries=0
while [ "$tries" -lt 200 ]; do
  for serial in /sys/block/*/serial /sys/block/*/device/serial; do
    [ -r "$serial" ] || continue
    [ "$(cat "$serial")" = scratch-db ] || continue
    block=${serial#/sys/block/}
    candidate="/dev/${block%%/*}"
    [ -b "$candidate" ] && { device="$candidate"; break 2; }
  done
  tries=$((tries + 1))
  sleep 0.01
done
[ -n "$device" ] || { echo "scratch-db device not ready" >&2; exit 1; }
printf '%s\n' "$device"

Then mount it however the workload needs, retrying a mount that reports not a valid block device for a short time. A ro entry is attached --readonly and stays read-only at the guest block layer no matter who mounts it, so the guarantee does not depend on your mount flags.

The whole consistency contract moves to the caller with the mount:

What still protects other claims is unchanged: admission excludes an attach-only rw claim against every other claim of the name (and readers against a writer), an attach-only ro claim of a marker-bearing image is refused with 409, and a claim with volumes still refuses checkpoint, fork and hibernate. An attach-only claim costs one disk attach per volume — no device settle, no mount round-trip — and release costs nothing at all: removing the VM closes the devices.

Redirects (mutually exclusive with the fields above) name peers to retry at — sent on a warm miss with warm peers, when the node lacks a golden for the key but gossip names a template owner, and when a claim without volumes, no_redirect or require_promoted meets an admission refusal and a peer reports warm capacity or, failing that, serves the pool:

{"redirect": ["10.0.0.6:7777", "10.0.0.7:7777"],
 "require_promoted": true}

Retry the same body (+no_redirect: true) at each candidate until one answers. Preserve require_promoted: true when the redirect carries it, as a redirect to a template’s owner does; warm-candidate redirects omit the field.

A volume claim may consume an ordinary warm VM; candidate ranking, the promoted-template intersection, and the no_redirect/require_promoted retry follow the fleet-wide rule in cluster. Redirect responses never carry volumes.

A tenant token claims the same way; the sandbox is stamped with the tenant name (attributed in the usage journal and counted against the tenant’s max_claims). A catalog access list may restrict an entry to named tenants; an unknown and a forbidden volume return the same error text.

Errors: 400 unknown template axis, claim_ref or metadata over a bound (the message names it), a bad env entry, an unknown on_expire, invalid/duplicate volumes, an attach-only entry with a nonempty mount, mode: "rw" against a non-writable entry, or a volume that is unknown or forbidden (the latter two are deliberately indistinguishable), or bad body; 401 bad api token; 409 egress requested on a node without an egress attachment, a writable name already claimed in a conflicting mode (volume busy — a live writer excludes every other claim for that name, live readers exclude a writer), or a ro claim against a writable image left dirty by an unclean rw release (needs recovery — one rw claim must replay and cleanly release before ro claims resume; see deploy); 429 an admission refusal; 500 provisioning failed.

GET /v1/volumes

Auth: node API token (root or tenant). Lists the fleet catalog entries the caller may use, without host paths or holder addresses:

{"volumes": [{"name": "imagenet", "default_mount": "/volumes/imagenet",
              "size_bytes": 214748364800, "available": true, "nodes": 3}]}

Root sees the gossiped union, including peer-only entries (available: false); a tenant sees only entries this node declares locally and whose access list permits it. nodes counts members advertising the name; size_bytes and available are a best-effort stat of the answering node’s image. Membership is eventually consistent by one gossip tick. writable is the entry’s catalog configuration, fleet-uniform like the access list; the field is emitted (as true) only for a writable entry this node declares and omitted otherwise (peer-only rows never carry it).

PUT /v1/volumes/{name}

Requires the configured root API token. A tenant or operator token or a node without an api_token returns 403. Sandbox, unknown, and missing tokens return 401 when api_token is configured. The URL supplies the name. Unknown body fields, duplicate fields, and trailing JSON return 400.

{"path": "/srv/datasets/data.img", "directio": "off",
 "writable": false, "tenants": ["acme"]}

Success is 204. An identical normalized registration is idempotent. A different binding, changed fields, a static name or cleaned path collision, or a name a claim still holds returns 409. A PUT that adds a name and every successful DELETE, also of an absent name, write an op:"volumes" audit record naming the volume. Invalid fields return 400. GET keeps its existing projection without host paths or tenant lists. Runtime entries support the existing live volume API.

Persistence failure returns 503 with Retry-After. Retry the same request; after a restart, either state can load.

The node stores metadata in data_dir/runtime-volumes.json independently of sync_claims and loads it before claim recovery. New records use mode 0600. Malformed or conflicting metadata, including a saved path outside volume_roots, prevents startup. The catalog is node-local with one sandboxd writer.

Sandboxd does not create, check, or delete backings. Keep them outside data_dir: sandboxd clears its readers, goldens, moves, fork-*, checkpoint-staging and template-staging entries.

Volume access lists

Static and runtime entries share one tenants rule. An omitted or empty list permits every authenticated scope. A nonempty list permits only claims owned by the listed tenants, and root-owned claims can use any entry. A listed name must match the tenant API syntax: runtime registration rejects one that does not, and static config fails to load. A valid name for a tenant that does not exist yet grants nothing until the tenant is created.

DELETE /v1/volumes/{name}

Auth and name validation match PUT. Deletes only runtime registration metadata. Static names and names referenced by a claim, an in-flight operation, or cleanup return 409. Otherwise DELETE writes the absence and returns 204, also for an absent name. Persistence failure returns 503 with Retry-After. Retry the same DELETE, even if GET no longer lists the name; a restart can restore the old registration.

GET /v1/sandboxes/{id}/volumes

Returns the persisted desired set (volumes_want), confirmed applied set (volumes), per-volume states, and converging. Auth requires a client API token. Root can access every sandbox; a tenant can access only its own. Sandbox tokens are rejected when authentication is enabled. The owning node handles the request without redirecting.

{
  "volumes_want": [{"name": "imagenet", "mount": "/datasets/training"}],
  "volumes": [{"name": "imagenet", "mount": "/datasets/training"}],
  "states": [{"name": "imagenet", "state": "attached"}],
  "converging": false
}

All three arrays are present, including when empty. A state is attached when the volume is in both sets, pending when desired but not yet confirmed, or detaching when applied but no longer desired. An interrupted device call can leave the actual guest state different from the confirmed set. A failed PUT or PATCH adds a top-level error naming the refusal, and the affected state may carry its own error. converging is true only while a request worker is running. pending with converging: false requires an explicit retry; there is no background convergence. GET retains the last safe per-volume error for the current daemon lifetime. After daemon restart, unfinished states report an interrupted operation that needs retry. Successful retries clear resolved errors. The sandbox index exposes both sets for operator inspection.

PUT /v1/sandboxes/{id}/volumes

Replaces the full desired set and converges synchronously. Auth matches GET. The owner tenant’s catalog access list applies to newly admitted names, including for root. Retaining, repairing, or removing held names does not reauthorize them. A new image must exist on the owning node. The request must include a non-null volumes array. An explicit empty array removes all volumes.

{"volumes": [{"name": "imagenet", "mount": "/datasets/training"}]}

Mounted read-only, mounted writable, attach-only read-only, and attach-only writable entries can change live, including mixed lists. The claim-time name, mount, and eight-volume limits apply. An omitted mount defaults to /volumes/<name> unless that entry has attach_only: true. Responses carry that flag directly, so a GET target can be sent back with PUT. Omitting the flag or sending false does not inherit an existing attachment’s shape.

To change a held name’s mount, mode, or attach_only, first remove the name and wait for successful convergence. An identical target can be retried. Claims with on_expire: archive must change that policy before adding a volume.

A claim that holds volumes is not replicated. Its replica retires on the next replication pass, and a replica capture that races a live addition is refused. The sandbox must be running. Paused, archived, and failed sandboxes return 409 without waking. A failed sandbox holding volumes is release-only after its VMM exits; PUT with an empty target cannot restore its processes. Missing, invalid, or sandbox bearer tokens return 401. Another tenant’s sandbox and an unknown id return 404. Invalid, inaccessible, or non-local entries return 400. Removed tenant tokens are rejected at authentication. Root can still repair or remove held entries, but admitting a new name for a removed tenant returns 403. A conflicting writer or dirty image returns 409.

PATCH /v1/sandboxes/{id}/volumes

Changes the desired set by name, then converges synchronously. Auth, validation, and responses match PUT. Omitted add or remove arrays leave that part unchanged. An empty patch retries the current desired set.

{"add": [{"name": "imagenet", "mount": "/datasets/training"}], "remove": ["old-data"]}

Adding an identical entry or removing an absent valid name is harmless. Duplicate names, a name in both arrays, or changes to a held entry’s mount, mode, or attach_only return 400. Removal refuses busy or foreign mounts and uses neither lazy nor forced unmount. A foreign mount, another bind mount, or a completed ordinary unmount with nonzero exit returns 409 with the guest mount path and a safe per-volume reason. Guest transport failures remain server errors; timeouts return 504 and cancellation returns 408.

A mounted writable attachment marks the image dirty before the guest can write. Live removal and release clear the marker only when the unmount of the mount at the volume’s path succeeds, the mount was the filesystem’s root and writable in both its mount and superblock options (ext4’s emergency_ro after a filesystem error counts as read-only), and no other mount of its device remains. Live removal finds the device by its serial and refuses any other mount of it; release identifies the mount by its path alone and does not check which device it is. These checks read silkd’s own mount namespace, so a mount of the device in another namespace, or one moved there, is not seen. Live removal also needs a confirmed host detach. A missing device or pre-unmounted filesystem does not prove clean unmount. The marker remains on uncertainty and blocks subsequent readers. Release waits up to 30 seconds for each writable volume’s unmount, and a guest that does not answer within that keeps every remaining marker. Attach-only writers have no automatic dirty-marker protection. The caller owns mounting, flushing, unmounting, recovery, and fencing before reuse.

Partial failure and recovery

PUT and PATCH return the GET response shape on success. A convergence error returns a non-2xx status with the same view, so callers can inspect desired, applied, and per-volume state; semantic validation 400s carry it too. A malformed body, a missing volumes array, authentication and an unknown sandbox carry only error. A failed request may already have persisted its target and completed some device changes. GET reads the last confirmed set, rather than probing the devices. Resolve guest mount references or runtime failures before retrying. Retry the same PUT to converge, or send a new target to abandon unfinished additions. Concurrent requests serialize per sandbox without blocking renewals or new exec, file, and port connections.

Before applying a target, sandboxd journals VolumesWant beside Volumes, storing it only while it differs from Volumes. A superseding request may first clean up an older desired-only attachment while the old target stays recorded. If that cleanup fails, the response retains the old view; retry the same PUT to finish accepting the new target. Admission holds cover the deduplicated union of both sets, including during restart and teardown; a teardown keeps them until the VM is confirmed gone. Failed or uncertain operations retain their holds until successful cleanup. Startup reserves those holds without replaying attachments. Release cancels convergence and transfers the held union to teardown without waiting for a blocked device call. Capture operations (hibernate, fork, checkpoint, and promote) are refused while either set is nonempty or a convergence request still owns admitted volume holds.

Guest device calls honor cancellation and have a 30-second ceiling per volume, and a cocoon attach or detach can take 30 more to exit; a caller that disconnects cancels the run, leaving interrupted states to retry. Journal writes are atomic. Host-crash durability follows sync_claims.

Live convergence requires Cocoon v0.7.2 or newer for idempotent matching attach and absent detach. PUT and PATCH reject an older release, or a runtime whose version cannot be queried, with 503 before changing intent, holds, or devices; a dev build is assumed current. sandboxd caches the first successful version read, so restart it after upgrading cocoon. GET and release remain available.

Catalog names must keep the same backing image while held, including pending holds. Publish changes under versioned names such as dataset-v1 and dataset-v2. The journal cannot detect a catalog name repointed to a new image.

A replacement first unmounts the old disk, then mounts the new disk. Pause reads and close files, mmap, cwd, and bind mount references during this gap. Existing independent guest processes and sessions continue. See the same-sandbox replacement tutorial.

POST /v1/sandboxes/{id}/release

Auth: the sandbox’s own token, or the root or operator token for cleanup by id. Destroys the VM. 204 on success, 404 for an unknown id or wrong token, 500 when the VM survives removal: the claim is already gone and the node retries the removal, 503 archive registry down; retry for a claim that holds a registry row while the meta store is down; resend it. Releasing an already-gone sandbox is 404 — the SDK treats it as success.

POST /v1/sandboxes/{id}/hibernate

Auth: the sandbox’s own token, or the root or operator token by id (like release). Atomically snapshots the VM and stops it, freeing its memory; the next agent access restores it transparently (sessions, processes, and memory state intact — cocoon’s hibernate keeps the snapshot point and the stop coincident), except a read-only file call that a reader serves without waking it. Idempotent on an already-hibernated sandbox. The TTL keeps running: a hibernated sandbox is still reaped (VM and snapshot) at its deadline, unless it claimed on_expire: archive or its pool archives. When to hibernate is the caller’s policy — the node only provides the transition. 204 on success, 404 unknown id or wrong token, 409 on the egress lane or when volumes are attached (neither kind of sandbox hibernates; see egress) or on a failed sandbox, 429 while the disk is under disk_reserve, 503 lease lost; retry for a replicated claim that is not archived while its node’s lease is overdue, even one already hibernated.

POST /v1/sandboxes/{id}/wake

Auth: the sandbox’s own token, or the root or operator token by id. Restores a hibernated (or archived) sandbox and leaves it running — waking is otherwise only a side effect of the next agent access, so this is the explicit form for warming a sandbox ahead of use. Idempotent on one already running. A hibernated sandbox retains its existing deadline, unless it is past its lease and archives at expiry, which starts a fresh lease. An archived sandbox instead uses archive_delete_after_seconds as its retention deadline while stored, and waking it starts a fresh lease of the length the claim was granted (the server default when it asked for none). A failed sandbox is cold-booted from its own disk (memory is lost) and its restart budget resets. A hibernated sandbox whose guest does not answer within the wake deadline after two restores of its snapshot is cold-booted from its own disk instead: memory is lost, restarts moves, and the snapshot is dropped. The boot records vmm_restart when it starts and wake, with the lease, when the guest answers; a boot that fails after its start records hibernate, and the claim stays hibernated. A restored VM that cannot be stopped is not cold-booted: the wake fails and the claim stays hibernated until the next wake restores the snapshot again. An archived sandbox whose guest does not answer after two clones of its archive is cold-booted from the second clone’s disk: memory is lost, restarts moves, and the wake records unarchive once it commits; a cold boot that fails removes the clone, and the claim stays archived. The same holds for a wake that an agent access triggers. 204 on success, 404 unknown id or wrong token, or an archived sandbox the node had no room for that it handed to a peer (look it up again), 409 a failed egress-lane or volume sandbox, or one whose VM record is gone, 429 when max_running or disk_reserve refuses the wake, 500 when the cold boot or a restore fails, and the guest errors.

POST /v1/sandboxes/{id}/renew

Auth: the sandbox’s own token, or the root or operator token by id. Resets the lease to ttl_seconds from now, so a client holding a sandbox longer than its claim asked for does not have to claim a new one:

{"ttl_seconds": 3600, "on_expire": "archive"}

→ 200 {"deadline": "2026-09-22T12:00:00Z"}. The grant, not the request, is authoritative: ttl_seconds 0 means the node default and anything past the 24 h cap is clamped, and the reply always carries what was granted. An absent body means the same as ttl_seconds 0. The new lease also becomes the one a wake from the archive grants again. A renew can shorten a lease as well as extend it. on_expire switches the expiry action to archive or back to destroy; absent keeps the current one. It is refused as on a claim for a sandbox that cannot hibernate. A renew stamps no activity, so it neither restarts the idle clock nor stops an idle hibernate.

An archived sandbox is refused: off-node, its deadline is the store’s retention window rather than a lease, and a TTL written over it would schedule the checkpoint for deletion. Wake it first. A claim whose tenant was removed runs to its deadline but cannot extend it: its own token gets 403, the root or operator token still renews. 400 a malformed body; 401 missing bearer token; 403 the claim’s tenant was removed; 404 unknown id or wrong token; 409 archived, or being archived right now; 503 the archive registry is down, for a hibernated claim with a replica under archive_failover: the node already holds the new lease, and a resend replaces the row; 503 tenant store down; retry for a tenant the node has not cached while the meta store is down.

POST /v1/sandboxes/{id}/fork

Auth: the node api_token (Bearer) — forking creates node resources, like a claim. The sandbox’s own token rides in the body as the ownership proof; the root api_token with an empty body token acts by id as root:

{"token": "…", "count": 2, "ttl_seconds": 300, "claim_ref_prefix": "team-a/"}

claim_ref_prefix, when set, records each child under prefix + child id as its claim_ref, so a control plane that names claims <namespace>/<name> addresses the children by name; without it children carry no claim_ref. The prefix is at most 512 bytes of printable UTF-8, as claim_ref is. Children inherit the parent’s metadata, on_expire and "egress": false; an on_expire in the body overrides the inherited action. Children do not inherit the claim env: a guest env file captured from the parent is rewritten empty in each child, and the parent’s host-only entries are not copied; set one with PUT /v1/sandboxes/{id}/env. Clones the sandbox into count fresh claims (1 up to the node’s max_fork_count, default 16). Memory, disk, and guest state (sessions, processes, tmpfs) duplicate at the fork point; cocoon’s clone reseed gives every child a distinct machine identity. Children get their own lease — ttl_seconds (0 = server default), never the parent’s remainder. A running parent is snapshotted in a brief pause window; a hibernated parent forks from its existing memory image without waking. All-or-nothing: on error no child survived. 200 with one claim per child:

{"children": [{"id": "sb_…", "token": "…", "deadline": "…", "owner_addr": "…"}]}

Children inherit the parent’s tenant and count against its max_claims, whoever calls. 400 invalid count or body, a claim_ref_prefix over its bound or an unknown on_expire; 401 bad api token; 403 the parent’s tenant was removed (sandbox token); 404 unknown id or wrong sandbox token; 409 egress-lane or volume parent (neither forks, checkpoints, or promotes; see egress), a failed parent, or an archived parent (POST /wake or an exec wakes it first); 429 an admission refusal; 503 lease lost; retry for a replicated parent while its node’s lease is overdue, tenant store down; retry for a sandbox token whose tenant the node has not cached while the meta store is down.

POST /v1/sandboxes/{id}/promote

Auth: like fork — node api_token in the header, the sandbox’s own token in the body; the root api_token with an empty body token acts by id as root:

{"token": "…", "template": "myproj:v1"}

Publishes the sandbox’s current state as a node-local template under (template, this sandbox’s net, its size): later claims for that key clone from it, provision-on-demand, with no warm pool. Re-promoting to the same name replaces the template. A hibernated sandbox is promoted from its memory image without waking. 200 returns the template’s full key and immutable content identity. On the default local-disk backend a template is node-local, so a cluster client claims from and deletes on this node (name-based calls route via gossip); a shared checkpoint store makes every node resolve it. Under exactly this key:

{"key": {"template": "myproj:v1", "net": "none", "size": "small"},
 "content_digest": "sha256:…"}

content_digest is SHA-256 over a versioned canonical stream of the published export’s regular files: slash-relative path, byte length, and the SHA-256 of each 16 MiB chunk of its bytes, ordered lexically. Directory entries, modes, mtimes, and the template’s ownership/creation metadata do not affect it. The digest is computed once while promoting, stored beside meta.json (a sibling digest file on the directory backend, object metadata on S3), and therefore has identical semantics on both. Re-promoting unchanged export bytes keeps the digest; changing any exported path or bytes changes it. A re-promote starts the template with no labels: they describe the export they were set on, which the new one replaces.

400 invalid name, 401 bad api token, 409 when the name collides with a configured pool, the template is owned by another tenant, or the sandbox is on the egress lane, has volumes attached (see egress), has its tun turned off, has failed, or is archived (POST /wake or an exec wakes it first), 404 unknown id or wrong sandbox token, 429 while the disk is under disk_reserve, 503 lease lost; retry for a replicated sandbox while its node’s lease is overdue.

DELETE /v1/templates?template=…&net=…&size=…

Auth: node API token. Removes a promoted template (the query parameters default like a claim’s: net=none, size=small). A tenant may delete only templates it promoted — anything else is 404, root deletes anything. 204 on success, 404 unknown template, 409 when the key belongs to a configured pool (those goldens are owned by the node config). digest=<content digest> deletes only while the record still has that digest and answers 412 otherwise, so a caller that observed an older build never removes the newer one that replaced it on this node; on a shared store a promote committing on another node at the same moment is not covered. On a cluster, a node that does not hold the template but sees an owner in gossip answers 200 {"redirect": [addrs]} — the claim redirect shape — and the SDK retries the delete at the owner. The retry carries no_redirect=1, mirroring the claim protocol: a node answering a no_redirect delete speaks only for itself.

PUT /v1/templates/labels?template=…&net=…&size=…

Auth: node API token. Replaces a promoted template’s labels on this node (the query parameters default like a claim’s):

{"labels": {"v1": "sha256:…", "team": "a"}}

Labels follow the claim metadata rules — at most 16 pairs, the same key and value grammar, at most 4 KiB as JSON — and replace the whole map; {} clears it. digest=<content digest> writes only while the record still has that digest and answers 412 otherwise, so a writer that observed an older build never relabels the newer one on this node. They are stored beside the template’s record, so they survive a restart, are listed by the node that promoted the template or loaded it at startup, and go with the template when it is deleted; a re-promote starts with none. A tenant may label only templates it promoted — anything else is 404, root labels anything. 204 on success, 400 bad labels or body, 404 unknown template, 409 when the key belongs to a configured pool. The call speaks only for the node it reaches; it does not follow a redirect.

PUT /v1/pools

Auth: root only (tenant and operator tokens get 403). Replaces the node’s desired warm targets online — no restart, and no live claim’s VM is touched. With meta_store.cell set, it replaces them on every node of the cell. On a node with egress policies, whether a key is pooled decides which policy layers a claim of it gets; that is settled when the claim is made, so adding or dropping a pool here changes only claims made afterwards — a key that gains a pool with no egress block leaves new tenant claims without egress, a key that loses such a pool gives new tenant claims their egress class’s policy alone:

{"pools": [{"template": "base:24.04", "net": "none", "size": "small",
            "warm": 4, "warm_max": 16, "idle_hibernate_seconds": 0}]}

Pools omitted from the list are drained: their unclaimed warm VMs are destroyed and the pool entry retires. Once such a pool is idle, its golden is deleted; a pool kept with warm: 0 keeps its golden. net/size default like a claim’s. Answers the fresh GET /v1/info payload. 400 bad key, negative warm/idle, warm_max below warm, a reader pool with warm below 1, idle_hibernate_seconds on an egress pool, a negative archive duration or an archive_after_seconds not above the pool’s idle_hibernate_seconds, duplicate pool, or a config-owned egress/warmup/capture_trim/storage/replicate_seconds/reader/egress_upstream_env/tun field; 401 bad api token; 409 egress pool on a node without an egress attachment; 503 the meta_store cannot take a cell’s write.

Tenants (/v1/tenants)

Auth: root or operator (tenant tokens get 403); a node without api_token answers 400 to a write, since tenants need it. The API is the only way in: config.json has no tenants field, and a node starts with no tenants. The set changes at runtime, with no restart:

A name follows the claim name grammar (^[a-zA-Z0-9][a-zA-Z0-9._:/-]{0,62}$; escape / as %2F in the path) and is the tenant’s identity: claims, checkpoints and promoted templates belong to a tenant by name, so a name reused after a removal inherits what the old tenant left. Use an id the control plane never reissues, such as a user UUID. Tokens must be unique and differ from api_token. egress_class names one of the node’s egress_classes (400 when unknown) and gives the tenant’s claims that egress layer; without one they reach nothing on a pool with a policy. The API references a class and never defines one. A claim keeps the class its tenant had when it was made.

A change applies to the next request: a removed or rotated token gets 401 at once. max_claims is a per-node cap (0 = unlimited), so a tenant’s cluster limit is max_claims × nodes; every usage event of a tenant’s claim carries the tenant, for per-tenant billing. Lowering max_claims below the live count refuses new claims and evicts none. A removed tenant’s claims keep running and stay listed for root (GET /v1/sandboxes?tenant=<name>) until they expire or root releases them, but they cannot extend: renew and fork answer 403 to the sandbox’s own token, a hibernated one is never archived, one archived before the removal is not woken (403) and is deleted when its retention ends, and at the deadline they are destroyed even under on_expire: archive. The verbs that need a tenant token (promote, checkpoint, preview, branch claims) are closed to them already.

The node writes the applied set to <data_dir>/tenants.json (0600, each token kept only as its SHA-256) before it serves, and loads that file at boot. Each change is an op:"tenants" audit record naming the tenants added, changed and removed. On a cluster every node needs the change (see cluster). 400 a bad entry, a duplicate name or token, a token equal to api_token, api_token_next, operator_token or operator_token_next, an unknown field, or an unknown egress_class.

POST /v1/config/reload

Auth: root only (tenant and operator tokens get 403). Re-reads the node’s config file and applies its reloadable settings at once, or nothing. SIGHUP runs the same reload and logs the result. →

{"changed": ["egress_internal_allow", "pools[desktop:v13 none small].warmup"],
 "ignored": ["pools[desktop:v13 none small] targets (PUT /v1/pools owns them)"]}

changed lists the settings that now apply. ignored lists differences the file holds for pool targets, which PUT /v1/pools owns. Which fields reload, and what happens to live claims and goldens, is described in deploy. The status codes:

A refused reload changes nothing.

POST /v1/drain

Auth: root only (tenant and operator tokens get 403). Empties the node for maintenance: claim/fork/branch answer 429 with an error ending node draining (on a cluster a non-volume claim tries a peer redirect first, and peers stop redirecting here within a tick, because this node gossips no pools), a live move to this node is refused, unclaimed warm VMs are destroyed, and live claims keep serving until release or TTL. Pool ownership is untouched — no pools.json write, no config change. Answers the fresh GET /v1/info payload; poll claimed to zero to know the node is empty. Deliberately not persisted: a restarted node serves again. To keep new claims off a node across restarts, cordon it; a cordoned node still adopts existing claims, so to keep it empty after a restart, drain it with ?evacuate=1 again; the admission table compares the states.

POST /v1/drain?evacuate=1 drains the same way, then moves the claims to the peers with their state, so the node empties without waiting for the leases to run out (see evacuating a node). It needs archive_failover. In the background, with bounded concurrency, each claim is hibernated if it runs, archived to the checkpoint store if it is not archived, its registry row is released, and a peer is asked to adopt it through GET /v1/sandboxes/{id}/owner, so that peer lists it, and root’s GET /v1/sandboxes/{id} there returns its token once that node has seen it. A claim the root adopted, whose token this node has not seen, is offered with the root token; a peer that does not accept the root on /owner refuses it, and the claim stays here. Open streams to a claim end when it hibernates. Until a peer holds it, a call that would wake the claim answers 503 sandbox moving to another node; retry with Retry-After; after that this node answers 404 and the client looks the claim up again. With no peer to adopt it, the node takes the row back and the claim stays. A claim that cannot move stays here and keeps serving: an egress-lane claim, one with volumes, a failed one, one whose tenant was removed, one archived before archive_failover was on, one whose step failed, and every claim of a node without archive_failover. The answer and GET /v1/info count the claims left in evacuating; once it is gone, claimed counts the ones that stayed, and the log names each and why. A claim of a pool without an archive tier, whose on_expire is not archive, keeps its lease deadline across the move until its first wake, which starts a fresh lease of the claim’s length; any other keeps the later of its lease and archive_delete_after_seconds, or no deadline when that is 0. A call while an evacuation runs changes nothing; a call after one finished starts a new one over the claims still here.

DELETE /v1/drain

Auth: root only (tenant and operator tokens get 403). Lifts the drain, stops an evacuation, and kicks an immediate refill. Answers the fresh info payload. A claim an evacuation is archiving finishes its archive and stays here archived; the next call wakes it. A cordon stays in place.

POST /v1/cordon

Auth: root only (tenant and operator tokens get 403). Keeps new claims off the node until DELETE /v1/cordon, across restarts: claim/fork/branch answer 429 with an error ending node cordoned, whatever max_claims and max_running allow (on a cluster a non-volume claim tries a peer redirect first, and peers stop redirecting here within a tick, because this node gossips no pools), refill stops and unclaimed warm VMs are destroyed. Live claims keep serving, and unlike a drain, a root move to this node is admitted, and so is the adoption of an archived claim, since neither is a new claim. The admission table lists what each state allows; a cordon and a drain are independent, and each DELETE lifts only its own.

The cordon is the file <data_dir>/cordoned; it survives a restart, a reboot and a config reload, and a node that starts cordoned starts no warm VM. With audit_log on, audit.jsonl records op cordon. Answers the fresh GET /v1/info payload with "cordoned": true; 500 when the file cannot be written, and the node stays as it was.

DELETE /v1/cordon

Auth: root only (tenant and operator tokens get 403). Removes <data_dir>/cordoned, lifts the cordon and kicks an immediate refill; audit.jsonl records op uncordon. A drain stays in place. Answers the fresh info payload; 500 when the file cannot be removed.

POST /v1/sandboxes/{id}/move

Auth: root only (tenant and operator tokens get 403). Moves one running claim to a peer with its guest’s processes and memory (see moving a running claim). Body: {"to": "<peer>", "max_downtime_ms": 2000}, to as GET /v1/peers lists it and never this node (else 400), max_downtime_ms optional with a default of 2000 and a floor: 400 or less answers 400 at once, because the target’s receive and start alone are allowed that long. Answers {"owner_addr": "<peer>", "rounds": n, "downtime_ms": d, "capture_ms": c, "send_ms": s, "commit_ms": m} once the peer took the claim and this node dropped it; a guest the peer cannot start is failed there. downtime_ms runs from the final capture’s start, including its wait for a free dump slot, until the peer answers the commit, after it started the guest; of it, capture_ms, send_ms and commit_ms are the final capture, the final round transfer and the commit. A final round whose capture time and size no longer fit max_downtime_ms is aborted before it is sent and the guest resumes here, so with the meta store up a capture that alone overruns pauses the guest for the capture plus the abort and its record write. While the move runs, release, fork, checkpoint, promote, hibernate and volume changes of the claim answer 503 sandbox moving to another node; retry. Env writes, renew and data-plane calls keep working until the final round and answer the same 503 during it, because the peer would miss them. Every error leaves the claim running here, except one after the final round paused the guest whose abort did not land (a 503 with the registry down, or a failed cocoon vm migrate abort): the guest stays paused until the abort is retried through, the guest then resumes or, when the record reads committed, the node drops the claim (see moving a running claim):

The target serves root-only peer routes for a move, POST /v1/moves/{id}, PUT /v1/moves/{id}/rounds/{n}, POST /v1/moves/{id}/commit and DELETE /v1/moves/{id}; they are no public API.

POST /v1/sandboxes/{id}/preview

Auth: node api_token in the header, the sandbox’s own token in the body. Unlike fork, promote and checkpoint — where the root token with an empty body token acts by id as root — preview has no root path: the body token is required. Mints a signed URL serving a guest HTTP port from a browser: body {"token": "...", "port": 8080, "ttl_seconds": 0} → {"url": "<preview_advertise>/p/<token>/"} (http:// is prefixed when preview_advertise names no scheme). The URL’s life is clamped to the claim’s remaining lease; an archived claim kept forever (archive_delete_after_seconds: 0) has no lease, so there the requested ttl_seconds stands unclamped and ttl_seconds: 0 mints a one-hour URL; releasing the sandbox ends it early either way. 400 a port of 0, 404 unknown id or wrong sandbox token, 501 when the node has no preview_listen. The signed token embeds the sandbox id, port, and owner advertise_addr, so any node’s preview listener can serve it (forwarding to the owner’s main listener) and a released sandbox’s URL answers 404 — no revocation list. A browser request to the minted URL answers 503 lease lost; retry for a replicated claim while its node’s lease is overdue, 503 sandbox moving to another node; retry while an evacuation has yet to move it, and 429 when it has to wake a sandbox while the disk reserve is low or the node is at max_running; a wake that hands an archived claim to a peer answers 404, as after a move. See deploy.

POST /v1/sandboxes/{id}/checkpoint

Auth: node API token; body {"token": "<sandbox token>", "name": "..."} (name optional); the root api_token with an empty body token acts by id as root. Captures the sandbox’s full state without stopping it and answers 200 {"checkpoint": {id, name?, sandbox_id, key, tenant?, created_at, policy_source?, no_egress?, tun_off?, tun_profile?, egress_limit?, guest_env?}} — tenant records the calling tenant, absent for root. 400 bad body or name, 401 bad api token, 404 unknown id or wrong sandbox token, 409 egress-lane sandbox, one with volumes attached (see egress), a failed one, or an archived one (POST /wake or an exec wakes it first), 429 while the disk is under disk_reserve, 503 lease lost; retry for a replicated sandbox while its node’s lease is overdue.

POST /v1/checkpoints/{id}/claim

Auth: node API token; body {"ttl_seconds": 0, "no_redirect": false, "metadata": {}, "on_expire": "destroy", "env": {}}. Claims a fresh sandbox branched from the checkpoint (a normal claim response, attributed to the caller); the checkpoint’s recorded key applies. A tenant token branches only its own tenant’s checkpoints; any other id answers 404, as for list and delete. On a node that does not hold the record, the redirect and the peer heal below run before that check, so another tenant’s id can redirect to its holders, or be pulled here before the 404; checkpoint ids are random and no cross-tenant read serves them. The branch carries this request’s metadata, on_expire and env, under the claim rules, never the source sandbox’s. 400 metadata over a bound, an unknown on_expire, or a bad env entry.

Checkpoints are node-local (unless the store is shared — see Configuration). On a miss a node with a node-local store probes its peers (a shared s3 store skips the probe: every node already resolves every record) and answers with a redirect shaped like a warm-miss POST /v1/claim, at most 3 addresses; retry the same body (+no_redirect: true) at each candidate. With nothing to redirect to (or no_redirect set) and checkpoint_peer_heal on, it pulls the record and serves the claim. See the placement lifecycle.

404 for an unknown checkpoint (locally, and after redirect and heal both miss), 429 an admission refusal, 503 when the node’s concurrent-heal cap is already full — retryable, and the response carries a Retry-After hint.

GET/HEAD /v1/checkpoints/{id}/blob

The peer-transfer route behind the probe and heal above — internal, not part of the public API; an SDK caller has no reason to call it directly.

GET /v1/checkpoints

Auth: node API token. Lists this node’s checkpoints, newest first. A tenant sees only its own records; root sees everything. A checkpoint backing an archived claim is that claim’s wake image (an ar_ id), not a listable record. Answers 200 {"checkpoints": [{id, name?, sandbox_id, key, tenant?, created_at, policy_source?, no_egress?, tun_off?, tun_profile?, egress_limit?, guest_env?}]}, the record shape POST /v1/sandboxes/{id}/checkpoint returns.

DELETE /v1/checkpoints/{id}

Auth: node API token. A tenant may delete only its own records — anything else is 404, never a hint the id exists; root deletes anything. 204 on success, 404 unknown — including the checkpoint behind an archived claim, which only that claim’s release or retention window removes.

Delete removes the local record, then best-effort broadcasts to peers so a healed replica does not outlive it — eventual cleanup, not a fleet-wide revocation. A peer offline during the broadcast keeps its copy until the checkpoint TTL ages it out, so an id-holder can still branch it for that window; placement lifecycle has the bound and why heal requires a nonzero, fleet-matching TTL. The s3 checkpoint store skips the broadcast: every node already resolves every record directly, so there is no replica to chase. A mesh without a cluster_key skips it too: it has no healed replica, and the broadcast would send the api_token to addresses learned from unauthenticated gossip. ?no_forward=1 marks a delete already arriving from another node’s own broadcast, so it is not itself re-broadcast (loop prevention); it is an internal parameter, not one an SDK caller should set.

GET /v1/sandboxes

Auth: node API token or the operator token. Root and the operator see every live claim; a tenant sees only its own. The index is {"sandboxes": [{id, key, tenant?, deadline, claimed_at?, hibernated, archived?, from_checkpoint?, claim_ref?, metadata?, on_expire?, cpu_count, mem_total_bytes, volumes?: [{name, mount?, mode?, attach_only?}], volumes_want?: [{name, mount?, mode?, attach_only?}], restarts?, restarted_at?, failed?, env_missing?}], "next": "…"} — deadline is 0001-01-01T00:00:00Z on an archive kept forever; cpu_count and mem_total_bytes are the size tier’s allocation; restarts counts the cold boots that replaced an exited VMM or a hibernate snapshot or archive whose guest did not answer a wake, and restarted_at stamps the last one (each loses guest memory and the instance-metadata document, so a caller re-establishes guest state when restarts moves; the claim’s env is written to the guest again); failed names why the VMM is down and was not restarted (see dead VMMs); env_missing is true while a guest: false env entry has no value on the node (claim env); mode is omitted for ro, matching the claim echo; never sandbox tokens, volume host paths, or catalog access lists. claimed_at is the first grant (renew and wake move deadline only). ?claim_ref=<ref> keeps only the claims recorded under exactly that reference, so a caller that named its claim reads its row without the rest of the index; an empty value lists everything. ?metadata=key=value, repeatable, keeps only the claims holding every pair; 400 a pair without =, with an empty key, or with a key repeated. tenant names the owning tenant (absent for root’s own claims); ?tenant=<name> lets root list one tenant’s claims, a removed tenant’s included, and is ignored for a tenant token.

GET /v1/sandboxes/{id}

Auth: root or operator (tenant tokens get 403). One live claim in the index-row shape above, so a reconcile loop can read a single sandbox without scanning the whole node listing. When the caller presents the configured api_token or operator_token the row also carries token, the sandbox’s own bearer token, so a control plane can hand a returning client its data-plane credential without keeping a copy; a node without an api_token never grants per-sandbox authority and answers without it. The row also carries the advanced network mode state, each field omitted when unset: tun_addr (the claim’s overlay address), tun_off (true when the claim is disabled) and tun_profile (the exit label; absent means default), and egress_limit, the claim’s bandwidth cap, omitted when it has none. 404 unknown id.

GET /v1/sandboxes/{id}/stats

Auth: root or the operator token (tenant tokens get 403). One sandbox’s resource usage — the per-sandbox counterpart to the node-scoped /metrics:

{"id": "sb_…", "cpu_count": 2, "mem_total_bytes": 1073741824,
 "mem_used_bytes": 187654144, "mem_used_measured": true,
 "cpu_seconds": 5321.47, "cpu_started_at": "…", "cpu_measured": true,
 "hibernated": false, "measured_at": "…"}

cpu_count/mem_total_bytes come from the size tier. hibernated marks a claim whose memory snapshot is on this node, and archived, present only when true, one whose state is in the checkpoint store; neither has a VMM to read. mem_used_bytes is the host VMM process’s resident set — the only memory usage signal available without a guest agent. cpu_seconds is the same process’s user plus system CPU time, all threads, guest time included, and cpu_started_at is when that process started. The counter starts over with the process (a wake, a restart or a move), so take the rate between two reads only while cpu_started_at is the same. mem_used_measured and cpu_measured are false when there is no VMM process to read (hibernated or archived, the PID not yet known, or the engine lookup failed), so a zero is never mistaken for idle; measured_at is taken after the reads. One read costs one engine lookup; to read every claim, use GET /v1/stats. 404 unknown id.

GET /v1/stats

Auth: root or the operator token (tenant tokens get 403). Every live claim’s stats on this node, sorted by id, in the shape of GET /v1/sandboxes/{id}/stats:

{"sandboxes": [{"id": "sb_…", "cpu_seconds": 5321.47, "cpu_started_at": "…", "cpu_measured": true, …}]}

The whole read costs one engine listing and one /proc/stat read, whatever the claim count, plus two /proc reads per running claim. 503 vm listing failed; retry when the engine cannot list its VMs.

GET/PUT/PATCH /v1/sandboxes/{id}/env

Auth: node API token (root or tenant) or the operator token, which acts as root. A tenant reaches only claims it owns; anything else is 404, like an unknown id. Each call speaks only for the node it reaches; on a cluster, send it to the claim’s owner_addr.

PUT replaces the claim’s whole env with the body’s map, under the claim rules; {"env": {}} clears it. PATCH merges: each entry in the body is set, an entry of null removes that name, and every other entry is kept as stored, host-only values included, so a control plane that cannot read those values back sends them again only when the node reports them missing (below):

{"env": {"CB_META_wake": {"value": "w2"}, "OLD": null}}

The size rules apply to the merged env, so a patch that takes it past 64 entries or 64 KiB answers 400 and changes nothing. GET answers the same shape with every guest: false value blanked, so a control plane sees the names without the values:

{"env": {"MODE": {"value": "prod"}, "GW_KEY": {"value": "", "guest": false}}}

An entry with inject comes back with its hosts, header or query, body flag and placeholder, value blanked. An inject host the claim’s pool does not intercept, or its tenant class does not allow, and a header the pool rule’s own secret already sets on that host, answer 400 and store nothing.

An entry the node holds no value for comes back with "missing": true, and the claim’s index row carries env_missing. Under host_env_memory_only that happens when the claim hibernates, after a restart and after an adoption. A missing entry injects nothing and blocks the node’s value for its secret; a missing upstream entry falls back to the class or pool upstream. Send it again once the claim runs: a PATCH replaces the entry whole, so repeat guest and inject:

{"env": {"GW_KEY": {"value": "Bearer …", "guest": false, "inject": {"hosts": ["api.example.com"], "header": "Authorization"}}}}

A value sent to a hibernated claim stays in that node’s memory until the claim hibernates again.

The env is persisted with the claim, its guest: false values only without host_env_memory_only. A guest: false change takes effect on the egress proxy’s next request at any time, a hibernated or archived claim included, and never touches the guest. A change to the guest entries is written to the guest at once, so it needs a running guest: on a hibernated, archived, failed or transitioning sandbox it answers 409 (this verb never wakes one). A PUT or PATCH whose result equals the stored env rewrites a running guest’s file, so resending repairs a failed write, including a removal or a change from guest to host-only. This also applies to an unchanged host-only patch; the file contains only the guest entries. An unchanged result does not rewrite the claims journal. A 500 after the store step means the env is stored and its guest: false values are already live, but the guest write failed; resend the same request to deliver it. The guest side is described in silkd. PUT/PATCH answer 204, GET 200; 400 a bad entry, an unknown body field or an inject outside the claim’s intercepted hosts; 404 unknown id or another tenant’s claim; 409 a guest change on a paused or failed sandbox; 503 the archive registry is down, for an archived claim or a hibernated claim with a replica under archive_failover: the node already holds the new env, and a resend replaces the row.

PUT /v1/sandboxes/{id}/instance-metadata

Auth: root only (tenant and operator tokens get 403). Sets the sandbox’s instance-metadata document: the JSON a guest agent reads from 169.254.169.254 in the IMDSv2 shape, as cloud-init and similar agents do (PUT /latest/api/token, then GET / with the X-metadata-token header). It is unrelated to the claim’s metadata labels. The body is one JSON object of at most 4 KiB, stored and served verbatim; a guest with no document answers {}, and a fork child, checkpoint branch or template clone starts with the document its source held. The node writes it to /run/silkd/instance-metadata.json in the guest atomically, so repeating a call is a no-op. silkd serves the address only in images that alias it on lo.

204 on success; 400 when the body is not one JSON object within the bound; 404 unknown id; 409 when the sandbox is failed, hibernated, mid-transition or archived (this verb never wakes one) or its image serves no instance metadata.

PUT /v1/sandboxes/{id}/tun

Auth: root only (tenant and operator tokens get 403). Switches a claim’s advanced network mode and sets its exit profile label:

{"enabled": false, "profile": "residential"}

Both fields are optional; an omitted field keeps the claim’s value. profile names an entry of tun.exits; an empty string returns the claim to default, and a label the exit table does not hold yet is accepted, since the table reloads (until it appears, the claim’s packets are dropped as having no exit). The state is persisted with the claim, follows it through hibernate, archive, move and an archive row, and is accepted on a hibernated or archived claim without waking it; its next wake applies it. A running claim’s replica row under archive_failover picks the change up at its next replica capture, as an env change does, so an adoption before then restores the switch with the rest of that replica. On a running claim the change applies at once: disabling closes the claim’s packet stream and its door, drops its flows and stops its ARP answers, then sends the guest tun_down, except while a checkpoint, fork or replica capture holds the claim, when the guest keeps a tun0 that carries nothing; enabling, or changing the profile, re-arms the packet door and sends tun_up. Enabling a claim of a tun pool that holds no overlay address, because the address store failed at its claim, allocates one. A fork child takes its parent’s switch and label, with its own address.

204 on success; 400 an unknown field or a profile that does not match the name rule; 404 unknown id; 409 an enable or profile change while a transition (a capture, wake, hibernate or archive) holds the claim (retry; a disable never waits), or another tun request to the claim committed first (retry); 503 the claim is in a live move’s final round or not yet started on its target (retry); 500 the address store failed or the claims journal could not be written (nothing changed); 503 the archive registry is down for an archived claim or one with a replica under archive_failover: the node already holds the change, and a resend of the same body replaces the row.

POST /v1/sandboxes/{id}/balloon

Auth: root or operator; a tenant token answers 403. Sets a running claim’s virtio-balloon target, the guest memory the balloon withholds from the guest, through one cocoon vm balloon call, and answers with cocoon’s report:

{"target": "50%"}

→ 200 {"target": 2147483648, "actual": 1073741824, "default": 1073741824}

target is what cocoon vm balloon accepts: NN% of guest memory (0 to 99) or a byte size (2G, 512M, 1073741824), below guest memory. In the reply, target is the size cocoon set in bytes, actual what the balloon holds now when the hypervisor reports it, and default the boot target. Send default back as the target to restore the claim. The call returns before the guest settles. sandboxd keeps no balloon state and never changes a target on its own: the caller sets it and the caller restores it. A hibernate wake and a live move keep the target; an archive wake, a cold VMM restart, a fork and a clone start at the boot target. cocoon serializes the call with captures and live-move rounds.

200 with cocoon’s report; 400 a target cocoon rejects or an unknown field; 404 unknown id; 409 cocoon on this node has no vm balloon, the sandbox's VM has no balloon device, or a hibernated, archived, failed or waking claim or one a hibernate is capturing; 503 sandbox moving to another node; retry during a live move; 500 the cocoon call failed or outran its 30 s budget, or cocoon found the guest paused (a capture or a move round that began after the node’s checks).

PUT /v1/sandboxes/{id}/egress-limit

Auth: root only. Caps a claim’s egress bandwidth in bytes per second, each direction apart, tx guest to destination and rx back:

{"tx_bytes_per_second": 1250000, "rx_bytes_per_second": 2500000}

Zero or an absent field leaves that direction unlimited, so {} removes the cap. It covers the claim’s proxy doors and, in advanced network mode, replaces tun.limits.bytes_per_second for the claim in each direction it sets. Preview, port relay and exec traffic is not capped. On a running claim the change applies at once to open connections, without a reconnect; an uncapped direction can pass one more splice, up to 1 MiB at the old rate. The cap is persisted with the claim, follows it through hibernate, archive, a live move, an archive row and an adoption, and is accepted on a hibernated or archived claim without waking it; its next wake applies it. A running claim’s replica row under archive_failover picks the change up at its next replica capture, which for a claim that hibernates first comes after its next wake. A fork child and a checkpoint branch take their source’s cap. A new claim starts without one; no pool setting gives a default.

204 on success; 400 an unknown field or a negative rate; 404 unknown id; 503 sandbox moving to another node; retry while a live move pauses the claim for its final round; 500 the claims journal could not be written (nothing changed) or the claim’s row could not be rewritten (the node holds the change; resend the same body); 503 the archive registry is down for an archived claim or a hibernated one with a replica under archive_failover: the node already holds the change, and a resend of the same body replaces the row.

GET /metrics

Auth: root only (tenant and operator tokens get 403). Prometheus text format, hand-rendered: pool warm/target gauges, claimed/running/max_running/hibernated/archived/failed/draining/cordoned gauges, sandboxd_evacuating (the claims an evacuation has yet to move), sandboxd_memory_low and sandboxd_anon_bytes (see memory_limit), a per-tenant live-claim gauge (sandboxd_tenant_claims{tenant="…"}, tenants holding live claims only, so the label set is bounded by the node’s claims), sandboxd_config_digest_mismatch on a mesh, claims by tier (warm/clone/cold), wake/hibernate/fork/checkpoint/promote/release/reap counters plus archive/unarchive/archive-delete counters, sandboxd_dns_malformed_total (guest DNS queries dropped as malformed), claim/wake *_seconds_total for average latency, sandboxd_claim_duration_seconds{tier}, a histogram of pool claims by tier, sandboxd_claim_redirects_total, claims this node handed to a peer, sandboxd_wakes_refused_total{reason}, wakes max_running (max_running), disk_reserve (disk) and memory_limit (memory) refused here, whether or not a hand-off ran, sandboxd_wake_handoffs_total, archived claims this node handed to a peer with room instead, and sandboxd_wake_memory_fallbacks_total, archived wakes over memory_limit this node woke itself because no hand-off ran or no peer adopted. On a node with a tun block it adds the packet path’s counters: sandboxd_tun_frames_total{dir} (frames on the claims’ vsock streams, out from the guest and in to it), sandboxd_tun_datagrams_total{dir} (VXLAN datagrams), sandboxd_tun_arp_total{kind} (answered replies to exits, announced gratuitous ARPs) and sandboxd_tun_drops_total{reason} (packets not carried: malformed, not_ipv4, options, fragment, proto, gso, policy, no_exit, unknown_claim, queue_full, size (a packet or GSO segment over tun.mtu, or a GSO frame of more than 128 packets), limit (a claim’s rate, flow or quota limit), send (the underlay refused a send) and sender (VXLAN from an address that is no exit)). The warm hit rate is sandboxd_claim_duration_seconds_count{tier="warm"} over the three tiers’ counts: sandboxd_claims_total also counts forked children and checkpoint branches under clone, and the histogram does not. A claim with volumes includes their attach and mount in its tier. /metrics is a derived ops view; the billing source of truth is the usage journal below.

Usage journal (usage.jsonl)

Always on: every lifecycle transition appends one JSONL event to <data_dir>/usage.jsonl — {"t": <RFC3339>, "ev": "claim|renew|hibernate|wake|fork|checkpoint|adopt|promote|release|reap|archive|unarchive|archive_delete|egress|egress_bytes|vmm_exit|vmm_restart|vmm_failed|move_in|move_out", "id": "sb_…"} plus vm (the VM’s name; absent on renew, adopt, archive, unarchive, archive_delete, egress, egress_bytes, and the reap or release of an archived claim), key (the pool key’s stable hash, claim events), template, net and size (the claim’s template, lane and size tier, claim events, so a consumer can price a sandbox by its shape without the claim record, which is gone after release; size sets the CPU and memory, and template is a label, since promote lets the caller name it), deadline (the deadline in force after this event: claim, renew, wake, unarchive and move_in carry the lease, archive and adopt the retention end, or nothing for an archive kept forever, and reap the deadline it ended under), tenant (the owning tenant, on every event of a tenant’s claim), children (fork) and ref (the promoted template / checkpoint id, the egress host, a vmm_failed reason, or why reconcile ended a claim). egress events, one per egress decision, are written only with egress_usage_events on (default off), and egress_bytes events only with egress_usage_bytes on. An egress event that injected credentials names them in secret (gh,claim:<entry>, never a value). An egress event routed through an upstream proxy carries upstream (its host:port), and with egress_usage_bytes on, egress_bytes events add the connection’s payload tx/rx bytes. With egress_usage_interval_seconds set, a connection or flow that lasts longer writes an egress_bytes event each interval with the bytes since its previous one, then one with the rest, if any, when it ends. Events carry no connection key: sums per claim and per upstream match the single events an interval of 0 writes. A sandbox records at most 10 denied or rate-limited egress decisions per second after a burst of 100, so egress events do not count every refusal. A volume claim also carries volumes, the applied catalog names, and — omitted when empty — volumes_rw, the subset of those names claimed rw, on its claim event, so billing can discriminate write access at claim (mounts and host paths are not billing dimensions); a live attach or detach records no usage event. The file rotates at 1 GiB keeping one .1 backup, so a tailing collector never loses a window silently. A last line a crash cut short is ended with a newline when the node reopens the journal, so it never swallows the next event. A live move records move_in on the target when it starts the guest and move_out on the source when it drops the claim, each with its own node’s vm. Folding rules: fold a sandbox’s events of all nodes in t order. An event ends the current span at its t and enters a state: claim, wake, unarchive, vmm_restart and move_in enter running, hibernate enters hibernated, archive enters archived, and vmm_exit, archive_delete, release and reap enter ended. Other events, vmm_failed and move_out included, change no state, and reap, adopt and archive_delete have the exceptions below. Billable compute, hibernated storage and archived storage seconds are the time in running, hibernated and archived, a cheaper store tier than a hibernated VM’s RAM. vmm_exit is stamped when the node first finds the VMM gone, so an outage before sandboxd runs again (a host that was down) stays billed up to that point. A reap instead ends the current span at min(t, its deadline), or t with none, because t can follow the true end by the reaper’s tick, the VM’s teardown or the node’s downtime. An archive_delete ends the current span at min(t, the last deadline): in archived that is the retention deadline, which a purge trails by the sweep’s tick or the node’s downtime, and a purge can follow a claim that has no end event by a whole retention window. An adopt names a claim a node took over, from a lost node or from its own lost record, with ref the checkpoint it holds. In archived it changes nothing; otherwise it ends the current span at min(t, the last deadline before it) and enters archived at its t. A span is never shorter than zero. A span still open ends at min(now, the last deadline), and one entered by an archive or adopt without a deadline stays open. A claim reconcile drops because its VM record is gone ends with a reap whose ref is vm_gone, and one that cannot re-lock its egress NIC with quarantined; a claim another node holds is dropped with no event, and neither has a replicated claim dropped while the registry is down. The vm name joins cocoon’s machine-level metering ledger for audit cross-checks. With meta_store.usage, the node also lands every line in the cell’s shared usage table, read through GET /v1/usage.

GET /v1/usage

Auth: root only (tenant and operator tokens get 403). Reads the raw usage events of every node of the cell from the shared usage table.

GET /v1/sandboxes/{id}/agent

Auth: the sandbox’s own token; the operator token gets 403. Requires Upgrade: silkd + Connection: Upgrade; answers 101 Switching Protocols and from then on the connection is a byte-for-byte relay to the guest’s silkd, carrying RPCs back to back (see silkd). An open relay holds the sandbox’s idle clock, so a client that keeps a connection warm must close it when idle for idle_hibernate_seconds to apply. A relay whose client vanished without a FIN ends after about two minutes of failed TCP keepalives. Frames sent right behind the request, before the 101, reach silkd in order; the Go and Python SDKs send their first frames that way over plain HTTP, so a proxy in front of a plain-HTTP endpoint must forward them too (Go’s httputil.ReverseProxy drops them; Caddy forwards them). 426 without the upgrade header, 404 unknown sandbox, 409 a failed sandbox, 500 a failed restore, 502 guest unreachable, and the guest errors. A relay of a claim of a replicating pool that is already open when the lease turns overdue is closed, whether or not it has a replica yet.

A hibernated or archived sandbox wakes when the upgrade request arrives, before the 101, unless its last sleep is reader-safe, which the pool’s reader setting provides; what a reader is, what it needs and when the call wakes instead are there. With it, the node answers the 101 before it touches the sandbox, reads the first frame, and serves fs_stat, fs_list, fs_read and fs_pull from a reader VM: the sandbox stays hibernated or archived, no wake usage event is recorded, and the read stamps neither the idle nor the archive clock. An info frame wakes the sandbox, since a reader would answer with its own processes and uptime. The first frame on the connection that is anything else, or any later one, wakes the sandbox and hands the same connection to its silkd with the unread bytes replayed, so a client sees one silkd and no close. When the sandbox sleeps again meanwhile (a new hibernate or an archive), the connection’s next read moves to a fresh reader of the new sleep; when it wakes, the connection moves to the woken sandbox, so reads never come from a stale sleep. Pipelining a request behind a read that has not yet sent its terminal frame is unsupported: when that request moves the connection, the reader closes and the rest of the read’s reply is lost. A read that no reader can serve wakes the sandbox as if the feature were off. Because the 101 has gone out by then, a failure on this path (unknown sandbox, a refused wake, an unreachable guest, an overdue node lease) arrives as a silkd error frame (not_found for an unknown sandbox, else internal) rather than an HTTP status, and the connection ends. A connection served by a reader, or handed over to the woken sandbox, of a claim of a replicating pool is closed when the lease turns overdue, as an open relay is.

GET /v1/sandboxes/{id}/ports/{port}

Auth: the sandbox’s own token, or the root or operator token for a control-plane read (below). Send Upgrade: tcp + Connection: Upgrade (the gate reads Upgrade alone, as /agent’s does); answers 101 Switching Protocols and from then on the connection is a raw byte relay to 127.0.0.1:{port} inside the guest, over the same vsock port_forward preview uses. Nothing of ours frames the stream, so HTTP/1.1, HTTP/2 and any other protocol pass through unchanged. This is how an edge proxy reaches a guest listener on a net=none sandbox, which has no NIC.

Half-close works in both directions: a client write shutdown reaches the guest socket, and a guest write shutdown reaches the client without ending the client’s writes. Each direction stays open until its own EOF. A relay copy error closes both connections and releases the sandbox hold.

With the sandbox’s token, an open relay holds the sandbox’s idle clock exactly like the silkd relay, and a hibernated sandbox wakes transparently — which is why the upgrade header is mandatory: a bare GET must not consume a hibernate snapshot. With the root or operator token the relay is passive, for control-plane reads such as metrics: it never wakes the sandbox and never stamps activity, so a poll neither undoes a pause nor keeps a sandbox from idling. A hibernated or archived sandbox, or one with a hibernate, wake, checkpoint, promote or fork in flight, answers 409 at once; an open passive relay still holds the sandbox against the idle sweep. 400 a port outside 1-65535; 401 missing bearer token; 404 unknown sandbox or wrong token; 409 a failed sandbox or a passive relay to a paused one; 426 without the upgrade header; 502 when the guest cannot be reached on that port — no listener, or a hibernated sandbox whose restore failed. The node log carries which; and the guest errors.

POST /v1/sandboxes/{id}/exec

Auth: the sandbox’s own token, or the root or operator token by id, which wakes a hibernated or archived sandbox like the sandbox’s token. A buffered exec for clients that cannot hold an upgraded connection — plain JSON in, plain JSON out, so it multiplexes over HTTP/2 through a TLS proxy:

{"argv": ["node", "-v"], "cwd": "/work", "env": {"CI": "1"}, "timeout_seconds": 60}

→ 200 {"exit_code": 0, "stdout": "v22.23.2\n", "stderr": ""}. The command runs to completion (no stdin, no streaming, no detach — use the relay for those); timeout_seconds 0 means no limit beyond the request itself. When the node gives up on a started command (timeout, client gone, output cap, buffer budget) it sends the child a kill through silkd, bounded to 5 s, before answering. Output comes back as JSON strings: bytes that are not valid UTF-8 are replaced with U+FFFD, so binary output belongs on the relay. 400 empty argv, negative timeout_seconds, an unknown field, or a silkd bad_request — a command that is missing or not executable and a cwd that does not exist are the caller’s, not 502s; 401 missing bearer token; 404 unknown sandbox or wrong token; 413 when stdout+stderr exceed 8 MiB; 503 with Retry-After: 1 when the node’s buffered-exec memory is exhausted (64 MiB of output across every buffered exec in flight; the command is killed); 502 guest unreachable or any other silkd error; 409 a failed sandbox; 500 a failed restore; the guest errors; 504 when the command outlives timeout_seconds. A client that hangs up first gets no reply and the command is killed. A hibernated sandbox wakes transparently like on the relay.

GET /v1/sandboxes/{id}/owner

Auth: the sandbox’s own token, or the root or operator token by id. Answers {"owner_addr": "host:port"} when this node owns the sandbox, 404 otherwise; a tenant token is a wrong sandbox token here. With archive_failover, a node that does not hold the sandbox adopts it, when the token’s SHA-256 matches the row’s or the caller is root or the operator, and its row names an owner whose node lease expired, or whose owner released it in an evacuation or a wake it had no room for, and answers 200; with ?room=1, which the owner of such a wake sends, it adopts only while it could admit the wake itself; 503 with Retry-After when the meta store is down. A row whose owner’s lease is live is never adopted unless the owner released it: 404. A draining or fenced node, or one that lifted its fence less than 8 seconds ago, does not adopt: 404; a cordoned one adopts. A node whose lease is overdue answers 404 for the replicated claims it holds, so the Lookup scatter moves on. With client_advertise configured, owner_addr is that node’s full HTTP(S) origin instead; the same contract applies to claim and fork responses. Used by the SDK’s Lookup scatter. A claim the root or the operator adopted keeps the row’s token hash in place of its token: its own token still authorizes it on this node and any later adopter, an empty token never does, and the first call with its token remembers it (persisted at the next claims commit), after which root’s or the operator’s GET /v1/sandboxes/{id} returns it. The hash is never served.

GET /v1/info

Auth: root or operator (tenant tokens get 403). Node pools, promoted templates, claim count, the node’s own address, and mesh peers:

{"pools": [{"key": {"template": "base:24.04", "net": "none", "size": "small"},
            "warm": 4, "refilling": 0, "target": 4, "golden": true}],
 "templates": [{"key": {"template": "myproj:v1", "net": "none", "size": "small"},
                "content_digest": "sha256:…", "tenant": "acme",
                "created_at": "2026-07-06T00:00:00Z", "cpu_count": 1,
                "mem_total_bytes": 536870912, "labels": {"v1": "sha256:…"}}],
 "claimed": 2,
 "running": 1,
 "max_running": 12,
 "hibernated": 1,
 "archived": 0,
 "failed": 0,
 "at_capacity": true,
 "at_capacity_reason": "exchange full",
 "advertise_addr": "10.0.0.5:7777",
 "peers": ["10.0.0.6:7777"]}

advertise_addr is the address the node hands clients: its client_advertise when set, else its advertise_addr. It is the same value a claim reports as owner_addr, and it is omitted when the node advertises no host.

hibernated counts claims whose VM is currently hibernated, archived those checkpointed to the store with the local VM dropped (see archive tiers), failed those whose VMM is down and was not restarted (omitted at zero); all are included in claimed. running counts claims that hold a VM, with a wake in flight counted, and max_running is the cap on it (omitted when off). A node drained via POST /v1/drain additionally reports "draining": true, a node cordoned via POST /v1/cordon "cordoned": true, and during an evacuation evacuating (claims left to move, omitted at zero). When refill is parked because the node cannot start another VM, "at_capacity": true distinguishes a full node from one still filling its warm target, and at_capacity_reason carries the engine’s capacity reason, or disk below reserve or memory above limit, which stay stable for callers that tell them apart. Both capacity fields are omitted while refill is not capacity-blocked. Each VM holds a console pty, so a node stops at kernel.pty.max (4096 by default) with no space left on device; raise it past the VM count you plan to run.

golden reports whether the pool’s snapshot exists (refill can clone); warm at target with golden: true means warm claims are served in sub-millisecond time.

templates lists the promoted templates this node holds: each one’s full key, the content_digest its promote returned, created_at of its last promote, the cpu_count and mem_total_bytes of its size tier, its labels when any are set, and tenant when a tenant promoted it (omitted for root). A key a configured pool now owns is left out, since claims of it clone the pool’s golden. On a shared checkpoint store a node lists what it promoted plus what was in the store when it started, and a template another node deletes later stays listed, and may redirect a claim to a 404, until this node restarts.

GET /v1/peers

Auth: node API token (root or tenant) or the operator token. Answers {"peers": [addr, …]} — the cluster’s other node addresses, for the SDK’s redirect follow and Lookup scatter. Cluster topology, not operator state, so a tenant token may read it (unlike GET /v1/info).

GET /healthz

Unauthenticated liveness probe; answers ok.

Operational notes