All bodies are JSON. Four token kinds:
api_token: every endpoint except the agent relay
and the preview mint, which need the sandbox’s own token. A set
api_token_next is root too, for a
rotation: the same
endpoints, and the audit caller stays root.operator_token,
for a public control plane that must not hold cluster configuration power.
It acts with root’s authority, with no tenant scoping, on exactly
POST /v1/claim, GET /v1/sandboxes, GET /v1/sandboxes/{id},
GET /v1/sandboxes/{id}/stats, GET /v1/stats,
POST /v1/sandboxes/{id}/release|hibernate|wake|renew|balloon,
GET/PUT/PATCH /v1/sandboxes/{id}/env, GET /v1/sandboxes/{id}/ports/{port},
POST /v1/sandboxes/{id}/exec, GET /v1/sandboxes/{id}/owner, the
tenant API, GET /v1/info and GET /v1/peers. Every
other authenticated endpoint answers it 403
operator token not allowed on this route; it is never a sandbox token, so
the agent relay refuses it too, and a claim that names volumes answers it
403 operator token cannot claim volumes. Unset or empty, it is off. A set
operator_token_next is the operator too, on the
same endpoints, and the audit caller stays operator.
Scope: it reaches every claim’s data — it reads any claim’s token by id and
execs in any claim. Through the tenant API it holds every tenant’s
authority: it can mint or re-key any tenant and then act as it, including a
claim that names volumes that tenant may mount and fork, checkpoint and
preview with a claim token. It holds no node or cluster configuration power. With audit_log on, its by-id token read records op token.PUT /v1/sandboxes/{id}/instance-metadata, PUT /v1/sandboxes/{id}/tun, PUT /v1/sandboxes/{id}/egress-limit, POST /v1/sandboxes/{id}/balloon, GET /v1/info, GET /v1/stats,
PUT /v1/pools, POST /v1/config/reload, the tenant API,
POST /v1/sandboxes/{id}/move and the peer /v1/moves/{id} calls, PUT/DELETE /v1/volumes/{name}, POST/DELETE /v1/drain, POST/DELETE /v1/cordon,
GET /metrics, GET /v1/usage, GET /v1/checkpoints/{id}/blob)
answer a tenant token 403 root token required — authenticated but not authorized; an unknown
token stays 401. On a node with a shared
meta_store, a tenant token the node
has not cached answers 503 while the database is unreachable; retry.Endpoints below that say “node API token” accept root or tenant unless
marked root-only; the operator token only where the endpoint names it. Errors are {"error": "message"} with the status codes
listed per endpoint. Request bodies are capped at 1 MiB; a larger one is
rejected with 400 and an error beginning invalid request body, and so is
a body with a field its endpoint does not declare.
PUT /v1/sandboxes/{id}/instance-metadata
is the exception: it takes one raw JSON object of at most 4 KiB.
Wake, the agent relay, the port relay and exec restore a sleeping sandbox before they reach its guest, so they share these answers:
max_runninglease lost; retry for a replicated claim while its node’s lease is
overdue (running claims)sandbox moving to another node; retry while an
evacuation has yet to move it, or while a
live move pauses it for its final roundarchive registry down; retry for an archived claim under
archive_failover while the meta store is downtenant store down; retry for an archived claim whose tenant the node has
not cached while the meta store is downA claim, fork or checkpoint branch answers 429 when the node refuses it for capacity, with the pool state unharmed. The causes:
max_claims or max_runningmax_claims: the caller’s
for a claim or branch, the parent’s for a forkPOST /v1/drain) or cordoned
(POST /v1/cordon)disk_reservememory_limitreplicate_secondsOn a cluster, a claim without volumes, no_redirect or require_promoted
that meets one of these redirects to a peer with warm capacity or, failing
that, one that serves the pool; see POST /v1/claim. Wake and
the guest-reaching verbs have their own 429 causes, listed with each.
Auth: Authorization: Bearer <api_token> (when configured), a tenant token,
or the operator token, which claims as root but never with volumes (403).
{"template": "base:24.04", "net": "none", "size": "small",
"ttl_seconds": 300,
"volumes": [{"name": "imagenet"}, {"name": "weights", "mount": "/models"},
{"name": "scratch-db", "mode": "rw"}],
"claim_ref": "namespace/workload", "metadata": {"team": "a"},
"on_expire": "archive", "no_redirect": false, "require_promoted": false,
"egress": true,
"env": {"MODE": {"value": "prod"}, "GW_KEY": {"value": "Bearer …", "guest": false}}}
net defaults to none, size to small; the pool key is
(template, net, size)ttl_seconds 0 means the server default (5 minutes); capped at 24h. The
owning node reaps the sandbox after the TTL even if the client vanishesclaim_ref is an optional opaque caller reference echoed by the scoped
sandbox index, at most 512 bytes of printable UTF-8 like a metadata value;
the aggregated apiserver uses <namespace>/<name>metadata is an optional map of caller labels, echoed and filterable by
the sandbox index: at most 16 pairs, each key 1 to 128
bytes of printable ASCII without = or & (list filters encode pairs as
key=value, and a query string joins pairs with &), each value at most
512 bytes of printable UTF-8, and at most 4 KiB as JSON, so quotes and
backslashes count twice. Labels are fixed at claim time; no verb edits themon_expire is what the node does when the lease ends: destroy, the
default, or archive. archive hibernates a running sandbox and archives it
to the checkpoint store, whatever the pool’s archive_after_seconds. The
archive is kept for archive_delete_after_seconds, or forever when that is
0, and the next call that reaches the guest restores it with a fresh lease
of the claim’s length, so its next expiry archives it again; a call that
wakes it between the hibernate and the export grants that fresh lease too.
A failed hibernate or export at expiry keeps the claim and retries it a
minute later; it never falls back to destroy. A failed claim (failed set, see
dead VMMs) is destroyed at expiry even
under archive. A volume claim or an egress-lane claim
cannot hibernate, so archive on one answers 409 with the same error as a
hibernateno_redirect is set by the SDK when retrying at a redirect targetrequire_promoted provisions only from a promoted template: a claim carrying
it never takes a warm VM or cold-boots, and answers 404 unknown template when
no record holds the key, a key a pool serves included. A redirect to a template’s owner sets it; copy it into the
no_redirect retry so a target whose gossip view is stale refuses instead of
cold-booting an image named like the templateenv is the claim’s own environment, at most 64 entries: each name a shell
identifier of at most 128 bytes, each value at most 8 KiB, a value and an
inject query or placeholder with no control characters other than tab, and names, values and inject fields at most 64 KiB in total. An entry holds only value, guest and inject; any other
member, "guest": null, a guest: false entry with an empty value, or
missing (the node’s own marker) answers 400, so a mistyped flag never lands
a host-only value in the guest and a read sent back never blanks one. guest (default true) delivers the entry into the guest; a
guest: false entry never enters it and only feeds the node’s egress
secrets, or, with inject
({"hosts": [...], "header": "...", "placeholder": "..."} with placeholder
optional, or {"hosts": [...], "query": "...", "body": true, "placeholder": "..."}
with body optional), becomes a
claim credential the proxy sends to those
hosts. A bad entry answers 400, and an inject host the claim’s egress does
not intercept answers 400 with the claim’s VM released; see
claim envegress false claims with no egress policy, whatever the
pool’s (or, for a promoted template, its source pool’s): the guest’s doors are
never bound. Omitted or true keeps the policy the claim would getvolumes is an ordered list of at most eight unique catalog names. mount
defaults to /volumes/<name>; a custom value must be absolute and clean,
outside the guest OS tree, unique, and non-nesting within the request.
mode is "ro" (default, omitted) or "rw"; "rw" requires the catalog
entry’s writable: true (see deploy).
attach_only: true attaches that entry without mounting it and requires an
empty or omitted mount. Mounted and attach-only entries can share a listSuccess:
{"id": "sb_…", "token": "…", "deadline": "2026-07-06T00:05:00Z",
"owner_addr": "10.0.0.5:7777", "template_digest": "sha256:…", "net_route": "relay",
"volumes": [{"name": "imagenet", "mount": "/volumes/imagenet"},
{"name": "weights", "mount": "/models"},
{"name": "scratch-db", "mount": "/volumes/scratch-db", "mode": "rw"}]}
A claim cloned from a promoted template carries template_digest, the exact
export generation fetched for that clone. It is absent for configured pools,
cold image boots, forks and checkpoints.
net_route says how the guest reaches the network: relay through the host
proxy behind the guest’s bound doors, direct over an egress-lane NIC the node
does not lock, or none (a claim with no effective egress policy, a claim with
"egress": false, or a locked egress-lane NIC with no policy). silkd hands its children the
http_proxy, https_proxy and no_proxy of its unit unless the lane is
direct; a guest agent of another kind reads net_route and hands its processes
http://127.0.0.1:3128 when it is relay. "egress": false closes the doors, not an unlocked NIC, so
such a claim on a direct lane still reaches out directly.
A claim branched directly from a checkpoint additionally carries
"from_checkpoint": "ck_…" — the lineage edge for reconstructing the
checkpoint tree. Fork children do not repeat that edge: they branch from the
parent sandbox, not directly from its checkpoint.
volumes reports the names, effective mounts, and (rw only) mode applied
and persisted at finalization — mode is omitted from the echo for ro
entries, matching the request shape. sandboxd attaches each disk, waits for
the device settle — a serial match under /sys/block, then the /dev/<name>
node itself, since the kernel publishes sysfs before devtmpfs creates it — up
to 2 seconds total, and mounts the filesystem — as ext4 first, then with mount’s
own probe if that is refused; read-only, unless the entry requested and was
granted rw — before returning. A mount that reports
not a valid block device is retried within the 10-second setup budget,
since the new node can exist before the kernel opens the disk. A custom mount
may shadow an existing populated guest directory for the claim’s life.
"attach_only": true stops after the attach for that entry. The device appears
in the guest without a mount. Responses preserve attach_only: true and omit
mount:
{"id": "sb_…", "token": "…",
"volumes": [{"name": "imagenet", "attach_only": true},
{"name": "scratch-db", "mode": "rw", "attach_only": true}]}
Find each device by its virtio serial, which is the catalog name — poll this,
not a one-shot lookup: the claim can return before the guest enumerates the
device (typically within ~100ms, never guaranteed), and the /dev/<blk> node
can lag the serial match the same way it does for an eager mount’s device
settle. Confirm both before mounting:
device=
tries=0
while [ "$tries" -lt 200 ]; do
for serial in /sys/block/*/serial /sys/block/*/device/serial; do
[ -r "$serial" ] || continue
[ "$(cat "$serial")" = scratch-db ] || continue
block=${serial#/sys/block/}
candidate="/dev/${block%%/*}"
[ -b "$candidate" ] && { device="$candidate"; break 2; }
done
tries=$((tries + 1))
sleep 0.01
done
[ -n "$device" ] || { echo "scratch-db device not ready" >&2; exit 1; }
printf '%s\n' "$device"
Then mount it however the workload needs, retrying a mount that reports
not a valid block device for a short time. A ro entry is attached
--readonly and stays read-only at the guest block layer no matter who
mounts it, so the guarantee does not depend on your mount flags.
The whole consistency contract moves to the caller with the mount:
rw claim and clears
none, because it cannot verify your unmount and therefore promises nothing.
The flush is yours too: release runs no unmount and no sync, so whatever
the workload left unsynced is discarded with the VM. Releasing without a
clean unmount of your own leaves the image exactly as a filesystem crash
would. The next eager ro claim then fails at mount time with a 500 —
loud and attributable to the claim that skipped its unmount — instead of
the marker’s 409. Recovery is any rw cycle that replays the journal.rw crash is not cleared by attach-only
cycles, and still refuses eager ro claims with 409 until a rw claim
releases cleanly.What still protects other claims is unchanged: admission excludes an
attach-only rw claim against every other claim of the name (and readers
against a writer), an attach-only ro claim of a marker-bearing image is
refused with 409, and a claim with volumes still refuses checkpoint, fork and
hibernate. An attach-only claim costs one disk attach per volume — no device
settle, no mount round-trip — and release costs nothing at all: removing the
VM closes the devices.
Redirects (mutually exclusive with the fields above) name peers to retry
at — sent on a warm miss with warm peers, when the node lacks a golden for
the key but gossip names a template owner, and when a claim without volumes,
no_redirect or require_promoted meets an admission refusal
and a peer reports warm capacity or, failing that, serves the pool:
{"redirect": ["10.0.0.6:7777", "10.0.0.7:7777"],
"require_promoted": true}
Retry the same body (+no_redirect: true) at each candidate until one
answers. Preserve require_promoted: true when the redirect carries it, as a
redirect to a template’s owner does; warm-candidate redirects omit the field.
A volume claim may consume an ordinary warm VM; candidate ranking, the
promoted-template intersection, and the no_redirect/require_promoted retry
follow the fleet-wide rule in cluster.
Redirect responses never carry volumes.
A tenant token claims the same way; the sandbox is stamped with the tenant
name (attributed in the usage journal and counted against the tenant’s
max_claims). A catalog access list may restrict an entry to named tenants;
an unknown and a forbidden volume return the same error text.
Errors: 400 unknown template axis, claim_ref or metadata over a bound (the
message names it), a bad env entry, an unknown on_expire, invalid/duplicate volumes,
an attach-only entry with a nonempty mount,
mode: "rw"
against a non-writable entry, or a volume that is unknown or forbidden (the
latter two are deliberately indistinguishable), or
bad body; 401 bad api token; 409 egress requested on a node without an egress
attachment, a writable name already claimed in a conflicting mode (volume
busy — a live writer excludes every other claim for that name, live readers
exclude a writer), or a ro claim against a writable image left dirty by an
unclean rw release (needs recovery — one rw claim must replay and cleanly
release before ro claims resume; see
deploy); 429 an admission refusal; 500 provisioning failed.
Auth: node API token (root or tenant). Lists the fleet catalog entries the caller may use, without host paths or holder addresses:
{"volumes": [{"name": "imagenet", "default_mount": "/volumes/imagenet",
"size_bytes": 214748364800, "available": true, "nodes": 3}]}
Root sees the gossiped union, including peer-only entries (available: false);
a tenant sees only entries this node declares locally and whose access list
permits it. nodes counts members advertising the name; size_bytes and
available are a best-effort stat of the answering node’s image. Membership is
eventually consistent by one gossip tick.
writable is the entry’s catalog configuration, fleet-uniform like the access
list; the field is emitted (as true) only for a writable entry this node
declares and omitted otherwise (peer-only rows never carry it).
Requires the configured root API token. A tenant or operator token or a node without an
api_token returns 403. Sandbox, unknown, and missing tokens return 401 when
api_token is configured. The URL supplies the name. Unknown body fields,
duplicate fields, and trailing JSON return 400.
{"path": "/srv/datasets/data.img", "directio": "off",
"writable": false, "tenants": ["acme"]}
name matches [a-z][a-z0-9_-]{0,19} and cannot start with cocoon-.path uses static config’s absolute-path validation. When the node sets
volume_roots, the cleaned path must equal or sit
under one of them. Registration does not open the backing or check its type
or availability. A missing backing keeps its registration but cannot be
attached.directio is on, off, or auto, and defaults to off.writable defaults to false and controls writable claim admission.tenants is the entry’s access list.Success is 204. An identical normalized registration is idempotent. A different
binding, changed fields, a static name or cleaned path collision, or a name a
claim still holds returns 409. A PUT that adds a name and every successful
DELETE, also of an absent name, write an op:"volumes" audit record naming the
volume.
Invalid fields return 400. GET keeps its existing projection without host paths
or tenant lists. Runtime entries support the existing live volume API.
Persistence failure returns 503 with Retry-After. Retry the same request;
after a restart, either state can load.
The node stores metadata in data_dir/runtime-volumes.json independently of
sync_claims and loads it before claim recovery. New records use mode 0600.
Malformed or conflicting metadata, including a saved path outside volume_roots,
prevents startup. The catalog is node-local with one sandboxd writer.
Sandboxd does not create, check, or delete backings. Keep them outside
data_dir: sandboxd clears its readers, goldens, moves, fork-*,
checkpoint-staging and template-staging entries.
Static and runtime entries share one tenants rule. An omitted or empty list
permits every authenticated scope. A nonempty list permits only claims owned by
the listed tenants, and root-owned claims can use any entry. A listed name must
match the tenant API syntax: runtime registration rejects one that does not,
and static config fails to load. A valid name for a tenant that does not exist
yet grants nothing until the tenant is created.
Auth and name validation match PUT. Deletes only runtime registration metadata.
Static names and names referenced by a claim, an in-flight operation, or
cleanup return 409. Otherwise DELETE writes the absence and returns 204, also
for an absent name.
Persistence failure returns 503 with Retry-After. Retry the same DELETE,
even if GET no longer lists the name; a restart can restore the old registration.
Returns the persisted desired set (volumes_want), confirmed applied set
(volumes), per-volume states, and converging. Auth requires a client API
token. Root can access every sandbox; a tenant can access only its own.
Sandbox tokens are rejected when authentication is enabled. The owning node
handles the request without redirecting.
{
"volumes_want": [{"name": "imagenet", "mount": "/datasets/training"}],
"volumes": [{"name": "imagenet", "mount": "/datasets/training"}],
"states": [{"name": "imagenet", "state": "attached"}],
"converging": false
}
All three arrays are present, including when empty. A state is attached
when the volume is in both sets, pending when desired but not yet confirmed,
or detaching when applied but no longer desired. An interrupted device call
can leave the actual guest state different from the confirmed set. A failed
PUT or PATCH adds a top-level error naming the refusal, and the affected
state may carry its own error. converging is true only while a request worker is running.
pending with converging: false requires an explicit retry; there is no
background convergence. GET retains the last safe per-volume error for the
current daemon lifetime. After daemon restart, unfinished states report an
interrupted operation that needs retry. Successful retries clear resolved errors.
The sandbox index exposes both sets for operator inspection.
Replaces the full desired set and converges synchronously. Auth matches GET.
The owner tenant’s catalog access list applies to newly admitted names, including
for root. Retaining, repairing, or removing held names does not reauthorize
them. A new image must exist on the owning node. The request must include a non-null volumes
array. An explicit empty array removes all volumes.
{"volumes": [{"name": "imagenet", "mount": "/datasets/training"}]}
Mounted read-only, mounted writable, attach-only read-only, and attach-only
writable entries can change live, including mixed lists. The claim-time name,
mount, and eight-volume limits apply. An omitted mount defaults to
/volumes/<name> unless that entry has attach_only: true. Responses carry
that flag directly, so a GET target can be sent back with PUT. Omitting the flag
or sending false does not inherit an existing attachment’s shape.
To change a held name’s mount, mode, or attach_only, first remove the name and
wait for successful convergence. An identical target can be retried. Claims
with on_expire: archive must change that policy before adding a volume.
A claim that holds volumes is not replicated. Its replica retires on the next replication pass, and a replica capture that races a live addition is refused. The sandbox must be running. Paused, archived, and failed sandboxes return 409 without waking. A failed sandbox holding volumes is release-only after its VMM exits; PUT with an empty target cannot restore its processes. Missing, invalid, or sandbox bearer tokens return 401. Another tenant’s sandbox and an unknown id return 404. Invalid, inaccessible, or non-local entries return 400. Removed tenant tokens are rejected at authentication. Root can still repair or remove held entries, but admitting a new name for a removed tenant returns 403. A conflicting writer or dirty image returns 409.
Changes the desired set by name, then converges synchronously. Auth, validation,
and responses match PUT. Omitted add or remove arrays leave that part
unchanged. An empty patch retries the current desired set.
{"add": [{"name": "imagenet", "mount": "/datasets/training"}], "remove": ["old-data"]}
Adding an identical entry or removing an absent valid name is harmless.
Duplicate names, a name in both arrays, or changes to a held entry’s mount,
mode, or attach_only return 400. Removal refuses busy or foreign mounts and uses neither
lazy nor forced unmount. A foreign mount, another bind mount, or a completed
ordinary unmount with nonzero exit returns 409 with the guest mount path and
a safe per-volume reason. Guest transport failures remain server errors;
timeouts return 504 and cancellation returns 408.
A mounted writable attachment marks the image dirty before the guest can write.
Live removal and release clear the marker only when the unmount of the mount at
the volume’s path succeeds, the mount was the filesystem’s root and writable in
both its mount and superblock options (ext4’s emergency_ro after a filesystem
error counts as read-only), and no other mount of its device remains. Live
removal finds the device by its serial and refuses any other mount of it;
release identifies the mount by its path alone and does not check which device
it is. These checks read silkd’s own mount namespace, so a mount of the device
in another namespace, or one moved there, is not seen. Live removal also needs
a confirmed host detach. A missing device or pre-unmounted filesystem does not
prove clean unmount. The marker remains on uncertainty and blocks subsequent
readers. Release waits up to 30 seconds for each writable volume’s unmount, and
a guest that does not answer within that keeps every remaining marker.
Attach-only writers have no automatic dirty-marker protection. The caller owns
mounting, flushing, unmounting, recovery, and fencing before reuse.
PUT and PATCH return the GET response shape on success. A convergence error
returns a non-2xx status with the same view, so callers can inspect desired,
applied, and per-volume state; semantic validation 400s carry it too. A
malformed body, a missing volumes array, authentication and an unknown sandbox
carry only error. A failed request may already have persisted its target and
completed some device changes. GET reads the last confirmed set, rather than
probing the devices. Resolve guest mount references or runtime failures before
retrying. Retry the same PUT to converge, or send a new
target to abandon unfinished additions. Concurrent requests serialize per
sandbox without blocking renewals or new exec, file, and port connections.
Before applying a target, sandboxd journals VolumesWant beside Volumes,
storing it only while it differs from Volumes. A
superseding request may first clean up an older desired-only attachment while
the old target stays recorded. If that cleanup fails, the response retains the
old view; retry the same PUT to finish accepting the new target. Admission
holds cover the deduplicated union of both sets, including during restart and
teardown; a teardown keeps them until the VM is confirmed gone. Failed or
uncertain operations retain their holds until successful cleanup. Startup
reserves those holds without replaying attachments. Release cancels convergence and transfers the held union to teardown without waiting
for a blocked device call. Capture operations (hibernate, fork, checkpoint, and
promote) are refused while either set is nonempty or a convergence request
still owns admitted volume holds.
Guest device calls honor cancellation and have a 30-second ceiling per volume, and a
cocoon attach or detach can take 30 more to exit; a
caller that disconnects cancels the run, leaving interrupted states to retry. Journal writes
are atomic. Host-crash durability follows sync_claims.
Live convergence requires Cocoon v0.7.2 or newer for idempotent matching attach and absent detach. PUT and PATCH reject an older release, or a runtime whose version cannot be queried, with 503 before changing intent, holds, or devices; a dev build is assumed current. sandboxd caches the first successful version read, so restart it after upgrading cocoon. GET and release remain available.
Catalog names must keep the same backing image while held, including pending
holds. Publish changes under versioned names such as dataset-v1 and
dataset-v2. The journal cannot detect a catalog name repointed to a new
image.
A replacement first unmounts the old disk, then mounts the new disk. Pause reads and close files, mmap, cwd, and bind mount references during this gap. Existing independent guest processes and sessions continue. See the same-sandbox replacement tutorial.
Auth: the sandbox’s own token, or the root or operator token for cleanup by id.
Destroys the VM. 204 on success, 404 for an unknown id or wrong token, 500 when
the VM survives removal: the claim is already gone and the node retries the
removal, 503 archive registry down; retry for a claim that holds a registry
row while the meta store is down; resend it.
Releasing an already-gone sandbox is 404 — the SDK treats it as success.
Auth: the sandbox’s own token, or the root or operator token by id (like
release). Atomically snapshots the VM and stops it,
freeing its memory; the next agent access restores it transparently
(sessions, processes, and memory state intact — cocoon’s hibernate keeps the
snapshot point and the stop coincident), except a read-only file call that a
reader serves without waking it. Idempotent on an already-hibernated
sandbox. The TTL keeps running: a hibernated sandbox is still reaped (VM and
snapshot) at its deadline, unless it claimed on_expire: archive or its pool
archives. When to hibernate is the caller’s policy — the
node only provides the transition. 204 on success, 404 unknown id or wrong
token, 409 on the egress lane or when volumes are attached (neither kind of
sandbox hibernates; see egress) or on a failed sandbox, 429 while the disk is under
disk_reserve, 503 lease lost; retry for a replicated claim that
is not archived while its node’s lease is overdue, even one already hibernated.
Auth: the sandbox’s own token, or the root or operator token by id. Restores
a hibernated (or archived) sandbox and leaves it running — waking is
otherwise only a side effect of the next agent access, so this is the
explicit form for warming a sandbox ahead of use. Idempotent on one already
running. A hibernated sandbox retains its existing deadline, unless it is past its lease
and archives at expiry, which starts a fresh lease. An archived
sandbox instead uses archive_delete_after_seconds as its retention deadline
while stored, and waking it starts a fresh lease of the length the claim was granted (the server default when it asked for none). A
failed sandbox is cold-booted from
its own disk (memory is lost) and its restart budget resets. A hibernated
sandbox whose guest does not answer within the wake deadline after two restores
of its snapshot is cold-booted from its own disk instead: memory is lost,
restarts moves, and the snapshot is dropped. The boot records vmm_restart
when it starts and wake, with the lease, when the guest answers; a boot that
fails after its start records hibernate, and the claim stays hibernated. A
restored VM that cannot be stopped is not cold-booted: the wake fails and the
claim stays hibernated until the next wake restores the snapshot again. An
archived sandbox whose guest does not answer after two clones of its archive is
cold-booted from the second clone’s disk: memory is lost, restarts moves, and
the wake records unarchive once it commits; a cold boot that fails removes
the clone, and the claim stays archived. The
same holds for a wake that an agent access triggers. 204 on
success, 404 unknown id or wrong token, or an archived sandbox the node had no
room for that it handed to a peer
(look it up again), 409 a failed egress-lane or volume sandbox, or one whose VM
record is gone, 429 when max_running or disk_reserve refuses the wake, 500
when the cold boot or a restore fails, and the guest errors.
Auth: the sandbox’s own token, or the root or operator token by id. Resets the
lease to ttl_seconds from now, so a client holding a sandbox longer than its
claim asked for does not have to claim a new one:
{"ttl_seconds": 3600, "on_expire": "archive"}
→ 200 {"deadline": "2026-09-22T12:00:00Z"}. The grant, not the request, is
authoritative: ttl_seconds 0 means the node default and anything past the
24 h cap is clamped, and the reply always carries what was granted. An absent
body means the same as ttl_seconds 0. The new lease also becomes the one a
wake from the archive grants again. A renew can shorten a lease as well as
extend it. on_expire switches the expiry action to archive or back to
destroy; absent keeps the current one. It is refused as on a claim for a
sandbox that cannot hibernate. A renew stamps no activity, so it neither restarts
the idle clock nor stops an idle hibernate.
An archived sandbox is refused: off-node, its deadline is the store’s retention
window rather than a lease, and a TTL written over it would schedule the
checkpoint for deletion. Wake it first. A claim whose tenant was
removed runs to its deadline but cannot extend it: its own
token gets 403, the root or operator token still renews. 400 a malformed body; 401 missing
bearer token; 403 the claim’s tenant was removed; 404 unknown id or wrong
token; 409 archived, or being archived right now; 503 the archive registry is
down, for a hibernated claim with a replica under archive_failover: the node
already holds the new lease, and a resend replaces the row; 503 tenant store
down; retry for a tenant the node has not cached while the meta store is down.
Auth: the node api_token (Bearer) — forking creates node resources, like a
claim. The sandbox’s own token rides in the body as the ownership proof; the
root api_token with an empty body token acts by id as root:
{"token": "…", "count": 2, "ttl_seconds": 300, "claim_ref_prefix": "team-a/"}
claim_ref_prefix, when set, records each child under prefix + child id as
its claim_ref, so a control plane that names claims <namespace>/<name>
addresses the children by name; without it children carry no claim_ref.
The prefix is at most 512 bytes of printable UTF-8, as claim_ref is.
Children inherit the parent’s metadata, on_expire and "egress": false; an
on_expire in the body overrides the inherited action.
Children do not inherit the claim env: a guest env file captured from the parent
is rewritten empty in each child, and the parent’s host-only entries are not
copied; set one with PUT /v1/sandboxes/{id}/env.
Clones the sandbox into count fresh claims (1 up to the node’s
max_fork_count, default 16). Memory, disk, and guest
state (sessions, processes, tmpfs) duplicate at the fork point; cocoon’s
clone reseed gives every child a distinct machine identity. Children get
their own lease — ttl_seconds (0 = server default), never the parent’s
remainder. A running parent is snapshotted in a brief pause window; a
hibernated parent forks from its existing memory image without waking.
All-or-nothing: on error no child survived. 200 with one claim per child:
{"children": [{"id": "sb_…", "token": "…", "deadline": "…", "owner_addr": "…"}]}
Children inherit the parent’s tenant and count against its max_claims,
whoever calls. 400 invalid count or body, a claim_ref_prefix over its bound or
an unknown on_expire; 401 bad api token; 403 the parent’s tenant was removed
(sandbox token); 404 unknown id or wrong sandbox token; 409 egress-lane or volume
parent (neither forks, checkpoints, or promotes; see egress), a
failed parent, or an archived parent (POST /wake or an exec wakes it first);
429 an admission refusal; 503 lease lost; retry for a replicated parent while its node’s lease is overdue, tenant store down; retry for a sandbox token whose tenant the node has not cached while the meta store is down.
Auth: like fork — node api_token in the header, the sandbox’s own token in
the body; the root api_token with an empty body token acts by id as
root:
{"token": "…", "template": "myproj:v1"}
Publishes the sandbox’s current state as a node-local template under (template, this sandbox’s net, its size): later claims for that key clone from it, provision-on-demand, with no warm pool. Re-promoting to the same name replaces the template. A hibernated sandbox is promoted from its memory image without waking. 200 returns the template’s full key and immutable content identity. On the default local-disk backend a template is node-local, so a cluster client claims from and deletes on this node (name-based calls route via gossip); a shared checkpoint store makes every node resolve it. Under exactly this key:
{"key": {"template": "myproj:v1", "net": "none", "size": "small"},
"content_digest": "sha256:…"}
content_digest is SHA-256 over a versioned canonical stream of the published
export’s regular files: slash-relative path, byte length, and the SHA-256 of
each 16 MiB chunk of its bytes, ordered lexically. Directory entries, modes,
mtimes, and the template’s ownership/creation metadata do not affect it. The
digest is computed once while promoting, stored beside meta.json (a sibling
digest file on the directory backend, object metadata on S3), and therefore has
identical semantics on both. Re-promoting unchanged export bytes keeps the
digest; changing any exported path or bytes changes it. A re-promote starts the
template with no labels: they describe
the export they were set on, which the new one replaces.
400 invalid name, 401 bad api token, 409 when the name collides with a
configured pool, the template is owned by another tenant, or the sandbox is
on the egress lane, has volumes attached (see egress), has its
tun turned off, has failed, or is archived (POST /wake or an exec wakes it first), 404 unknown id or wrong
sandbox token, 429 while the disk is under disk_reserve, 503
lease lost; retry for a replicated sandbox while its node’s lease is overdue.
Auth: node API token. Removes a promoted template (the query parameters
default like a claim’s: net=none, size=small). A tenant may delete only
templates it promoted — anything else is 404, root deletes anything. 204 on
success, 404 unknown template, 409 when the key belongs to a configured pool
(those goldens are owned by the node config). digest=<content digest> deletes
only while the record still has that digest and answers 412 otherwise, so a
caller that observed an older build never removes the newer one that replaced
it on this node; on a shared store a promote committing on another node at the
same moment is not covered. On a cluster, a node that does not
hold the template but sees an owner in gossip answers 200
{"redirect": [addrs]} — the claim redirect shape — and the SDK retries the
delete at the owner. The retry carries no_redirect=1, mirroring the claim
protocol: a node answering a no_redirect delete speaks only for itself.
Auth: node API token. Replaces a promoted template’s labels on this node (the query parameters default like a claim’s):
{"labels": {"v1": "sha256:…", "team": "a"}}
Labels follow the claim metadata rules — at most 16 pairs,
the same key and value grammar, at most 4 KiB as JSON — and replace the whole
map; {} clears it. digest=<content digest> writes only while the record
still has that digest and answers 412 otherwise, so a writer that observed an
older build never relabels the newer one on this node. They are stored beside the template’s record, so they
survive a restart, are listed by the node that promoted the template or loaded
it at startup, and go with the template when it is deleted; a re-promote starts with none. A tenant may label
only templates it promoted — anything else is 404, root labels anything. 204 on
success, 400 bad labels or body, 404 unknown template, 409 when the key belongs
to a configured pool. The call speaks only for the node it reaches; it does not
follow a redirect.
Auth: root only (tenant and operator tokens get 403). Replaces the node’s desired warm
targets online — no restart, and no live claim’s VM is touched. With
meta_store.cell set, it replaces them on every
node of the cell. On a node with egress policies,
whether a key is pooled decides which policy layers a claim of it
gets; that is settled when the claim is made, so adding or dropping a pool here
changes only claims made afterwards — a key that gains a pool with no egress
block leaves new tenant claims without egress, a key that loses such a pool
gives new tenant claims their egress class’s policy alone:
{"pools": [{"template": "base:24.04", "net": "none", "size": "small",
"warm": 4, "warm_max": 16, "idle_hibernate_seconds": 0}]}
Pools omitted from the list are drained: their unclaimed warm VMs are
destroyed and the pool entry retires. Once such a pool is idle, its golden
is deleted; a pool kept with warm: 0 keeps its golden.
net/size default like a claim’s.
Answers the fresh GET /v1/info payload. 400 bad key, negative warm/idle,
warm_max below warm, a reader pool with warm below 1, idle_hibernate_seconds on an egress pool, a
negative archive duration or an archive_after_seconds not above the pool’s
idle_hibernate_seconds, duplicate pool, or a config-owned egress/warmup/capture_trim/storage/replicate_seconds/reader/egress_upstream_env/tun
field; 401 bad api
token; 409 egress pool on a node without an egress attachment; 503 the
meta_store cannot take a cell’s write.
Auth: root or operator (tenant tokens get 403); a node without api_token answers 400
to a write, since tenants need it. The API is the only way in: config.json has no
tenants field, and a node starts with no tenants. The set changes at
runtime, with no restart:
PUT /v1/tenants/{name} {"token": "…", "max_claims": 50, "egress_class": "desk"}
adds a tenant or changes one; an omitted token keeps the tenant’s stored
one, so a cap change never resends it, while max_claims and egress_class
take the values sent. 204.DELETE /v1/tenants/{name} removes one. 204; 404 unknown tenant.PUT /v1/tenants {"tenants": [{"name": "…", "token": "…", "max_claims": 0, "egress_class": "desk"}]}
replaces the whole set: a tenant left out is removed, and an entry without
token keeps that tenant’s. It answers the GET body. Use it to reconcile
and to import many tenants at once, since every change rewrites the whole
set; a sign-up is a single-tenant PUT, so concurrent sign-ups never
overwrite each other. The 1 MiB body cap holds roughly 8,000 entries; import
a larger set with single-tenant PUTs.GET /v1/tenants?after=<name>&limit=<n> → {"tenants": [{"name": "…", "max_claims": 50, "egress_class": "desk", "claims": 3},
{"name": "…", "claims": 1, "removed": true}], "digest": "…", "next": "…"}.
limit defaults to 1000, at most 10000 (400 outside);next as the following page’s after; it is absent on the last page.digest hold the same set. A node on a
shared meta_store reports none.A name follows the claim name grammar (^[a-zA-Z0-9][a-zA-Z0-9._:/-]{0,62}$;
escape / as %2F in the path) and is the tenant’s identity: claims,
checkpoints and promoted templates belong to a tenant by name, so a name reused
after a removal inherits what the old tenant left. Use an id the control plane
never reissues, such as a user UUID. Tokens must be unique and differ from
api_token. egress_class names one of the node’s egress_classes (400 when
unknown) and gives the tenant’s claims that egress layer; without one they
reach nothing on a pool with a policy. The API references a class and never
defines one. A claim keeps the class its tenant had when it was made.
A change applies to the next request: a removed or rotated token gets 401 at
once. max_claims is a per-node cap (0 = unlimited), so a tenant’s cluster
limit is max_claims × nodes; every usage event of a tenant’s claim carries the
tenant, for per-tenant billing. Lowering max_claims below the live count
refuses new claims and evicts none. A removed tenant’s claims keep running and stay listed for root
(GET /v1/sandboxes?tenant=<name>) until they expire or root releases them,
but they cannot extend: renew and fork answer 403 to the sandbox’s own token,
a hibernated one is never archived, one archived before the removal is not
woken (403) and is deleted when its retention ends, and at the deadline they
are destroyed even under on_expire: archive. The verbs that need a tenant
token (promote, checkpoint, preview, branch claims) are closed to them already.
The node writes the applied set to <data_dir>/tenants.json (0600, each token
kept only as its SHA-256) before it serves, and loads that file at boot. Each
change is an op:"tenants" audit record naming the
tenants added, changed and removed. On a cluster every node needs the change
(see cluster). 400 a bad entry, a duplicate name or token,
a token equal to api_token, api_token_next, operator_token or
operator_token_next, an unknown field, or an unknown egress_class.
Auth: root only (tenant and operator tokens get 403). Re-reads the node’s config file and applies its reloadable settings at once, or nothing. SIGHUP runs the same reload and logs the result. →
{"changed": ["egress_internal_allow", "pools[desktop:v13 none small].warmup"],
"ignored": ["pools[desktop:v13 none small] targets (PUT /v1/pools owns them)"]}
changed lists the settings that now apply. ignored lists differences the
file holds for pool targets, which PUT /v1/pools owns.
Which fields reload, and what happens to live claims and goldens, is described
in deploy. The status codes:
A refused reload changes nothing.
Auth: root only (tenant and operator tokens get 403). Empties the node for maintenance: claim/fork/branch answer
429 with an error ending node draining (on a cluster a non-volume claim tries a peer redirect first, and
peers stop redirecting here within a tick, because this node gossips no pools),
a live move to this node is refused,
unclaimed warm VMs are destroyed, and live claims keep serving until release
or TTL. Pool ownership is untouched — no pools.json write, no config change.
Answers the fresh GET /v1/info payload; poll claimed to zero to know the
node is empty. Deliberately not persisted: a restarted node serves again. To
keep new claims off a node across restarts, cordon it; a
cordoned node still adopts existing claims, so to keep it empty after a
restart, drain it with ?evacuate=1 again; the
admission table compares the states.
POST /v1/drain?evacuate=1 drains the same way, then moves the claims to the
peers with their state, so the node empties without waiting for the leases to
run out (see evacuating a node). It needs
archive_failover. In the
background, with bounded concurrency, each claim is hibernated if it runs,
archived to the checkpoint store if it is not archived, its registry row is
released, and a peer is asked to adopt it through GET /v1/sandboxes/{id}/owner,
so that peer lists it, and root’s GET /v1/sandboxes/{id} there returns its token once that
node has seen it. A claim the root adopted, whose token this node has not seen, is offered with
the root token; a peer that does not accept the root on /owner refuses it, and the claim stays
here. Open streams to a
claim end when it hibernates. Until a peer holds it, a call that would wake the
claim answers 503 sandbox moving to another node; retry with Retry-After;
after that this node answers 404 and the client looks the claim up again. With
no peer to adopt it, the node takes the row back and the claim stays. A claim
that cannot move stays here
and keeps serving: an egress-lane claim, one with volumes, a failed one, one
whose tenant was removed, one archived before archive_failover was on, one
whose step failed, and every claim of a node without archive_failover. The
answer and GET /v1/info count the claims left in evacuating; once it is
gone, claimed counts the ones that stayed, and the log names each and why. A
claim of a pool without an archive tier, whose on_expire is not archive,
keeps its lease deadline across the move until its first wake, which starts a
fresh lease of the claim’s length; any other keeps the later of its
lease and archive_delete_after_seconds, or no deadline when that is 0. A call
while an evacuation runs changes nothing; a call after one finished starts a
new one over the claims still here.
Auth: root only (tenant and operator tokens get 403). Lifts the drain, stops an evacuation, and kicks an immediate refill. Answers the fresh info payload. A claim an evacuation is archiving finishes its archive and stays here archived; the next call wakes it. A cordon stays in place.
Auth: root only (tenant and operator tokens get 403). Keeps new claims off the node until
DELETE /v1/cordon, across restarts: claim/fork/branch answer 429 with an
error ending node cordoned, whatever max_claims and max_running allow (on
a cluster a non-volume claim tries a peer redirect first, and peers stop
redirecting here within a tick, because this node gossips no pools), refill
stops and unclaimed warm VMs are destroyed. Live claims keep serving, and unlike
a drain, a root move to this node is admitted, and so
is the adoption of an archived claim, since neither is a new claim. The
admission table lists what each state allows; a
cordon and a drain are independent, and each DELETE lifts only its own.
The cordon is the file <data_dir>/cordoned; it survives a restart, a reboot
and a config reload, and a node that starts cordoned starts no warm VM. With
audit_log on, audit.jsonl records op cordon. Answers the fresh
GET /v1/info payload with "cordoned": true; 500 when the file cannot be
written, and the node stays as it was.
Auth: root only (tenant and operator tokens get 403). Removes <data_dir>/cordoned, lifts
the cordon and kicks an immediate refill; audit.jsonl records op uncordon.
A drain stays in place. Answers the fresh info payload; 500 when the file cannot
be removed.
Auth: root only (tenant and operator tokens get 403). Moves one running claim to a peer with
its guest’s processes and memory (see
moving a running claim). Body:
{"to": "<peer>", "max_downtime_ms": 2000}, to as GET /v1/peers lists it
and never this node (else 400), max_downtime_ms optional with a default of
2000 and a floor: 400 or less answers 400 at once, because the target’s receive
and start alone are allowed that long. Answers {"owner_addr": "<peer>", "rounds": n, "downtime_ms": d, "capture_ms": c, "send_ms": s, "commit_ms": m} once the
peer took the claim and this node dropped it; a guest the peer cannot start is
failed there. downtime_ms runs from the final capture’s start, including its
wait for a free dump slot, until the peer answers the commit, after it started the guest; of it, capture_ms, send_ms and commit_ms are the final capture, the final round transfer and the commit. A final round whose capture time and size no longer fit max_downtime_ms is aborted before it is sent and the guest resumes here, so with the meta store up a capture that alone overruns pauses the guest for the capture plus the abort and its record write. While the move runs, release, fork,
checkpoint, promote, hibernate and volume changes of the claim answer 503
sandbox moving to another node; retry. Env writes, renew and data-plane calls keep working until
the final round and answer the same 503 during it, because the peer would miss
them. Every error leaves the claim running here, except one after the final round
paused the guest whose abort did not land (a 503 with the registry down, or a
failed cocoon vm migrate abort): the guest stays paused until the abort is
retried through, the guest then resumes or, when the record reads committed, the
node drops the claim (see moving a running claim):
target <peer>: <reason>), or the node lacks
a meta_store cell or mesh.cluster_keymove_concurrency reached here or on the target, or the target’s
admission refused (target <peer>: <reason>)sandbox moving to another node; retry), or the
final round would not fit max_downtime_ms, after at most 4 deltas or once
its capture overran it with the guest paused (the guest resumes here)The target serves root-only peer routes for a move, POST /v1/moves/{id},
PUT /v1/moves/{id}/rounds/{n}, POST /v1/moves/{id}/commit and
DELETE /v1/moves/{id}; they are no public API.
Auth: node api_token in the header, the sandbox’s own token in the body.
Unlike fork, promote and checkpoint — where the root token with an empty body
token acts by id as root — preview has no root path: the body
token is required. Mints a signed URL serving a
guest HTTP port from a browser: body {"token": "...", "port": 8080,
"ttl_seconds": 0} → {"url": "<preview_advertise>/p/<token>/"} (http:// is
prefixed when preview_advertise names no scheme).
The URL’s life is clamped to the claim’s remaining lease; an archived claim
kept forever (archive_delete_after_seconds: 0) has no lease, so there the
requested ttl_seconds stands unclamped and ttl_seconds: 0 mints a one-hour
URL; releasing the sandbox ends it early either way.
400 a port of 0, 404 unknown id or wrong sandbox token, 501 when the node
has no preview_listen. The signed token embeds the sandbox id, port, and
owner advertise_addr, so any node’s preview listener can serve it (forwarding
to the owner’s main listener) and a released sandbox’s URL answers 404 —
no revocation list. A browser request to the minted URL answers 503
lease lost; retry for a replicated claim while its node’s lease is overdue, 503 sandbox moving to another node; retry while an evacuation has yet to move it, and 429 when it has to wake a sandbox while the disk reserve is low or the node is at max_running; a wake that hands an archived claim to a peer answers 404, as after a move. See deploy.
Auth: node API token; body {"token": "<sandbox token>", "name": "..."}
(name optional); the root api_token with an empty body token acts by id as
root. Captures the sandbox’s full state without stopping it and
answers 200 {"checkpoint": {id, name?, sandbox_id, key, tenant?, created_at,
policy_source?, no_egress?, tun_off?, tun_profile?, egress_limit?, guest_env?}} — tenant records the calling tenant, absent for root.
400 bad body or name, 401 bad api token, 404 unknown id or wrong sandbox
token, 409 egress-lane sandbox, one with volumes attached (see
egress), a failed one, or an archived one (POST /wake or an exec
wakes it first), 429 while the disk is under disk_reserve, 503 lease lost; retry
for a replicated sandbox while its node’s lease is overdue.
Auth: node API token; body {"ttl_seconds": 0, "no_redirect": false,
"metadata": {}, "on_expire": "destroy", "env": {}}. Claims a fresh sandbox branched from the checkpoint (a
normal claim response, attributed to the caller); the checkpoint’s recorded
key applies. A tenant token branches only its own tenant’s checkpoints; any
other id answers 404, as for list and delete. On a node that does not hold the
record, the redirect and the peer heal below run before that check, so another
tenant’s id can redirect to its holders, or be pulled here before the 404;
checkpoint ids are random and no cross-tenant read serves them. The branch
carries this request’s metadata, on_expire and env, under the claim rules,
never the source sandbox’s. 400 metadata over a bound, an unknown
on_expire, or a bad env entry.
Checkpoints are node-local (unless the store is shared — see
Configuration). On a miss a node with a node-local
store probes its peers (a shared s3 store skips the probe: every node already
resolves every record) and answers with a redirect shaped like a warm-miss
POST /v1/claim, at most 3 addresses; retry the same body
(+no_redirect: true) at each candidate. With nothing to redirect to (or
no_redirect set) and checkpoint_peer_heal on, it pulls the record and serves
the claim. See the placement lifecycle.
404 for an unknown checkpoint (locally, and after redirect and heal both
miss), 429 an admission refusal, 503 when the node’s
concurrent-heal cap is already full — retryable, and the response carries a
Retry-After hint.
The peer-transfer route behind the probe and heal above — internal, not part of the public API; an SDK caller has no reason to call it directly.
GET streams the whole record — guest memory, disk, and meta — as a tar
in which every export file travels as its size plus its non-zero 16 MiB chunks,
each with its SHA-256, so holes cost nothing on the wire.
Root only: the stream carries no tenant scoping, and its only
real caller is a peer’s heal pull, which presents the fleet api_token.
401 missing or unrecognized token, 403 a valid tenant or operator token
(authenticated but not root), 404 unknown checkpoint.HEAD is the ownership probe: 200 when this node holds a branchable
(non-archive) copy, 404 otherwise, and 401 on a keyed mesh when
X-Cocoon-Probe is absent or expired. On a mesh with cluster_key set the
request must carry X-Cocoon-Probe, an HMAC over the id and a coarse time
bucket keyed off a probe-specific derivation of the cluster key — verified
before any disk is touched, replayable for roughly ninety seconds at most. On a
keyless mesh (redirect-only fleets have no shared secret to sign with) the
id itself remains the only capability, matching the mesh’s own posture.
Repeat probes for one id are answered from a short positive cache on the
asking node, which a local delete evicts immediately.Auth: node API token. Lists this node’s checkpoints, newest first. A tenant
sees only its own records; root sees everything. A checkpoint backing an
archived claim is that claim’s wake image (an ar_ id), not a listable record. Answers
200 {"checkpoints": [{id, name?, sandbox_id, key, tenant?, created_at, policy_source?, no_egress?, tun_off?, tun_profile?, egress_limit?, guest_env?}]}, the
record shape POST /v1/sandboxes/{id}/checkpoint returns.
Auth: node API token. A tenant may delete only its own records — anything else is 404, never a hint the id exists; root deletes anything. 204 on success, 404 unknown — including the checkpoint behind an archived claim, which only that claim’s release or retention window removes.
Delete removes the local record, then best-effort broadcasts to peers so a
healed replica does not outlive it — eventual cleanup, not a fleet-wide
revocation. A peer offline during the broadcast keeps its copy until the
checkpoint TTL ages it out, so an id-holder can still branch it for that
window; placement lifecycle
has the bound and why heal requires a nonzero, fleet-matching TTL. The s3
checkpoint store skips the broadcast: every node already resolves every
record directly, so there is no replica to chase. A mesh without a
cluster_key skips it too: it has no healed replica, and the broadcast would
send the api_token to addresses learned from unauthenticated gossip.
?no_forward=1 marks a
delete already arriving from another node’s own broadcast, so it is not
itself re-broadcast (loop prevention); it is an internal parameter, not one
an SDK caller should set.
Auth: node API token or the operator token. Root and the operator see every live claim; a tenant sees only its own.
The index is {"sandboxes": [{id, key, tenant?, deadline, claimed_at?, hibernated,
archived?, from_checkpoint?, claim_ref?, metadata?, on_expire?, cpu_count, mem_total_bytes,
volumes?: [{name, mount?, mode?, attach_only?}], volumes_want?: [{name, mount?, mode?, attach_only?}], restarts?, restarted_at?, failed?, env_missing?}], "next": "…"} — deadline is 0001-01-01T00:00:00Z on an archive kept forever; cpu_count and mem_total_bytes are the
size tier’s allocation; restarts counts the cold boots that replaced an
exited VMM or a hibernate snapshot or archive whose guest did not answer a
wake, and restarted_at stamps the last one (each loses guest memory and
the instance-metadata document, so a caller re-establishes guest state when
restarts moves; the claim’s env is written to the guest again); failed names why the VMM is down and was not restarted
(see dead VMMs);
env_missing is true while a guest: false env entry has no value on the node
(claim env);
mode is omitted for ro, matching the claim echo; never sandbox tokens,
volume host paths, or catalog access lists. claimed_at is the first grant
(renew and wake move deadline only). ?claim_ref=<ref> keeps only the claims recorded under
exactly that reference, so a caller that named its claim reads its row without
the rest of the index; an empty value lists everything. ?metadata=key=value,
repeatable, keeps only the claims holding every pair; 400 a pair without =,
with an empty key, or with a key repeated. tenant names the owning tenant
(absent for root’s own claims); ?tenant=<name> lets root list one tenant’s
claims, a removed tenant’s included, and is ignored for a tenant token.
id order;limit the reply is the whole listing and carries no next;limit (1 to 10000, 400 outside) caps the rows of a page, after every
filter and the tenant scope apply;after=<id> returns only claims with a larger id;next is the last row’s id when a limited page is full, absent
otherwise; pass it as the following page’s after;id sorts after the
cursor of the page that reads it.Auth: root or operator (tenant tokens get 403). One live claim in the index-row shape above, so a reconcile
loop can read a single sandbox without scanning the whole node listing. When
the caller presents the configured api_token or operator_token the row also carries token,
the sandbox’s own bearer token, so a control plane can hand a returning client
its data-plane credential without keeping a copy; a node without an
api_token never grants per-sandbox authority and answers without it. The
row also carries the advanced network mode
state, each field omitted when unset: tun_addr (the claim’s overlay address),
tun_off (true when the claim is disabled) and tun_profile (the exit label;
absent means default), and egress_limit, the claim’s
bandwidth cap, omitted when it has none.
404 unknown id.
Auth: root or the operator token (tenant tokens get 403). One sandbox’s resource usage — the per-sandbox counterpart
to the node-scoped /metrics:
{"id": "sb_…", "cpu_count": 2, "mem_total_bytes": 1073741824,
"mem_used_bytes": 187654144, "mem_used_measured": true,
"cpu_seconds": 5321.47, "cpu_started_at": "…", "cpu_measured": true,
"hibernated": false, "measured_at": "…"}
cpu_count/mem_total_bytes come from the size tier. hibernated marks a
claim whose memory snapshot is on this node, and archived, present only
when true, one whose state is in the checkpoint store; neither has a VMM to read. mem_used_bytes is
the host VMM process’s resident set — the only memory usage signal
available without a guest agent. cpu_seconds is the same process’s user plus system
CPU time, all threads, guest time included, and cpu_started_at is when that
process started. The counter starts over with the process (a wake, a restart
or a move), so take the rate between two reads only while cpu_started_at is
the same. mem_used_measured and cpu_measured are false when there is no
VMM process to read (hibernated or archived, the PID not yet known, or the engine lookup
failed), so a zero is never mistaken for idle; measured_at is taken after
the reads. One read costs one engine lookup; to read every claim, use
GET /v1/stats. 404 unknown id.
Auth: root or the operator token (tenant tokens get 403). Every live claim’s
stats on this node, sorted by id, in the shape of
GET /v1/sandboxes/{id}/stats:
{"sandboxes": [{"id": "sb_…", "cpu_seconds": 5321.47, "cpu_started_at": "…", "cpu_measured": true, …}]}
The whole read costs one engine listing and one /proc/stat read, whatever
the claim count, plus two /proc reads per running claim. 503
vm listing failed; retry when the engine cannot list its VMs.
Auth: node API token (root or tenant) or the operator token, which acts as root. A tenant reaches only claims it owns;
anything else is 404, like an unknown id. Each call speaks only for the node
it reaches; on a cluster, send it to the claim’s owner_addr.
PUT replaces the claim’s whole env with the body’s map, under the claim
rules; {"env": {}} clears it. PATCH merges: each entry in the body is set,
an entry of null removes that name, and every other entry is kept as stored,
host-only values included, so a control plane that cannot read those values
back sends them again only when the node reports them missing (below):
{"env": {"CB_META_wake": {"value": "w2"}, "OLD": null}}
The size rules apply to the merged env, so a patch that takes it past 64
entries or 64 KiB answers 400 and changes nothing. GET answers the same shape with every
guest: false value blanked, so a control plane sees the names without the
values:
{"env": {"MODE": {"value": "prod"}, "GW_KEY": {"value": "", "guest": false}}}
An entry with inject comes back with its hosts, header or query, body flag and placeholder, value blanked.
An inject host the claim’s pool does not intercept, or its tenant class does
not allow, and a header the pool rule’s own secret already sets on that host,
answer 400 and store nothing.
An entry the node holds no value for comes back with "missing": true, and
the claim’s index row carries env_missing. Under
host_env_memory_only that happens when the claim
hibernates, after a restart and after an adoption. A missing entry injects
nothing and blocks the node’s value for its secret; a missing upstream entry
falls back to the class or pool upstream. Send it again once the claim runs:
a PATCH replaces the entry whole, so repeat guest and inject:
{"env": {"GW_KEY": {"value": "Bearer …", "guest": false, "inject": {"hosts": ["api.example.com"], "header": "Authorization"}}}}
A value sent to a hibernated claim stays in that node’s memory until the claim hibernates again.
The env is persisted with the claim, its guest: false values only without
host_env_memory_only. A guest: false change takes effect on
the egress proxy’s next request at any time, a hibernated or archived claim
included, and never touches the guest. A change to the guest entries is
written to the guest at once, so it needs a running guest: on a hibernated,
archived, failed or transitioning sandbox it answers 409 (this verb never wakes one).
A PUT or PATCH whose result equals the stored env rewrites a running
guest’s file, so resending repairs a failed write, including a removal or
a change from guest to host-only. This also applies to an unchanged
host-only patch; the file contains only the guest entries. An unchanged
result does not rewrite the claims journal. A 500
after the store step means the env is stored and its guest: false values
are already live, but the guest write failed; resend the same request to
deliver it. The guest side is described in silkd.
PUT/PATCH answer 204, GET 200; 400 a bad entry, an unknown body
field or an inject outside the claim’s intercepted hosts; 404 unknown id or another tenant’s claim; 409 a guest change on a
paused or failed sandbox; 503 the archive registry is down, for an archived
claim or a hibernated claim with a replica under archive_failover: the node
already holds the new env, and a resend replaces the row.
Auth: root only (tenant and operator tokens get 403). Sets the sandbox’s instance-metadata document: the JSON a guest
agent reads from 169.254.169.254 in the IMDSv2 shape, as cloud-init and
similar agents do (PUT /latest/api/token, then GET / with the
X-metadata-token header). It is unrelated to the claim’s metadata labels.
The body is one JSON object of at most 4 KiB, stored and served verbatim; a
guest with no document answers {}, and a fork child, checkpoint branch or
template clone starts with the document its source held. The node writes it to
/run/silkd/instance-metadata.json in the guest atomically, so repeating a call
is a no-op. silkd serves the address only in images that alias it on lo.
204 on success; 400 when the body is not one JSON object within the bound; 404 unknown id; 409 when the sandbox is failed, hibernated, mid-transition or archived (this verb never wakes one) or its image serves no instance metadata.
Auth: root only (tenant and operator tokens get 403). Switches a claim’s advanced network mode and sets its exit profile label:
{"enabled": false, "profile": "residential"}
Both fields are optional; an omitted field keeps the claim’s value. profile
names an entry of tun.exits; an empty string returns the claim to default,
and a label the exit table does not hold yet is accepted, since the table
reloads (until it appears, the claim’s packets are dropped as having no exit).
The state is persisted with the claim, follows it through hibernate, archive,
move and an archive row, and is accepted on a hibernated or archived claim
without waking it; its next wake applies it. A running claim’s replica row
under archive_failover picks the change up at its next replica capture, as
an env change does, so an adoption before then restores the switch with the
rest of that replica. On a running claim the change
applies at once: disabling closes the claim’s packet stream and its door, drops
its flows and stops its ARP answers, then sends the guest tun_down, except
while a checkpoint, fork or replica capture holds the claim, when the guest
keeps a tun0 that carries nothing; enabling, or changing the profile, re-arms the packet door and sends tun_up.
Enabling a claim of a tun pool that holds no overlay address, because the
address store failed at its claim, allocates one. A fork child takes its
parent’s switch and label, with its own address.
204 on success; 400 an unknown field or a profile that does not match the
name rule; 404 unknown id; 409 an enable or profile change while a
transition (a capture, wake, hibernate or archive) holds the claim (retry; a
disable never waits), or another tun request to the claim committed first
(retry); 503 the claim is in a live move’s final round or not yet
started on its target (retry); 500 the address store failed or the claims
journal could not be written (nothing changed); 503 the archive registry is
down for an archived claim or one with a replica under archive_failover:
the node already holds the change, and a resend of the same body replaces the
row.
Auth: root or operator; a tenant token answers 403. Sets a running claim’s
virtio-balloon target, the guest memory the balloon withholds from the guest,
through one cocoon vm balloon call, and answers with cocoon’s report:
{"target": "50%"}
→ 200 {"target": 2147483648, "actual": 1073741824, "default": 1073741824}
target is what cocoon vm balloon accepts: NN% of guest memory (0 to 99)
or a byte size (2G, 512M, 1073741824), below guest memory. In the reply,
target is the size cocoon set in bytes, actual what the balloon holds now
when the hypervisor reports it, and default the boot target. Send default
back as the target to restore the claim. The call returns before the guest
settles. sandboxd keeps no balloon state and never changes a target on its
own: the caller sets it and the caller restores it. A hibernate wake and a
live move keep the target; an archive wake, a cold VMM restart, a fork and a
clone start at the boot target. cocoon serializes the call with captures and
live-move rounds.
200 with cocoon’s report; 400 a target cocoon rejects or an unknown field;
404 unknown id; 409 cocoon on this node has no vm balloon,
the sandbox's VM has no balloon device, or a hibernated, archived, failed or
waking claim or one a hibernate is capturing; 503
sandbox moving to another node; retry during a live move; 500 the cocoon
call failed or outran its 30 s budget, or cocoon found the guest paused (a
capture or a move round that began after the node’s checks).
Auth: root only. Caps a claim’s egress bandwidth in bytes per second, each
direction apart, tx guest to destination and rx back:
{"tx_bytes_per_second": 1250000, "rx_bytes_per_second": 2500000}
Zero or an absent field leaves that direction unlimited, so {} removes the
cap. It covers the claim’s proxy doors and, in
advanced network mode, replaces
tun.limits.bytes_per_second for the claim in each direction it sets. Preview,
port relay and exec traffic is not capped. On a running claim the change
applies at once to open connections, without a reconnect; an uncapped direction
can pass one more splice, up to 1 MiB at the old rate. The cap is persisted
with the claim, follows it through hibernate, archive, a live move, an archive
row and an adoption, and is accepted on a hibernated or archived claim without
waking it; its next wake applies it. A running claim’s replica row under
archive_failover picks the change up at its next replica capture, which for a
claim that hibernates first comes after its next wake. A fork child and a
checkpoint branch take their source’s cap. A new claim starts without one; no
pool setting gives a default.
204 on success; 400 an unknown field or a negative rate; 404 unknown id; 503
sandbox moving to another node; retry while a live move pauses the claim for
its final round; 500 the claims journal could not be written (nothing
changed) or the claim’s row could not be rewritten (the node holds the change;
resend the same body); 503 the archive registry is down for an archived claim
or a hibernated one with a replica under archive_failover: the node already
holds the change, and a resend of the same body replaces the row.
Auth: root only (tenant and operator tokens get 403). Prometheus text format,
hand-rendered: pool warm/target gauges, claimed/running/max_running/hibernated/archived/failed/draining/cordoned
gauges, sandboxd_evacuating (the claims an evacuation has yet to move), sandboxd_memory_low and sandboxd_anon_bytes (see memory_limit), a per-tenant live-claim gauge (sandboxd_tenant_claims{tenant="…"},
tenants holding live claims only, so the label set is bounded by the node’s claims), sandboxd_config_digest_mismatch on a mesh, claims
by tier (warm/clone/cold),
wake/hibernate/fork/checkpoint/promote/release/reap counters plus
archive/unarchive/archive-delete counters, sandboxd_dns_malformed_total (guest DNS queries dropped as malformed), claim/wake *_seconds_total
for average latency, sandboxd_claim_duration_seconds{tier}, a histogram of pool
claims by tier, sandboxd_claim_redirects_total, claims this node handed to a
peer, sandboxd_wakes_refused_total{reason}, wakes max_running (max_running),
disk_reserve (disk) and memory_limit (memory)
refused here, whether or not a hand-off ran, sandboxd_wake_handoffs_total,
archived claims this node handed to a peer with room
instead, and sandboxd_wake_memory_fallbacks_total, archived wakes over
memory_limit this node woke itself because no hand-off ran or no peer adopted. On a node with a tun block it adds the packet path’s counters:
sandboxd_tun_frames_total{dir} (frames on the claims’ vsock streams, out
from the guest and in to it), sandboxd_tun_datagrams_total{dir} (VXLAN
datagrams), sandboxd_tun_arp_total{kind} (answered replies to exits,
announced gratuitous ARPs) and sandboxd_tun_drops_total{reason} (packets
not carried: malformed, not_ipv4, options, fragment, proto, gso,
policy, no_exit, unknown_claim, queue_full, size (a packet or GSO
segment over tun.mtu, or a GSO frame of more than 128 packets), limit (a claim’s rate, flow or quota limit), send
(the underlay refused a send) and sender (VXLAN from an address that is no
exit)). The warm hit rate is sandboxd_claim_duration_seconds_count{tier="warm"}
over the three tiers’ counts: sandboxd_claims_total also counts forked children
and checkpoint branches under clone, and the histogram does not. A claim with
volumes includes their attach and mount in its tier. /metrics is a derived ops view; the billing source of
truth is the usage journal below.
Always on: every lifecycle transition appends one JSONL event to
<data_dir>/usage.jsonl — {"t": <RFC3339>, "ev":
"claim|renew|hibernate|wake|fork|checkpoint|adopt|promote|release|reap|archive|unarchive|archive_delete|egress|egress_bytes|vmm_exit|vmm_restart|vmm_failed|move_in|move_out",
"id": "sb_…"} plus vm (the VM’s name; absent on renew, adopt, archive,
unarchive, archive_delete, egress, egress_bytes, and the reap or
release of an archived claim), key (the pool key’s stable hash, claim
events), template, net and size (the claim’s template, lane and size
tier, claim events, so a consumer can price a sandbox by its shape without the
claim record, which is gone after release; size sets the CPU and memory,
and template is a label, since promote lets the caller name it), deadline (the deadline in force after this event: claim,
renew, wake, unarchive and move_in carry the lease, archive and adopt the retention
end, or nothing for an archive kept forever, and reap the deadline it ended under), tenant (the owning tenant, on every event of a
tenant’s claim), children (fork) and ref (the promoted template / checkpoint
id, the egress host, a vmm_failed reason, or why reconcile ended a claim). egress events, one per egress decision, are written only with
egress_usage_events on (default off), and egress_bytes events only with
egress_usage_bytes on. An egress event that injected credentials names
them in secret (gh,claim:<entry>, never a value). An egress event routed through an
upstream proxy carries upstream (its host:port), and with
egress_usage_bytes on, egress_bytes events add the connection’s payload tx/rx bytes. With
egress_usage_interval_seconds set, a connection or flow that lasts longer
writes an egress_bytes event each interval with the bytes since its previous one, then one with the
rest, if any, when it ends. Events carry no connection key: sums per claim and per upstream match the
single events an interval of 0 writes. A sandbox records at most 10 denied or rate-limited egress decisions per second after a burst of 100, so egress events do not count every refusal. A volume claim also carries
volumes, the applied catalog names, and — omitted when empty — volumes_rw,
the subset of those names claimed rw, on its claim event, so billing can
discriminate write access at claim (mounts and host paths are not billing
dimensions); a live attach or detach records no usage event. The file rotates at
1 GiB keeping one .1 backup, so a tailing collector never loses a window
silently. A last line a crash cut short is ended with a newline when the node
reopens the journal, so it never swallows the next event. A live move
records move_in on the target when it starts the guest and move_out on the
source when it drops the claim, each with its own node’s vm. Folding rules: fold a sandbox’s events of all nodes in t order.
An event ends the current span at its t and enters a state: claim, wake,
unarchive, vmm_restart and move_in enter running, hibernate enters hibernated,
archive enters archived, and vmm_exit, archive_delete, release and
reap enter ended. Other events, vmm_failed and move_out included, change no state, and
reap, adopt and archive_delete have the exceptions below.
Billable compute, hibernated storage and archived storage seconds are the time
in running, hibernated and archived, a cheaper store tier than a hibernated
VM’s RAM. vmm_exit is stamped when the node first finds the VMM gone, so an
outage before sandboxd runs again (a host that was down) stays billed up to that
point. A reap instead ends the current span at min(t, its deadline), or t
with none, because t can follow the true end by the reaper’s tick, the VM’s
teardown or the node’s downtime. An archive_delete ends the current span at
min(t, the last deadline): in archived that is the retention deadline, which a
purge trails by the sweep’s tick or the node’s downtime, and a purge can follow a
claim that has no end event by a whole retention window. An adopt names a claim
a node took over, from a lost node or from its own lost record, with ref the
checkpoint it holds. In archived it changes nothing; otherwise it ends the
current span at min(t, the last deadline before it) and enters archived at
its t. A span is never shorter than zero. A span still open ends at min(now,
the last deadline), and one entered by an archive or adopt without a
deadline stays open. A claim reconcile drops because its VM record is gone
ends with a reap whose ref is vm_gone, and one that cannot re-lock its
egress NIC with quarantined; a claim another node holds is dropped with no
event, and neither has a replicated claim dropped while the registry is down. The vm name joins
cocoon’s machine-level metering ledger for audit cross-checks. With
meta_store.usage, the node also lands every line in the cell’s
shared usage table, read through
GET /v1/usage.
Auth: root only (tenant and operator tokens get 403). Reads the raw usage events of every node of the cell from the shared usage table.
GET /v1/usage?tenant=<name>&since=<time>&until=<time>&after=<cursor>&limit=<n> →
{"events": [{"cursor": 41, "node": "n1", "t": "2026-10-30T08:00:00.123456Z", "ev": "claim", "id": "sb_…", "vm": "sbx-…", "key": "…", "template": "…", "net": "none", "size": "large", "tenant": "acme", "deadline": "2026-10-30T09:00:00.123456Z"}], "next": 41}.t in UTC to the microsecond, plus
cursor (its place in the table; one node’s events follow its journal
order) and node (the node that recorded it). A journal written by
v0.1.20 or earlier has no renew or adopt events and no deadline, a claim
reconcile dropped has no end event, and its claim lines have no template,
net or size.tenant keeps one tenant’s events; since (inclusive) and
until (exclusive) are RFC 3339 times compared with t. Each is optional.
A journal written by v0.1.20 or earlier stamps tenant only on claim,
archive, unarchive, archive_delete, egress, egress_bytes and
vmm_*; join its other events on id to the claim.cursor order, the order they landed, which is not t
order: a node that was cut off from the database lands its backlog later,
and a renew carries the time its lease changed, so its t can precede
that of an earlier line of the same journal;after returns only events with a larger cursor (default 0);limit defaults to 1000, at most 10000;next is the last event’s cursor when the page is full, absent otherwise;
pass it as the following page’s after;after returns every event exactly once, backfill included.meta_store.usage_retention_hours,
each node deletes its rows older than the retention once an hour, so page
within the retention: an after below the oldest kept row returns from
the oldest kept row, with no error, and the deleted events are gone. A
backlog a node lands with t older than the retention can be deleted at
its next sweep, before a consumer reads it.after, limit, since or until; 501
no shared usage store configured on a node without meta_store.usage; 503
usage store down; retry while the database is unreachable.Auth: the sandbox’s own token; the operator token gets 403. Requires Upgrade: silkd +
Connection: Upgrade; answers 101 Switching Protocols and from then on
the connection is a byte-for-byte relay to the guest’s silkd, carrying RPCs
back to back (see silkd). An open relay holds the sandbox’s idle
clock, so a client that keeps a connection warm must close it when idle for
idle_hibernate_seconds to apply. A relay whose client vanished without a FIN
ends after about two minutes of failed TCP keepalives. Frames sent right behind the request,
before the 101, reach silkd in order; the Go and Python SDKs send their first
frames that way over plain HTTP, so a proxy in front of a plain-HTTP endpoint
must forward them too (Go’s httputil.ReverseProxy drops them; Caddy forwards
them). 426 without the upgrade header, 404 unknown sandbox, 409 a failed sandbox,
500 a failed restore, 502 guest unreachable, and the
guest errors. A relay of a
claim of a replicating pool that is already open when the lease turns overdue
is closed, whether or not it has a replica yet.
A hibernated or archived sandbox wakes when the upgrade request arrives, before the 101,
unless its last sleep is reader-safe, which the pool’s reader setting provides;
what a reader is, what it needs and when the call wakes instead are there.
With it, the node answers the 101 before it touches the sandbox, reads the
first frame, and serves fs_stat, fs_list, fs_read and fs_pull from a
reader VM: the sandbox stays hibernated or archived, no wake usage event is
recorded, and the read stamps neither the idle nor the archive clock. An info frame wakes the sandbox, since a
reader would answer with its own processes and uptime. The first frame on the connection that
is anything else, or any later one, wakes the sandbox and hands the same
connection to its silkd with the unread bytes replayed, so a client sees one
silkd and no close. When the sandbox sleeps again meanwhile (a new hibernate or
an archive), the connection’s next read moves to a fresh reader of the new sleep;
when it wakes, the connection moves to the woken sandbox, so reads never come
from a stale sleep. Pipelining a request behind a read that has not yet sent
its terminal frame is unsupported: when that request moves the connection, the
reader closes and the rest of the read’s reply is lost. A read that no reader
can serve wakes the sandbox as if the feature were off. Because the 101 has
gone out by then, a failure on this path
(unknown sandbox, a refused wake, an unreachable guest, an overdue node lease)
arrives as a silkd error frame (not_found for an unknown sandbox, else
internal) rather than an HTTP status, and the connection ends. A connection
served by a reader, or handed over to the woken sandbox, of a claim of a
replicating pool is closed when the lease turns overdue, as an open relay is.
Auth: the sandbox’s own token, or the root or operator token for a control-plane
read (below). Send Upgrade: tcp + Connection: Upgrade
(the gate reads Upgrade alone, as /agent’s does); answers
101 Switching Protocols and from then on
the connection is a raw byte relay to 127.0.0.1:{port} inside the guest,
over the same vsock port_forward preview uses. Nothing of ours frames the
stream, so HTTP/1.1, HTTP/2 and any other protocol pass through unchanged.
This is how an edge proxy reaches a guest listener on a net=none sandbox,
which has no NIC.
Half-close works in both directions: a client write shutdown reaches the guest socket, and a guest write shutdown reaches the client without ending the client’s writes. Each direction stays open until its own EOF. A relay copy error closes both connections and releases the sandbox hold.
With the sandbox’s token, an open relay holds the sandbox’s idle clock exactly
like the silkd relay, and a hibernated sandbox wakes transparently — which is
why the upgrade header is mandatory: a bare GET must not consume a hibernate
snapshot. With the root or operator token the relay is passive, for control-plane
reads such as metrics: it never wakes the sandbox and never stamps activity,
so a poll neither undoes a pause nor keeps a sandbox from idling. A hibernated
or archived sandbox, or one with a hibernate, wake, checkpoint, promote or
fork in flight, answers 409 at once; an open passive relay still holds the sandbox against the idle sweep.
400 a port outside 1-65535; 401 missing bearer token; 404 unknown sandbox or
wrong token; 409 a failed sandbox or a passive relay to a paused one; 426 without the upgrade
header; 502 when the guest cannot be reached on that port — no listener, or a
hibernated sandbox whose restore failed. The node log carries which; and the
guest errors.
Auth: the sandbox’s own token, or the root or operator token by id, which wakes a hibernated or archived sandbox like the sandbox’s token. A buffered exec for clients that cannot hold an upgraded connection — plain JSON in, plain JSON out, so it multiplexes over HTTP/2 through a TLS proxy:
{"argv": ["node", "-v"], "cwd": "/work", "env": {"CI": "1"}, "timeout_seconds": 60}
→ 200 {"exit_code": 0, "stdout": "v22.23.2\n", "stderr": ""}. The command
runs to completion (no stdin, no streaming, no detach — use the relay for
those); timeout_seconds 0 means no limit beyond the request itself. When
the node gives up on a started command (timeout, client gone, output cap,
buffer budget) it sends the child a kill through silkd, bounded to 5 s, before
answering. Output comes back as JSON strings: bytes that are not valid
UTF-8 are replaced with U+FFFD, so binary output belongs on the relay. 400
empty argv, negative timeout_seconds, an unknown field, or a silkd
bad_request — a command that is missing or not executable and a cwd that
does not exist are the caller’s, not 502s; 401 missing bearer token; 404 unknown sandbox or wrong token;
413 when stdout+stderr exceed 8 MiB; 503 with Retry-After: 1 when the
node’s buffered-exec memory is exhausted (64 MiB of output across every
buffered exec in flight; the command is killed); 502 guest unreachable or any other
silkd error; 409 a failed sandbox; 500 a failed restore; the
guest errors; 504 when the command outlives timeout_seconds. A client that hangs up first gets no reply and the command is killed. A hibernated sandbox wakes transparently like on the relay.
Auth: the sandbox’s own token, or the root or operator token by id. Answers {"owner_addr": "host:port"} when
this node owns the sandbox, 404 otherwise; a tenant token is a wrong sandbox token here. With
archive_failover, a node that
does not hold the sandbox adopts it, when the token’s SHA-256 matches the row’s or the caller is root or the operator, and its row names an owner whose node
lease expired, or whose owner released it in an
evacuation or a wake it had no room for, and answers 200; with ?room=1, which the owner of such a wake sends, it adopts only while it could admit the wake itself; 503 with Retry-After when the meta store is
down. A row whose owner’s lease is live is never adopted unless the owner released it: 404. A draining or fenced
node, or one that lifted its fence less than 8 seconds ago, does not adopt: 404; a cordoned one adopts. A node whose
lease is overdue answers 404 for the replicated claims it holds, so the
Lookup scatter moves on. With client_advertise configured,
owner_addr is that node’s full HTTP(S) origin instead; the same contract applies
to claim and fork responses. Used by the SDK’s Lookup scatter. A claim the root or the operator
adopted keeps the row’s token hash in place of its token: its own token still
authorizes it on this node and any later adopter, an empty token never does,
and the first call with its token remembers it (persisted at the next claims
commit), after which root’s or the operator’s GET /v1/sandboxes/{id} returns
it. The hash is never served.
Auth: root or operator (tenant tokens get 403). Node pools, promoted templates, claim count, the node’s own address, and mesh peers:
{"pools": [{"key": {"template": "base:24.04", "net": "none", "size": "small"},
"warm": 4, "refilling": 0, "target": 4, "golden": true}],
"templates": [{"key": {"template": "myproj:v1", "net": "none", "size": "small"},
"content_digest": "sha256:…", "tenant": "acme",
"created_at": "2026-07-06T00:00:00Z", "cpu_count": 1,
"mem_total_bytes": 536870912, "labels": {"v1": "sha256:…"}}],
"claimed": 2,
"running": 1,
"max_running": 12,
"hibernated": 1,
"archived": 0,
"failed": 0,
"at_capacity": true,
"at_capacity_reason": "exchange full",
"advertise_addr": "10.0.0.5:7777",
"peers": ["10.0.0.6:7777"]}
advertise_addr is the address the node hands clients: its client_advertise
when set, else its advertise_addr. It is the same value a claim reports as
owner_addr, and it is omitted when the node advertises no host.
hibernated counts claims whose VM is currently hibernated, archived those
checkpointed to the store with the local VM dropped (see
archive tiers), failed those whose VMM is down
and was not restarted (omitted at zero); all are included in claimed.
running counts claims that hold a VM, with a wake in flight counted, and
max_running is the cap on it (omitted when off).
A node drained via POST /v1/drain additionally reports
"draining": true, a node cordoned via POST /v1/cordon
"cordoned": true, and during an evacuation evacuating (claims left to
move, omitted at zero). When refill is parked because the node cannot start
another VM, "at_capacity": true distinguishes a full node from one still
filling its warm target, and at_capacity_reason carries the engine’s capacity
reason, or disk below reserve or memory above limit, which stay stable for callers that tell them apart. Both capacity fields are omitted while refill is not capacity-blocked. Each VM holds a console pty, so a node stops at kernel.pty.max (4096 by default) with no space left on device; raise it past the VM count you plan to run.
golden reports whether the pool’s snapshot exists (refill can clone);
warm at target with golden: true means warm claims are served in
sub-millisecond time.
templates lists the promoted templates this
node holds: each one’s full key, the content_digest its promote returned,
created_at of its last promote, the cpu_count and mem_total_bytes of its
size tier, its labels when any are
set, and tenant when a tenant promoted it (omitted for root). A key
a configured pool now owns is left out, since claims of it clone the pool’s
golden. On a shared checkpoint store a node lists what it promoted plus what
was in the store when it started, and a template another node deletes later
stays listed, and may redirect a claim to a 404, until this node restarts.
Auth: node API token (root or tenant) or the operator token. Answers {"peers": [addr, …]}
— the cluster’s other node addresses, for the SDK’s redirect follow and
Lookup scatter. Cluster topology, not operator state, so a tenant token
may read it (unlike GET /v1/info).
Unauthenticated liveness probe; answers ok.
ReadTimeout/WriteTimeout at zero if you
front it with your own server: cold claims legitimately block for the cold
probe window and relays stream indefinitely. sandboxd itself uses
ReadHeaderTimeout for slowloris protection, and closes a keep-alive
connection that sends no request for 15 minutes, longer than the backend
idle timeout of common load balancers (600 s on Google Cloud), so a balancer
never reuses a connection sandboxd just closed.