# sandboxd HTTP API

All bodies are JSON. Four token kinds:

- **root** — the node-level `api_token`: every endpoint except the agent relay
  and the preview mint, which need the sandbox's own token. A set
  [`api_token_next`](deploy.md#configuration) is root too, for a
  [rotation](deploy.md#rotating-api_token-and-operator_token): the same
  endpoints, and the audit `caller` stays `root`.
- **operator** — the node-level [`operator_token`](deploy.md#configuration),
  for a public control plane that must not hold cluster configuration power.
  It acts with root's authority, with no tenant scoping, on exactly
  `POST /v1/claim`, `GET /v1/sandboxes`, `GET /v1/sandboxes/{id}`,
  `GET /v1/sandboxes/{id}/stats`, `GET /v1/stats`,
  `POST /v1/sandboxes/{id}/release|hibernate|wake|renew|balloon`,
  `GET/PUT/PATCH /v1/sandboxes/{id}/env`, `GET /v1/sandboxes/{id}/ports/{port}`,
  `POST /v1/sandboxes/{id}/exec`, `GET /v1/sandboxes/{id}/owner`, the
  [tenant API](#tenants-v1tenants), `GET /v1/info` and `GET /v1/peers`. Every
  other authenticated endpoint answers it `403`
  `operator token not allowed on this route`; it is never a sandbox token, so
  the agent relay refuses it too, and a claim that names `volumes` answers it
  `403` `operator token cannot claim volumes`. Unset or empty, it is off. A set
  [`operator_token_next`](deploy.md#configuration) is the operator too, on the
  same endpoints, and the audit `caller` stays `operator`.
  Scope: it reaches every claim's data — it reads any claim's token by id and
  execs in any claim. Through the tenant API it holds every tenant's
  authority: it can mint or re-key any tenant and then act as it, including a
  claim that names volumes that tenant may mount and fork, checkpoint and
  preview with a claim token. It holds no node or cluster configuration power. With `audit_log` on, its by-id token read records op `token`.
- **tenant** — a token the [tenant API](#tenants-v1tenants) added: accepted by the
  resource-creating verbs (claim, fork, promote, checkpoint create/claim,
  preview mint) and the tenant-scoped listings/deletes below; everything a
  tenant creates is stamped with its name. Root surfaces
  (the per-id sandbox reads, `PUT /v1/sandboxes/{id}/instance-metadata`, `PUT /v1/sandboxes/{id}/tun`, `PUT /v1/sandboxes/{id}/egress-limit`, `POST /v1/sandboxes/{id}/balloon`, `GET /v1/info`, `GET /v1/stats`,
  `PUT /v1/pools`, `POST /v1/config/reload`, the [tenant API](#tenants-v1tenants),
  `POST /v1/sandboxes/{id}/move` and the peer `/v1/moves/{id}` calls, `PUT/DELETE /v1/volumes/{name}`, `POST/DELETE /v1/drain`, `POST/DELETE /v1/cordon`,
  `GET /metrics`, `GET /v1/usage`, `GET /v1/checkpoints/{id}/blob`)
  answer a tenant token `403` `root token required` — authenticated but not authorized; an unknown
  token stays `401`. On a node with a shared
  [`meta_store`](deploy.md#shared-tenant-database), a tenant token the node
  has not cached answers `503` while the database is unreachable; retry.
- **sandbox** — every claimed sandbox carries its own bearer token guarding
  the sandbox-scoped endpoints; each endpoint says whether root also acts by id.

Endpoints below that say "node API token" accept root or tenant unless
marked root-only; the operator token only where the endpoint names it. Errors are `{"error": "message"}` with the status codes
listed per endpoint. Request bodies are capped at 1 MiB; a larger one is
rejected with `400` and an error beginning `invalid request body`, and so is
a body with a field its endpoint does not declare.
[`PUT /v1/sandboxes/{id}/instance-metadata`](#put-v1sandboxesidinstance-metadata)
is the exception: it takes one raw JSON object of at most 4 KiB.

## Guest errors

Wake, the agent relay, the port relay and exec restore a sleeping sandbox before
they reach its guest, so they share these answers:

- 403 a removed tenant's archive
- 429 while the disk reserve is low or the node is at
  [`max_running`](deploy.md#configuration)
- 503 `lease lost; retry` for a replicated claim while its node's lease is
  overdue ([running claims](cluster.md#running-claims-across-node-loss))
- 503 `sandbox moving to another node; retry` while an
  [evacuation](#post-v1drain) has yet to move it, or while a
  [live move](#post-v1sandboxesidmove) pauses it for its final round
- 503 `archive registry down; retry` for an archived claim under
  `archive_failover` while the meta store is down
- 503 `tenant store down; retry` for an archived claim whose tenant the node has
  not cached while the meta store is down

## Admission refusals

A claim, fork or checkpoint branch answers 429 when the node refuses it for
capacity, with the pool state unharmed. The causes:

- the node is at [`max_claims`](deploy.md#configuration) or `max_running`
- the tenant the claim counts against is at its own `max_claims`: the caller's
  for a claim or branch, the parent's for a fork
- the node is draining ([`POST /v1/drain`](#post-v1drain)) or cordoned
  ([`POST /v1/cordon`](#post-v1cordon))
- the disk is under `disk_reserve`
- memory is over `memory_limit`
- the node is [fenced](cluster.md#running-claims-across-node-loss) and the
  claim's policy pool has `replicate_seconds`

On a cluster, a claim without `volumes`, `no_redirect` or `require_promoted`
that meets one of these redirects to a peer with warm capacity or, failing
that, one that serves the pool; see [`POST /v1/claim`](#post-v1claim). Wake and
the guest-reaching verbs have their own 429 causes, listed with each.

## POST /v1/claim

Auth: `Authorization: Bearer <api_token>` (when configured), a tenant token,
or the operator token, which claims as root but never with `volumes` (403).

```json
{"template": "base:24.04", "net": "none", "size": "small",
 "ttl_seconds": 300,
 "volumes": [{"name": "imagenet"}, {"name": "weights", "mount": "/models"},
             {"name": "scratch-db", "mode": "rw"}],
 "claim_ref": "namespace/workload", "metadata": {"team": "a"},
 "on_expire": "archive", "no_redirect": false, "require_promoted": false,
 "egress": true,
 "env": {"MODE": {"value": "prod"}, "GW_KEY": {"value": "Bearer …", "guest": false}}}
```

- `net` defaults to `none`, `size` to `small`; the pool key is
  `(template, net, size)`
- `ttl_seconds` 0 means the server default (5 minutes); capped at 24h. The
  owning node reaps the sandbox after the TTL even if the client vanishes
- `claim_ref` is an optional opaque caller reference echoed by the scoped
  sandbox index, at most 512 bytes of printable UTF-8 like a metadata value;
  the aggregated apiserver uses `<namespace>/<name>`
- `metadata` is an optional map of caller labels, echoed and filterable by
  the [sandbox index](#get-v1sandboxes): at most 16 pairs, each key 1 to 128
  bytes of printable ASCII without `=` or `&` (list filters encode pairs as
  `key=value`, and a query string joins pairs with `&`), each value at most
  512 bytes of printable UTF-8, and at most 4 KiB as JSON, so quotes and
  backslashes count twice. Labels are fixed at claim time; no verb edits them
- `on_expire` is what the node does when the lease ends: `destroy`, the
  default, or `archive`. `archive` hibernates a running sandbox and archives it
  to the checkpoint store, whatever the pool's `archive_after_seconds`. The
  archive is kept for `archive_delete_after_seconds`, or forever when that is
  0, and the next call that reaches the guest restores it with a fresh lease
  of the claim's length, so its next expiry archives it again; a call that
  wakes it between the hibernate and the export grants that fresh lease too.
  A failed hibernate or export at expiry keeps the claim and retries it a
  minute later; it never falls back to destroy. A failed claim (`failed` set, see
  [dead VMMs](deploy.md#dead-vmms-and-host-restarts)) is destroyed at expiry even
  under `archive`. A volume claim or an egress-lane claim
  cannot hibernate, so `archive` on one answers 409 with the same error as a
  hibernate
- `no_redirect` is set by the SDK when retrying at a redirect target
- `require_promoted` provisions only from a promoted template: a claim carrying
  it never takes a warm VM or cold-boots, and answers 404 unknown template when
  no record holds the key, a key a pool serves included. A redirect to a template's owner sets it; copy it into the
  `no_redirect` retry so a target whose gossip view is stale refuses instead of
  cold-booting an image named like the template
- `env` is the claim's own environment, at most 64 entries: each name a shell
  identifier of at most 128 bytes, each value at most 8 KiB, a value and an
  inject `query` or `placeholder` with no control characters other than tab, and names, values and inject fields at most 64 KiB in total. An entry holds only `value`, `guest` and `inject`; any other
  member, `"guest": null`, a `guest: false` entry with an empty value, or
  `missing` (the node's own marker) answers 400, so a mistyped flag never lands
  a host-only value in the guest and a read sent back never blanks one. `guest` (default `true`) delivers the entry into the guest; a
  `guest: false` entry never enters it and only feeds the node's egress
  [secrets](egress.md#claim-env-and-secrets), or, with `inject`
  (`{"hosts": [...], "header": "...", "placeholder": "..."}` with `placeholder`
  optional, or `{"hosts": [...], "query": "...", "body": true, "placeholder": "..."}`
  with `body` optional), becomes a
  [claim credential](egress.md#claim-credentials) the proxy sends to those
  hosts. A bad entry answers 400, and an `inject` host the claim's egress does
  not intercept answers 400 with the claim's VM released; see
  [claim env](#getputpatch-v1sandboxesidenv)
- `egress` `false` claims with no [egress policy](egress.md), whatever the
  pool's (or, for a promoted template, its source pool's): the guest's doors are
  never bound. Omitted or `true` keeps the policy the claim would get
- `volumes` is an ordered list of at most eight unique catalog names. `mount`
  defaults to `/volumes/<name>`; a custom value must be absolute and clean,
  outside the guest OS tree, unique, and non-nesting within the request.
  `mode` is `"ro"` (default, omitted) or `"rw"`; `"rw"` requires the catalog
  entry's `writable: true` (see [deploy](deploy.md#dataset-volumes)).
  `attach_only: true` attaches that entry without mounting it and requires an
  empty or omitted `mount`. Mounted and attach-only entries can share a list

Success:

```json
{"id": "sb_…", "token": "…", "deadline": "2026-07-06T00:05:00Z",
 "owner_addr": "10.0.0.5:7777", "template_digest": "sha256:…", "net_route": "relay",
 "volumes": [{"name": "imagenet", "mount": "/volumes/imagenet"},
             {"name": "weights", "mount": "/models"},
             {"name": "scratch-db", "mount": "/volumes/scratch-db", "mode": "rw"}]}
```

A claim cloned from a promoted template carries `template_digest`, the exact
export generation fetched for that clone. It is absent for configured pools,
cold image boots, forks and checkpoints.

`net_route` says how the guest reaches the network: `relay` through the host
proxy behind the guest's bound doors, `direct` over an egress-lane NIC the node
does not lock, or `none` (a claim with no effective egress policy, a claim with
`"egress": false`, or a locked egress-lane NIC with no policy). silkd hands its children the
`http_proxy`, `https_proxy` and `no_proxy` of its unit unless the lane is
direct; a guest agent of another kind reads `net_route` and hands its processes
`http://127.0.0.1:3128` when it is `relay`. `"egress": false` closes the doors, not an unlocked NIC, so
such a claim on a `direct` lane still reaches out directly.

A claim branched directly from a checkpoint additionally carries
`"from_checkpoint": "ck_…"` — the lineage edge for reconstructing the
checkpoint tree. Fork children do not repeat that edge: they branch from the
parent sandbox, not directly from its checkpoint.

`volumes` reports the names, effective mounts, and (`rw` only) mode applied
and persisted at finalization — `mode` is omitted from the echo for `ro`
entries, matching the request shape. sandboxd attaches each disk, waits for
the device settle — a serial match under `/sys/block`, then the `/dev/<name>`
node itself, since the kernel publishes sysfs before devtmpfs creates it — up
to 2 seconds total, and mounts the filesystem — as ext4 first, then with mount's
own probe if that is refused; read-only, unless the entry requested and was
granted `rw` — before returning. A mount that reports
`not a valid block device` is retried within the 10-second setup budget,
since the new node can exist before the kernel opens the disk. A custom mount
may shadow an existing populated guest directory for the claim's life.

### Attach-only volumes

`"attach_only": true` stops after the attach for that entry. The device appears
in the guest without a mount. Responses preserve `attach_only: true` and omit
`mount`:

```json
{"id": "sb_…", "token": "…",
 "volumes": [{"name": "imagenet", "attach_only": true},
             {"name": "scratch-db", "mode": "rw", "attach_only": true}]}
```

Find each device by its virtio serial, which is the catalog name — poll this,
not a one-shot lookup: the claim can return before the guest enumerates the
device (typically within ~100ms, never guaranteed), and the `/dev/<blk>` node
can lag the serial match the same way it does for an eager mount's device
settle. Confirm both before mounting:

```sh
device=
tries=0
while [ "$tries" -lt 200 ]; do
  for serial in /sys/block/*/serial /sys/block/*/device/serial; do
    [ -r "$serial" ] || continue
    [ "$(cat "$serial")" = scratch-db ] || continue
    block=${serial#/sys/block/}
    candidate="/dev/${block%%/*}"
    [ -b "$candidate" ] && { device="$candidate"; break 2; }
  done
  tries=$((tries + 1))
  sleep 0.01
done
[ -n "$device" ] || { echo "scratch-db device not ready" >&2; exit 1; }
printf '%s\n' "$device"
```

Then mount it however the workload needs, retrying a mount that reports
`not a valid block device` for a short time. A `ro` entry is attached
`--readonly` and stays read-only at the guest block layer no matter who
mounts it, so the guarantee does not depend on your mount flags.

The whole consistency contract moves to the caller with the mount:

- sandboxd writes no dirty marker for an attach-only `rw` claim and clears
  none, because it cannot verify your unmount and therefore promises nothing.
  The flush is yours too: release runs no unmount and no `sync`, so whatever
  the workload left unsynced is discarded with the VM. Releasing without a
  clean unmount of your own leaves the image exactly as a filesystem crash
  would. The next *eager* `ro` claim then fails at mount time with a 500 —
  loud and attributable to the claim that skipped its unmount — instead of
  the marker's 409. Recovery is any `rw` cycle that replays the journal.
- a marker left by an earlier *eager* `rw` crash is not cleared by attach-only
  cycles, and still refuses eager `ro` claims with 409 until a `rw` claim
  releases cleanly.
- writes straight to the block device bypass the filesystem journal entirely
  and are outside marker protection in either mode.

What still protects other claims is unchanged: admission excludes an
attach-only `rw` claim against every other claim of the name (and readers
against a writer), an attach-only `ro` claim of a marker-bearing image is
refused with 409, and a claim with volumes still refuses checkpoint, fork and
hibernate. An attach-only claim costs one disk attach per volume — no device
settle, no mount round-trip — and release costs nothing at all: removing the
VM closes the devices.

Redirects (mutually exclusive with the fields above) name peers to retry
at — sent on a warm miss with warm peers, when the node lacks a golden for
the key but gossip names a template owner, and when a claim without `volumes`,
`no_redirect` or `require_promoted` meets an [admission refusal](#admission-refusals)
and a peer reports warm capacity or, failing that, serves the pool:

```json
{"redirect": ["10.0.0.6:7777", "10.0.0.7:7777"],
 "require_promoted": true}
```

Retry the same body (+`no_redirect: true`) at each candidate until one
answers. Preserve `require_promoted: true` when the redirect carries it, as a
redirect to a template's owner does; warm-candidate redirects omit the field.

A volume claim may consume an ordinary warm VM; candidate ranking, the
promoted-template intersection, and the `no_redirect`/`require_promoted` retry
follow the fleet-wide rule in [cluster](cluster.md#volumes-and-placement).
Redirect responses never carry `volumes`.

A tenant token claims the same way; the sandbox is stamped with the tenant
name (attributed in the usage journal and counted against the tenant's
`max_claims`). A catalog access list may restrict an entry to named tenants;
an unknown and a forbidden volume return the same error text.

Errors: 400 unknown template axis, `claim_ref` or metadata over a bound (the
message names it), a bad `env` entry, an unknown `on_expire`, invalid/duplicate volumes,
an attach-only entry with a nonempty `mount`,
`mode: "rw"`
against a non-writable entry, or a volume that is unknown or forbidden (the
latter two are deliberately indistinguishable), or
bad body; 401 bad api token; 409 egress requested on a node without an egress
attachment, a writable name already claimed in a conflicting mode (volume
busy — a live writer excludes every other claim for that name, live readers
exclude a writer), or a `ro` claim against a writable image left dirty by an
unclean `rw` release (needs recovery — one `rw` claim must replay and cleanly
release before `ro` claims resume; see
[deploy](deploy.md#writable-dataset-volumes)); 429 an [admission refusal](#admission-refusals); 500 provisioning failed.

## GET /v1/volumes

Auth: node API token (root or tenant). Lists the fleet catalog entries the
caller may use, without host paths or holder addresses:

```json
{"volumes": [{"name": "imagenet", "default_mount": "/volumes/imagenet",
              "size_bytes": 214748364800, "available": true, "nodes": 3}]}
```

Root sees the gossiped union, including peer-only entries (`available: false`);
a tenant sees only entries this node declares locally and whose access list
permits it. `nodes` counts members advertising the name; `size_bytes` and
`available` are a best-effort stat of the answering node's image. Membership is
eventually consistent by one gossip tick.
`writable` is the entry's catalog configuration, fleet-uniform like the access
list; the field is emitted (as `true`) only for a writable entry this node
declares and omitted otherwise (peer-only rows never carry it).

## PUT /v1/volumes/{name}

Requires the configured root API token. A tenant or operator token or a node without an
`api_token` returns 403. Sandbox, unknown, and missing tokens return 401 when
`api_token` is configured. The URL supplies the name. Unknown body fields,
duplicate fields, and trailing JSON return 400.

```json
{"path": "/srv/datasets/data.img", "directio": "off",
 "writable": false, "tenants": ["acme"]}
```

- `name` matches `[a-z][a-z0-9_-]{0,19}` and cannot start with `cocoon-`.
- `path` uses static config's absolute-path validation. When the node sets
  [`volume_roots`](deploy.md#configuration), the cleaned path must equal or sit
  under one of them. Registration does not open the backing or check its type
  or availability. A missing backing keeps its registration but cannot be
  attached.
- `directio` is `on`, `off`, or `auto`, and defaults to `off`.
- `writable` defaults to `false` and controls writable claim admission.
- `tenants` is the entry's [access list](#volume-access-lists).

Success is 204. An identical normalized registration is idempotent. A different
binding, changed fields, a static name or cleaned path collision, or a name a
claim still holds returns 409. A PUT that adds a name and every successful
DELETE, also of an absent name, write an `op:"volumes"` audit record naming the
volume.
Invalid fields return 400. GET keeps its existing projection without host paths
or tenant lists. Runtime entries support the existing live volume API.

Persistence failure returns 503 with `Retry-After`. Retry the same request;
after a restart, either state can load.

The node stores metadata in `data_dir/runtime-volumes.json` independently of
`sync_claims` and loads it before claim recovery. New records use mode 0600.
Malformed or conflicting metadata, including a saved path outside `volume_roots`,
prevents startup. The catalog is node-local with one sandboxd writer.

Sandboxd does not create, check, or delete backings. Keep them outside
`data_dir`: sandboxd clears its `readers`, `goldens`, `moves`, `fork-*`,
`checkpoint-staging` and `template-staging` entries.

### Volume access lists

Static and runtime entries share one `tenants` rule. An omitted or empty list
permits every authenticated scope. A nonempty list permits only claims owned by
the listed tenants, and root-owned claims can use any entry. A listed name must
match the tenant API syntax: runtime registration rejects one that does not,
and static config fails to load. A valid name for a tenant that does not exist
yet grants nothing until the tenant is created.

## DELETE /v1/volumes/{name}

Auth and name validation match PUT. Deletes only runtime registration metadata.
Static names and names referenced by a claim, an in-flight operation, or
cleanup return 409. Otherwise DELETE writes the absence and returns 204, also
for an absent name.
Persistence failure returns 503 with `Retry-After`. Retry the same DELETE,
even if GET no longer lists the name; a restart can restore the old registration.

## GET /v1/sandboxes/{id}/volumes

Returns the persisted desired set (`volumes_want`), confirmed applied set
(`volumes`), per-volume `states`, and `converging`. Auth requires a client API
token. Root can access every sandbox; a tenant can access only its own.
Sandbox tokens are rejected when authentication is enabled. The owning node
handles the request without redirecting.

```json
{
  "volumes_want": [{"name": "imagenet", "mount": "/datasets/training"}],
  "volumes": [{"name": "imagenet", "mount": "/datasets/training"}],
  "states": [{"name": "imagenet", "state": "attached"}],
  "converging": false
}
```

All three arrays are present, including when empty. A state is `attached`
when the volume is in both sets, `pending` when desired but not yet confirmed,
or `detaching` when applied but no longer desired. An interrupted device call
can leave the actual guest state different from the confirmed set. A failed
PUT or PATCH adds a top-level `error` naming the refusal, and the affected
state may carry its own `error`. `converging` is true only while a request worker is running.
`pending` with `converging: false` requires an explicit retry; there is no
background convergence. GET retains the last safe per-volume error for the
current daemon lifetime. After daemon restart, unfinished states report an
interrupted operation that needs retry. Successful retries clear resolved errors.
The sandbox index exposes both sets for operator inspection.

## PUT /v1/sandboxes/{id}/volumes

Replaces the full desired set and converges synchronously. Auth matches GET.
The owner tenant's catalog access list applies to newly admitted names, including
for root. Retaining, repairing, or removing held names does not reauthorize
them. A new image must exist on the owning node. The request must include a non-null `volumes`
array. An explicit empty array removes all volumes.

```json
{"volumes": [{"name": "imagenet", "mount": "/datasets/training"}]}
```

Mounted read-only, mounted writable, attach-only read-only, and attach-only
writable entries can change live, including mixed lists. The claim-time name,
mount, and eight-volume limits apply. An omitted mount defaults to
`/volumes/<name>` unless that entry has `attach_only: true`. Responses carry
that flag directly, so a GET target can be sent back with PUT. Omitting the flag
or sending `false` does not inherit an existing attachment's shape.

To change a held name's mount, mode, or `attach_only`, first remove the name and
wait for successful convergence. An identical target can be retried. Claims
with `on_expire: archive` must change that policy before adding a volume.

A claim that holds volumes is not replicated. Its replica retires on the next
replication pass, and a replica capture that races a live addition is refused.
The sandbox must be running. Paused, archived, and failed sandboxes return 409
without waking. A failed sandbox holding volumes is release-only after its VMM
exits; PUT with an empty target cannot restore its processes. Missing, invalid,
or sandbox bearer tokens return 401. Another tenant's sandbox and an unknown id
return 404. Invalid, inaccessible, or non-local entries return 400. Removed
tenant tokens are rejected at authentication. Root can still repair or remove
held entries, but admitting a new name for a removed tenant returns 403. A conflicting
writer or dirty image returns 409.

## PATCH /v1/sandboxes/{id}/volumes

Changes the desired set by name, then converges synchronously. Auth, validation,
and responses match PUT. Omitted `add` or `remove` arrays leave that part
unchanged. An empty patch retries the current desired set.

```json
{"add": [{"name": "imagenet", "mount": "/datasets/training"}], "remove": ["old-data"]}
```

Adding an identical entry or removing an absent valid name is harmless.
Duplicate names, a name in both arrays, or changes to a held entry's mount,
mode, or `attach_only` return 400. Removal refuses busy or foreign mounts and uses neither
lazy nor forced unmount. A foreign mount, another bind mount, or a completed
ordinary unmount with nonzero exit returns 409 with the guest mount path and
a safe per-volume reason. Guest transport failures remain server errors;
timeouts return 504 and cancellation returns 408.

A mounted writable attachment marks the image dirty before the guest can write.
Live removal and release clear the marker only when the unmount of the mount at
the volume's path succeeds, the mount was the filesystem's root and writable in
both its mount and superblock options (ext4's `emergency_ro` after a filesystem
error counts as read-only), and no other mount of its device remains. Live
removal finds the device by its serial and refuses any other mount of it;
release identifies the mount by its path alone and does not check which device
it is. These checks read silkd's own mount namespace, so a mount of the device
in another namespace, or one moved there, is not seen. Live removal also needs
a confirmed host detach. A missing device or pre-unmounted filesystem does not
prove clean unmount. The marker remains on uncertainty and blocks subsequent
readers. Release waits up to 30 seconds for each writable volume's unmount, and
a guest that does not answer within that keeps every remaining marker.
Attach-only writers have no automatic dirty-marker protection. The caller owns
mounting, flushing, unmounting, recovery, and fencing before reuse.

### Partial failure and recovery

PUT and PATCH return the GET response shape on success. A convergence error
returns a non-2xx status with the same view, so callers can inspect desired,
applied, and per-volume state; semantic validation 400s carry it too. A
malformed body, a missing `volumes` array, authentication and an unknown sandbox
carry only `error`. A failed request may already have persisted its target and
completed some device changes. GET reads the last confirmed set, rather than
probing the devices. Resolve guest mount references or runtime failures before
retrying. Retry the same PUT to converge, or send a new
target to abandon unfinished additions. Concurrent requests serialize per
sandbox without blocking renewals or new exec, file, and port connections.

Before applying a target, sandboxd journals `VolumesWant` beside `Volumes`,
storing it only while it differs from `Volumes`. A
superseding request may first clean up an older desired-only attachment while
the old target stays recorded. If that cleanup fails, the response retains the
old view; retry the same PUT to finish accepting the new target. Admission
holds cover the deduplicated union of both sets, including during restart and
teardown; a teardown keeps them until the VM is confirmed gone. Failed or
uncertain operations retain their holds until successful cleanup. Startup
reserves those holds without replaying attachments. Release cancels convergence and transfers the held union to teardown without waiting
for a blocked device call. Capture operations (hibernate, fork, checkpoint, and
promote) are refused while either set is nonempty or a convergence request
still owns admitted volume holds.

Guest device calls honor cancellation and have a 30-second ceiling per volume, and a
`cocoon` attach or detach can take 30 more to exit; a
caller that disconnects cancels the run, leaving interrupted states to retry. Journal writes
are atomic. Host-crash durability follows `sync_claims`.

Live convergence requires Cocoon v0.7.2 or newer for idempotent matching
attach and absent detach. PUT and PATCH reject an older release, or a runtime
whose version cannot be queried, with 503 before changing intent, holds, or
devices; a dev build is assumed current. sandboxd caches the first successful
version read, so restart it after upgrading cocoon. GET and release remain
available.

Catalog names must keep the same backing image while held, including pending
holds. Publish changes under versioned names such as `dataset-v1` and
`dataset-v2`. The journal cannot detect a catalog name repointed to a new
image.

A replacement first unmounts the old disk, then mounts the new disk. Pause reads
and close files, mmap, cwd, and bind mount references during this gap. Existing
independent guest processes and sessions continue. See the
[same-sandbox replacement tutorial](live-volumes.md).

## POST /v1/sandboxes/{id}/release

Auth: the sandbox's own token, or the root or operator token for cleanup by id.
Destroys the VM. 204 on success, 404 for an unknown id or wrong token, 500 when
the VM survives removal: the claim is already gone and the node retries the
removal, 503 `archive registry down; retry` for a claim that holds a registry
row while the meta store is down; resend it.
Releasing an already-gone sandbox is 404 — the SDK treats it as success.

## POST /v1/sandboxes/{id}/hibernate

Auth: the sandbox's own token, or the root or operator token by id (like
release). Atomically snapshots the VM and stops it,
freeing its memory; the next agent access restores it transparently
(sessions, processes, and memory state intact — cocoon's hibernate keeps the
snapshot point and the stop coincident), except a read-only file call that a
[reader](#get-v1sandboxesidagent) serves without waking it. Idempotent on an already-hibernated
sandbox. The TTL keeps running: a hibernated sandbox is still reaped (VM and
snapshot) at its deadline, unless it claimed `on_expire: archive` or its pool
archives. When to hibernate is the caller's policy — the
node only provides the transition. 204 on success, 404 unknown id or wrong
token, 409 on the egress lane or when volumes are attached (neither kind of
sandbox hibernates; see [egress](egress.md)) or on a failed sandbox, 429 while the disk is under
[`disk_reserve`](deploy.md), 503 `lease lost; retry` for a replicated claim that
is not archived while its node's lease is overdue, even one already hibernated.

## POST /v1/sandboxes/{id}/wake

Auth: the sandbox's own token, or the root or operator token by id. Restores
a hibernated (or archived) sandbox and leaves it running — waking is
otherwise only a side effect of the next agent access, so this is the
explicit form for warming a sandbox ahead of use. Idempotent on one already
running. A hibernated sandbox retains its existing deadline, unless it is past its lease
and archives at expiry, which starts a fresh lease. An archived
sandbox instead uses `archive_delete_after_seconds` as its retention deadline
while stored, and waking it starts a fresh lease of the length the claim was granted (the server default when it asked for none). A
[failed](deploy.md#dead-vmms-and-host-restarts) sandbox is cold-booted from
its own disk (memory is lost) and its restart budget resets. A hibernated
sandbox whose guest does not answer within the wake deadline after two restores
of its snapshot is cold-booted from its own disk instead: memory is lost,
`restarts` moves, and the snapshot is dropped. The boot records `vmm_restart`
when it starts and `wake`, with the lease, when the guest answers; a boot that
fails after its start records `hibernate`, and the claim stays hibernated. A
restored VM that cannot be stopped is not cold-booted: the wake fails and the
claim stays hibernated until the next wake restores the snapshot again. An
archived sandbox whose guest does not answer after two clones of its archive is
cold-booted from the second clone's disk: memory is lost, `restarts` moves, and
the wake records `unarchive` once it commits; a cold boot that fails removes
the clone, and the claim stays archived. The
same holds for a wake that an agent access triggers. 204 on
success, 404 unknown id or wrong token, or an archived sandbox the node had no
room for that it [handed to a peer](cluster.md#waking-an-archived-claim-on-a-full-node)
(look it up again), 409 a failed egress-lane or volume sandbox, or one whose VM
record is gone, 429 when `max_running` or `disk_reserve` refuses the wake, 500
when the cold boot or a restore fails, and the [guest errors](#guest-errors).

## POST /v1/sandboxes/{id}/renew

Auth: the sandbox's own token, or the root or operator token by id. Resets the
lease to `ttl_seconds` from now, so a client holding a sandbox longer than its
claim asked for does not have to claim a new one:

```json
{"ttl_seconds": 3600, "on_expire": "archive"}
```

→ `200 {"deadline": "2026-09-22T12:00:00Z"}`. The grant, not the request, is
authoritative: `ttl_seconds` 0 means the node default and anything past the
24 h cap is clamped, and the reply always carries what was granted. An absent
body means the same as `ttl_seconds` 0. The new lease also becomes the one a
wake from the archive grants again. A renew can shorten a lease as well as
extend it. `on_expire` switches the expiry action to `archive` or back to
`destroy`; absent keeps the current one. It is refused as on a claim for a
sandbox that cannot hibernate. A renew stamps no activity, so it neither restarts
the idle clock nor stops an idle hibernate.

An archived sandbox is refused: off-node, its deadline is the store's retention
window rather than a lease, and a TTL written over it would schedule the
checkpoint for deletion. Wake it first. A claim whose [tenant was
removed](#tenants-v1tenants) runs to its deadline but cannot extend it: its own
token gets 403, the root or operator token still renews. 400 a malformed body; 401 missing
bearer token; 403 the claim's tenant was removed; 404 unknown id or wrong
token; 409 archived, or being archived right now; 503 the archive registry is
down, for a hibernated claim with a replica under `archive_failover`: the node
already holds the new lease, and a resend replaces the row; 503 `tenant store
down; retry` for a tenant the node has not cached while the meta store is down.

## POST /v1/sandboxes/{id}/fork

Auth: the node `api_token` (Bearer) — forking creates node resources, like a
claim. The sandbox's own token rides in the body as the ownership proof; the
root `api_token` with an empty body `token` acts by id as root:

```json
{"token": "…", "count": 2, "ttl_seconds": 300, "claim_ref_prefix": "team-a/"}
```

`claim_ref_prefix`, when set, records each child under `prefix + child id` as
its `claim_ref`, so a control plane that names claims `<namespace>/<name>`
addresses the children by name; without it children carry no `claim_ref`.
The prefix is at most 512 bytes of printable UTF-8, as `claim_ref` is.
Children inherit the parent's `metadata`, `on_expire` and `"egress": false`; an
`on_expire` in the body overrides the inherited action.
Children do not inherit the claim env: a guest env file captured from the parent
is rewritten empty in each child, and the parent's host-only entries are not
copied; set one with `PUT /v1/sandboxes/{id}/env`.
Clones the sandbox into `count` fresh claims (1 up to the node's
`max_fork_count`, default 16). Memory, disk, and guest
state (sessions, processes, tmpfs) duplicate at the fork point; cocoon's
clone reseed gives every child a distinct machine identity. Children get
their own lease — `ttl_seconds` (0 = server default), never the parent's
remainder. A running parent is snapshotted in a brief pause window; a
hibernated parent forks from its existing memory image without waking.
All-or-nothing: on error no child survived. 200 with one claim per child:

```json
{"children": [{"id": "sb_…", "token": "…", "deadline": "…", "owner_addr": "…"}]}
```

Children inherit the parent's tenant and count against its `max_claims`,
whoever calls. 400 invalid count or body, a `claim_ref_prefix` over its bound or
an unknown `on_expire`; 401 bad api token; 403 the parent's tenant was removed
(sandbox token); 404 unknown id or wrong sandbox token; 409 egress-lane or volume
parent (neither forks, checkpoints, or promotes; see [egress](egress.md)), a
failed parent, or an archived parent (`POST /wake` or an exec wakes it first);
429 an [admission refusal](#admission-refusals); 503 `lease lost; retry` for a replicated parent while its node's lease is overdue, `tenant store down; retry` for a sandbox token whose tenant the node has not cached while the meta store is down.

## POST /v1/sandboxes/{id}/promote

Auth: like fork — node `api_token` in the header, the sandbox's own token in
the body; the root `api_token` with an empty body `token` acts by id as
root:

```json
{"token": "…", "template": "myproj:v1"}
```

Publishes the sandbox's current state as a node-local template under
(template, this sandbox's net, its size): later claims for that key clone
from it, provision-on-demand, with no warm pool.
Re-promoting to the same name replaces the template. A hibernated sandbox is
promoted from its memory image without waking. 200 returns the template's
full key and immutable content identity. On the default local-disk backend a
template is node-local, so a cluster client claims from and deletes on this
node (name-based calls route via gossip); a shared checkpoint store makes
every node resolve it. Under exactly this key:

```json
{"key": {"template": "myproj:v1", "net": "none", "size": "small"},
 "content_digest": "sha256:…"}
```

`content_digest` is SHA-256 over a versioned canonical stream of the published
export's regular files: slash-relative path, byte length, and the SHA-256 of
each 16 MiB chunk of its bytes, ordered lexically. Directory entries, modes,
mtimes, and the template's ownership/creation metadata do not affect it. The
digest is computed once while promoting, stored beside `meta.json` (a sibling
digest file on the directory backend, object metadata on S3), and therefore has
identical semantics on both. Re-promoting unchanged export bytes keeps the
digest; changing any exported path or bytes changes it. A re-promote starts the
template with no [labels](#put-v1templateslabelstemplatenetsize): they describe
the export they were set on, which the new one replaces.

400 invalid name, 401 bad api token, 409 when the name collides with a
configured pool, the template is owned by another tenant, or the sandbox is
on the egress lane, has volumes attached (see [egress](egress.md)), has its
[tun](#put-v1sandboxesidtun) turned off, has failed, or is archived (`POST /wake` or an exec wakes it first), 404 unknown id or wrong
sandbox token, 429 while the disk is under [`disk_reserve`](deploy.md), 503
`lease lost; retry` for a replicated sandbox while its node's lease is overdue.

## DELETE /v1/templates?template=…&net=…&size=…

Auth: node API token. Removes a promoted template (the query parameters
default like a claim's: `net=none`, `size=small`). A tenant may delete only
templates it promoted — anything else is 404, root deletes anything. 204 on
success, 404 unknown template, 409 when the key belongs to a configured pool
(those goldens are owned by the node config). `digest=<content digest>` deletes
only while the record still has that digest and answers 412 otherwise, so a
caller that observed an older build never removes the newer one that replaced
it on this node; on a shared store a promote committing on another node at the
same moment is not covered. On a cluster, a node that does not
hold the template but sees an owner in gossip answers `200
{"redirect": [addrs]}` — the claim redirect shape — and the SDK retries the
delete at the owner. The retry carries `no_redirect=1`, mirroring the claim
protocol: a node answering a `no_redirect` delete speaks only for itself.

## PUT /v1/templates/labels?template=…&net=…&size=…

Auth: node API token. Replaces a promoted template's labels on this node (the
query parameters default like a claim's):

```json
{"labels": {"v1": "sha256:…", "team": "a"}}
```

Labels follow the [claim `metadata`](#post-v1claim) rules — at most 16 pairs,
the same key and value grammar, at most 4 KiB as JSON — and replace the whole
map; `{}` clears it. `digest=<content digest>` writes only while the record
still has that digest and answers 412 otherwise, so a writer that observed an
older build never relabels the newer one on this node. They are stored beside the template's record, so they
survive a restart, are listed by the node that promoted the template or loaded
it at startup, and go with the template when it is deleted; a re-promote starts with none. A tenant may label
only templates it promoted — anything else is 404, root labels anything. 204 on
success, 400 bad labels or body, 404 unknown template, 409 when the key belongs
to a configured pool. The call speaks only for the node it reaches; it does not
follow a redirect.

## PUT /v1/pools

Auth: root only (tenant and operator tokens get 403). Replaces the node's desired warm
targets online — no restart, and no live claim's VM is touched. With
[`meta_store.cell`](deploy.md#shared-pool-set) set, it replaces them on every
node of the cell. On a node with egress policies,
whether a key is pooled decides which [policy layers](egress.md) a claim of it
gets; that is settled when the claim is made, so adding or dropping a pool here
changes only claims made afterwards — a key that gains a pool with no `egress`
block leaves new tenant claims without egress, a key that loses such a pool
gives new tenant claims their egress class's policy alone:

```json
{"pools": [{"template": "base:24.04", "net": "none", "size": "small",
            "warm": 4, "warm_max": 16, "idle_hibernate_seconds": 0}]}
```

Pools omitted from the list are drained: their unclaimed warm VMs are
destroyed and the pool entry retires. Once such a pool is idle, its golden
is deleted; a pool kept with `warm: 0` keeps its golden.
`net`/`size` default like a claim's.
Answers the fresh `GET /v1/info` payload. 400 bad key, negative warm/idle,
`warm_max` below `warm`, a reader pool with `warm` below 1, `idle_hibernate_seconds` on an egress pool, a
negative archive duration or an `archive_after_seconds` not above the pool's
`idle_hibernate_seconds`, duplicate pool, or a config-owned `egress`/`warmup`/`capture_trim`/`storage`/`replicate_seconds`/`reader`/`egress_upstream_env`/`tun`
field; 401 bad api
token; 409 egress pool on a node without an egress attachment; 503 the
`meta_store` cannot take a cell's write.

## Tenants (/v1/tenants)

Auth: root or operator (tenant tokens get 403); a node without `api_token` answers 400
to a write, since tenants need it. The API is the only way in: `config.json` has no
`tenants` field, and a node starts with no tenants. The set changes at
runtime, with no restart:

- `PUT /v1/tenants/{name}` `{"token": "…", "max_claims": 50, "egress_class": "desk"}`
  adds a tenant or changes one; an omitted `token` keeps the tenant's stored
  one, so a cap change never resends it, while `max_claims` and `egress_class`
  take the values sent. 204.
- `DELETE /v1/tenants/{name}` removes one. 204; 404 unknown tenant.
- `PUT /v1/tenants` `{"tenants": [{"name": "…", "token": "…", "max_claims": 0, "egress_class": "desk"}]}`
  replaces the whole set: a tenant left out is removed, and an entry without
  `token` keeps that tenant's. It answers the `GET` body. Use it to reconcile
  and to import many tenants at once, since every change rewrites the whole
  set; a sign-up is a single-tenant `PUT`, so concurrent sign-ups never
  overwrite each other. The 1 MiB body cap holds roughly 8,000 entries; import
  a larger set with single-tenant `PUT`s.
- `GET /v1/tenants?after=<name>&limit=<n>` → `{"tenants": [{"name": "…", "max_claims": 50, "egress_class": "desk", "claims": 3},
  {"name": "…", "claims": 1, "removed": true}], "digest": "…", "next": "…"}`.
  - **Paging:**
    - one page of tenants in name order, each with its live claims on this node;
    - `limit` defaults to 1000, at most 10000 (400 outside);
    - pass `next` as the following page's `after`; it is absent on the last page.
  - **Removed tenants:** the first page also lists each removed tenant that
    still owns claims here.
  - **Digest:** two nodes with the same `digest` hold the same set. A node on a
    shared [`meta_store`](deploy.md#shared-tenant-database) reports none.
  - It never carries a token.

A name follows the claim name grammar (`^[a-zA-Z0-9][a-zA-Z0-9._:/-]{0,62}$`;
escape `/` as `%2F` in the path) and is the tenant's identity: claims,
checkpoints and promoted templates belong to a tenant by name, so a name reused
after a removal inherits what the old tenant left. Use an id the control plane
never reissues, such as a user UUID. Tokens must be unique and differ from
`api_token`. `egress_class` names one of the node's `egress_classes` (400 when
unknown) and gives the tenant's claims that egress layer; without one they
reach nothing on a pool with a policy. The API references a class and never
defines one. A claim keeps the class its tenant had when it was made.

A change applies to the next request: a removed or rotated token gets 401 at
once. `max_claims` is a per-node cap (0 = unlimited), so a tenant's cluster
limit is `max_claims` × nodes; every usage event of a tenant's claim carries the
tenant, for per-tenant billing. Lowering `max_claims` below the live count
refuses new claims and evicts none. A removed tenant's claims keep running and stay listed for root
(`GET /v1/sandboxes?tenant=<name>`) until they expire or root releases them,
but they cannot extend: renew and fork answer 403 to the sandbox's own token,
a hibernated one is never archived, one archived before the removal is not
woken (403) and is deleted when its retention ends, and at the deadline they
are destroyed even under `on_expire: archive`. The verbs that need a tenant
token (promote, checkpoint, preview, branch claims) are closed to them already.

The node writes the applied set to `<data_dir>/tenants.json` (0600, each token
kept only as its SHA-256) before it serves, and loads that file at boot. Each
change is an `op:"tenants"` audit record naming the
tenants added, changed and removed. On a cluster every node needs the change
(see [cluster](cluster.md#tenants)). 400 a bad entry, a duplicate name or token,
a token equal to `api_token`, `api_token_next`, `operator_token` or
`operator_token_next`, an unknown field, or an unknown `egress_class`.

## POST /v1/config/reload

Auth: root only (tenant and operator tokens get 403). Re-reads the node's config file and
applies its reloadable settings at once, or nothing. SIGHUP runs the same
reload and logs the result. →

```json
{"changed": ["egress_internal_allow", "pools[desktop:v13 none small].warmup"],
 "ignored": ["pools[desktop:v13 none small] targets (PUT /v1/pools owns them)"]}
```

`changed` lists the settings that now apply. `ignored` lists differences the
file holds for pool targets, which `PUT /v1/pools` owns.
Which fields reload, and what happens to live claims and goldens, is described
in [deploy](deploy.md#reloading-the-config). The status codes:

- **400**: the file does not parse or does not validate.
- **409**: a field needs a restart (the message names it), intercept is
  turned on for a key this node has served or without a CA loaded at boot, or
  a pool's egress policy is removed while live claims still use it.

A refused reload changes nothing.

## POST /v1/drain

Auth: root only (tenant and operator tokens get 403). Empties the node for maintenance: claim/fork/branch answer
429 with an error ending `node draining` (on a cluster a non-volume claim tries a peer redirect first, and
peers stop redirecting here within a tick, because this node gossips no pools),
a live [move](#post-v1sandboxesidmove) to this node is refused,
unclaimed warm VMs are destroyed, and live claims keep serving until release
or TTL. Pool ownership is untouched — no pools.json write, no config change.
Answers the fresh `GET /v1/info` payload; poll `claimed` to zero to know the
node is empty. Deliberately not persisted: a restarted node serves again. To
keep new claims off a node across restarts, [cordon](#post-v1cordon) it; a
cordoned node still adopts existing claims, so to keep it empty after a
restart, drain it with `?evacuate=1` again; the
[admission table](cluster.md#node-admission) compares the states.

`POST /v1/drain?evacuate=1` drains the same way, then moves the claims to the
peers with their state, so the node empties without waiting for the leases to
run out (see [evacuating a node](cluster.md#evacuating-a-node)). It needs
[`archive_failover`](cluster.md#archived-claims-across-node-loss). In the
background, with bounded concurrency, each claim is hibernated if it runs,
archived to the checkpoint store if it is not archived, its registry row is
released, and a peer is asked to adopt it through `GET /v1/sandboxes/{id}/owner`,
so that peer lists it, and root's `GET /v1/sandboxes/{id}` there returns its token once that
node has seen it. A claim the root adopted, whose token this node has not seen, is offered with
the root token; a peer that does not accept the root on `/owner` refuses it, and the claim stays
here. Open streams to a
claim end when it hibernates. Until a peer holds it, a call that would wake the
claim answers 503 `sandbox moving to another node; retry` with `Retry-After`;
after that this node answers 404 and the client looks the claim up again. With
no peer to adopt it, the node takes the row back and the claim stays. A claim
that cannot move stays here
and keeps serving: an egress-lane claim, one with volumes, a failed one, one
whose tenant was removed, one archived before `archive_failover` was on, one
whose step failed, and every claim of a node without `archive_failover`. The
answer and `GET /v1/info` count the claims left in `evacuating`; once it is
gone, `claimed` counts the ones that stayed, and the log names each and why. A
claim of a pool without an archive tier, whose `on_expire` is not `archive`,
keeps its lease deadline across the move until its first wake, which starts a
fresh lease of the claim's length; any other keeps the later of its
lease and `archive_delete_after_seconds`, or no deadline when that is 0. A call
while an evacuation runs changes nothing; a call after one finished starts a
new one over the claims still here.

## DELETE /v1/drain

Auth: root only (tenant and operator tokens get 403). Lifts the drain, stops an evacuation, and kicks an immediate refill. Answers the
fresh info payload. A claim an evacuation is archiving finishes its archive and
stays here archived; the next call wakes it. A cordon stays in place.

## POST /v1/cordon

Auth: root only (tenant and operator tokens get 403). Keeps new claims off the node until
`DELETE /v1/cordon`, across restarts: claim/fork/branch answer 429 with an
error ending `node cordoned`, whatever `max_claims` and `max_running` allow (on
a cluster a non-volume claim tries a peer redirect first, and peers stop
redirecting here within a tick, because this node gossips no pools), refill
stops and unclaimed warm VMs are destroyed. Live claims keep serving, and unlike
a drain, a root [move](#post-v1sandboxesidmove) to this node is admitted, and so
is the [adoption](#get-v1sandboxesidowner) of an archived claim, since neither is a new claim. The
[admission table](cluster.md#node-admission) lists what each state allows; a
cordon and a drain are independent, and each `DELETE` lifts only its own.

The cordon is the file `<data_dir>/cordoned`; it survives a restart, a reboot
and a config reload, and a node that starts cordoned starts no warm VM. With
`audit_log` on, `audit.jsonl` records op `cordon`. Answers the fresh
`GET /v1/info` payload with `"cordoned": true`; 500 when the file cannot be
written, and the node stays as it was.

## DELETE /v1/cordon

Auth: root only (tenant and operator tokens get 403). Removes `<data_dir>/cordoned`, lifts
the cordon and kicks an immediate refill; `audit.jsonl` records op `uncordon`.
A drain stays in place. Answers the fresh info payload; 500 when the file cannot
be removed.

## POST /v1/sandboxes/{id}/move

Auth: root only (tenant and operator tokens get 403). Moves one running claim to a peer with
its guest's processes and memory (see
[moving a running claim](cluster.md#moving-a-running-claim)). Body:
`{"to": "<peer>", "max_downtime_ms": 2000}`, `to` as `GET /v1/peers` lists it
and never this node (else 400), `max_downtime_ms` optional with a default of
2000 and a floor: 400 or less answers 400 at once, because the target's receive
and start alone are allowed that long. Answers `{"owner_addr": "<peer>", "rounds": n, "downtime_ms": d, "capture_ms": c, "send_ms": s, "commit_ms": m}` once the
peer took the claim and this node dropped it; a guest the peer cannot start is
failed there. `downtime_ms` runs from the final capture's start, including its
wait for a free dump slot, until the peer answers the commit, after it started the guest; of it, `capture_ms`, `send_ms` and `commit_ms` are the final capture, the final round transfer and the commit. A final round whose capture time and size no longer fit `max_downtime_ms` is aborted before it is sent and the guest resumes here, so with the meta store up a capture that alone overruns pauses the guest for the capture plus the abort and its record write. While the move runs, release, fork,
checkpoint, promote, hibernate and volume changes of the claim answer 503
`sandbox moving to another node; retry`. Env writes, renew and data-plane calls keep working until
the final round and answer the same 503 during it, because the peer would miss
them. Every error leaves the claim running here, except one after the final round
paused the guest whose abort did not land (a 503 with the registry down, or a
failed `cocoon vm migrate abort`): the guest stays paused until the abort is
retried through, the guest then resumes or, when the record reads `committed`, the
node drops the claim (see [moving a running claim](cluster.md#moving-a-running-claim)):

- 403 the claim's tenant was removed
- 404 unknown sandbox
- 409 the claim cannot move (hibernated, archived, failed, waking, with
  volumes, egress lane, a move already running), cocoon or its run dir cannot
  migrate, a capture failed (the VMM died mid-move), the target refused (`target <peer>: <reason>`), or the node lacks
  a `meta_store` cell or `mesh.cluster_key`
- 429 `move_concurrency` reached here or on the target, or the target's
  admission refused (`target <peer>: <reason>`)
- 502 the target is unreachable or failed
- 503 the move or archive registry is down, the node lease is overdue, the node
  is evacuating the claim (`sandbox moving to another node; retry`), or the
  final round would not fit `max_downtime_ms`, after at most 4 deltas or once
  its capture overran it with the guest paused (the guest resumes here)

The target serves root-only peer routes for a move, `POST /v1/moves/{id}`,
`PUT /v1/moves/{id}/rounds/{n}`, `POST /v1/moves/{id}/commit` and
`DELETE /v1/moves/{id}`; they are no public API.

## POST /v1/sandboxes/{id}/preview

Auth: node `api_token` in the header, the sandbox's own token in the body.
Unlike fork, promote and checkpoint — where the root token with an empty body
token acts by id as root — preview has no root path: the body
token is required. Mints a signed URL serving a
guest HTTP port from a browser: body `{"token": "...", "port": 8080,
"ttl_seconds": 0}` → `{"url": "<preview_advertise>/p/<token>/"}` (`http://` is
prefixed when `preview_advertise` names no scheme).
The URL's life is clamped to the claim's remaining lease; an archived claim
kept forever (`archive_delete_after_seconds: 0`) has no lease, so there the
requested `ttl_seconds` stands unclamped and `ttl_seconds: 0` mints a one-hour
URL; releasing the sandbox ends it early either way.
400 a port of 0, 404 unknown id or wrong sandbox token, 501 when the node
has no `preview_listen`. The signed token embeds the sandbox id, port, and
owner `advertise_addr`, so any node's preview listener can serve it (forwarding
to the owner's main listener) and a released sandbox's URL answers 404 —
no revocation list. A browser request to the minted URL answers 503
`lease lost; retry` for a replicated claim while its node's lease is overdue, 503 `sandbox moving to another node; retry` while an [evacuation](#post-v1drain) has yet to move it, and 429 when it has to wake a sandbox while the disk reserve is low or the node is at `max_running`; a wake that [hands an archived claim to a peer](cluster.md#waking-an-archived-claim-on-a-full-node) answers 404, as after a move. See [deploy](deploy.md#preview-urls).

## POST /v1/sandboxes/{id}/checkpoint

Auth: node API token; body `{"token": "<sandbox token>", "name": "..."}`
(name optional); the root `api_token` with an empty body `token` acts by id as
root. Captures the sandbox's full state without stopping it and
answers `200 {"checkpoint": {id, name?, sandbox_id, key, tenant?, created_at,
policy_source?, no_egress?, tun_off?, tun_profile?, egress_limit?, guest_env?}}` — `tenant` records the calling tenant, absent for root.
400 bad body or name, 401 bad api token, 404 unknown id or wrong sandbox
token, 409 egress-lane sandbox, one with volumes attached (see
[egress](egress.md)), a failed one, or an archived one (`POST /wake` or an exec
wakes it first), 429 while the disk is under [`disk_reserve`](deploy.md), 503 `lease lost; retry`
for a replicated sandbox while its node's lease is overdue.

## POST /v1/checkpoints/{id}/claim

Auth: node API token; body `{"ttl_seconds": 0, "no_redirect": false,
"metadata": {}, "on_expire": "destroy", "env": {}}`. Claims a fresh sandbox branched from the checkpoint (a
normal claim response, attributed to the caller); the checkpoint's recorded
key applies. A tenant token branches only its own tenant's checkpoints; any
other id answers 404, as for list and delete. On a node that does not hold the
record, the redirect and the peer heal below run before that check, so another
tenant's id can redirect to its holders, or be pulled here before the 404;
checkpoint ids are random and no cross-tenant read serves them. The branch
carries this request's `metadata`, `on_expire` and `env`, under the claim rules,
never the source sandbox's. 400 metadata over a bound, an unknown
`on_expire`, or a bad `env` entry.

Checkpoints are node-local (unless the store is shared — see
[Configuration](deploy.md#configuration)). On a miss a node with a node-local
store probes its peers (a shared s3 store skips the probe: every node already
resolves every record) and answers with a redirect shaped like a warm-miss
[`POST /v1/claim`](#post-v1claim), at most 3 addresses; retry the same body
(+`no_redirect: true`) at each candidate. With nothing to redirect to (or
`no_redirect` set) and `checkpoint_peer_heal` on, it pulls the record and serves
the claim. See the [placement lifecycle](cluster.md#checkpoints-on-a-cluster).

404 for an unknown checkpoint (locally, and after redirect and heal both
miss), 429 an [admission refusal](#admission-refusals), 503 when the node's
concurrent-heal cap is already full — retryable, and the response carries a
`Retry-After` hint.

## GET/HEAD /v1/checkpoints/{id}/blob

The peer-transfer route behind the probe and heal above — internal, not
part of the public API; an SDK caller has no reason to call it directly.

- `GET` streams the whole record — guest memory, disk, and meta — as a tar
  in which every export file travels as its size plus its non-zero 16 MiB chunks,
  each with its SHA-256, so holes cost nothing on the wire.
  Root only: the stream carries no tenant scoping, and its only
  real caller is a peer's heal pull, which presents the fleet `api_token`.
  401 missing or unrecognized token, 403 a valid tenant or operator token
  (authenticated but not root), 404 unknown checkpoint.
- `HEAD` is the ownership probe: 200 when this node holds a branchable
  (non-archive) copy, 404 otherwise, and 401 on a keyed mesh when
  `X-Cocoon-Probe` is absent or expired. On a mesh with `cluster_key` set the
  request must carry `X-Cocoon-Probe`, an HMAC over the id and a coarse time
  bucket keyed off a probe-specific derivation of the cluster key — verified
  before any disk is touched, replayable for roughly ninety seconds at most. On a
  keyless mesh (redirect-only fleets have no shared secret to sign with) the
  id itself remains the only capability, matching the mesh's own posture.
  Repeat probes for one id are answered from a short positive cache on the
  asking node, which a local delete evicts immediately.

## GET /v1/checkpoints

Auth: node API token. Lists this node's checkpoints, newest first. A tenant
sees only its own records; root sees everything. A checkpoint backing an
archived claim is that claim's wake image (an `ar_` id), not a listable record. Answers
`200 {"checkpoints": [{id, name?, sandbox_id, key, tenant?, created_at, policy_source?, no_egress?, tun_off?, tun_profile?, egress_limit?, guest_env?}]}`, the
record shape `POST /v1/sandboxes/{id}/checkpoint` returns.

## DELETE /v1/checkpoints/{id}

Auth: node API token. A tenant may delete only its own records — anything
else is 404, never a hint the id exists; root deletes anything. 204 on
success, 404 unknown — including the checkpoint behind an archived claim,
which only that claim's release or retention window removes.

**Delete removes the local record, then best-effort broadcasts to peers so a
healed replica does not outlive it — eventual cleanup, not a fleet-wide
revocation.** A peer offline during the broadcast keeps its copy until the
checkpoint TTL ages it out, so an id-holder can still branch it for that
window; [placement lifecycle](cluster.md#delete-is-eventual-not-a-fleet-wide-revocation)
has the bound and why heal requires a nonzero, fleet-matching TTL. The s3
checkpoint store skips the broadcast: every node already resolves every
record directly, so there is no replica to chase. A mesh without a
`cluster_key` skips it too: it has no healed replica, and the broadcast would
send the `api_token` to addresses learned from unauthenticated gossip.
`?no_forward=1` marks a
delete already arriving from another node's own broadcast, so it is not
itself re-broadcast (loop prevention); it is an internal parameter, not one
an SDK caller should set.

## GET /v1/sandboxes

Auth: node API token or the operator token. Root and the operator see every live claim; a tenant sees only its own.
The index is `{"sandboxes": [{id, key, tenant?, deadline, claimed_at?, hibernated,
archived?, from_checkpoint?, claim_ref?, metadata?, on_expire?, cpu_count, mem_total_bytes,
volumes?: [{name, mount?, mode?, attach_only?}], volumes_want?: [{name, mount?, mode?, attach_only?}], restarts?, restarted_at?, failed?, env_missing?}], "next": "…"}` — `deadline` is `0001-01-01T00:00:00Z` on an archive kept forever; `cpu_count` and `mem_total_bytes` are the
size tier's allocation; `restarts` counts the cold boots that replaced an
exited VMM or a hibernate snapshot or archive whose guest did not answer a
[wake](#post-v1sandboxesidwake), and `restarted_at` stamps the last one (each loses guest memory and
the instance-metadata document, so a caller re-establishes guest state when
`restarts` moves; the claim's env is written to the guest again); `failed` names why the VMM is down and was not restarted
(see [dead VMMs](deploy.md#dead-vmms-and-host-restarts));
`env_missing` is true while a `guest: false` env entry has no value on the node
([claim env](#getputpatch-v1sandboxesidenv));
`mode` is omitted for `ro`, matching the claim echo; never sandbox tokens,
volume host paths, or catalog access lists. `claimed_at` is the first grant
(renew and wake move `deadline` only). `?claim_ref=<ref>` keeps only the claims recorded under
exactly that reference, so a caller that named its claim reads its row without
the rest of the index; an empty value lists everything. `?metadata=key=value`,
repeatable, keeps only the claims holding every pair; 400 a pair without `=`,
with an empty key, or with a key repeated. `tenant` names the owning tenant
(absent for root's own claims); `?tenant=<name>` lets root list one tenant's
claims, a removed tenant's included, and is ignored for a tenant token.

- **Order and paging:**
  - rows come in `id` order;
  - without `limit` the reply is the whole listing and carries no `next`;
  - `limit` (1 to 10000, 400 outside) caps the rows of a page, after every
    filter and the tenant scope apply;
  - `after=<id>` returns only claims with a larger `id`;
  - `next` is the last row's `id` when a limited page is full, absent
    otherwise; pass it as the following page's `after`;
  - a claim that stays live through the walk appears exactly once;
  - a claim released while its page is built can still appear on it with its
    last state; one released earlier is absent;
  - a claim added during the walk appears only when its `id` sorts after the
    cursor of the page that reads it.

## GET /v1/sandboxes/{id}

Auth: root or operator (tenant tokens get 403). One live claim in the index-row shape above, so a reconcile
loop can read a single sandbox without scanning the whole node listing. When
the caller presents the configured `api_token` or `operator_token` the row also carries `token`,
the sandbox's own bearer token, so a control plane can hand a returning client
its data-plane credential without keeping a copy; a node without an
`api_token` never grants per-sandbox authority and answers without it. The
row also carries the [advanced network mode](advanced-network.md#host-side)
state, each field omitted when unset: `tun_addr` (the claim's overlay address),
`tun_off` (true when the claim is disabled) and `tun_profile` (the exit label;
absent means `default`), and `egress_limit`, the claim's
[bandwidth cap](#put-v1sandboxesidegress-limit), omitted when it has none.
404 unknown id.

## GET /v1/sandboxes/{id}/stats

Auth: root or the operator token (tenant tokens get 403). One sandbox's resource usage — the per-sandbox counterpart
to the node-scoped `/metrics`:

```json
{"id": "sb_…", "cpu_count": 2, "mem_total_bytes": 1073741824,
 "mem_used_bytes": 187654144, "mem_used_measured": true,
 "cpu_seconds": 5321.47, "cpu_started_at": "…", "cpu_measured": true,
 "hibernated": false, "measured_at": "…"}
```

`cpu_count`/`mem_total_bytes` come from the size tier. `hibernated` marks a
claim whose memory snapshot is on this node, and `archived`, present only
when true, one whose state is in the checkpoint store; neither has a VMM to read. `mem_used_bytes` is
the host VMM process's resident set — the only memory usage signal
available without a guest agent. `cpu_seconds` is the same process's user plus system
CPU time, all threads, guest time included, and `cpu_started_at` is when that
process started. The counter starts over with the process (a wake, a restart
or a move), so take the rate between two reads only while `cpu_started_at` is
the same. `mem_used_measured` and `cpu_measured` are false when there is no
VMM process to read (hibernated or archived, the PID not yet known, or the engine lookup
failed), so a zero is never mistaken for idle; `measured_at` is taken after
the reads. One read costs one engine lookup; to read every claim, use
[`GET /v1/stats`](#get-v1stats). 404 unknown id.

## GET /v1/stats

Auth: root or the operator token (tenant tokens get 403). Every live claim's
stats on this node, sorted by id, in the shape of
[`GET /v1/sandboxes/{id}/stats`](#get-v1sandboxesidstats):

```json
{"sandboxes": [{"id": "sb_…", "cpu_seconds": 5321.47, "cpu_started_at": "…", "cpu_measured": true, …}]}
```

The whole read costs one engine listing and one `/proc/stat` read, whatever
the claim count, plus two `/proc` reads per running claim. 503
`vm listing failed; retry` when the engine cannot list its VMs.

## GET/PUT/PATCH /v1/sandboxes/{id}/env

Auth: node API token (root or tenant) or the operator token, which acts as root. A tenant reaches only claims it owns;
anything else is 404, like an unknown id. Each call speaks only for the node
it reaches; on a cluster, send it to the claim's `owner_addr`.

`PUT` replaces the claim's whole env with the body's map, under the claim
rules; `{"env": {}}` clears it. `PATCH` merges: each entry in the body is set,
an entry of `null` removes that name, and every other entry is kept as stored,
host-only values included, so a control plane that cannot read those values
back sends them again only when the node reports them missing (below):

```json
{"env": {"CB_META_wake": {"value": "w2"}, "OLD": null}}
```

The size rules apply to the merged env, so a patch that takes it past 64
entries or 64 KiB answers 400 and changes nothing. `GET` answers the same shape with every
`guest: false` value blanked, so a control plane sees the names without the
values:

```json
{"env": {"MODE": {"value": "prod"}, "GW_KEY": {"value": "", "guest": false}}}
```

An entry with `inject` comes back with its hosts, header or query, body flag and placeholder, value blanked.
An `inject` host the claim's pool does not intercept, or its tenant class does
not allow, and a header the pool rule's own secret already sets on that host,
answer 400 and store nothing.

An entry the node holds no value for comes back with `"missing": true`, and
the claim's index row carries `env_missing`. Under
[`host_env_memory_only`](deploy.md#configuration) that happens when the claim
hibernates, after a restart and after an adoption. A missing entry injects
nothing and blocks the node's value for its secret; a missing upstream entry
falls back to the class or pool upstream. Send it again once the claim runs:
a `PATCH` replaces the entry whole, so repeat `guest` and `inject`:

```json
{"env": {"GW_KEY": {"value": "Bearer …", "guest": false, "inject": {"hosts": ["api.example.com"], "header": "Authorization"}}}}
```

A value sent to a hibernated claim stays in that node's memory until the
claim hibernates again.

The env is persisted with the claim, its `guest: false` values only without
`host_env_memory_only`. A `guest: false` change takes effect on
the egress proxy's next request at any time, a hibernated or archived claim
included, and never touches the guest. A change to the guest entries is
written to the guest at once, so it needs a running guest: on a hibernated,
archived, failed or transitioning sandbox it answers 409 (this verb never wakes one).
A `PUT` or `PATCH` whose result equals the stored env rewrites a running
guest's file, so resending repairs a failed write, including a removal or
a change from guest to host-only. This also applies to an unchanged
host-only patch; the file contains only the guest entries. An unchanged
result does not rewrite the claims journal. A 500
after the store step means the env is stored and its `guest: false` values
are already live, but the guest write failed; resend the same request to
deliver it. The guest side is described in [silkd](silkd.md#claim-env).
`PUT`/`PATCH` answer 204, `GET` 200; 400 a bad entry, an unknown body
field or an `inject` outside the claim's intercepted hosts; 404 unknown id or another tenant's claim; 409 a guest change on a
paused or failed sandbox; 503 the archive registry is down, for an archived
claim or a hibernated claim with a replica under `archive_failover`: the node
already holds the new env, and a resend replaces the row.

## PUT /v1/sandboxes/{id}/instance-metadata

Auth: root only (tenant and operator tokens get 403). Sets the sandbox's instance-metadata document: the JSON a guest
agent reads from `169.254.169.254` in the IMDSv2 shape, as cloud-init and
similar agents do (`PUT /latest/api/token`, then `GET /` with the
`X-metadata-token` header). It is unrelated to the claim's `metadata` labels.
The body is one JSON object of at most 4 KiB, stored and served verbatim; a
guest with no document answers `{}`, and a fork child, checkpoint branch or
template clone starts with the document its source held. The node writes it to
`/run/silkd/instance-metadata.json` in the guest atomically, so repeating a call
is a no-op. silkd serves the address only in images that alias it on `lo`.

204 on success; 400 when the body is not one JSON object within the bound; 404
unknown id; 409 when the sandbox is failed, hibernated, mid-transition or archived (this
verb never wakes one) or its image serves no instance metadata.

## PUT /v1/sandboxes/{id}/tun

Auth: root only (tenant and operator tokens get 403). Switches a claim's [advanced network mode](advanced-network.md)
and sets its exit profile label:

```json
{"enabled": false, "profile": "residential"}
```

Both fields are optional; an omitted field keeps the claim's value. `profile`
names an entry of `tun.exits`; an empty string returns the claim to `default`,
and a label the exit table does not hold yet is accepted, since the table
reloads (until it appears, the claim's packets are dropped as having no exit).
The state is persisted with the claim, follows it through hibernate, archive,
move and an archive row, and is accepted on a hibernated or archived claim
without waking it; its next wake applies it. A running claim's replica row
under `archive_failover` picks the change up at its next replica capture, as
an env change does, so an adoption before then restores the switch with the
rest of that replica. On a running claim the change
applies at once: disabling closes the claim's packet stream and its door, drops
its flows and stops its ARP answers, then sends the guest `tun_down`, except
while a checkpoint, fork or replica capture holds the claim, when the guest
keeps a `tun0` that carries nothing; enabling, or changing the profile, re-arms the packet door and sends `tun_up`.
Enabling a claim of a `tun` pool that holds no overlay address, because the
address store failed at its claim, allocates one. A fork child takes its
parent's switch and label, with its own address.

204 on success; 400 an unknown field or a `profile` that does not match the
name rule; 404 unknown id; 409 an enable or profile change while a
transition (a capture, wake, hibernate or archive) holds the claim (retry; a
disable never waits), or another tun request to the claim committed first
(retry); 503 the claim is in a live move's final round or not yet
started on its target (retry); 500 the address store failed or the claims
journal could not be written (nothing changed); 503 the archive registry is
down for an archived claim or one with a replica under `archive_failover`:
the node already holds the change, and a resend of the same body replaces the
row.

## POST /v1/sandboxes/{id}/balloon

Auth: root or operator; a tenant token answers 403. Sets a running claim's
virtio-balloon target, the guest memory the balloon withholds from the guest,
through one `cocoon vm balloon` call, and answers with cocoon's report:

```json
{"target": "50%"}
```

→ `200 {"target": 2147483648, "actual": 1073741824, "default": 1073741824}`

`target` is what `cocoon vm balloon` accepts: `NN%` of guest memory (0 to 99)
or a byte size (`2G`, `512M`, `1073741824`), below guest memory. In the reply,
`target` is the size cocoon set in bytes, `actual` what the balloon holds now
when the hypervisor reports it, and `default` the boot target. Send `default`
back as the target to restore the claim. The call returns before the guest
settles. sandboxd keeps no balloon state and never changes a target on its
own: the caller sets it and the caller restores it. A hibernate wake and a
live move keep the target; an archive wake, a cold VMM restart, a fork and a
clone start at the boot target. cocoon serializes the call with captures and
live-move rounds.

200 with cocoon's report; 400 a target cocoon rejects or an unknown field;
404 unknown id; 409 `cocoon on this node has no vm balloon`,
`the sandbox's VM has no balloon device`, or a hibernated, archived, failed or
waking claim or one a hibernate is capturing; 503
`sandbox moving to another node; retry` during a live move; 500 the cocoon
call failed or outran its 30 s budget, or cocoon found the guest paused (a
capture or a move round that began after the node's checks).

## PUT /v1/sandboxes/{id}/egress-limit

Auth: root only. Caps a claim's egress bandwidth in bytes per second, each
direction apart, `tx` guest to destination and `rx` back:

```json
{"tx_bytes_per_second": 1250000, "rx_bytes_per_second": 2500000}
```

Zero or an absent field leaves that direction unlimited, so `{}` removes the
cap. It covers the claim's [proxy doors](egress.md#bandwidth-limit) and, in
[advanced network mode](advanced-network.md#host-side), replaces
`tun.limits.bytes_per_second` for the claim in each direction it sets. Preview,
port relay and exec traffic is not capped. On a running claim the change
applies at once to open connections, without a reconnect; an uncapped direction
can pass one more splice, up to 1 MiB at the old rate. The cap is persisted
with the claim, follows it through hibernate, archive, a live move, an archive
row and an adoption, and is accepted on a hibernated or archived claim without
waking it; its next wake applies it. A running claim's replica row under
`archive_failover` picks the change up at its next replica capture, which for a
claim that hibernates first comes after its next wake. A fork child and a
checkpoint branch take their source's cap. A new claim starts without one; no
pool setting gives a default.

204 on success; 400 an unknown field or a negative rate; 404 unknown id; 503
`sandbox moving to another node; retry` while a live move pauses the claim for
its final round; 500 the claims journal could not be written (nothing
changed) or the claim's row could not be rewritten (the node holds the change;
resend the same body); 503 the archive registry is down for an archived claim
or a hibernated one with a replica under `archive_failover`: the node already
holds the change, and a resend of the same body replaces the row.

## GET /metrics

Auth: root only (tenant and operator tokens get 403). Prometheus text format,
hand-rendered: pool warm/target gauges, claimed/running/max_running/hibernated/archived/failed/draining/cordoned
gauges, `sandboxd_evacuating` (the claims an evacuation has yet to move), `sandboxd_memory_low` and `sandboxd_anon_bytes` (see [`memory_limit`](deploy.md#configuration)), a per-tenant live-claim gauge (`sandboxd_tenant_claims{tenant="…"}`,
tenants holding live claims only, so the label set is bounded by the node's claims), `sandboxd_config_digest_mismatch` on a mesh, claims
by tier (warm/clone/cold),
wake/hibernate/fork/checkpoint/promote/release/reap counters plus
archive/unarchive/archive-delete counters, `sandboxd_dns_malformed_total` (guest DNS queries dropped as malformed), claim/wake `*_seconds_total`
for average latency, `sandboxd_claim_duration_seconds{tier}`, a histogram of pool
claims by tier, `sandboxd_claim_redirects_total`, claims this node handed to a
peer, `sandboxd_wakes_refused_total{reason}`, wakes `max_running` (`max_running`),
`disk_reserve` (`disk`) and `memory_limit` (`memory`)
refused here, whether or not a hand-off ran, `sandboxd_wake_handoffs_total`,
archived claims this node [handed to a peer with room](cluster.md#waking-an-archived-claim-on-a-full-node)
instead, and `sandboxd_wake_memory_fallbacks_total`, archived wakes over
`memory_limit` this node woke itself because no hand-off ran or no peer adopted. On a node with a `tun` block it adds the packet path's counters:
`sandboxd_tun_frames_total{dir}` (frames on the claims' vsock streams, `out`
from the guest and `in` to it), `sandboxd_tun_datagrams_total{dir}` (VXLAN
datagrams), `sandboxd_tun_arp_total{kind}` (`answered` replies to exits,
`announced` gratuitous ARPs) and `sandboxd_tun_drops_total{reason}` (packets
not carried: `malformed`, `not_ipv4`, `options`, `fragment`, `proto`, `gso`,
`policy`, `no_exit`, `unknown_claim`, `queue_full`, `size` (a packet or GSO
segment over `tun.mtu`, or a GSO frame of more than 128 packets), `limit` (a claim's rate, flow or quota limit), `send`
(the underlay refused a send) and `sender` (VXLAN from an address that is no
exit)). The warm hit rate is `sandboxd_claim_duration_seconds_count{tier="warm"}`
over the three tiers' counts: `sandboxd_claims_total` also counts forked children
and checkpoint branches under `clone`, and the histogram does not. A claim with
volumes includes their attach and mount in its tier. /metrics is a derived ops view; the billing source of
truth is the usage journal below.

## Usage journal (usage.jsonl)

Always on: every lifecycle transition appends one JSONL event to
`<data_dir>/usage.jsonl` — `{"t": <RFC3339>, "ev":
"claim|renew|hibernate|wake|fork|checkpoint|adopt|promote|release|reap|archive|unarchive|archive_delete|egress|egress_bytes|vmm_exit|vmm_restart|vmm_failed|move_in|move_out",
"id": "sb_…"}` plus `vm` (the VM's name; absent on `renew`, `adopt`, `archive`,
`unarchive`, `archive_delete`, `egress`, `egress_bytes`, and the `reap` or
`release` of an archived claim), `key` (the pool key's stable hash, claim
events), `template`, `net` and `size` (the claim's template, lane and size
tier, claim events, so a consumer can price a sandbox by its shape without the
claim record, which is gone after release; `size` sets the CPU and memory,
and `template` is a label, since promote lets the caller name it), `deadline` (the deadline in force after this event: `claim`,
`renew`, `wake`, `unarchive` and `move_in` carry the lease, `archive` and `adopt` the retention
end, or nothing for an archive kept forever, and `reap` the deadline it ended under), `tenant` (the owning tenant, on every event of a
tenant's claim), `children` (fork) and `ref` (the promoted template / checkpoint
id, the egress host, a `vmm_failed` reason, or why reconcile ended a claim). `egress` events, one per egress decision, are written only with
[`egress_usage_events`](deploy.md#configuration) on (default off), and `egress_bytes` events only with
`egress_usage_bytes` on. An egress event that injected credentials names
them in `secret` (`gh,claim:<entry>`, never a value). An egress event routed through an
[upstream proxy](egress.md#upstream-proxies) carries `upstream` (its `host:port`), and with
`egress_usage_bytes` on, `egress_bytes` events add the connection's payload `tx`/`rx` bytes. With
[`egress_usage_interval_seconds`](deploy.md#configuration) set, a connection or flow that lasts longer
writes an `egress_bytes` event each interval with the bytes since its previous one, then one with the
rest, if any, when it ends. Events carry no connection key: sums per claim and per upstream match the
single events an interval of 0 writes. A sandbox records at most 10 denied or rate-limited egress decisions per second after a burst of 100, so `egress` events do not count every refusal. A volume claim also carries
`volumes`, the applied catalog names, and — omitted when empty — `volumes_rw`,
the subset of those names claimed `rw`, on its `claim` event, so billing can
discriminate write access at claim (mounts and host paths are not billing
dimensions); a live attach or detach records no usage event. The file rotates at
1 GiB keeping one `.1` backup, so a tailing collector never loses a window
silently. A last line a crash cut short is ended with a newline when the node
reopens the journal, so it never swallows the next event. A [live move](#post-v1sandboxesidmove)
records `move_in` on the target when it starts the guest and `move_out` on the
source when it drops the claim, each with its own node's `vm`. Folding rules: fold a sandbox's events of all nodes in `t` order.
An event ends the current span at its `t` and enters a state: `claim`, `wake`,
`unarchive`, `vmm_restart` and `move_in` enter `running`, `hibernate` enters `hibernated`,
`archive` enters `archived`, and `vmm_exit`, `archive_delete`, `release` and
`reap` enter `ended`. Other events, `vmm_failed` and `move_out` included, change no state, and
`reap`, `adopt` and `archive_delete` have the exceptions below.
Billable compute, hibernated storage and archived storage seconds are the time
in `running`, `hibernated` and `archived`, a cheaper store tier than a hibernated
VM's RAM. `vmm_exit` is stamped when the node first finds the VMM gone, so an
outage before sandboxd runs again (a host that was down) stays billed up to that
point. A `reap` instead ends the current span at min(`t`, its `deadline`), or `t`
with none, because `t` can follow the true end by the reaper's tick, the VM's
teardown or the node's downtime. An `archive_delete` ends the current span at
min(`t`, the last `deadline`): in `archived` that is the retention deadline, which a
purge trails by the sweep's tick or the node's downtime, and a purge can follow a
claim that has no end event by a whole retention window. An `adopt` names a claim
a node took over, from a lost node or from its own lost record, with `ref` the
checkpoint it holds. In `archived` it changes nothing; otherwise it ends the
current span at min(`t`, the last `deadline` before it) and enters `archived` at
its `t`. A span is never shorter than zero. A span still open ends at min(now,
the last `deadline`), and one entered by an `archive` or `adopt` without a
`deadline` stays open. A claim reconcile drops because its VM record is gone
ends with a `reap` whose `ref` is `vm_gone`, and one that cannot re-lock its
egress NIC with `quarantined`; a claim another node holds is dropped with no
event, and neither has a replicated claim dropped while the registry is down. The `vm` name joins
cocoon's machine-level metering ledger for audit cross-checks. With
`meta_store.usage`, the node also lands every line in the cell's
[shared usage table](deploy.md#shared-usage-table), read through
[`GET /v1/usage`](#get-v1usage).

## GET /v1/usage

Auth: root only (tenant and operator tokens get 403). Reads the raw usage events of every
node of the cell from the [shared usage table](deploy.md#shared-usage-table).

- `GET /v1/usage?tenant=<name>&since=<time>&until=<time>&after=<cursor>&limit=<n>` →
  `{"events": [{"cursor": 41, "node": "n1", "t": "2026-10-30T08:00:00.123456Z", "ev": "claim", "id": "sb_…", "vm": "sbx-…", "key": "…", "template": "…", "net": "none", "size": "large", "tenant": "acme", "deadline": "2026-10-30T09:00:00.123456Z"}], "next": 41}`.
- **Events:** each is one journal line with its keys as in
  [usage.jsonl](#usage-journal-usagejsonl), `t` in UTC to the microsecond, plus
  `cursor` (its place in the table; one node's events follow its journal
  order) and `node` (the node that recorded it). A journal written by
  v0.1.20 or earlier has no `renew` or `adopt` events and no `deadline`, a claim
  reconcile dropped has no end event, and its `claim` lines have no `template`,
  `net` or `size`.
- **Filters:** `tenant` keeps one tenant's events; `since` (inclusive) and
  `until` (exclusive) are RFC 3339 times compared with `t`. Each is optional.
  A journal written by v0.1.20 or earlier stamps `tenant` only on `claim`,
  `archive`, `unarchive`, `archive_delete`, `egress`, `egress_bytes` and
  `vmm_*`; join its other events on `id` to the `claim`.
- **Order and paging:**
  - events come in `cursor` order, the order they landed, which is not `t`
    order: a node that was cut off from the database lands its backlog later,
    and a `renew` carries the time its lease changed, so its `t` can precede
    that of an earlier line of the same journal;
  - `after` returns only events with a larger cursor (default 0);
  - `limit` defaults to 1000, at most 10000;
  - `next` is the last event's cursor when the page is full, absent otherwise;
    pass it as the following page's `after`;
  - the cell's lands commit in cursor order, so polling with the last cursor seen
    as `after` returns every event exactly once, backfill included.
  - with [`meta_store.usage_retention_hours`](deploy.md#shared-usage-table),
    each node deletes its rows older than the retention once an hour, so page
    within the retention: an `after` below the oldest kept row returns from
    the oldest kept row, with no error, and the deleted events are gone. A
    backlog a node lands with `t` older than the retention can be deleted at
    its next sweep, before a consumer reads it.
- 400 on a malformed `after`, `limit`, `since` or `until`; 501
  `no shared usage store configured` on a node without `meta_store.usage`; 503
  `usage store down; retry` while the database is unreachable.

## GET /v1/sandboxes/{id}/agent

Auth: the sandbox's own token; the operator token gets 403. Requires `Upgrade: silkd` +
`Connection: Upgrade`; answers `101 Switching Protocols` and from then on
the connection is a byte-for-byte relay to the guest's silkd, carrying RPCs
back to back (see [silkd](silkd.md)). An open relay holds the sandbox's idle
clock, so a client that keeps a connection warm must close it when idle for
`idle_hibernate_seconds` to apply. A relay whose client vanished without a FIN
ends after about two minutes of failed TCP keepalives. Frames sent right behind the request,
before the 101, reach silkd in order; the Go and Python SDKs send their first
frames that way over plain HTTP, so a proxy in front of a plain-HTTP endpoint
must forward them too (Go's `httputil.ReverseProxy` drops them; Caddy forwards
them). 426 without the upgrade header, 404 unknown sandbox, 409 a failed sandbox,
500 a failed restore, 502 guest unreachable, and the
[guest errors](#guest-errors). A relay of a
claim of a replicating pool that is already open when the lease turns overdue
is closed, whether or not it has a replica yet.

A hibernated or archived sandbox wakes when the upgrade request arrives, before the `101`,
unless its last sleep is reader-safe, which the pool's [`reader`](deploy.md#reading-sleeping-claims-without-waking-them) setting provides;
what a reader is, what it needs and when the call wakes instead are there.
With it, the node answers the `101` before it touches the sandbox, reads the
first frame, and serves `fs_stat`, `fs_list`, `fs_read` and `fs_pull` from a
reader VM: the sandbox stays hibernated or archived, no `wake` usage event is
recorded, and the read stamps neither the idle nor the archive clock. An `info` frame wakes the sandbox, since a
reader would answer with its own processes and uptime. The first frame on the connection that
is anything else, or any later one, wakes the sandbox and hands the same
connection to its silkd with the unread bytes replayed, so a client sees one
silkd and no close. When the sandbox sleeps again meanwhile (a new hibernate or
an archive), the connection's next read moves to a fresh reader of the new sleep;
when it wakes, the connection moves to the woken sandbox, so reads never come
from a stale sleep. Pipelining a request behind a read that has not yet sent
its terminal frame is unsupported: when that request moves the connection, the
reader closes and the rest of the read's reply is lost. A read that no reader
can serve wakes the sandbox as if the feature were off. Because the `101` has
gone out by then, a failure on this path
(unknown sandbox, a refused wake, an unreachable guest, an overdue node lease)
arrives as a silkd `error` frame (`not_found` for an unknown sandbox, else
`internal`) rather than an HTTP status, and the connection ends. A connection
served by a reader, or handed over to the woken sandbox, of a claim of a
replicating pool is closed when the lease turns overdue, as an open relay is.

## GET /v1/sandboxes/{id}/ports/{port}

Auth: the sandbox's own token, or the root or operator token for a control-plane
read (below). Send `Upgrade: tcp` + `Connection: Upgrade`
(the gate reads `Upgrade` alone, as `/agent`'s does); answers
`101 Switching Protocols` and from then on
the connection is a raw byte relay to `127.0.0.1:{port}` inside the guest,
over the same vsock `port_forward` preview uses. Nothing of ours frames the
stream, so HTTP/1.1, HTTP/2 and any other protocol pass through unchanged.
This is how an edge proxy reaches a guest listener on a `net=none` sandbox,
which has no NIC.

Half-close works in **both directions**: a client write shutdown reaches the
guest socket, and a guest write shutdown reaches the client without ending
the client's writes. Each direction stays open until its own EOF. A relay
copy error closes both connections and releases the sandbox hold.

With the sandbox's token, an open relay holds the sandbox's idle clock exactly
like the silkd relay, and a hibernated sandbox wakes transparently — which is
why the upgrade header is mandatory: a bare `GET` must not consume a hibernate
snapshot. With the root or operator token the relay is passive, for control-plane
reads such as metrics: it never wakes the sandbox and never stamps activity,
so a poll neither undoes a pause nor keeps a sandbox from idling. A hibernated
or archived sandbox, or one with a hibernate, wake, checkpoint, promote or
fork in flight, answers `409` at once; an open passive relay still holds the sandbox against the idle sweep.
400 a port outside 1-65535; 401 missing bearer token; 404 unknown sandbox or
wrong token; 409 a failed sandbox or a passive relay to a paused one; 426 without the upgrade
header; 502 when the guest cannot be reached on that port — no listener, or a
hibernated sandbox whose restore failed. The node log carries which; and the
[guest errors](#guest-errors).

## POST /v1/sandboxes/{id}/exec

Auth: the sandbox's own token, or the root or operator token by id, which
wakes a hibernated or archived sandbox like the sandbox's token. A buffered exec for clients that cannot hold
an upgraded connection — plain JSON in, plain JSON out, so it multiplexes over
HTTP/2 through a TLS proxy:

```json
{"argv": ["node", "-v"], "cwd": "/work", "env": {"CI": "1"}, "timeout_seconds": 60}
```

→ `200 {"exit_code": 0, "stdout": "v22.23.2\n", "stderr": ""}`. The command
runs to completion (no stdin, no streaming, no detach — use the relay for
those); `timeout_seconds` 0 means no limit beyond the request itself. When
the node gives up on a started command (timeout, client gone, output cap,
buffer budget) it sends the child a `kill` through silkd, bounded to 5 s, before
answering. Output comes back as JSON strings: bytes that are not valid
UTF-8 are replaced with U+FFFD, so binary output belongs on the relay. 400
empty `argv`, negative `timeout_seconds`, an unknown field, or a silkd
`bad_request` — a command that is missing or not executable and a `cwd` that
does not exist are the caller's, not 502s; 401 missing bearer token; 404 unknown sandbox or wrong token;
413 when stdout+stderr exceed 8 MiB; 503 with `Retry-After: 1` when the
node's buffered-exec memory is exhausted (64 MiB of output across every
buffered exec in flight; the command is killed); 502 guest unreachable or any other
silkd error; 409 a failed sandbox; 500 a failed restore; the
[guest errors](#guest-errors); 504 when the command outlives `timeout_seconds`. A client that hangs up first gets no reply and the command is killed. A hibernated sandbox wakes transparently like on the relay.

## GET /v1/sandboxes/{id}/owner

Auth: the sandbox's own token, or the root or operator token by id. Answers `{"owner_addr": "host:port"}` when
this node owns the sandbox, 404 otherwise; a tenant token is a wrong sandbox token here. With
[`archive_failover`](cluster.md#archived-claims-across-node-loss), a node that
does not hold the sandbox adopts it, when the token's SHA-256 matches the row's or the caller is root or the operator, and its row names an owner whose node
lease expired, or whose owner released it in an
[evacuation](#post-v1drain) or a [wake it had no room for](cluster.md#waking-an-archived-claim-on-a-full-node), and answers 200; with `?room=1`, which the owner of such a wake sends, it adopts only while it could admit the wake itself; 503 with `Retry-After` when the meta store is
down. A row whose owner's lease is live is never adopted unless the owner released it: 404. A draining or fenced
node, or one that lifted its fence less than 8 seconds ago, does not adopt: 404; a cordoned one adopts. A node whose
lease is overdue answers 404 for the replicated claims it holds, so the
`Lookup` scatter moves on. With `client_advertise` configured,
`owner_addr` is that node's full HTTP(S) origin instead; the same contract applies
to claim and fork responses. Used by the SDK's `Lookup` scatter. A claim the root or the operator
adopted keeps the row's token hash in place of its token: its own token still
authorizes it on this node and any later adopter, an empty token never does,
and the first call with its token remembers it (persisted at the next claims
commit), after which root's or the operator's `GET /v1/sandboxes/{id}` returns
it. The hash is never served.

## GET /v1/info

Auth: root or operator (tenant tokens get 403). Node pools, promoted templates, claim
count, the node's own address, and mesh peers:

```json
{"pools": [{"key": {"template": "base:24.04", "net": "none", "size": "small"},
            "warm": 4, "refilling": 0, "target": 4, "golden": true}],
 "templates": [{"key": {"template": "myproj:v1", "net": "none", "size": "small"},
                "content_digest": "sha256:…", "tenant": "acme",
                "created_at": "2026-07-06T00:00:00Z", "cpu_count": 1,
                "mem_total_bytes": 536870912, "labels": {"v1": "sha256:…"}}],
 "claimed": 2,
 "running": 1,
 "max_running": 12,
 "hibernated": 1,
 "archived": 0,
 "failed": 0,
 "at_capacity": true,
 "at_capacity_reason": "exchange full",
 "advertise_addr": "10.0.0.5:7777",
 "peers": ["10.0.0.6:7777"]}
```

`advertise_addr` is the address the node hands clients: its `client_advertise`
when set, else its `advertise_addr`. It is the same value a claim reports as
`owner_addr`, and it is omitted when the node advertises no host.

`hibernated` counts claims whose VM is currently hibernated, `archived` those
checkpointed to the store with the local VM dropped (see
[archive tiers](deploy.md#configuration)), `failed` those whose VMM is down
and was not restarted (omitted at zero); all are included in `claimed`.
`running` counts claims that hold a VM, with a wake in flight counted, and
`max_running` is the [cap](deploy.md#configuration) on it (omitted when off).
A node drained via [`POST /v1/drain`](#post-v1drain) additionally reports
`"draining": true`, a node cordoned via [`POST /v1/cordon`](#post-v1cordon)
`"cordoned": true`, and during an evacuation `evacuating` (claims left to
move, omitted at zero). When refill is parked because the node cannot start
another VM, `"at_capacity": true` distinguishes a full node from one still
filling its warm target, and `at_capacity_reason` carries the engine's capacity
reason, or `disk below reserve` or `memory above limit`, which stay stable for callers that tell them apart. Both capacity fields are omitted while refill is not capacity-blocked. Each VM holds a console pty, so a node stops at `kernel.pty.max` (4096 by default) with `no space left on device`; raise it past the VM count you plan to run.

`golden` reports whether the pool's snapshot exists (refill can clone);
`warm` at `target` with `golden: true` means warm claims are served in
sub-millisecond time.

`templates` lists the [promoted templates](#post-v1sandboxesidpromote) this
node holds: each one's full key, the `content_digest` its promote returned,
`created_at` of its last promote, the `cpu_count` and `mem_total_bytes` of its
size tier, its [`labels`](#put-v1templateslabelstemplatenetsize) when any are
set, and `tenant` when a tenant promoted it (omitted for root). A key
a configured pool now owns is left out, since claims of it clone the pool's
golden. On a shared checkpoint store a node lists what it promoted plus what
was in the store when it started, and a template another node deletes later
stays listed, and may redirect a claim to a 404, until this node restarts.

## GET /v1/peers

Auth: node API token (root **or** tenant) or the operator token. Answers `{"peers": [addr, …]}`
— the cluster's other node addresses, for the SDK's redirect follow and
`Lookup` scatter. Cluster topology, not operator state, so a tenant token
may read it (unlike `GET /v1/info`).

## GET /healthz

Unauthenticated liveness probe; answers `ok`.

## Operational notes

- The HTTP server must keep `ReadTimeout`/`WriteTimeout` at zero if you
  front it with your own server: cold claims legitimately block for the cold
  probe window and relays stream indefinitely. sandboxd itself uses
  `ReadHeaderTimeout` for slowloris protection, and closes a keep-alive
  connection that sends no request for 15 minutes, longer than the backend
  idle timeout of common load balancers (600 s on Google Cloud), so a balancer
  never reuses a connection sandboxd just closed.
- Shutdown force-closes in-flight relays and leaves VMs running for the next
  reconcile.
