cocoon sandbox

Clusters

A cluster is a set of sandboxd nodes joined through a hashicorp/memberlist SWIM mesh. Gossip carries only placement hints — per-pool warm counts, promoted-template hashes, available volume names, each node’s data-plane address, and a digest of the cluster-invariant config. Per-sandbox state never leaves its owning node, so a stale view costs at most one extra redirect, never correctness. A single node with no seeds is a valid mesh of one.

Joining

Give every node a mesh block:

"mesh": {
  "node_id": "node-b",
  "bind": "10.0.0.6:7946",
  "join": ["10.0.0.5:7946"],
  "cluster_key": "MDEyMzQ1Njc4OWFiY2RlZg=="
}

Two constraints, fine for a homogeneous cluster: all nodes share the same api_token and tenant set (the SDK replays whichever token authorized a call across a redirect; a peer missing that tenant answers 401), and only egress-capable nodes redirect egress claims.

How placement works

A claim always enters at whatever node the client dialed:

  1. Warm hit — the node has a warm sandbox for the pool key: ownership transfers in sub-millisecond time.
  2. Warm miss with peers — the node answers with a MOVED-style redirect: up to two peer addresses that gossip says hold warm sandboxes, chosen power-of-two-choices to avoid herding. The SDK retries there with no_redirect set, so the target warms-or-provisions locally and a stale view can never ping-pong. The same redirect fires when the node lacks a golden for the key but gossip names a template owner, and when the node sits at max_claims while a peer reports warm capacity.
  3. No candidates — the node provisions locally: a golden clone (tens of ms) or a cold boot for an unpooled key.

The data plane is never proxied between nodes: the claim response carries owner_addr and all sandbox traffic dials the owner directly. Behind a TLS edge, set each node’s client_advertise to its own public HTTPS origin. Owner, redirect, and peer responses use these origins; gossip, peer probes, and preview forwarding retain the internal advertise_addr. See TLS termination for the edge setup.

Node death is honest: a dead node’s sandboxes die with it (memory state is node-local by design). SWIM detects the death and peers stop redirecting to it.

Volumes and placement

A volume name has one fleet-wide meaning and access list, while catalog membership is node-local and deliberately excluded from the cluster config digest. Nodes gossip only their currently available catalog names: host paths and access lists never leave the node. After config load the set appears on the next gossip tick; later image distribution or removal is detected the same way. The node epoch bumps only when one of the gossiped sets — warm counts, template hashes, volume names — actually changes.

A writable name (writable: true) still needs its catalog entry — name, access list, and the writable flag — declared identically on every node meant to serve it, the same rule the read-only case already needs: gossip carries only currently-available names, never ACL or writable metadata, so a tenant claim landing on a node with no local entry for that name is a hard error (unknown/forbidden), not a gossip redirect. Root tokens carry no ACL to enforce and are exempt — they can still redirect off gossip alone. What a writable entry adds is a constraint on the backing image file, not the entry: exactly one of those nodes should actually have the file present at the configured path (the operator contract in deploy); the others log a missing-path warning and never advertise the name, so an ordinary claim still funnels to that one holder. Two nodes both holding the writable file is the operator error the fleet cannot detect — two nodes both declaring the entry is normal and expected.

A volume claim may consume an ordinary warm VM because attach happens after the pop and before finalization. Warm candidates retain their normal ranking, but a candidate must advertise every requested volume. If the entry node cannot serve all requested names, it redirects once to a peer advertising their intersection. A promoted-template claim uses the intersection of template owners and volume owners first. If that advertised intersection is empty, a volume holder gets one chance to prove it can resolve the template from a shared store. The target retries with no_redirect plus the carried require_promoted intent and validates both resources before provisioning, so a node-local template still fails without a second hop even while template gossip is one tick stale.

One key should be pooled or promoted fleet-wide, not both — the same class of operator error as two nodes holding one writable image. promote refuses a key the answering node pools, but nothing coordinates that across the mesh, so a peer can promote what another node pools and the two resolve to different content. Each node then answers with its own: a node holding a pool golden serves it to every claim for that key and ignores peer template advertisements, so volume and non-volume claims there never disagree. A claim that sets require_promoted, volume or plain, is refused with 404 instead of being handed the pool golden or a cold boot, which lets the SDK walk on to a node that has the template.

GET /v1/volumes and the SDK discovery calls return the gossiped union filtered through the answering node’s fleet-uniform access lists. nodes counts members advertising each name, while available and size_bytes describe only the answering node’s image, and writable is the entry’s catalog configuration, uniform fleet-wide. No node address or dataset-to-host mapping is returned; claim placement resolves the holder.

Querying members

GET /v1/info (root api_token) reports this node’s pools plus the peer data-plane addresses it currently sees:

curl -s -H "Authorization: Bearer $TOKEN" http://node-a:7777/v1/info | jq .

{
  "pools": [
    {"key": {"template": "base:24.04", "net": "none", "size": "small"},
     "warm": 4, "refilling": 0, "target": 4, "golden": true}
  ],
  "claimed": 2,
  "hibernated": 0,
  "archived": 0,
  "peers": ["10.0.0.6:7777"]
}

peers, served by GET /v1/peers (root or tenant token), is what the SDK scatters across in Lookup. An empty/absent peers means a mesh of one.

Relocating a handle

If a client kept a sandbox’s id and token but lost the owner address (process restart, handle passed between services), Client.Lookup asks the entry node, then queries every peer concurrently (GET /v1/sandboxes/{id}/owner — the token is both authorization and ownership proof) and returns a handle bound to whichever node confirms ownership first.

// Persist sb.ID and sb.Token() before the process exits; dial any node to
// rebind — Lookup finds the owner wherever the sandbox actually lives.
client, _ := sandbox.Connect("10.0.0.6:7777", sandbox.WithAPIToken(token))
sb, err := client.Lookup(ctx, savedID, savedToken)

Templates on a cluster

A promoted template (see the SDK guide) lives in its node’s checkpoint store. On the default local-disk backend that means only on its owner node, and the mesh gossips each node’s template set alongside its warm counts; on a shared store (a FUSE-mounted checkpoint_dir or the s3 backend) every node resolves every template directly, so the gossip routing simply never fires. An export carries only the sandbox’s writable state — the base image blobs resolve from the claiming node’s local store, pulled by digest when missing (vm clone --pull). Name-based calls route cluster-wide:

Gossip is eventually consistent: a template promoted a moment ago may be invisible to name-based calls for about a gossip tick (the claim fails cold — retry), and one deleted a moment ago may still redirect to a 404. Correctness is never violated; only the name-based convenience lags. The handle Sandbox.Promote returns is still owner-bound. template.Delete and a template.New without volumes dial the owner directly; a volume claim may use one placement redirect to a node that can resolve both resources.

Checkpoints on a cluster

A checkpoint lives on whichever node captured it (the default checkpoint_dir) unless the store is shared (a FUSE mount or the s3 backend), in which case every node resolves every checkpoint directly and nothing below applies.

On the default per-node store, a branch claim (Checkpoint.New / POST /v1/checkpoints/{id}/claim) at a node that does not hold the record runs a tier order:

  1. Local claim — the fast path when the claiming node already holds it.
  2. Probe + redirect — a live HEAD fan-out (HMAC-signed when the mesh carries a cluster_key) to every mesh peer in parallel, redirecting to the first owners that answer — the same claim-redirect contract a warm miss already uses. The answer is capped at 3 addresses: a hint, not an exhaustive list of every owner. The probe and the follow-up claim are not atomic: a peer can answer the probe, then lose the record — a delete’s broadcast lands, or its own TTL sweep runs (below) — before the retry reaches it, so a redirect can go stale between the two calls.
  3. Heal — when checkpoint_peer_heal is on and neither of the above answered, the node pulls the record from a probed peer, validates its shape and id before trusting it, and publishes it locally so later branches are local — paid once per node. The pull is bounded to one overall time budget across however many peers it tries (healBudget in store/peer), and a node accepts only so many heals at once (maxConcurrentHeals in pool) — past that cap it answers 503 rather than queuing indefinitely.

Delete is eventual, not a fleet-wide revocation

Checkpoint.Delete (DELETE /v1/checkpoints/{id}) removes the local record, then best-effort broadcasts the delete to every peer this node currently sees, so a replica a heal pulled earlier is cleaned up too, not just the original. A peer that is offline or partitioned during that broadcast keeps its copy until the checkpoint TTL ages it out. A healed replica carries the source checkpoint’s original CreatedAt, so it becomes eligible for expiry at the same instant everywhere; each node then removes it on its own hourly sweep, which is independently phased and retries on failure. The window in which a deleted checkpoint stays branchable by an id-holder is therefore normally checkpoint_ttl_hours plus the wait for the next hourly sweep; a sweep that fails retries on a later one, so persistent sweep failure extends retention until one succeeds — the TTL is the eligibility point, not a hard ceiling. This is why heal requires a nonzero, fleet-matching TTL (see cluster-invariant config). With TTL disabled it could never close at all, so that combination is rejected at config load.

State ownership

Each kind of state has exactly one source of truth, and a restart rebuilds fully from it:

state source of truth survives restart
operator config (volumes, secrets, egress policies, bridges/networks, mesh, preview_secret, egress_ca) config.json (human/deploy-tool owned) re-read at boot; its reloadable part on a per-node reload
API-applied pool targets (PUT /v1/pools) <data_dir>/pools.json (machine owned), or the cell’s row in the shared meta_store when the node sets meta_store.cell yes
API-applied tenants (/v1/tenants) <data_dir>/tenants.json (machine owned), or the shared meta_store table yes
claims the claims journal + Reconcile yes
placement hints (warm counts, template and volume sets) gossip rebuilt
checkpoint ownership a live per-request probe (no gossip); a healed replica is this node’s own persisted copy, aged out by checkpoint_ttl_hours yes

Pools are managed API-first. The first time a node takes PUT /v1/pools, it writes the applied set to <data_dir>/pools.json and from then on that file seeds the pools at boot, overriding the pools section of config.json (a loud log line notes this; if the two disagree because someone edited config.json afterward, a second warning names it). The file’s presence is the ownership marker — delete pools.json to return the node to config-owned pools. Egress stays config-owned regardless: the API rejects egress specs, so pools.json never carries them, and egress policies re-merge from config.json by pool key at boot.

With meta_store.cell, the set belongs to the node’s cell: one PUT /v1/pools on any node of the cell reaches every node in it, and no fan-out is needed; every node of the cell runs the same targets. This pool cell is only a name for nodes that share one pool set in the database. It is not a mesh cell (see scale): a pool cell may cover part of a mesh or nodes of several meshes, and gossip, redirects and heals ignore it.

Without it, cluster-wide pool changes are a client-side fan-out, not a gossiped desired state (pools are legitimately heterogeneous per node): Client.SetPoolsCluster PUTs the set to the entry node and every peer and returns a per-node result; retrying the failed nodes is the whole protocol, because the apply is an idempotent declarative replace. A non-nil error means peer discovery itself failed and the fan-out reached only the entry node — an incomplete apply to retry, kept distinct from a genuine single-node cluster (nil error). It fits a homogeneous cluster — a spec set that names an egress-lane pool is refused on a node without an attachment, so nodes that differ in capacity or attachment take a per-node Client.SetPools.

Tenants

Tenants are API-only: config.json has no tenants field, a node starts with an empty set, and each tenant API change rewrites <data_dir>/tenants.json, which the node loads at boot. Unlike pools, the set must be equal on every node, because the SDK replays the authorizing token across a redirect. The change is still a client-side fan-out: Client.PutTenantCluster, Client.DeleteTenantCluster and Client.SetTenantsCluster apply it to the entry node and every peer and return a per-node result, and retrying the failed nodes converges because each apply is idempotent (a cluster delete counts a node that no longer holds the tenant as done). Until every node has it, a redirect to a node that lacks the tenant answers 401, which the SDK treats as transient and resolves at the origin. GET /v1/tenants reports each node’s set digest, and the gossiped config digest below covers the live set, so a node that missed a change shows in sandboxd_config_digest_mismatch. A node that is down during a fan-out has left the peer list, so it is absent from the results rather than failed: when it rejoins with its old tenants.json, that mismatch names the divergence, and one apply to that node closes it.

Without a shared store, every node holds the whole tenant set in memory and rewrites tenants.json on each change, so the set suits thousands to tens of thousands of tenants: control planes, organisations, services. Map end users onto them in the control plane (claim_ref, metadata) rather than giving each user a tenant.

With meta_store, the tenant set lives in one PostgreSQL table that every node reads, so one tenant per end user works:

Cluster-invariant config

Some config must match on every node or the cluster fails in confusing ways:

config what breaks on mismatch
api_token, the tenant set the SDK replays the authorizing token across a redirect; a mid-rotation peer missing that tenant/token answers 401, which the SDK treats as transient — the claim falls back to the origin node (which already authorized it) and resolves there
preview_secret a preview URL signed on one node fails verification on another
mesh.cluster_key nodes cannot join / decrypt gossip at all
egress_ca cluster root a guest checkpointed/redirected across nodes trusts the root; a divergent root fails interception
egress: egress_classes, each pool’s egress and egress_upstream_env, secrets names, headers and value_env, egress_internal_allow, egress_upstream a claim or tenant redirected to a divergent node reaches different hosts, and its claim credentials are refused or not injected there
the engine root path (cocoon’s root_dir) a checkpoint or template export pins the base image blobs’ absolute path under the root that captured it; a node whose root sits at another path refuses to clone a record it healed or pulled (untrusted storage path in snapshot metadata) — the heal succeeds, the branch fails. Not part of the gossiped digest below

Each node gossips a digest of these over its live tenant set and quotas (HMAC-keyed by cluster_key when set, covering each tenant token’s SHA-256; otherwise a token-free digest of tenant names and quotas + the CA root + checkpoint_ttl_hours, so nothing brute-forceable rides cleartext gossip) plus the egress config above in an order-free form, never a secret value, and republishes it after every tenant change and every config reload. A mismatch logs a warning at the moment the divergent node appears — not at the first unlucky redirect — and raises the sandboxd_config_digest_mismatch gauge. It is warn-only: a rolling credential rotation is a legitimate transient mismatch, so a divergence never partitions the mesh.

Scale

Every memberlist push/pull carries the whole cluster’s placement state: each node’s warm counts, template set and volume set, as one JSON document. Its size grows with nodes × templates per node:

nodes templates per node state per push/pull
1,000 5 0.6 MB
10,000 5 6.3 MB
10,000 500 353 MB

memberlist refuses a push/pull state over 20 MiB, and above 32 nodes it stretches the push/pull interval with log2 of the cluster size (10 s at 10,000 nodes), so a change takes minutes to reach every node while warm counts move with every claim.

Cluster checklist

This site is open source. Improve this page.