cocoon sandbox

Clusters

A cluster is a set of sandboxd nodes joined through a hashicorp/memberlist SWIM mesh. Gossip carries only placement hints — per-pool warm counts, promoted-template hashes, available volume names, each node’s data-plane and client addresses, and a digest of the cluster-invariant config. Gossip carries no per-sandbox state, so a stale view costs at most one extra redirect, never correctness. A single node with no seeds is a valid mesh of one.

Cells

A cell is one mesh and the backends its nodes share: one PostgreSQL database for meta_store, one checkpoint store location, and one set of secrets (api_token, mesh.cluster_key, preview_secret, the egress root). Every node of a cell holds the cell’s credentials, so a host compromise on one node reaches its whole cell. A cell that shares nothing with another bounds that compromise; security has the contract.

Joining

Give every node a mesh block:

"mesh": {
  "node_id": "node-b",
  "bind": "10.0.0.6:7946",
  "join": ["10.0.0.5:7946", "10.0.0.6:7946"],
  "cluster_key": "MDEyMzQ1Njc4OWFiY2RlZg=="
}

Two constraints, fine for a homogeneous cluster: all nodes of a cell share the same api_token and tenant set (the SDK replays whichever token authorized a call across a redirect; a peer missing that tenant answers 401), and only egress-capable nodes redirect egress claims.

How placement works

A claim always enters at whatever node the client dialed:

  1. Warm hit — the node has a warm sandbox for the pool key: ownership transfers in sub-millisecond time.
  2. Warm miss with peers — the node answers with a MOVED-style redirect: up to two peer addresses that gossip says hold warm sandboxes, chosen power-of-two-choices to avoid herding. The SDK retries there with no_redirect set, so the target warms-or-provisions locally and a stale view can never ping-pong. The same redirect fires when the node lacks a golden for the key but gossip names a template owner, and when the node refuses admission (any admission refusal): a warm peer first, else a peer that serves the pool. A node that would refuse a claim stops gossiping its pools within a tick. A stale view can still name one; it answers 429, and the SDK then tries the next one and the origin.
  3. No candidates — the node provisions locally: a golden clone (tens of ms) or a cold boot for an unpooled key.

The data plane is never proxied between nodes: the claim response carries owner_addr and all sandbox traffic dials the owner directly. Behind a TLS edge, set each node’s client_advertise to its own public HTTPS origin. Owner, redirect, and peer responses use these origins; gossip, peer probes, and preview forwarding retain the internal advertise_addr. See TLS termination for the edge setup.

Node death is honest: a dead node’s sandboxes die with it (memory state is node-local by design), except archived and replicated claims. SWIM detects the death and peers stop redirecting to it.

Volumes and placement

A volume name has one fleet-wide meaning and access list, while catalog membership is node-local and deliberately excluded from the cluster config digest. Nodes gossip only their currently available catalog names: host paths and access lists never leave the node. After config load the set appears on the next gossip tick; later image distribution or removal is detected the same way. The node epoch bumps only when one of the gossiped sets — warm counts, template hashes, volume names — or the config digest actually changes.

A writable name (writable: true) still needs its catalog entry — name, access list, and the writable flag — declared identically on every node meant to serve it, the same rule the read-only case already needs: gossip carries only currently-available names, never ACL or writable metadata, so a tenant claim landing on a node with no local entry for that name is a hard error (unknown/forbidden), not a gossip redirect. Root tokens carry no ACL to enforce and are exempt — they can still redirect off gossip alone. What a writable entry adds is a constraint on the backing image file, not the entry: exactly one of those nodes should actually have the file present at the configured path (the operator contract in deploy); the others log a missing-path warning and never advertise the name, so an ordinary claim still funnels to that one holder. Two nodes both holding the writable file is the operator error the fleet cannot detect — two nodes both declaring the entry is normal and expected.

A volume claim may consume an ordinary warm VM because attach happens after the pop and before finalization. Warm candidates retain their normal ranking, but a candidate must advertise every requested volume. If the entry node cannot serve all requested names, it redirects once to a peer advertising their intersection. A promoted-template claim uses the intersection of template owners and volume owners first. If that advertised intersection is empty, a volume holder gets one chance to prove it can resolve the template from a shared store. The target retries with no_redirect plus the carried require_promoted intent and validates both resources before provisioning, so a node-local template still fails without a second hop even while template gossip is one tick stale.

One key should be pooled or promoted fleet-wide, not both — the same class of operator error as two nodes holding one writable image. promote refuses a key the answering node pools, but nothing coordinates that across the mesh, so a peer can promote what another node pools and the two resolve to different content. Each node then answers with its own: a node holding a pool golden serves it to every claim for that key and ignores peer template advertisements, so volume and non-volume claims there never disagree. A claim that sets require_promoted, volume or plain, is refused with 404 instead of being handed the pool golden or a cold boot, which lets the SDK walk on to a node that has the template.

GET /v1/volumes and the SDK discovery calls return the gossiped union filtered through the answering node’s fleet-uniform access lists. nodes counts members advertising each name, while available and size_bytes describe only the answering node’s image, and writable is the entry’s catalog configuration, uniform fleet-wide. No node address or dataset-to-host mapping is returned; claim placement resolves the holder.

Querying members

GET /v1/info (root or operator token) reports this node’s pools plus the peer data-plane addresses it currently sees:

curl -s -H "Authorization: Bearer $TOKEN" http://node-a:7777/v1/info | jq .

The API page describes the payload.

peers, served by GET /v1/peers (root, operator or tenant token), is what the SDK scatters across in Lookup; a node with client_advertise set answers the client addresses. An empty/absent peers means a mesh of one.

Relocating a handle

If a client kept a sandbox’s id and token but lost the owner address (process restart, handle passed between services), Client.Lookup asks the entry node, then queries every peer concurrently (GET /v1/sandboxes/{id}/owner — the token is both authorization and ownership proof) and returns a handle bound to whichever node confirms ownership first. With archive_failover, a node may adopt a claim whose owner’s node lease expired and answer for it (see archived claims across node loss).

// Persist sb.ID and sb.Token() before the process exits; dial any node to
// rebind — Lookup finds the owner wherever the sandbox actually lives.
client, _ := sandbox.Connect("10.0.0.6:7777", sandbox.WithAPIToken(token))
sb, err := client.Lookup(ctx, savedID, savedToken)

Node admission

A node is in one of these states. POST /v1/cordon and POST /v1/drain set them, and each DELETE lifts only its own; both are root-only (see the API). A cap of 0 or an omitted cap means no cap; to take no new claims, cordon the node.

State New claim, fork, branch Existing claims Root live move in Adoption of archived claims Refill and pool gossip Survives restart and reboot
serving yes, unless an admission refusal applies; a 429 for a claim without volumes redirects to a peer yes; a wake is bound by max_running and disk_reserve; an archived claim’s refused wake, or one over memory_limit, first tries a peer with room yes, under the same caps and move_concurrency yes; no claim cap binds, max_running binds at the wake yes —
cordoned no: 429 node cordoned; peers stop redirecting here yes yes, under the same caps and move_concurrency yes, as serving: an adopted claim is no new claim no; warm VMs are destroyed yes, as <data_dir>/cordoned
draining no: 429 node draining; peers stop redirecting here yes no: admission and commit refuse no no; warm VMs are destroyed no
draining with evacuate=1 no, as draining yes until the claim’s turn, then 503 until a peer holds it and 404 after; a claim that cannot move stays and serves no no no no: a restart stops the evacuation
cordoned and draining no: 429 node cordoned yes no no no the cordon only

Archived claims across node loss

A claim lives on one node, so a lost node takes its running and hibernated claims with it, unless the claim is archived or replicated. An archived claim keeps only a pointer on the node; with archive_failover on every node (it needs meta_store with a cell, an s3 checkpoint_store and a mesh), the archive also writes one row to the shared PostgreSQL, scoped by the cell: id, tenant, token SHA-256, owner node, epoch, a consumed flag, checkpoint id, retention deadline and the claim record without its token. Archives, replication rounds, and env writes and renews of a claim whose row must follow it while it sleeps write it, so the claim path is unchanged.

The node lease decides adoption, so a partitioned owner can lose an archived claim it still lists once its lease runs out; the epoch fences it out. A claim that has entered its wake counts as running. Running and hibernated claims of a pool without replicate_seconds are lost with their node, and an owner that never returns leaves behind only the tombstones and checkpoints of the claims it was waking or releasing when it died, plus what a replicated claim leaves (see running claims).

Waking an archived claim on a full node

A wake that max_running or disk_reserve refuses answers 429 and is counted in sandboxd_wakes_refused_total{reason}. memory_limit sends only an archived claim’s wake to a hand-off, since only that one can move, and never answers 429 for it. For an archived claim with a row, the owner first tries to hand the claim to a peer that has room, with the release and adoption an evacuation uses:

The hand-off costs nothing on the claim and wake paths. A refused archived wake costs one registry release, one call per peer until one adopts, and, with no adopter, one take-back and one claims-journal write.

Evacuating a node

POST /v1/drain?evacuate=1 empties a node for maintenance or retirement without losing its claims. The node stops taking claims as a drain does, then moves each claim with bounded concurrency, on the maintenance budget, archived claims first and running ones last: it hibernates a running claim, archives a hibernated one through the archive path above, releases the row, and asks a peer to adopt it. A release is one compare-and-set: the row this node owns at the epoch it holds moves to no owner at a new epoch. A row with no owner has no lease to wait out, so a peer takes it with the same epoch compare-and-set as any adoption, and two peers adopting at once still leave one owner. This node keeps the claim listed until a peer answers GET /v1/sandboxes/{id}/owner, so a root listing finds the claim on at least one node at every moment, then forgets it under the claim’s transition lock and answers 404 for it; the checkpoint stays, since it belongs to the adopter. With no adopter (no peer, every peer draining, fenced or unreachable) the node takes the row back and the claim stays here. Until the hand-off, a wake of a claim the evacuation still has to move answers 503 with Retry-After, so a client call cannot wake it back onto the node. A running claim keeps serving until its turn. The run takes the budget one batch at a time, so crash recovery and reaps of the claims that stay keep their turn while it runs.

The release keeps the fencing intact. Every later write of the old owner is refused: a put is owner-guarded, and a replace, consume or second release needs the epoch the release moved on. A release that finds its row moved reads it: a row with no owner is one an outage cut short or a later owner released, so the node asks a peer to adopt it and never takes it back; a row another node took means the claim moved already, and the node drops it; a row this node still owns at another epoch keeps the claim, which is skipped. A crash between the release and a peer’s answer leaves a row no node lists: a restart drops the claim, and the next Lookup with its token adopts it, so a holder that reads the token back with root’s GET /v1/sandboxes/{id} loses that claim. A released row past its retention deadline is purged by any node’s hourly sweep, like the row of a departed owner.

An archive the sweeps already run for a claim is waited out and the claim retried every 5 seconds. Claims that cannot move are skipped and keep serving here; once evacuating is gone claimed counts them, and the log names each one with its reason. A client that holds a handle gets 404 from this node after the move, the same answer as for a released claim, or a connection error once the node is retired, and only a Lookup tells these apart: the SDKs run it on either answer and send the call again to the new owner when nothing of it reached the old one. A reader-served connection gets the retry as an error frame, not a 503, and a preview URL minted before the move stops working until it is minted again, except behind a shared preview_advertise, where it answers when it reaches the new owner. DELETE /v1/drain stops the evacuation: claims not yet started stay as they are, and one mid-archive finishes its archive but keeps its row.

Moving a running claim

POST /v1/sandboxes/{id}/move (root only) hands one running claim to a named peer with cocoon’s live migration, so its guest keeps its processes and memory. Evacuation stays the cold path:

  Cold: POST /v1/drain?evacuate=1 Hot: POST /v1/sandboxes/{id}/move
What moves every claim of a draining node one running claim
How hibernate, archive to the checkpoint store, release the row, a peer adopts memory and disk rounds straight to the target, which restores and starts the guest
Guest after the move stopped until the next call wakes it running, with its processes and memory
Target any peer that answers the one the caller names
When it cannot move the claim is skipped and keeps serving the call fails and the claim keeps running here

A move needs meta_store with a cell, which holds its record, and mesh.cluster_key. It does not need archive_failover or an S3 checkpoint_store, since its rounds go straight to the peer.

Only a none-lane claim without volumes moves; a hibernated, archived, failed or waking claim, an egress-lane claim and a claim with volumes answer 409. Open exec and pty streams, port forwards and egress connections end; the next call meets a 404 on the source, and the SDKs look the claim up and reach the target.

The record. Each move has a record of its own in the meta store (table sandboxd_moves), separate from the archive rows: id, from_node, to_node, state and epoch, each change a compare-and-set at a new epoch.

From To Who When
(none) pending source after the target admitted the move, before the baseline round
pending final source the final round paused the guest
final committed target after the receive; the point of no return
committed started target, or a later source that holds the claim after it journaled that it runs the claim
pending or final aborted source, or a target that holds a received claim or finds the record stale before the guest resumes

A new move of the claim overwrites only a started or aborted record; a committed one refuses it with 409 unless the node making the new move holds the claim, which proves the target started it, so that node marks it started first. Until a received claim starts, its target gates it like a claim in a move: sweeps skip it, calls that reach the guest or change the claim answer 503, and operator calls answer 409.

The target commits only from final, so it never starts a guest restored from a round the source kept running after. The source resumes its guest only on proof the move stayed here: a record it wrote, at or past the epoch in its journal, in pending or final (it writes aborted first) or aborted. A committed or started record, a missing one, or another node’s means the claim moved on, and the source drops it and its VM. While the meta store is down, a guest paused in the final round stays paused and the call answers 503; the abort is retried every 2 seconds.

The record outlives each side’s need for it. A target with a received claim starts it on committed or started and drops it on aborted or a missing record, which is safe because no record is deleted while the target still journals that claim: it clears its journal first and only then writes started. An abort runs in the order aborted, resume, target drop, journal clear, record delete, so a source holding a journal never finds its own abort missing. The source deletes a record only in started or aborted, never committed. Each node’s hourly sweep handles its records older than an hour that no move in flight holds: a target writes started over a committed record, either side writes aborted over a pending or final one, and the source deletes the ended ones.

Rounds. The source sends a baseline round, then up to 4 deltas, each over the control port with the root token. Once the last delta’s bytes on the wire at the rate the rounds reached, plus 400 ms for the receive and the start, fits max_downtime_ms (default 2000, and above 400 since those 400 ms alone would never fit), the source pauses the guest for the final round and the target receives the series and starts the guest. A move whose deltas never fit answers 503 and the guest runs on; from the second delta on, one that shrinks by under 10% and is still too large answers at once. The estimate cannot see the final capture, which runs with the guest paused, or the pages dirtied since the last delta, so once the guest is paused the source checks again: the final capture’s time plus the final round’s bytes on the wire (the last delta’s, plus the growth of the final round) at the rounds’ rate plus the 400 ms. Past max_downtime_ms it aborts the move before sending the final round: cocoon vm migrate abort resumes the guest on the source, the target drops what it received, the record ends aborted, and the move answers 503. The check cannot give back a pause already spent, so a final capture that alone overruns the budget still pauses the guest for the capture plus the abort and its record write; with the meta store up, that, not max_downtime_ms, is the worst case. Past the check, a slow transfer or commit still lengthens the pause, and downtime_ms reports it. downtime_ms in the answer is what the source measured from the final capture’s start, including its wait for a free dump slot, until the target answers the commit, after it started the guest; of which capture_ms, send_ms and commit_ms are the final capture, the final round’s transfer and the commit. Each round has one minute plus its size at 10 MB/s to arrive, so a target that stops reading fails the move. A replicated claim’s registry unrow, the final capture, its transfer and the target’s commit have 2 minutes together; past that the source resolves the move through the record, and the guest resumes on the source unless the target committed.

Counting. The target admits the move like a new claim: drain, disk_reserve, memory_limit, node max_claims and max_running, the tenant’s max_claims, and the node’s move_concurrency, and holds a reservation from admission to its commit. A cordon refuses only new claims, so a cordoned target admits the move. The reservation counts in every cap but is no claim: listings, reaps and gossip never see it. At the commit the reservation becomes the claim under one lock, so the target never counts it twice. The source counts the claim until it drops it after the commit, so the cluster counts a moving claim twice, which can make a node refuse earlier but never over-admit. memory_limit is checked again before the receive, and a target that started draining refuses the commit. A new admission from the source that holds the reservation replaces it and resets its staged files, so a move retried after a source crash does not wait out the idle drop; one from another source answers 409. A reservation with no round arriving and none finished or commit asked for 15 minutes, the longest a capture may take, is dropped with its staged files.

Same paths. Both nodes need the same path strings in cocoon’s root_dir and run_dir. The requirement is on the string, not the real directory: a symlink of one name can lead to different disks on each node, and sandboxd’s data_dir may differ. Set the string before the VM starts; a VM started under another string cannot move. A move between nodes with different strings fails at the receive, after the guest paused: cocoon vm migrate receive refuses the series, the call answers 502 and the guest resumes on the source.

Known limits.

Replicated claims. Before the final round the source consumes the claim’s registry row, since no node adopts a tombstone, then clears the claim’s replica fields and deletes the row, so no third node can adopt a stale replica while the target runs the claim. It keeps the replica until the move ends. The target’s first replica round puts a new row when the claim’s carried interval comes due, from its replica_at, at once when the last round is older than the interval, and an aborted move makes the claim due at the source’s next replica sweep, which puts it again. A source cut off from the registry changes nothing and its move fails: it keeps the row’s epoch, so the fence still finds a peer that adopted the claim during the outage, and the source drops it. A consume whose reply was lost leaves the source’s own tombstone, which the hourly sweep puts back on the replica. A claim that holds no row can still have a tombstone of the source, left by a wake whose checkpoint delete failed or by a replica retirement. The source deletes it before the final round too, since the target would read another node’s tombstone as an adoption and drop the guest; with archive_failover on, a registry that cannot answer that read fails the move.

Failures. Any failure before the commit aborts: the source resumes the guest, the target drops its reservation, files and any received VM, and the call answers the error. A target answer lost after its commit is resolved through the record, and the call answers 200. A restarted node resolves a move in its journal by the same rules, so a source back from an outage never resumes a claim that moved on, and a target that crashed after its commit starts the guest. A guest the target cannot start fails there through the failed path; it never goes back.

Node lease

With archive_failover every node renews one row of sandboxd_nodes (cell, node, lease_until) every 5 seconds, with a single upsert on the database clock that covers the whole node: lease_until = now() + 15 s. A renewal that fails once does not lose the lease or trip the fence, as long as the next one answers within 2 seconds of its start. The table is created at start when it is missing, and a node with no row counts as expired. Adoption and the departed-owner sweep consult the lease, never mesh membership, so a node the mesh lost but the database still hears keeps its claims, and a node the mesh still lists but the database no longer hears loses them. The cost is one statement per node every 5 seconds; no claim path or data plane reads it.

Running claims across node loss

A pool entry with replicate_seconds (a none-lane pool on a node with archive_failover) snapshots its running claims to the checkpoint store and keeps each claim’s registry row on the latest snapshot:

A replica costs, per claim per interval, one capture pause, an upload of the changed chunks plus a manifest and a meta, one registry read, one write and one delete of the previous replica; the node’s own cost is one scan of its claims every 5 seconds. The delete is soft: the chunks only the previous replica names stay in the store until the operator’s offline reclamation removes them. With replication this is the fastest-growing part of the store: roughly the per-round churn times the rounds since the last reclamation, per replicated claim, on top of the live replica. Later rounds dedup; the first after a wake or adoption does not, because the wake deletes the record it restored from. Resolving a claim for a stream of a pool that does not replicate reads one plain field and no atomic; a replicated claim adds one atomic load of the fence state and its stream one watcher goroutine. Adding a claim to a failover node gains one map lookup. Nothing runs per frame.

Templates on a cluster

A promoted template (see the SDK guide) lives in its node’s checkpoint store. On the default local-disk backend that means only on its owner node, and the mesh gossips each node’s template set alongside its warm counts; on a shared store (a FUSE-mounted checkpoint_dir or the s3 backend) every node resolves every template directly, so the gossip routing fires only for a template another node deleted after this one started, which stays listed until this node restarts. An export carries only the sandbox’s writable state — the base image blobs resolve from the claiming node’s local store, pulled by digest when missing (vm clone --pull). Name-based calls route cluster-wide:

Gossip is eventually consistent: a template promoted a moment ago may be invisible to name-based calls for about a gossip tick (the claim fails cold — retry), and one deleted a moment ago may still redirect to a 404. Correctness is never violated; only the name-based convenience lags. The handle Sandbox.Promote returns is still owner-bound. template.Delete and a template.New without volumes dial the owner directly; a volume claim may use one placement redirect to a node that can resolve both resources.

Checkpoints on a cluster

A checkpoint lives on whichever node captured it (the default checkpoint_dir) unless the store is shared (a FUSE mount or the s3 backend), in which case every node resolves every checkpoint directly and nothing below applies; on a FUSE mount, which the node cannot tell from a local directory, a miss still probes the peers, which find nothing.

On the default per-node store, a branch claim (Checkpoint.New / POST /v1/checkpoints/{id}/claim) at a node that does not hold the record runs a tier order:

  1. Local claim — the fast path when the claiming node already holds it.
  2. Probe + redirect — a live HEAD fan-out (HMAC-signed when the mesh carries a cluster_key) to every mesh peer in parallel, redirecting to the first owners that answer — the same claim-redirect contract a warm miss already uses. The answer is capped at 3 addresses: a hint, not an exhaustive list of every owner. The probe and the follow-up claim are not atomic: a peer can answer the probe, then lose the record — a delete’s broadcast lands, or its own TTL sweep runs (below) — before the retry reaches it, so a redirect can go stale between the two calls.
  3. Heal — when checkpoint_peer_heal is on and neither of the above answered, the node pulls the record from a probed peer, validates its shape and id before trusting it, and publishes it locally so later branches are local — paid once per node. The pull is bounded to one 30-minute budget across however many peers it tries, and a node accepts at most 4 heals at once — past that cap it answers 503 rather than queuing indefinitely.

Delete is eventual, not a fleet-wide revocation

Checkpoint.Delete (DELETE /v1/checkpoints/{id}) removes the local record, then best-effort broadcasts the delete to every peer this node currently sees, so a replica a heal pulled earlier is cleaned up too, not just the original. A mesh without a cluster_key sends no broadcast: the request carries the api_token, and heal, the only source of a replica, needs the key. A peer that is offline, partitioned or not yet joined during that broadcast keeps its copy until the checkpoint TTL ages it out. A healed replica carries the source checkpoint’s original CreatedAt, so it becomes eligible for expiry at the same instant everywhere; each node then removes it on its own hourly sweep, which is independently phased and retries on failure. The window in which a deleted checkpoint stays branchable by its tenant is therefore normally checkpoint_ttl_hours plus the wait for the next hourly sweep; a sweep that fails retries on a later one, so persistent sweep failure extends retention until one succeeds — the TTL is the eligibility point, not a hard ceiling. This is why heal requires a nonzero, fleet-matching TTL (see cluster-invariant config). With TTL disabled it could never close at all, so that combination is rejected at config load. A node later run with healing off and that TTL back at 0 keeps such a replica until an explicit delete.

State ownership

Each kind of state has exactly one source of truth, and a restart rebuilds fully from it:

state source of truth survives restart
operator config (volumes, secrets, egress policies, bridges/networks, mesh, preview_secret, egress_ca) config.json (human/deploy-tool owned) re-read at boot; its reloadable part on a per-node reload
API-applied pool targets (PUT /v1/pools) <data_dir>/pools.json (machine owned), or the cell’s row in the shared meta_store when the node sets meta_store.cell yes
API-applied tenants (/v1/tenants) <data_dir>/tenants.json (machine owned), or the shared meta_store table yes
API-registered volumes (PUT /v1/volumes/{name}) <data_dir>/runtime-volumes.json (machine owned, per node) yes
claims the claims journal + Reconcile yes
placement hints (warm counts, template and volume sets) gossip rebuilt
checkpoint ownership a live per-request probe (no gossip); a healed replica is this node’s own persisted copy, aged out by checkpoint_ttl_hours yes

Pools are managed API-first. The first time a node takes PUT /v1/pools, it writes the applied set to <data_dir>/pools.json and from then on that file seeds the pools at boot, overriding the pools section of config.json (a loud log line notes this; if the two disagree because someone edited config.json afterward, a second warning names it). The file’s presence is the ownership marker — delete pools.json to return the node to config-owned pools. Egress stays config-owned regardless: the API rejects egress specs, so pools.json never carries them, and egress policies re-merge from config.json by pool key at boot.

With meta_store.cell, the set belongs to the node’s cell: one PUT /v1/pools on any node of the cell reaches every node in it, and no fan-out is needed; every node of the cell runs the same targets. The meta_store.cell name partitions the database’s shared rows, not the cell: it may cover all or part of a cell’s nodes, and gossip, redirects and heals ignore it.

Without it, cluster-wide pool changes are a client-side fan-out, not a gossiped desired state (pools are legitimately heterogeneous per node): Client.SetPoolsCluster PUTs the set to the entry node and every peer and returns a per-node result; retrying the failed nodes is the whole protocol, because the apply is an idempotent declarative replace. A non-nil error means peer discovery itself failed and the fan-out reached only the entry node — an incomplete apply to retry, kept distinct from a genuine single-node cluster (nil error). It fits a homogeneous cluster — a spec set that names an egress-lane pool is refused on a node without an attachment, so nodes that differ in capacity or attachment take a per-node Client.SetPools.

Tenants

Tenants are API-only: config.json has no tenants field, a node starts with an empty set, and each tenant API change rewrites <data_dir>/tenants.json, which the node loads at boot. Unlike pools, the set must be equal on every node, because the SDK replays the authorizing token across a redirect. The change is still a client-side fan-out: Client.PutTenantCluster, Client.DeleteTenantCluster and Client.SetTenantsCluster apply it to the entry node and every peer and return a per-node result, and retrying the failed nodes converges because each apply is idempotent (a cluster delete counts a node that no longer holds the tenant as done). Until every node has it, a redirect to a node that lacks the tenant answers 401, which the SDK treats as transient and resolves at the origin. GET /v1/tenants reports each node’s set digest, and the gossiped config digest below covers the live set, so a node that missed a change shows in sandboxd_config_digest_mismatch. A node that is down during a fan-out has left the peer list, so it is absent from the results rather than failed: when it rejoins with its old tenants.json, that mismatch names the divergence, and one apply to that node closes it.

Without a shared store, every node holds the whole tenant set in memory and rewrites tenants.json on each change, so the set suits thousands to tens of thousands of tenants: control planes, organisations, services. Map end users onto them in the control plane (claim_ref, metadata) rather than giving each user a tenant.

With meta_store, the tenant set lives in one PostgreSQL table that every node reads, so one tenant per end user works:

Cluster-invariant config

Some config must match on every node of a cell or the cell fails in confusing ways:

config what breaks on mismatch
api_token, operator_token, the tenant set the SDK replays the authorizing token across a redirect; a mid-rotation peer missing that tenant/token answers 401, which the SDK treats as transient — the claim falls back to the origin node (which already authorized it) and resolves there
preview_secret a preview URL signed on one node fails verification on another
mesh.cluster_key nodes cannot join / decrypt gossip at all
egress_ca root a guest checkpointed/redirected across nodes trusts the cell’s root; a divergent root fails interception
egress: egress_classes, each pool’s egress and egress_upstream_env, secrets names, headers and value_env, egress_internal_allow, egress_upstream a claim or tenant redirected to a divergent node reaches different hosts, and its claim credentials are refused or not injected there
tun.subnet every node of a meta_store.cell partition draws overlay addresses from that partition, and a claim keeps its address across a move or an adoption, so a node with another subnet serves claims outside it (see advanced network mode). Not part of the gossiped digest below
checkpoint_ttl_hours a healed replica a delete broadcast missed ages out at a different time on each node, and nodes report a digest mismatch until every node has reloaded the same value
the engine root path (cocoon’s root_dir) a checkpoint or template export pins the base image blobs’ absolute path under the root that captured it; a node whose root sits at another path refuses to clone a record it healed or pulled (untrusted storage path in snapshot metadata) — the heal succeeds, the branch fails. Not part of the gossiped digest below

Each node gossips a digest of these over its live tenant set and quotas (HMAC-keyed by cluster_key when set, covering each tenant token’s SHA-256; otherwise a token-free digest of tenant names and quotas + the CA root + checkpoint_ttl_hours, so nothing brute-forceable rides cleartext gossip) plus the egress config above in an order-free form, never a secret value, and republishes it after every tenant change and every config reload. A mismatch logs a warning at the moment the divergent node appears — not at the first unlucky redirect — and raises the sandboxd_config_digest_mismatch gauge. It is warn-only: a rolling credential rotation is a legitimate transient mismatch, so a divergence never partitions the mesh.

Scale

Every memberlist push/pull carries the whole cluster’s placement state: each node’s warm counts, template set and volume set, as one JSON document. Its size grows with nodes × templates per node:

nodes templates per node state per push/pull
1,000 5 0.6 MB
10,000 5 6.3 MB
10,000 500 353 MB

memberlist refuses a push/pull state over 20 MiB, and above 32 nodes it stretches the push/pull interval with log2 of the cluster size (10 s at 10,000 nodes), so a change takes minutes to reach every node while warm counts move with every claim.

Cluster checklist