cocoon sandbox

Advanced network mode

The default network mode of a none-lane guest is the proxy path: silkd relays 127.0.0.1:3128 and 127.0.0.1:1080 to the host’s guarded-egress proxy over vsock. The advanced network mode adds a full L3 path beside it for clients the proxy cannot carry (UDP, QUIC, WebRTC, clients that ignore proxy variables): silkd creates tun0 in the guest and carries raw IP packets over one vsock stream to the host. The guest still has no NIC, so hibernate, archive, wake and move work as before.

This page is the contract between silkd and the host. The host side design is #471.

Guest side

Item Value
Device tun0, IFF_TUN \| IFF_NO_PI \| IFF_VNET_HDR, owned by silkd
Address 100.64.0.2/30 in every guest; the host tells guests apart by their vsock stream, never by address
Gateway 100.64.0.1, default route via tun0
MTU 1500 until the host hello names another
IPv6 none; silkd turns IPv6 off on tun0
Packet stream guest dials host vsock port 2051 (<vsock_socket>_2051 on the host)

Both modes run at once. silkd keeps the proxy variables in every exec, so a client that honours them still reaches 127.0.0.1:3128 over loopback; all other traffic takes the default route through tun0. Loopback never enters tun0, so the guest DNS relay on 127.0.0.153 is unaffected; the tun path does not intercept DNS.

Turning it on and off

The guest switch is advisory. The guest user can be root, create its own TUN and dial 2051 directly, so the host enforces: it accepts a packet stream only for a tun-enabled claim, rewrites every source address, and closes the stream to revoke.

Wire protocol (version 1)

All integers are little-endian.

Hello

Each side sends one 12-byte hello first; the guest sends first, the host answers.

Offset Size Field Guest Host
0 4 magic SLKT SLKT
4 2 version highest it speaks (1) the version both use; ≤ the guest’s
6 2 offloads offloads its tun accepts on write CSUM, TSO4, TSO_ECN, USO4 and USO6, always
8 2 mtu 0 MTU for tun0; 0 keeps the current one
10 2 flags 0 bit 0 ON: packet mode is on for this box

Flag bits 1 to 15 are reserved and sent as 0; a peer ignores a bit it does not know.

A host that answers without ON closes the stream after the hello; the guest removes tun0. A host that refuses a guest answers without ON rather than closing silently. A guest that does not speak the host’s version closes the stream. The guest gives up on a hello that does not arrive in 5 seconds.

Offload bits are the kernel’s TUN_F_* values: 0x01 CSUM, 0x02 TSO4, 0x04 TSO6, 0x08 TSO_ECN, 0x20 USO4, 0x40 USO6. The guest probes its kernel (USO needs Linux 6.2) and then enables, with TUNSETOFFLOAD, only the offloads both hellos carry; segmentation offloads are dropped when CSUM is not common, and USO is enabled only as the USO4 and USO6 pair. The host has no TUN and segments in software; it offers USO6 beside USO4 so a guest kernel with USO enables it, and drops an IPv6 frame as it drops any non-IPv4 packet. A side never builds a frame whose GSO type the peer did not offer; packets its kernel queued before a reconnect keep their old type, and a receiver may drop one it cannot take.

Frames

After the hellos, both directions carry frames back to back:

u32 len | virtio_net_hdr (10 bytes) | IP packet

len counts the header and the packet: 10 to 65545 bytes. A length outside that range is a protocol error and ends the stream. The header is the plain 10-byte struct virtio_net_hdr (flags, gso_type, hdr_len, gso_size, csum_start, csum_offset) exactly as IFF_VNET_HDR reads and writes it, so a GSO packet of up to 64 KiB crosses vsock as one frame and is segmented, if at all, by the receiving kernel. A packet the receiving kernel refuses is dropped like a packet lost on a link.

Writers batch: every frame ready at the moment goes out in one stream write. Readers parse every complete frame one stream read returns; the io_uring path hands them to the TUN in batched submissions.

Reconnect

There is no session state beyond the hello. After any stream loss the guest dials again and both sides renegotiate offloads and MTU, so a move to a host with a different kernel or MTU is safe. Packets in flight during the gap are lost; TCP retransmits.

Host side

sandboxd needs no TUN, no kernel routes and no NAT rules on the node. It carries each claim’s packets over VXLAN, on an ordinary UDP socket, to an exit: plain Linux configuration on the same host or on a gateway machine.

guest -> tun0 -> silkd --vsock 2051--> sandboxd --VXLAN--> exit (vxlan0, forward, NAT) -> uplink

Configuration

The node’s tun block turns the mode on; a pool opts in with "tun": true (the none lane only). Like egress, the flag is config-owned: PUT /v1/pools rejects it.

"tun": {
  "subnet": "100.80.0.0/12",
  "listen": "10.0.0.5:4789",
  "exits": {"default": {"vtep": "10.0.0.20", "mac": "02:ee:00:00:00:01"}},
  "limits": {"bytes_per_second": 125000000, "quota_bytes": 107374182400}
}
key default meaning
subnet required the overlay addresses, /4 to /24, outside 100.64.0.0/30; every node of a meta_store.cell partition sets the same one; different partitions need non-overlapping subnets or separate overlays (cells); a change needs a restart, and claims that still hold an old address keep it until released, so drain tun claims first
listen required the node’s VXLAN socket, a specific IPv4 ip:port; the exits send to it
vni 1 the VXLAN network identifier
port 4789 the exits’ VXLAN port
mtu 1450 the guest tun0 MTU, 576 to 9000
senders 4 send sockets; the outer source port follows the flow across them
exits required profile label to {vtep, mac}; a default entry is required
limits max_flows 16384, new_flows_per_second 1000 per claim: bytes_per_second, packets_per_second, new_flows_per_second, max_flows, quota_bytes; an unset bytes_per_second, packets_per_second or quota_bytes is unlimited; a claim’s egress-limit replaces bytes_per_second per direction
reuse_after_seconds 432000 how long a freed address waits; match the exits’ nf_conntrack_tcp_timeout_established, and on a mesh keep it above an hour, the interval at which a node re-reserves an address released while it still held it

Single-host exit

The exit’s VXLAN socket binds the wildcard address, so on the node itself the exit lives in its own network namespace, reached over a veth pair. This is the layout the CI test runs:

ip netns add exit
ip link add vh type veth peer name ve netns exit
ip addr add 10.99.0.1/24 dev vh && ip link set vh up          # tun.listen 10.99.0.1:4789
ip -n exit addr add 10.99.0.2/24 dev ve && ip -n exit link set ve up
# a veth leaves the inner checksum partial; the exit must compute it
ip netns exec exit ethtool -K ve tx off
ip -n exit link add vxlan0 address 02:ee:00:00:00:01 \
    type vxlan id 1 dstport 4789 local 10.99.0.2 dev ve learning
# the exit's own address sits outside tun.subnet, so no claim can be handed it
ip -n exit addr add 192.0.2.1/32 dev vxlan0 && ip -n exit link set vxlan0 up
ip -n exit route add 100.80.0.0/12 dev vxlan0
ip netns exec exit sysctl -w net.ipv4.ip_forward=1 \
    net.ipv4.conf.vxlan0.arp_accept=1 \
    net.ipv4.neigh.vxlan0.base_reachable_time_ms=300000
# uplink: route the exit namespace out through a second veth or a NIC moved into it
ip netns exec exit nft -f - <<'NFT'
table ip exit {
  chain overlay_nat { type nat hook postrouting priority srcnat; ip saddr 100.80.0.0/12 oifname != "vxlan0" masquerade; }
  chain overlay_forward { type filter hook forward priority filter; iifname "vxlan0" oifname "vxlan0" drop; }
  chain overlay_input { type filter hook input priority filter; iifname "vxlan0" drop; }
  chain underlay_input { type filter hook input priority filter; udp dport 4789 ip saddr != 10.99.0.1 drop; }
}
NFT
ip netns exec exit sysctl -w net.netfilter.nf_conntrack_max=1048576

Limits