The default network mode of a none-lane guest is the proxy path: silkd
relays 127.0.0.1:3128 and 127.0.0.1:1080 to the host’s
guarded-egress proxy over vsock. The advanced network mode
adds a full L3 path beside it for clients the proxy cannot carry (UDP, QUIC,
WebRTC, clients that ignore proxy variables): silkd creates tun0 in the
guest and carries raw IP packets over one vsock stream to the host. The guest
still has no NIC, so hibernate, archive, wake and move work as before.
This page is the contract between silkd and the host. The host side design is #471.
| Item | Value |
|---|---|
| Device | tun0, IFF_TUN \| IFF_NO_PI \| IFF_VNET_HDR, owned by silkd |
| Address | 100.64.0.2/30 in every guest; the host tells guests apart by their vsock stream, never by address |
| Gateway | 100.64.0.1, default route via tun0 |
| MTU | 1500 until the host hello names another |
| IPv6 | none; silkd turns IPv6 off on tun0 |
| Packet stream | guest dials host vsock port 2051 (<vsock_socket>_2051 on the host) |
Both modes run at once. silkd keeps the proxy variables in every exec, so a
client that honours them still reaches 127.0.0.1:3128 over loopback; all
other traffic takes the default route through tun0. Loopback never enters
tun0, so the guest DNS relay on
127.0.0.153 is unaffected; the tun path does not intercept DNS.
tun_up (control channel, port 2048): creates tun0 with its address
and route, starts the packet stream, answers done or an error saying why
tun0 could not be created. On a guest whose tun0 is already up it ends
the current stream and dials again. sandboxd sends it once a claim of a
tun-enabled pool is live, and again after every wake, unarchive and move,
so the guest drops its old stream at once whatever the VMM did with it (on
Cloud Hypervisor the guest also notices by itself within about 200 ms of a
restore). Warm VMs never get it, so goldens are unchanged.tun_down (control channel): stops the packet stream and removes tun0
with its routes before answering done, so clients fail at once with
“network unreachable”.tun0 is up, a refused dial or a failed hello is
retried with backoff (50 ms doubling to 1 s); an off answer removes tun0.The guest switch is advisory. The guest user can be root, create its own TUN and dial 2051 directly, so the host enforces: it accepts a packet stream only for a tun-enabled claim, rewrites every source address, and closes the stream to revoke.
All integers are little-endian.
Each side sends one 12-byte hello first; the guest sends first, the host answers.
| Offset | Size | Field | Guest | Host |
|---|---|---|---|---|
| 0 | 4 | magic | SLKT |
SLKT |
| 4 | 2 | version | highest it speaks (1) | the version both use; ≤ the guest’s |
| 6 | 2 | offloads | offloads its tun accepts on write | CSUM, TSO4, TSO_ECN, USO4 and USO6, always |
| 8 | 2 | mtu | 0 | MTU for tun0; 0 keeps the current one |
| 10 | 2 | flags | 0 | bit 0 ON: packet mode is on for this box |
Flag bits 1 to 15 are reserved and sent as 0; a peer ignores a bit it does not know.
A host that answers without ON closes the stream after the hello; the guest
removes tun0. A host that refuses a guest answers without ON rather than
closing silently. A guest that does not speak the host’s version closes the
stream. The guest gives up on a hello that does not arrive in 5 seconds.
Offload bits are the kernel’s TUN_F_* values: 0x01 CSUM, 0x02 TSO4,
0x04 TSO6, 0x08 TSO_ECN, 0x20 USO4, 0x40 USO6. The guest probes its
kernel (USO needs Linux 6.2) and then enables, with TUNSETOFFLOAD, only the
offloads both hellos carry; segmentation offloads are dropped when CSUM is not
common, and USO is enabled only as the USO4 and USO6 pair. The host has no TUN
and segments in software; it offers USO6 beside USO4 so a guest kernel with USO
enables it, and drops an IPv6 frame as it drops any non-IPv4 packet. A side
never builds a frame whose GSO type the peer did not offer; packets its kernel
queued before a reconnect keep their old type, and a receiver may drop one it
cannot take.
After the hellos, both directions carry frames back to back:
u32 len | virtio_net_hdr (10 bytes) | IP packet
len counts the header and the packet: 10 to 65545 bytes. A length outside
that range is a protocol error and ends the stream. The header is the plain
10-byte struct virtio_net_hdr (flags, gso_type, hdr_len, gso_size,
csum_start, csum_offset) exactly as IFF_VNET_HDR reads and writes it, so
a GSO packet of up to 64 KiB crosses vsock as one frame and is segmented, if
at all, by the receiving kernel. A packet the receiving kernel refuses is
dropped like a packet lost on a link.
Writers batch: every frame ready at the moment goes out in one stream write. Readers parse every complete frame one stream read returns; the io_uring path hands them to the TUN in batched submissions.
There is no session state beyond the hello. After any stream loss the guest dials again and both sides renegotiate offloads and MTU, so a move to a host with a different kernel or MTU is safe. Packets in flight during the gap are lost; TCP retransmits.
sandboxd needs no TUN, no kernel routes and no NAT rules on the node. It carries each claim’s packets over VXLAN, on an ordinary UDP socket, to an exit: plain Linux configuration on the same host or on a gateway machine.
guest -> tun0 -> silkd --vsock 2051--> sandboxd --VXLAN--> exit (vxlan0, forward, NAT) -> uplink
tun.subnet, kept across hibernate, archive and move and freed at release.
A single node allocates it in memory and keeps it in claims.json, with no
disk write on the claim path; each release is appended to
<data_dir>/tun-freed.log so its reuse window outlives a restart. A node
with a mesh must have meta_store.cell and allocates it in the cell’s
table, one round trip per candidate address, and stays off (proxy mode only) while the
meta store is unreachable. A freed address is reused after
tun.reuse_after_seconds, so the subnet’s size over that window bounds the
sustained claim rate: a /8 with the 5-day default carries about 38 new tun
claims a second across the cell. The claim’s MAC is 02:00: followed by the
address bytes.default when it has none). The outer source port is one of senders
sockets, picked by a hash of the protocol, the ports and the inner destination.100.64.0.2 and onto that claim’s stream;
anything else is dropped. Inbound passes only for a flow the claim opened,
except that a TCP segment other than a bare SYN reopens its flow when the
claim’s policy allows the peer, so a stream crosses a move, a wake or a
restart. In-order TCP segments of one flow whose checksums
verify are merged into one GSO frame for the guest. An ICMP error gets the
quoted packet’s source rewritten to 100.64.0.2 too, so path MTU discovery
and port-unreachable reach the guest’s socket.EvalFlow (bare host: "*" rules only) and the internal-address
guard. The verdict also refuses the overlay subnet and 100.64.0.0/30,
whatever egress_internal_allow re-admits.
Per-claim byte, packet and new-flow rates, a flow cap and a byte quota apply
before any further work on a packet; a GSO frame is charged for every packet
and header it puts on the wire. The limits and flows belong to the claim, so
a guest that reconnects its stream starts with what it had used. A claim’s
egress-limit replaces
bytes_per_second for its up (tx) and down (rx) byte buckets, above or
below the node’s value, at once on a running claim; a direction it leaves at
zero keeps the node’s value. Packets over the bucket are dropped as before.ON only for an armed,
tun-enabled claim; disabling a claim ends its stream, drops its flows and
stops its ARP answers. A guest of a tun pool that wakes, moves or reloads
into a state without a door (disabled, no address, the pool’s tun off) is
sent tun_down in the background. The claim switch and the profile label change only
with the root token, through
PUT /v1/sandboxes/{id}/tun.audit_log on, each flow writes one
op:"tun_flow" record when it ends (fields); with
egress_usage_bytes on, each allowed flow also appends an egress_bytes
usage event with its IP bytes both ways, the destination ip:port as its
reference, and upstream tun/<label>; with
egress_usage_interval_seconds a flow that lasts
longer also writes one with the bytes since its previous one, checked every
10 s. A TCP flow that saw a FIN or RST and stayed quiet for 10 s writes its
usage event at the next check; its entry stays two minutes so late segments
of the close still match, and its tun_flow record is written when the entry
goes. A clean stop ends every live flow,
so its records cover the bytes up to the stop, and persists the quota
counter. A disarm or a move copies the counter into the claim record, which
the next claims.json write persists (a move’s on the target); a crash
loses both since the last of these.tun block, a claim of a pool without tun pays
one map lookup and one short hold of the pool lock. A tun claim
adds, on the claim path, an in-memory address (or one meta-store round
trip), a socket bind and one gratuitous ARP; tun_up is sent in the
background. Per packet: one parse, one flow-table read, the rate checks, an
incremental checksum update and one copy into the send buffer; a 64 KiB GSO
frame costs about 9 µs to segment and checksum (BenchmarkForwardGSO), a
small packet about 100 ns, a delivered packet about 250 ns, all without
allocation.The node’s tun block turns the mode on; a pool opts in with "tun": true
(the none lane only). Like egress, the flag is config-owned: PUT /v1/pools rejects it.
"tun": {
"subnet": "100.80.0.0/12",
"listen": "10.0.0.5:4789",
"exits": {"default": {"vtep": "10.0.0.20", "mac": "02:ee:00:00:00:01"}},
"limits": {"bytes_per_second": 125000000, "quota_bytes": 107374182400}
}
| key | default | meaning |
|---|---|---|
subnet |
required | the overlay addresses, /4 to /24, outside 100.64.0.0/30; every node of a meta_store.cell partition sets the same one; different partitions need non-overlapping subnets or separate overlays (cells); a change needs a restart, and claims that still hold an old address keep it until released, so drain tun claims first |
listen |
required | the node’s VXLAN socket, a specific IPv4 ip:port; the exits send to it |
vni |
1 |
the VXLAN network identifier |
port |
4789 |
the exits’ VXLAN port |
mtu |
1450 |
the guest tun0 MTU, 576 to 9000 |
senders |
4 |
send sockets; the outer source port follows the flow across them |
exits |
required | profile label to {vtep, mac}; a default entry is required |
limits |
max_flows 16384, new_flows_per_second 1000 |
per claim: bytes_per_second, packets_per_second, new_flows_per_second, max_flows, quota_bytes; an unset bytes_per_second, packets_per_second or quota_bytes is unlimited; a claim’s egress-limit replaces bytes_per_second per direction |
reuse_after_seconds |
432000 |
how long a freed address waits; match the exits’ nf_conntrack_tcp_timeout_established, and on a mesh keep it above an hour, the interval at which a node re-reserves an address released while it still held it |
The exit’s VXLAN socket binds the wildcard address, so on the node itself the exit lives in its own network namespace, reached over a veth pair. This is the layout the CI test runs:
ip netns add exit
ip link add vh type veth peer name ve netns exit
ip addr add 10.99.0.1/24 dev vh && ip link set vh up # tun.listen 10.99.0.1:4789
ip -n exit addr add 10.99.0.2/24 dev ve && ip -n exit link set ve up
# a veth leaves the inner checksum partial; the exit must compute it
ip netns exec exit ethtool -K ve tx off
ip -n exit link add vxlan0 address 02:ee:00:00:00:01 \
type vxlan id 1 dstport 4789 local 10.99.0.2 dev ve learning
# the exit's own address sits outside tun.subnet, so no claim can be handed it
ip -n exit addr add 192.0.2.1/32 dev vxlan0 && ip -n exit link set vxlan0 up
ip -n exit route add 100.80.0.0/12 dev vxlan0
ip netns exec exit sysctl -w net.ipv4.ip_forward=1 \
net.ipv4.conf.vxlan0.arp_accept=1 \
net.ipv4.neigh.vxlan0.base_reachable_time_ms=300000
# uplink: route the exit namespace out through a second veth or a NIC moved into it
ip netns exec exit nft -f - <<'NFT'
table ip exit {
chain overlay_nat { type nat hook postrouting priority srcnat; ip saddr 100.80.0.0/12 oifname != "vxlan0" masquerade; }
chain overlay_forward { type filter hook forward priority filter; iifname "vxlan0" oifname "vxlan0" drop; }
chain overlay_input { type filter hook input priority filter; iifname "vxlan0" drop; }
chain underlay_input { type filter hook input priority filter; udp dport 4789 ip saddr != 10.99.0.1 drop; }
}
NFT
ip netns exec exit sysctl -w net.netfilter.nf_conntrack_max=1048576
overlay_forward keeps claims from reaching each other through the exit.overlay_input drops everything from vxlan0 addressed to the exit itself;
ARP is not IP traffic and is unaffected.underlay_input accepts VXLAN only from node addresses, because a sender
that reaches the port could claim an overlay address with a gratuitous ARP.
sandboxd in turn accepts VXLAN only from the exits’ VTEP addresses; the
underlay must not let other hosts spoof those addresses toward tun.listen.nf_conntrack_count against nf_conntrack_max; a full table drops
new flows for every claim.nft carries the same four rules in iptables inside the
exit’s namespace.tun.listen address through the same next
hop: all on its own link, or all through one gateway. Kernel VXLAN learning
moves a claim’s MAC to its new node but keeps the route it cached for the old
one, so after a live move to a node behind another next hop the exit sends
that claim’s packets the wrong way until the cache is invalidated;
ip route flush cache in the exit’s namespace clears it at once.ethtool line is only for veth.tun.mtu is 1450, on a
1460-byte cloud underlay 1410.SILKD_TUN_IO=epoll
in silkd’s environment, uses the epoll path.