Limitations, workarounds, and upstream tracking. Fixed entries are kept for operators running older binaries.
cocoon vm clone allocates the clone its own IP, but the guest still carries the source VM’s network configuration, so the clone does not answer on that IP until the guest is reconfigured. What the guest wakes up with differs by backend, because only Cloud Hypervisor changes the guest-visible MAC:
ens3 → ens8 on cloudimg, eth0 → eth1 on OCI). The snapshot’s MAC-matched configuration then matches nothing and the guest has no IPv4 address at all: it cannot reach the network, and it cannot collide with the source VM either.Consequence on Firecracker: if the source VM is still running, both VMs advertise the same IP with different MACs, the upstream gateway flaps between them, and both see intermittent connectivity loss until the clone is reconfigured. Nothing in the data path prevents this — it is L2 end to end (TC ingress redirect + bridge), no component checks the guest’s source IP against the allocated one, and bridge CNI does not enforce IP ↔ veth binding (host-local IPAM only tracks allocations in its control-plane state files, not in data-plane rules).
Mitigation: run the post-clone guest setup commands printed by cocoon vm clone (see Post-Clone Guest Setup). For cloudimg VMs they re-run cloud-init; for OCI VMs they replace the generated /etc/systemd/network/10-*.network files with per-MAC units for the new IP and restart systemd-networkd; on Firecracker they set the new MAC first. The clone answers on its allocated IP a few seconds after they run.
cocoon vm clone inherits CPU, memory, and storage from the snapshot — none of these can be grown at clone time on either backend. NIC count inherits by default; Cloud Hypervisor clones may override it with --nics N (cocoon hot-swaps the snapshot’s NICs for a fresh set after restore). MMIO Firecracker clones must keep the snapshot’s NIC topology — FC’s network_overrides only retargets existing interfaces, so --nics is rejected there (a --pci snapshot restores its NICs and hot-plugs the delta). On every Firecracker clone, MMIO and --pci alike, the target network’s MTU must equal the snapshot’s because the guest keeps the advertised MTU. cocoon vm restore is more restrictive: CPU, memory, and storage come from the snapshot (the persisted record is realigned to match), and NIC count must already match the target VM (mismatches are rejected, since restore reuses the existing network namespace). Use cocoon vm run to create a fresh VM with different sizing.
OCI VMs use the kernel ip= boot parameter for network configuration. While multiple ip= parameters can be specified, the Linux kernel only reliably configures one interface via this mechanism — subsequent ip= parameters may be silently ignored or produce inconsistent results depending on kernel version.
Consequence: on every boot of an OCI VM with multiple NICs, the first vm run included, only the last NIC receives its IP from the kernel ip= parameter (each ip= overrides the previous one). The other NICs come up without an IPv4 address until the guest init system configures them (e.g., systemd-networkd .network files written by the post-clone hints).
Workaround: persistent MAC-based systemd-networkd configs for all NICs configure every interface on every boot regardless of the kernel ip= limitation. The post-clone setup hints write them for a clone; a VM created with vm run --nics N has only the one the initramfs writes for the NIC its ip= configured, so write one .network file per other NIC in the guest, matching the MAC and address cocoon vm inspect reports.
When --user specifies a non-root username (e.g. --user admin), the user is created via cloud-init runcmd which runs in the final stage — after networking, SSH key generation, and other modules. This means:
cloud-init status shows done when the user is readyThe root user’s password is set via chpasswd (config stage, earlier) and is available sooner, but --user admin deliberately does not set a root password — root stays locked for security.
Workaround: wait for cloud-init status: done before attempting SSH. The default root/cocoon credentials use the faster chpasswd path and work on the console as soon as it is set; stock cloud images ship sshd with PermitRootLogin prohibit-password, so root over SSH needs a key or a custom image — use --user for password SSH.
Clone regenerates cidata for network reconfiguration only — it does not inject new user or password settings. The cloned VM’s credentials are whatever the source VM had in /etc/shadow at snapshot time.
--user/--password flags are not available on cocoon vm cloneThis is by design: clone restores the VM’s exact state including all account settings.
vm exec requires vsock — legacy VMs excludedcocoon vm exec dials the cocoon-agent inside the guest over hybrid vsock. Two caveats:
vm inspect omits vsock_socket and vm exec returns vsock not configured for this VM (recreate the VM to enable agent exec). Recreate the VM to gain exec capability.vsock_override field on PUT /snapshot/load was merged post-v1.15). Older FC rejects the field with unknown field vsock_override. Cocoon omits the field on same-VM restore, so FC < v1.16 still runs and restores fine — only vm clone needs the field.vm clone or vm restore the guest agent never answers hybrid-vsock CONNECT, with no recovery. Fixed upstream in FC v1.16.1 — the effective floor for any FC restore/clone lane. doctor/check.sh installs the fork’s rolling dev release, which carries the fix.Windows guests are supported as of cocoon-agent v0.1.3 (registered via SCM, runs as LocalSystem). Official ghcr.io/cocoonstack/windows/win11:* images bake the agent and a CocoonNicAutoHeal scheduled task that recovers chained-clone NDIS-stuck NICs in-guest. DIY Windows images need to install both pieces themselves; without the agent, vm exec returns read CONNECT reply: EOF.
Race window: cocoon vm run X && cocoon vm exec X -- cmd may fail with read CONNECT reply: EOF if the in-guest agent hasn’t started yet. The error includes the hint (cocoon-agent may still be starting; retry shortly). Wait ~5–10s after vm run returns.
On x86_64, cocoon uses rust-hypervisor-firmware (CLOUDHV.fd) for cloud image UEFI boot. This firmware implements a minimal EFI specification and does not support the InstallMultipleProtocolInterfaces() call required by newer distributions. aarch64 hosts boot through EDK2 (CLOUDHV_EFI.fd, installed by cocoon-check --upgrade), which is not affected.
Affected images (kernel panic on boot — GRUB loads kernel but not initrd):
Working images:
This is an upstream issue tracked in rust-hypervisor-firmware#333 and cloud-hypervisor#7356. As a workaround, use OCI VM images for Ubuntu 24.04 — OCI images use direct kernel boot and are not affected.
cocoon never removes a running VM’s named network namespace. If an operator or a foreign GC runs ip netns del, it removes the name under /var/run/netns; the running VMM still holds the original namespace and its TAPs. Existing traffic may continue, and vm list still shows the recorded IP. NIC hot-add cannot open the missing netns path, while cocoon vm exec can still work over vsock.
To restore the named path, stop and relaunch the VMM through the start-time recovery path (Verify → Prepare → re-plumb every NIC with its recorded MAC, IP and TAP name):
cocoon vm hibernate VM then cocoon vm restore VM <snapshot> — about 300 ms, processes and RAM untouchedcocoon vm stop --force VM then cocoon vm start VMDo not recreate the name by hand while the VM runs: it would point to a different namespace from the one holding the VMM and existing TAPs, so a later NIC hot-add would provision in the wrong place.
When using a DHCP-based network (e.g., macvlan attached to a network with an external DHCP server), the CNI conflist should not use the dhcp IPAM plugin. Instead, configure the CNI plugin with no IPAM (or "ipam": {}) and let the guest obtain its IP directly from the external DHCP server.
The dhcp IPAM plugin runs a host-side DHCP client that competes with the guest’s own DHCP client, causing:
dhcp daemon must remain running to renew leases; if it crashes or is restarted, the host-side lease expires while the guest keeps using the IP.This applies to all CNI plugins where the upstream network provides DHCP (bridge with external DHCP, macvlan, ipvlan, etc.). The correct approach is:
{
"cniVersion": "1.0.0",
"name": "my-dhcp-network",
"plugins": [
{
"type": "macvlan",
"master": "eth0",
"mode": "bridge",
"ipam": {}
}
]
}
Cocoon detects when CNI returns no IP allocation and automatically configures the guest for DHCP — cloudimg VMs get dhcp4: true in their Netplan network-config (plus MAC-matched DHCP=ipv4 systemd-networkd units written by cloud-init), and OCI VMs get DHCP systemd-networkd units generated by the initramfs cocoon-network script.
Note: the OCI initramfs uses IP=off to prevent the initramfs from running its own DHCP client during boot. DHCP is handled entirely by systemd-networkd after switch_root. The configure_networking function is only called when a kernel ip= parameter is present (static IP from CNI).
ipMasq leaks a masquerade rule per VMWith "ipMasq": true, the CNI bridge plugin adds a POSTROUTING rule for each VM address and a CNI-… chain for each VM on ADD. On DEL it removes them only for the addresses it reads off the VM’s interface in the netns. Cocoon gives that address to the guest and removes it from the netns interface when it sets up the TC redirect, so the plugin finds none, and every removed VM keeps its rule and chain. The plugin’s iptables backend also shares one chain across all NICs of a VM on one network, so a teardown of one NIC would clear it under the others.
Leave ipMasq off and masquerade the bridge subnet on the host once:
iptables -t nat -A POSTROUTING -s 10.22.0.0/16 ! -o cni0 -j MASQUERADE
The rules an earlier ipMasq network left carry its name in their comment, for a network named cocoon:
iptables-save -t nat | grep -F 'name: \"cocoon\"'
Status: FIXED in our fork and upstream.
Cloud Hypervisor v51.x had a regression (#7849) that caused Windows to BSOD (DRIVER_IRQL_NOT_LESS_OR_EQUAL in viostor.sys) when DISCARD/WRITE_ZEROES features were advertised with default-zero config values, violating virtio spec v1.2.
Fix: the DISCARD fix is included in our Cloud Hypervisor fork (cocoonstack/cloud-hypervisor dev branch). Upstream has also merged it (PR #7936). Cloud Hypervisor v52 and newer work correctly with Windows VMs.
Previous recommendation (no longer needed): use Cloud Hypervisor v50.2 for Windows VMs.
Status: FIXED upstream since Cloud Hypervisor v52.
virtio-win 0.1.271+ network drivers were incompatible with Cloud Hypervisor due to incomplete virtio-net control queue implementation (#7925). CH only handled CTRL_MQ and CTRL_GUEST_OFFLOADS; all other commands (CTRL_RX, CTRL_MAC, CTRL_VLAN, CTRL_ANNOUNCE) returned VIRTIO_NET_ERR.
| Version | Behavior on VIRTIO_NET_ERR |
|---|---|
| 0.1.240 | Tolerates error, continues working |
| 0.1.271 | May silently fail, NIC unusable |
| 0.1.285+ | Fail-fast: NdisMRemoveMiniport(), Problem Code 43 |
0.1.285 introduced commit 50e7db9 (“indicate driver error on unexpected CX behavior”) with zero-tolerance on control queue errors. Root cause was a CH bug — the correct fix is to return VIRTIO_NET_OK for unsupported commands and to report the correct used_len.
Fix: the ctrl_queue command tolerance (from @liuw) plus the used_len fix merged upstream in PR #7953 and ship in Cloud Hypervisor v52 and newer, including the cocoonstack fork build. virtio-win 0.1.285 now works.
Previous recommendation (no longer needed): use virtio-win 0.1.240 for Windows VMs on Cloud Hypervisor.
Status: FIXED in our firmware fork.
Cloud Hypervisor uses a GED (Generic Event Device, ACPI0013) to deliver power-button notifications on its hardware-reduced ACPI platform. While this mechanism works correctly for Linux guests, Windows guests did not respond to the vm.power-button API call — no power-button event appeared in the Windows event log (Event ID 109).
Root cause: the EFI ResetSystem runtime service in rust-hypervisor-firmware was a no-op. When Windows attempted a graceful shutdown via the UEFI reset path, nothing happened. Tracked in cloud-hypervisor/rust-hypervisor-firmware#422 and cloud-hypervisor/cloud-hypervisor#7929.
Fix: our firmware fork (cocoonstack/rust-hypervisor-firmware dev branch) implements ResetSystem properly. Upstream merged the same fix in cloud-hypervisor/rust-hypervisor-firmware#424 (replacing #423); no firmware release carries it yet, the latest being 0.5.0. With this fix, the ACPI power-button works for Windows guests, and cocoon vm stop completes in ~8-13 seconds on a fully booted VM.
Timing caveat: the Windows ACPI shutdown handler needs ~60 seconds from cold boot to fully initialize (SAC appears at ~30s, handler ready ~30s later). Stopping a Windows VM before the handler is ready triggers the 30s stop_timeout_seconds fallback and escalates to force-kill. Clone-restored VMs inherit the ready ACPI state and shut down in ~8-13s immediately.
Previous consequence (no longer applies with our firmware fork): cocoon vm stop always timed out on Windows VMs (default 30s), then fell back to vm.shutdown → SIGTERM → SIGKILL.
Previous workaround (no longer needed with our firmware fork): shut down Windows guests via SSH or WinRM before stopping:
ssh cocoon@<vm-ip> "shutdown /s /t 0"
cocoon vm stop <vm>
Or use cocoon vm stop --force to skip the ACPI timeout and immediately kill the process.
The Windows image’s autounattend.xml includes defensive power-button configuration (PBUTTONACTION=3) and shutdown optimization (WaitToKillServiceTimeout=5000, shutdownwithoutlogon=1) which remain useful for environments not using our firmware fork.
Status: WONTFIX — disabled by design.
Cocoon does not attach a virtio-balloon device to Windows VMs (--windows). The virtio-win balloon driver >= 0.1.262 (PR #1157, “cope with the unresponsive host”) changed balloon deflation behavior: when the host does not ACK a balloon page-batch within 1 second, the driver now retries indefinitely instead of giving up. This causes two problems during ACPI power-button shutdown:
Deflation CPU storm: Windows begins tearing down virtio devices during shutdown. The balloon driver attempts to deflate (return all reclaimed pages to the guest) one 512-page batch at a time, retrying on every host timeout. This pins all vCPUs at 100% for 2–3 minutes (virtio-win #1148, open, unresolved upstream).
Watchdog reboot: Cloud Hypervisor’s virtio-watchdog has a 20-second timeout. Windows stops petting the watchdog during shutdown, and the balloon deflation outlasts the timeout — the watchdog triggers a VM reset before Windows can call UEFI ResetSystem to write to the ACPI shutdown port (0x600). The VM reboots instead of powering off.
virtio-win 0.1.240 did not have this problem because its balloon driver gave up on the first host timeout, allowing Windows to proceed to ResetSystem quickly (~14 seconds total shutdown).
Workaround if balloon is needed: increase stop_timeout_seconds to 180+ and apply the watchdog-pause patch to Cloud Hypervisor (pause the watchdog timer when vm.power-button is received).
Workaround for guest-initiated reboot: if the guest driver leaves the watchdog armed, create the VM with --no-watchdog. This trades device-level hang reset for reboot safety and should be paired with an external liveness policy where automatic recovery is required.
See cocoonstack/windows for download and installation instructions.
Firecracker snapshots store absolute host paths in the vmstate binary (Rust serde format, not patchable). This means:
--force), and clone rejects a missing source drive whose recorded path falls outside the current managed run_dirroot_dir and run_dir (default: /var/lib/cocoon and /var/lib/cocoon/run) and have the same OCI image pulledsnapshot/load Cocoon bind-mounts the clone’s disks over the source-absolute paths inside a private mount namespace, holding a shared per-VM lease on the source so concurrent source operations wait; a dead source gets an empty placeholder file to bind over. The host namespace is never mutatedThis is a fundamental Firecracker design limitation. Cloud Hypervisor snapshots do not have this restriction because CH stores device config in a patchable JSON format (config.json).
Upstream fix in progress: Firecracker PR #5774 adds drive_overrides to the PUT /snapshot/load API, which would eliminate the bind-mount redirect and make FC snapshots natively portable. Track this PR for future simplification.
Firecracker does not support virtio-blk serial numbers. Cocoon’s OCI init script (overlay.sh) uses device paths (/dev/vdX) instead of serial names to identify disks when booting under Firecracker. OCI images built from os-image/ubuntu/overlay.sh (v0.3+) support both formats automatically. Older images must be rebuilt to work with --fc.
cidata.imgWhen a cloudimg VM is snapshotted after it has been stopped and started at least once, the resulting snapshot tar still contains a cidata.img file even though the snapshot’s cocoon.json sidecar and CH config.json no longer reference cidata. This is because hypervisor/cloudhypervisor/snapshot.go copies cidata.img whenever the source VM’s runDir has it (cloudimg + non-Windows), regardless of whether activeDisks(rec) would have re-attached it.
Consequence: ~1 MB of dead weight inside the tar. No correctness impact:
cidata.img via ensureCloneCidata and overwrites the orphan with the clone’s identity / network config.cleanSnapshotFiles removes only the memory ranges, config.json, state.json, cocoon.json and data-*.raw, so the staged cidata.img is merged back over the VM’s own copy — harmless, since restore targets the same VM.This predates the data-disk feature; the cocoon.json sidecar just makes the asymmetry visible. We are not patching the copy logic because the orphan is harmless and the cidata disk is small (sparse-copied, ~1 MB).
Firecracker does not support overriding the guest MAC address during snapshot/load. Cloned FC VMs retain the source VM’s guest MAC (baked into the vmstate binary). In Cocoon’s TC redirect architecture, each VM runs in an isolated network namespace, so MAC identity is not visible to other VMs or the host bridge — no MAC conflict occurs in practice.
On CNI plugins with strict per-veth MAC enforcement (Cilium eBPF, Calico eBPF), the guest MAC vs veth MAC mismatch could theoretically cause packet drops. This has not been observed in testing with the standard bridge CNI.
Upstream status: FC’s NetworkOverride struct only has iface_id and host_dev_name — no guest_mac field. Adding it would follow the existing VsockOverride pattern. No issue or PR exists yet.
Workaround: If MAC matching is required, run ip link set dev ethX address <new-mac> inside the guest after clone (the post-clone hints print the expected MAC values).
cocoon vm fs attach only works on CH VMs that were created with --shared-memory. CH’s memory shared=on is fixed at VM creation: backend processes (e.g. virtiofsd) need to mmap guest memory via the negotiated memfd, and the memory model cannot be flipped on a running VM. If --shared-memory was omitted at create time, the only path is to recreate the VM. Cocoon’s preflight reads vm.info and surfaces a clear error rather than letting CH return a vague rejection.
Cloud Hypervisor captures a snapshot of a VM that holds a vhost-user-fs share or a VFIO PCI passthrough device, but the result cannot be restored: it names a backend socket that is gone (restore fails after 60s) or hangs against a fresh one. Cocoon refuses the capture itself — the pre-flight reads the live vm.info before pausing the guest and fails with hot-attached vhost-user-fs "<tag>": detach before snapshot or hibernate (or hot-attached device "<path>": ...) before anything is captured or recorded. cocoon vm fs detach / cocoon vm device detach first to clear runtime devices, then snapshot.
Fixed in v0.6.8 by #253: snapshot rm unlinks the snapshot’s <root>/snapshot/localfile/<id>.lease when it releases the exclusive lease, and gc reclaims the leases of snapshots that no longer have a record or a data dir, so a host upgraded from an older binary converges on its first gc run (133 leftover files on one testbed went to the four live snapshots’ leases).
On v0.6.7 and older every snapshot save leaves that zero-byte file behind and snapshot rm and gc remove only the data dir, so the files accumulate one inode per snapshot ever created; delete them by hand while no snapshot verb runs.
--from-dir and snapshot import reject a snapshot rebuilt by a third-party tarcocoon vm clone --from-dir, cocoon vm restore --from-dir and cocoon snapshot import compare cow.raw’s on-disk size with the size recorded in snapshot.json. cocoon’s export tar stores sparse disks with private pax records that only its own extractor understands; a generic tar -x drops them and rebuilds a short, shifted disk that would restore into silent corruption. The snapshot is refused with cow.raw in <dir> is N bytes but the envelope records M. Use snapshot export --to-dir for the directory form, or snapshot import on the tar exactly as cocoon wrote it.
Attaches via cocoon vm fs attach and cocoon vm device attach are runtime-only — they live in the CH process state and are never written into the VM record, sidecar, or snapshot. After vm stop, vm clone, or vm restore, the user must re-run the attach commands. vm inspect reflects the live CH vm.info for running VMs and omits attached_devices for stopped VMs. This is by design: cocoon does not own the backend (virtiofsd / vfio-pci binding) and cannot guarantee the resource still exists across host events.
Upstream virtiofsd serves exactly one vhost-user client and exits when that client disconnects. Consequence: after cocoon vm fs detach, the daemon is gone — a follow-up cocoon vm fs attach against the same socket path will hang or time out until a fresh virtiofsd instance is launched. The same applies after cocoon vm stop (CH closes the socket on shutdown). Scripts that cycle attach/detach should respawn virtiofsd between calls. This is a virtiofsd behavior, not a cocoon limitation.
root:cocoon and PermitRootLogin yesEvery Ubuntu and Debian image under os-image/ enables openssh-server with PermitRootLogin yes and the default root:cocoon credentials baked in. This is convenient for development and matches the existing OCI-image behavior, but it is not safe for production exposure.
Mitigations for production users:
passwd root) and/or disable password auth (PasswordAuthentication no) inside the guest before exposing it.PermitRootLogin back to no.install-agent.sh and drop its sshd step (the PermitRootLogin sed and systemctl enable ssh); the script has no flag or env knob for it.Control-plane traffic from cocoon-managed hosts (vk-cocoon, cocoon vm exec) goes through cocoon-agent over vsock and never depends on SSH credentials.
cocoon vm net --nics N waits for CH’s device_tree to drop the removed device (polled via vm.info until the entry disappears, max 30 s) before tearing down host plumbing. CH only frees the PCI slot — and unregisters the device’s ioeventfds — when the guest writes to the ACPI hot-plug controller (B0EJ) in response to the SCI raised by remove-device. If the guest never ACKs (driver wedged, paused, NDIS halted), the wait times out, the cocoon record is left intact, and the command returns an error so the user can quiesce the guest and retry.
If the host plumbing teardown itself fails after a successful eject, cocoon truncates the record anyway and surfaces a warning — the orphan TAP / veth / CNI lease is cleaned up by cocoon vm rm or the next gc cycle. The user still has to consider:
Cannot register ioevent: File exists when CH next tries to reuse that slot (e.g. a subsequent hot-add).Quiesce the guest NIC (ip link set down + NetworkManager remove on Linux; Disable-PnpDevice + driver unbind on Windows) before reducing the count.
Firecracker VMs created with --pci resize through the same command; the guest receives no eject notification, so cocoon prints the rescan/remove steps instead of waiting (see networking). MMIO Firecracker VMs are rejected.
Plain cocoon vm net --nics N polls CH’s device_tree after vm.remove-device until the guest ACKs B0EJ via ACPI/SCI (typically < 1 s on Linux, 10–20 s on Windows). The clone path’s hotSwapNets runs while the VM is paused, so the guest cannot process SCI — eject stays pending until resume. The clone path therefore returns without waiting and adds the fresh NICs against half-removed device-tree state. This is the long-standing CH limitation cocoon was designed around; the new wait only applies to the running-VM resize.
os-image/android/{14.0,15.0,15.0-gms} install the cocoon-agent binary at /system/bin/cocoon-agent and register it via /system/etc/init/cocoon-agent.rc. Android’s SELinux policies don’t ship with a domain for cocoon-agent, so the service may run in init’s domain or be denied outright depending on the redroid build.
If cocoon vm exec against an Android VM returns dial agent: ..., check logcat | grep -i avc inside the guest. The fix is build-time — adjust the Android sepolicy to grant the new binary network/socket permissions — and is out of scope for the Dockerfile.