Hot-plug vhost-user-fs shares, VFIO PCI devices, and external raw disks on a running VM (Cloud Hypervisor; raw disks also on Firecracker VMs created with --pci).
Cocoon can hot-plug three classes of external resources onto a running VM:
virtiofsd (or compatible backend) over a Unix socket. Attach surfaces a virtio-fs device in the guest, accessible via mount -t virtiofs <tag> /mnt/....vfio-pci (GPU, NIC, NVMe). Attach hands the device to the guest with IOMMU isolation./dev/disk/by-id/virtio-<name>. Cocoon never creates or deletes the backing file: it can outlive any VM and be re-attached elsewhere (a persistent volume).All three attaches are runtime-only: the device lives only for the current VM process and is gone after stop/restart — re-attach after the next start. cocoon refuses snapshot save and vm hibernate while any of the three is attached — umount an external disk in-guest, detach, capture, then re-attach after the wake/restore; on both backends the refusal arrives before the guest is paused. Cloud Hypervisor does not enforce this itself: it captures a snapshot that names the vhost-user socket, and restoring that snapshot either fails once the backend is gone or hangs against a fresh one. External volumes are host state cocoon does not own: it never creates, copies, or deletes the backing file, and snapshot, restore, and clone carry no trace of them. All mutating verbs on one VM (attach/detach, net resize, snapshot, hibernate, restore, stop) serialize on a per-VM lock, so concurrent invocations cannot interleave with a capture window. While the VM runs, vm inspect reports the live attach set under .attached_devices (fs, devices, disks).
Prerequisite: the VM must have been created with --shared-memory. CH’s memory shared=on cannot be flipped on a running VM, and vhost-user-fs requires it to share guest memory with the backend process.
# 1) Boot VM with shared-memory enabled.
cocoon vm run --shared-memory --name share-host ghcr.io/cocoonstack/cocoon/ubuntu:24.04
# 2) On host, run virtiofsd against a directory.
virtiofsd --socket-path=/tmp/virtiofsd.sock --shared-dir=/srv/data --cache=never &
# 3) Attach to the VM (tag is the guest mount tag and detach key).
cocoon vm fs attach share-host --socket /tmp/virtiofsd.sock --tag data
# 4) Inside the guest:
mkdir -p /mnt/data && mount -t virtiofs data /mnt/data
# 5) Detach later:
cocoon vm fs detach share-host --tag data
Flags:
| Flag | Default | Description |
|---|---|---|
--socket |
required | Absolute path to the virtiofsd unix socket |
--tag |
required | Guest mount tag (also detach key) |
--num-queues |
1 |
Request queues |
--queue-size |
1024 |
Queue depth |
# Provision a raw disk file anywhere on the host (cocoon never touches its lifecycle).
dd if=/dev/zero of=/srv/volumes/vol1.raw bs=1M count=1024 && mkfs.ext4 /srv/volumes/vol1.raw
# Attach: name is the guest serial (/dev/disk/by-id/virtio-vol1) and the detach key.
cocoon vm disk attach my-vm --path /srv/volumes/vol1.raw --name vol1
# Inside the guest:
mount /dev/disk/by-id/virtio-vol1 /mnt/vol1
# Detach later (backing file kept; re-attach to another VM to see the same data — the same VM refuses a duplicate path or serial):
cocoon vm disk detach my-vm --name vol1
The backing file may live anywhere outside cocoon’s managed directories;
attach resolves symlinks and refuses a path under the root/run/log dirs
because vm rm would delete it along with them.
Both calls are idempotent. Attaching a disk that is already attached under the same name, path and mode succeeds and returns its existing id; the same name with a different path or mode is refused. Detaching a name that is not attached succeeds and logs a warning; on Cloud Hypervisor, a retry while the guest has not yet ejected an earlier removal waits for that eject.
Flags:
| Flag | Default | Description |
|---|---|---|
--path |
required | Absolute path to an existing raw disk file |
--name |
required | Guest serial and detach key (^[a-z][a-z0-9_-]{0,19}$; the cocoon- prefix is reserved) |
--readonly |
false |
Attach read-only |
--directio |
auto |
O_DIRECT for the disk: on/off/auto (use off for files on tmpfs); Cloud Hypervisor only, Firecracker ignores it with a warning |
On Cloud Hypervisor, detach blocks until the guest acks the ACPI eject (up to 30s — Windows can take 10–20s), so when it returns the slot, the name, and the backing file are free for immediate reuse; a guest that never acks fails the detach with an error naming the device. Firecracker detach returns as soon as the DELETE lands; run the printed hint so the guest drops the stale node.
The block device (/dev/vdX, serial visible in lsblk -o NAME,SERIAL) is
usable immediately after attach. The /dev/disk/by-id/virtio-<name> symlink
is created by guest udev and can lag or be skipped on minimal images —
udevadm trigger --action=add --subsystem-match=block && udevadm settle
materializes it; scripts should resolve by serial instead.
Prerequisite: host has intel_iommu=on (or amd_iommu=on) on the kernel command line and the target PCI device is bound to vfio-pci.
# Bind the device on the host (one-time per device).
echo 0000:01:00.0 > /sys/bus/pci/drivers/vfio-pci/bind # see https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF
# --pci accepts: short BDF (01:00.0), full BDF (0000:01:00.0), or
# sysfs path under /sys/bus/pci/devices/. Other absolute paths are
# rejected so cocoon does not forward a non-PCI directory to CH.
cocoon vm device attach my-vm --pci 01:00.0 --id mygpu
# Detach.
cocoon vm device detach my-vm --id mygpu
cocoon vm inspect VM includes an attached_devices field for running VMs that surfaces every attached vhost-user-fs share, VFIO device, and hot-attached disk, read live from CH vm.info. The field is omitted for stopped VMs and whenever the live attach set is empty.
--pci)Firecracker VMs created with --pci accept vm disk attach/detach (vhost-user-fs and VFIO stay Cloud Hypervisor only); MMIO VMs are rejected. Firecracker does not notify the guest, so cocoon prints the guest-side step after each call, and --output json carries it as hints:
cocoon vm disk attach my-vm --path /vols/scratch.raw --name scratch
# echo 1 > /sys/bus/pci/rescan # inside the guest: the disk appears as /dev/vdX (no virtio-blk serial on FC)
cocoon vm disk detach my-vm --name scratch # unmount inside the guest first
# echo 1 > /sys/block/<vdX>/device/../remove
Firecracker’s own guide removes the guest’s PCI node before the host unplugs; cocoon runs nothing inside the guest, so it unplugs first (after the caller has unmounted or downed the device) and prints the stale-node removal, which the guest kernel handles once the device is gone. Verified on the ubuntu 24.04 image for disks and NICs.
As on Cloud Hypervisor, a hot-attached disk blocks snapshot and hibernate until detached; cocoon vm inspect lists it from Firecracker’s /vm/config under the id cocoon_disk_<name> (Firecracker ids allow only letters, digits and underscores, so a name with - is rejected there).