From ce02f2ae2a48f92fa1166ea50fc7f007dca9aaac Mon Sep 17 00:00:00 2001 From: Komh Date: Wed, 22 Apr 2026 23:36:31 +0800 Subject: [PATCH] [virtualization] Live Storage Migration Fails When Target Block PV Is Smaller Than Source --- ..._Target_Block_PV_Is_Smaller_Than_Source.md | 95 +++++++++++++++++++ 1 file changed, 95 insertions(+) create mode 100644 docs/en/solutions/Live_Storage_Migration_Fails_When_Target_Block_PV_Is_Smaller_Than_Source.md diff --git a/docs/en/solutions/Live_Storage_Migration_Fails_When_Target_Block_PV_Is_Smaller_Than_Source.md b/docs/en/solutions/Live_Storage_Migration_Fails_When_Target_Block_PV_Is_Smaller_Than_Source.md new file mode 100644 index 0000000..793fbbb --- /dev/null +++ b/docs/en/solutions/Live_Storage_Migration_Fails_When_Target_Block_PV_Is_Smaller_Than_Source.md @@ -0,0 +1,95 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.1.0,4.2.x +--- +## Issue + +A running VirtualMachine is being migrated between storage classes. Both the source and destination PersistentVolumeClaims request the same capacity, yet the migration fails in QEMU with a size-mismatch error surfaced in the `virt-launcher` pod log: + +```text +-blockdev {"driver":"raw","file":"libvirt-1-storage","offset":0,"size":17482664378368,...}: + The sum of offset (0) and size (0) has to be smaller or equal + to the actual size of the containing file (17482235510784) +``` + +The numbers suggest the target block device is slightly smaller (~400 MiB, in this example) than the source. The PVC objects themselves look identical, so a quick `kubectl describe pvc` does not reveal the discrepancy. + +## Root Cause + +PVC `resources.requests.storage` is a *floor*, not an exact size. The Kubernetes storage contract lets a provisioner hand back a PV that is at least the requested size but is permitted to be larger. Different storage backends round differently: + +- Some backends snap to their allocation unit (e.g. 4 GiB, 1 GiB, 128 MiB) and return the rounded-up size on the PV. +- Some backends return exactly the requested bytes. + +For a VM running with `volumeMode: Block`, the QEMU process sees the **raw block device** — its byte count is the PV's actual size, not the PVC's request. When the source backend provisioned a PV slightly larger than requested (e.g. 16.00 TiB because its allocation unit is 1 GiB) and the target backend provisioned exactly the requested amount (15.999 TiB), the source disk address space extends past the end of the target device. QEMU's safety check in the `raw` block driver refuses to import data into a smaller device and aborts with the `offset + size` error. + +In filesystem-mode PVs this problem is hidden by the filesystem layer: the disk image file just ends where the data ends. Block mode has no such buffer. + +## Resolution + +Align the destination PV's **actual** byte count with the source's actual byte count, not just the PVC request. There are three options, in order of preference. + +1. **Request the source's actual size on the target PVC.** Read the source PV's byte capacity and set the target PVC request to at least that many bytes: + + ```bash + SRC_PV=$(kubectl get pvc -n -o jsonpath='{.spec.volumeName}') + kubectl get pv "$SRC_PV" \ + -o jsonpath='{.spec.capacity.storage}{"\n"}' + # -> "16297Gi" (example) + ``` + + Patch the destination PVC to request the same (or larger): + + ```bash + kubectl -n patch pvc --type=merge \ + -p '{"spec":{"resources":{"requests":{"storage":"17482664378368"}}}}' + ``` + + If the target StorageClass supports online expansion (most do), the provisioner will grow the PV and the live migration resumes on the next reconcile. + +2. **Pick a target StorageClass whose allocation unit matches the source.** If you are doing a bulk migration, this is more predictable than patching every PVC individually. Consult the StorageClass parameters or the driver docs to determine its rounding behaviour, then provision a target whose unit is ≥ the source's. + +3. **Convert to filesystem mode for the migration, then back.** Only viable for downtime-tolerant VMs; involves a cold migration through a filesystem-backed intermediate PVC. Use this only when neither of the above is possible. + +As a forward-looking prevention: when you know your storage plan in advance, always round the PVC request up to a multiple of the target backend's allocation unit. Asking for `17T` on a backend that allocates in 1 GiB blocks is fine; asking for `17482235510784` bytes on one backend and migrating to another that rounds differently is an accident waiting to happen. + +## Diagnostic Steps + +Identify the actual byte count on each side: + +```bash +SRC_PVC=; DST_PVC=; NS= +SRC_PV=$(kubectl -n $NS get pvc $SRC_PVC -o jsonpath='{.spec.volumeName}') +DST_PV=$(kubectl -n $NS get pvc $DST_PVC -o jsonpath='{.spec.volumeName}') + +kubectl get pv $SRC_PV -o jsonpath='{.spec.capacity.storage}{"\n"}' +kubectl get pv $DST_PV -o jsonpath='{.spec.capacity.storage}{"\n"}' +``` + +For block-mode PVs, confirm the raw device size visible to the virt-launcher pod: + +```bash +LAUNCHER=$(kubectl -n $NS get pod -l kubevirt.io=virt-launcher \ + -o jsonpath='{.items[0].metadata.name}') +kubectl -n $NS exec $LAUNCHER -- \ + sh -c 'for d in /dev/disk; do blockdev --getsize64 "$d" 2>/dev/null || true; done' +``` + +Inspect the QEMU error in context: + +```bash +kubectl -n $NS logs $LAUNCHER -c compute | grep -A3 'offset (0) and size (0)' +``` + +Confirm the target StorageClass permits expansion (some CSI drivers do not): + +```bash +kubectl get storageclass $(kubectl get pv $DST_PV -o jsonpath='{.spec.storageClassName}') \ + -o jsonpath='{.allowVolumeExpansion}{"\n"}' +``` + +If expansion is not allowed, you must delete and recreate the target PVC at the correct size; in-place expansion will not happen.