Recently I had a bunch of virtual machines that started generating this error during Veeam backups. I hadn’t bothered to really be checking my snaps because my daily job was supposed to be taking care of that for me. Unfortunately, this bit me and Here we are
As many of you have probably experienced, Veeam doesn’t always clean up after itself when it is finished backing up a VM. Sometimes a file lock or other operation prevents the snap cleanup, and you end up with a huge chain of snapshots.
Normally you would just be able to right click the VM, select Snapshots, then “delete all” and be done, however I was getting.
A general system error occurred: vim.fault.GenericVmConfigFault
So how do we fix this? Well it turns out there are two ways.
This resolution requires downtime, however is significantly faster. It does however have some caveats I will get to later.
- Shut down the Virtual Machine.
- SSH to the host where the VM was running.
- Change Directory to the volume where the guests disks are stored.
- Find all delta.vmdk for the VM
ls -ltrh | grep delta.vmdk
- If it looks something like this you’re good to go. Notice how of the 5 snapshot deltas, only 0x5.delta.vmdk actually has data, the others are empty?
- If it looks like this, you’re going to have to use Resolution 2. Notice how multiple of the snapshot deltas hold data?
- Create a new folder in the VMs folder.
- Move all of the EMPTY snapshot data to the /tmp folder. DO NOT DELETE IT. If the remaining steps do not work we have to restore these files.
mv VM_1-000001* ./tmp
- Once all Snapshots are moved, and you are left with your base disk and your last snapshot. Open VMware vSphere Client, and create a new VM. Make it identical to the old VM, however DO NOT add a disk. Remove the disk VMware wants to create, and select “add existing disk” Select the snapshot disk, NOT the base disk, and attach it to the VM. Do not select “Power On VM after Creation.”
- Once creation is complete you will notice that VMware will show “Virtual machine disks need consolidation.” Right click the VM and chose to consolidate the disk. Once consolidation is complete, boot the VM and verify functionality. You should be all set!
If your VM is not recoverable with Resolution 1, your alternate resolution is to use the VMware Standalone converter to convert a “Powered On Windows / Linux Machine.” So yes you are doing a P2V migration of a Virtual machine to a new virtual machine.
This isn’t the best solution in the world and its essentially the same as taking / restoring an OS based backup instead of a VM based one, however it allows your VM to be online during the entire operation which could be a consideration for those who have very small maintenance windows and may not be able to do a full consolidation in that window.