Salvaging corrupt VMware guest OS data with VMDK snapshots

Reverting to a snapshot can fix a crashed guest OS, but without a backup copy of the guest OS, you can still lose important data. By moving VMDK snapshots to a new guest OS, however, you can save the data.

Jon Owings

Snapshots are a great tool for VMware backup tasks, but if you make a change to a guest OS and the change causes unwanted behavior, you need to delete or revert the snapshot and return the VM to its previous state. Failure to do so can cause a guest OS failure. If you don't have a recent backup copy on hand, you may lose weeks' worth of data.

In this tip, I explain how to get the original guest OS working, and how to recover the lost data by adding the VMware Virtual Machine Disk Format (VMDK) file snapshots to a new virtual disk and adding the disk and its snapshots as an additional VMDK for a secondary guest OS.

How snapshots can contribute to guest OS failures

Generally speaking, a best practice for snapshots is to create one, test a patch or install new software, then delete or revert the snapshot. But this may not always be the practice. Occasionally, I forgot about a snapshot, or someone else created one and did not tell me. Then there would be a major failure in the guest OS.

My favorite real-life example involves a Windows Small Business Server (SBS) 2003 that died on a reboot. The Active Directory (AD) database was corrupt and no one knew the restore mode password. We could revert the snapshot to a day we knew the AD worked properly, but in doing so we would lose weeks of Microsoft Exchange and SQL Server data. To further complicate the situation, we didn't have a decent backup copy of the server, as two different groups of people mistakenly thought that the other group was in charge of backing up the server.

Click to enlarge.

You may find yourself in this situation of needing data but also of needing to revert a virtualized guest OS to a restore point created weeks prior. But with a simple process using the vmkfstools from the VMware ESX Service Console, you can import up-to-date data to a new VMDK, enabling you to revert to and manipulate a snapshot without losing the newest data.

To get started, make sure the guest is powered off and log in to the service console. It helps to change the directory to the path of your guest OS and the VMDK. By default, snapshots are kept in this same directory. You will see delta VMDK for the snapshot, and a flat VMDK for the pre-snapshot data. If you look at the files present in the directory, you will see two VMDK files for every virtual disk. The smaller file is the pointer VMDK. It contains settings that correspond to the disk.

One of these settings refers to the actual data file for the VMDK. It's best not to mess with the file unless you get guidance from VMware support. When you work with a snapshot version of a VM, all disk changes are written to this delta VMDK. In the screenshot below, the ubuntu-000001.vmdk file refers to the ubuntu-000001-delta.vmdk, and the ubuntu.vmdk refers to the ubuntu-flat.vmdk. When you have more than one snapshot, the numbers in the file names increase. For simplicity, let's suppose we have only one snapshot. (If you have several, you can import them to different files and track down when your problem occurred.)

Click to enlarge.

Importing the snapshot to a new VMDK

Now you can import the snapshot to a new VMDK. If you have more than one disk, you have to perform this step for each VMDK. Use the name of the pointer file in the syntax of the command. Here is how to import a snapshot create a new folder for your backup VMDK:


#mkdir /vmfs/volumes/DS-LUN1/ubuntubackup
vmkfstools –i [source file] [destination file]


#vmkfstools –i /vmfs/volumes/DS-LUN1/ubuntu/ubuntu-000001.vmdk /vmfs/volumes/DS-LUN1/ubuntubackup/ubuntubackup.vmdk

The file will begin cloning. Depending on the size of the original and the amount of data in the snapshot, itcould take a while for the cloning to complete, but at least you have a nice progress indicator to watch in the meantime.

Click to enlarge.

Now you are able to revert your snapshot from vCenter Server without losing the changes. In the vCenter client, go to the guest OS, right-click and choose snapshot, revert to snapshot, and click yes.

Click to enlarge.

You might wonder whether all changes are now being written to the original VMDK because you have reverted the snapshot. No. You have actually deleted all the changes in the original snapshot, but new changes go yet another delta file, 000002 in my case.

Click to enlarge.

Now you can either delete the snapshot from vCenter to start writing to the original disk or let the changes that occur during your troubleshooting go to the delta file. I prefer the latter, as you still have an potential restore point after everything is fixed.

Adding the VMDK as an additional disk to a guest OS

So what do we do with the VMDK we just created? Since it may not be bootable, I always add it as an additional disk to a guest VM and copy the needed data to the stable and working guest. To do so, edit your guest's settings and add a hard disk. Select the Use an Existing Disk option and browse for your backup VMDK. In my example, that would be ubuntubackup.vmdk. Now perform the task in your guest operating system necessary to find the new hard drive. In Windows, you would right-click and select rescan disks under the disk management snap-in, commonly found in the Computer Management MMC console.

Click to enlarge.

Following this process got me out of the potential disaster described previously. I was able to save current SQL and Exchange data while reverting the OS into a state that was once again usable. Having a snapshot available, however, does not guarantee a consistent database. I can easily recover the Exchange data, but a database administrator had to clean up the SQL database. The data was present, but there was more work to be done. A proper backup solution is the preferred alternative.

Jon Owings (VCP) has 12 years of experience as an IT professional. He has worked as a senior technical engineer for an IT consulting company, a network administrator and a Citrix administrator. He specializes in design, implementation and management of VMware, Cisco and SAN products. Owings also manages his own blog, 2 VCPs and a truck.

Dig Deeper on Backing up VMware host servers and guest OSes