VCB is VMware's attempt to provide an enterprise level backup solution for ESX. It is a very good effort, although it lacks in some fairly major areas. Some may compare it to the popular esxRanger software, but it may not be fair to make comparisons between a mature product and one that is still relatively new. Only time will tell what is made of VCB. For now, here are some tips that can help unravel some of this backup solution's Gordian knots."
So you've got VCB talking to your VirtualCenter Management Server, but you just cannot seem to get it back up your virtual machines (VM). The first step to figuring out what is going wrong is to increase the amount of debugging information that gets outputted to the VCB log files. Use notepad to open the following file, "C:\Program Files\VMware\VMware Consolidated Backup Framework\generic\glue.js". Search for the string "function PrepareForBackup". This will zoom to the function responsible for building the command that mounts the VMs for backup. Near the beginning of the function there is a line that reads:
cmd = StdCommandTemplate(vmMounterCmd);
cmd += " -L 6 ";
Save the file and close notepad. Subsequent VCB executions will now generate verbose logs located in the temporary directory of the user who is running the backup process. Since most backup processes run as SYSTEM (unless one is executed manually outside the scope of the backup software by an administrator) the log files will be located in %WINDIR%\Temp. The log files adhere to the following naming convention:
pre_ESX-(File|FullVM)MDDYYYYHHmmSS.log and post_ESX-(File|FullVM)MDDYYYYHHmmSS.log
The interesting information is going to be in the pre logs. Once the logging has been turned up, go ahead and attempt a backup of the VMs. After the backup job fails, it is time to examine the log file. Use the following guide as a way of determining where the problem is and how to fix it.
Error: Snapshot creation failed: This operation is not supported.
The VM in question appears to not have had its virtual hardware upgraded from an earlier version of ESX. Even the most vigilant ESX administrator can fall prey to this problem because although all VMs were upgraded when ESX was upgraded to 3.x, the template were not. If templates created in an earlier version of ESX are not upgraded as well, then VMs stamped out of those templates will still be using an previous virtual hardware version.
Power off the VM, right-click on it, and select "Upgrade Hardware."
Does not report an IP address.
The VM's guest operating system (OS) in question either does not have the VMware Tools installed, the VMware Tools are not running, or they are corrupted.
Install VMware Tools inside the VM's guest OS, ensure that they are running, or possibly reinstall them.
Could not resolve IP address.
VCB requires that all VMs have resolvable IP addresses. If the VM's name does not resolve to a valid IP address then VCB will fail to back it up.
Ensure that the VM is resolvable via DNS. If for some reason you cannot create a DNS entry for this VM then edit the Windows hosts file at %WINDIR%\system32\drivers\etc\hosts on the VCB backup proxy and create a static entry for this VM.
Failed to communicate with the remote host, either due to network errors or because the host is not responding.
This message can be indicative of a misconfigured or corrupted VMware Tools installation. This message can also occur when there is a problem reading the VM's existing snapshots.
Reinstall VMware Tools. If that does not fix the problem then it may be necessary to remove the VM's snapshots one at a time until the backup operation succeeds.
Error: Failed to open the disk: One LUN could not be opened.
This message means that the LUN IDs of the ESX LUNs as they are presented to the VCB backup proxy are not in the same order as they are on the ESX servers. VCB requires that the VCB backup proxy and the ESX servers each see the same LUNs in the same order. Adapter and Target IDs do not matter, only the LUN ID.
Remove all of the ESX LUNs from the VCB backup proxy and re-add them in the same order that they were added to the ESX servers.
Error: Could not find virtual machine specified by ipaddr
This message means that the a successful vcbMounter command has never been executed against the VM in question. The first time vcbMounter is executed successfully against a VM it will cache the VM's IP address. If this IP address is not in the cache or if it cannot be obtained then the backup operation will fail. This typically occurs when VCB is trying to back up a VM that is powered off and has no cached IP address, or VCB is attempting to backup a VM that does not have VMware Tools installed.
It may be necessary to power on the VM and execute the vcbMounter command so that the VM's IP address will be cached as well as making sure that the VM has the VMware Tools package installed.
Error: Backup snapshot already exists.
This is indicative of a previous backup attempt by VCB that did not clean up its snapshot after the backup operation.
Manually remove the snapshot "_VCB-BACKUP_" from the VM.
I hope this article has helped its readers figure out some of the harder-to-solve issues that can crop of with VCB. I will be writing an upcoming series for SearchServerVirtualization.com called "Getting Started with VMware Consolidated Backup (VCB)," where I detail how to start using VCB from start to finish. Stay tuned!
About the author: Andrew Kutz is deeply embedded in the dark, dangerous world of
virtualization. Andrew is an avid fan of .NET, Open Source, Terminal
Services, coding, and comics. He is a Microsoft Certified Solutions
Developer (MCSD), a SANS/GIAC Certified Windows Security Administrator
(GCWN), and a VMware Certified Professional (VCP) in VI3. Andrew also
recently submitted a security practical entitled "Sudo for Windows
(sudowin)" to the SANS organization where it was accepted, thereby
granting Andrew SANS GOLD status. Andrew graduated from the University
of Texas at Austin with a BA in Ancient History and Classical
Civilization and currently lives in Austin, TX with his wife Mandy and
their two puppies, Lucy and CJ.
This was first published in February 2007