At some point, every administrator needs to determine why his VMware Infrastructure 3 (VI3) environment is functioning...
a certain way or why a specific event occurred. When the standard interface tools do not provide the requisite information, it's time to go to the next level. The good news is that there are several methods available to gather more information about an issue. The bad news is that this information can reside in many places. In this tip, we'll discuss how the diagnostic data feature of VI3 can locate aiding information and how it can help with VI3 implementation administration.
Exporting diagnostic data from the VMware Infrastructure Client
The export diagnostic data task in the VMware Infrastructure Client (VI Client) is an incredibly thorough collection of information related to VirtualCenter, the VI Client and ESX hosts of the environment. If you've interacted with VMware's Support Services, you know that one of the first steps they ask you to take is to export diagnostic data. This task is shown in the scrolling log of the VI Client as Generate Diagnostic Bundles. This multidimensional bundle offers a great starting point for diagnosing issues for a selected group of resources. We'll explain the host portion of the diagnostic process later with the vm-support command, but the VirtualCenter and VI Client components of the diagnostic data bundle are relevant to the Administration task within the VI Client.
It is simple to generate the diagnostic data from the Administration menu. Select Export Diagnostic Data, and you will be prompted with a screen similar to the one shown below:
This task generally takes time to complete on a single host. If multiple hosts are selected, it could take a considerable amount of time to copy the files, as the exports can be upward of 15 MB per host. VirtualCenter exports can be much larger. The VI Client export is generally smaller than that of a host export from a file size perspective. When the process is under way, the following scrolling log entry will appear within the VI Client:
VirtualCenter data export
The VirtualCenter export is contained in a vcSupport compressed file as part of the process to export diagnostic data. This collection of data contains the Windows Event Logs, VirtualCenter host configuration data, vpxd-x.log log files, information on the critical VMware services and other configuration elements from the VirtualCenter server.
One of the critical areas of the export diagnostic data function is the category related to the VI Client. This is an important factor because as the VI plug-in architecture matures, having various installed plug-ins may interfere with the correct operation of the VI Client's calls to VirtualCenter. From an official perspective, plug-ins not published by VMware are not supported, but they are used frequently and many are quite handy for day-to-day administration.
In exporting the diagnostic data for a test system, the viclient-9.log file has an entry showing a plug-in that I am using on one installation shown below:
[viclient:QuickInf] 2008-10-29 19:21:59.808 \ Load plugin: net.sf.vip.svmotion.SVMotionPlugin
Beyond the plug-ins, the VI Client export provides a full compliment of the operating environment detailing the interaction with VirtualCenter.
Running vm-support directly on a host
The VI Client diagnostic data task includes host-based data. This can be run explicitly on a host with the vm-support command. The main folders of the export retain the same file and folder footprint when performed in the VI Client or directly on the ESX host with vm-support.
The vm-support command creates a compressed file with host-specific logs that VMware's support services can use to diagnose issues. The figure below shows the vm-support task running on an ESX server:
In regard to the ESX host, the most substantive content of vm-support and the VI Client initiated export diagnostic data task is contained in the /var/log/vmware path of the compressed file. This mimics the file system on the ESX server, so if you have navigated for files on the console or an SSH session, this will be familiar ground.
The /var/log/vmware hive of the compressed folder contains the core log files of the ESX environment. The active log files of ESX should never be touched directly, and the compressed folder makes the logging information available outside of the running ESX host system. In addition, the compressed file is transportable to VMware support if necessary. The hostd-x.log series of files contain an incredible amount of information, and going through these files can be cumbersome. It is ideal to search for a specific issue within the folders. Most objects in ESX or VirtualCenter that are host-specific are searchable. For example, here are a few entries (there are many more for this particular task) related to the Client-5 guest VM being cloned and initiated:
[2008-10-21 08:23:50.139 'vm:/vmfs/volumes/48e2ba85-22a32fd3-8f7d-00188b374d8b/ Client-5/Client-5.vmx' 3076449184 info] State Transition (VM_STATE_INITIALIZING -> VM_STATE_ON) [2008-10-21 08:23:50.161 'vm:/vmfs/volumes/48e2ba85-22a32fd3-8f7d-00188b374d8b/ Client-5/Client-5.vmx' 3076449184 info] Initialized virtual machine. [2008-10-21 08:23:50.173 'Vmsvc' 3076449184 info] Loaded virtual machine: /vmfs/volumes/48e2ba85-22a32fd3-8f7d-00188b374d8b/ Client-5/Client-5.vmx
Some of the searchable elements include the VM's name, the full datastore path, and a timeframe.
Establish a basis for comparison
Having access to the logging is great but, as illustrated here, the log files can be quite cumbersome. To add to the complexity, I have not even covered the tip of the iceberg of how much detail is contained in the diagnostic data exports. When support matters or some other diagnostic effort is engaged, having a known healthy export of a host's behavior from the vm-support or VI Client task will be beneficial. This can aid any level of support or system research by establishing a baseline or normal configuration. This additional information can potentially help distinguish between what is a benign error or identify a new issue that needs attention.
One way to provide an excellent baseline is to perform a monthly export for each host. This way, the vm-support or VI Client export is fully representative of the host configuration from storage, network drivers and the live workload. By having it performed regularly, if support is ever engaged, a history of error frequency can be determined to help diagnose an issue.
What other tools are available?
Outside of the base tools of VI3, the natural place to look is the VirtualCenter database. The VPX_EVENT table contains events that were initiated through VirtualCenter across all objects in VI3. This would include logon attempts, VMware Distributed Resource Scheduler (DRS) events, VMware High Availability (HA) events, host configuration activities, resource pool management and VM configuration. The VPX_EVENT table should not be interacted with while it is running. Instead, a good way to look at the events available in the table is to restore the VPX_EVENT table to a different database and do any queries on the separate database.
The log files can also be accessed directly on the ESX hosts, VirtualCenter server and the VI Client. For the files on the hosts and VirtualCenter server, exercise great caution with the live log file and always copy it off to a path not used for the ongoing operation of the server or application to avoid contention. On current versions, most ESX log files are kept in /var/log/vmware. The VirtualCenter log files are kept in C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Logs for default installations. The VI Client logs are kept in the C:\Documents and Settings\username\Local Settings\Application Data\VMware\vpx path for default installations.
The diagnostic data for VirtualCenter, ESX and the VI Client is quite information-heavy. The diagnostic data export is a useful support tool for VMware, but having a good understanding of the contents as well as a baseline of normal operation can help provide a good baseline of the VI3 environment.
ABOUT THE AUTHOR: Rick Vanover is a MCSA-certified system administrator for Belron p in Columbus, Ohio. Previous roles included software engineering roles with Siemens and Dematic in Grand Rapids, Michigan, working with complex supply chain execution systems. Rick has been working information technology for over 10 years and virtualization technologies for over seven years. He has been publishing articles online for over six years.