VMware ESX Server and vCenter Server automatically maintain more than 12 different log files and update them on
a daily or hourly basis. Fresh log files are created according to log rotation, which involves "dumping" existing log data into a different file and starting a new log after a designated period of time. The log rotation also defines how many logs files of a particular kind to keep before older ones are deleted.
Because log files are crucial for troubleshooting, security, and other administrative purposes, you need to have them available when the time comes. However, log files grow with your environment and you don't want logs taking up too much space. Additionally, some VMware log files, specifically hostd, vpxa and vpxd, may prove more valuable than the others, so you may want to designate more log space to these files and less to others. In this tip, I'll first explain what kind of information you can find within each log file and where you can find it, and then I'll go over how to tweak log rotation settings.
What VMware ESX log files are available and what do they do?
Below, I'll list the VMware ESX log files that you should most be familiar with. Eric Siebert lists out each of the log files, where they are and what they do in another SearchVMware.com article Troubleshooting common VMware ESX host server problems. I've made a few modifications in what follows.
VMkernel - /var/log/vmkernel
Records activities related to the virtual machines and ESX server. Rotated with a numeric extension; current log has no extension; most recent has a ".1" extension.
VMkernel warnings - /var/log/vmkwarning
Records virtual machine activities. It's a subset of the VMkernel log and uses the same rotation scheme.
VMkernel summary - /var/log/vmksummary
Used to determine uptime and availability statistics for ESX Server; readable summary found in /var/log/vmksummary.txt.
ESX Server host agent log - /var/log/vmware/hostd.log
Contains information on the agent that manages and configures the ESX Server host and its virtual machines. (Search the file date/time stamps to find the current log file or open hostd.log, which is linked to the current log file.)
ESX firewall log - /var/log/vmware/esxcfg-firewall.log
Logs all firewall rule events.
ESX update log - /var/log/vmware/esxupdate.log
Logs all updates executed by the esxupdate tool.
Service console - /var/log/messages
Contains all general log messages used to troubleshoot virtual machines or ESX server.
Web access - /var/log/vmware/webAccess/*.log
Logs inside this directory record information on web-based ESX server access.
Authentication log - /var/log/secure
Contains records of connections that require authentication, such as VMware daemons and actions initiated by the xinetd daemon.
Vpxa log - /var/log/vmware/vpx/*.log
The vpx directory contains logs with information on the agent that communicates with vCenter. Search the file date/time stamps to find the log file it is currently outputting to, or open vpxa.log which links to the current log file.
In my experience, the most important ESX server log files are hostd.log and vpxa.log. Of course, the more important log file varies depending on the technical issue you're trying to research, i.e., if you are having trouble with the ESX firewall, the ESX firewall log would be the most important file.
What VMware vCenter log files are available and what do they do?
The most important vCenter log files are the vpxd log files. Here is where they are located and what they do. (Note: I will am omitting typical Windows Server log files that you likely already know such as the system, security, and application logs inside the event viewer.)
Vpxd log - C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Logs\vpxd-X.log, where the X is a number 0 to 9.
The vpxd log files are the rotating logs of the vpx daemon (vpxd.exe) which is what the vpx agents on each ESX host are communicating with. You can find the most current vpxd log file by opening the vpxd-index file, getting the number from the file, and substituting the X in the vpd-X.log filename with that number.
How do you control the rotation and size of ESX server and vCenter log files?
Now that we know which log files are important, where they're located and what they do, how do we control log files rotation to ensure that less important logs don't take up too much space? Let's find out.
Several ESX host log files are controlled by the Linux logrotate program. With ESX server, logrotate is executed hourly using the /etc/cron.daily/logrotate command. From there, logrotate uses its configuration file /etc/logrotate.conf. By default, that file references the directory /etc/logrotate.d to find out which log files it needs to rotate.
In VMware ESX 3.5, the logrotate programs will rotate the following logs:
Here is the vmkernel logrotate configuration:
As you can see, the vmkernel log file will be rotated monthly, over 36 months (three years) with a maximum disk usage of 200 K per log. That's good. Now, what about the other two most important ESX Server log files, hostd and vpxa?
The log rotation and growth configuration for the ESX server Hostd.log file is found at /etc/vmware/hostd/config.xml. This is a large text file that you will have to open and find the section that starts with <log>, like this:
This section is small by default, but according to VMware KB Article 1004795, Changing the size and number of logs for hostd, vpxa and vpxd, you could use entries in this section to control the log growth. These lines would look like this:
<config> : <log> <directory>/var/log/vmware/</directory>
--- Logging path <level>verbose</level>
--- Log Level <name>hostd</name>
--- Prefix <outputToConsole>false</outputToConsole>
--- Console out <maxFileSize>5242880</maxFileSize>
--- Maximum size of each log file (in bytes) <maxFileNum>50</maxFileNum>
--- Maximum number of log files </log> : </config>
As for the vpx agent (or vpxa.log files), the configuration to control rotation and log growth is found at /etc/vmware/vpxa.cfg with ESX 3.0, and in /etc/opt/vmware/vpxa/vpxa.cfg for ESX server 3.5.
Here is what the log section of my ESX 3.5 vpxa.cfg file looked like:
According to the same VMware KB Article mentioned earlier, you could use entries in this section to control the log growth like this:
--- Maximum number of log files <maxFileSize>5242880</maxFileSize>
--- Maximum size of each log file (in bytes)
Finally, in order to control the log file growth of vpxd files on your vCenter server, open the configuration file at C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\vpxd.cfg.
Here is what it looks like:
The <vpxd> section of the log is where you could have log growth entries that look like this (according to the VMware knowledgebase article):
--- Log Level <maxFileNum>50</maxFileNum>
--- Number of log files (Approx. 5MB each) <maxFileSize>5242880</maxFileSize>
--- Maximum size of each log file (in bytes) <directory>C:\vpxlog\</directory>
--- Logging path <name>vpxd</name>
--- Prefix </log>
ABOUT THE AUTHOR: David Davis (CCIE #9369, VCP, CWNA, MCSE, CISSP, Linux+, CEH) is the Director of Infrastructure at Train Signal, Inc. He has written hundreds of articles and six video training courses. His websites are Happy Router.com and VMwareVideos.com.