Virtualization event logs provide insight on the health and status of an infrastructure. These event logs are especially important for understanding and remedying virtualization failure.
For the everyday administrator, accessing VMware vCenter event logs is pretty straightforward. VCenter runs within Windows, and it's easy to configure basic event forwarding in the operating system. But organizations that must meet strict compliance regulations -- such as PCI DSS or HIPAA -- may require a centralized logging system, which collects data from ESX or ESXi hosts.
For IT administrators and security auditors, a centralized logging system provides a repository for sensitive and pertinent data. It also streamlines international business operations in cases where various countries' retention and privacy laws differ.
This tip explores different ways to centralize event logs in VMware infrastructures, including how to set up basic event-forwarding systems and advanced centralized-logging mechanisms.
Event forwarding in Windows
Event forwarding is easy with the Event Log Service in Windows Server 2008, Windows Server 2008 R2, Windows Vista and Windows 7. In the properties window of each Windows log type (system, application and security), you can set up log forwarding.
But this configuration does not pull critical vCenter event log data, because vCenter doesn't use the Windows Event Log Service for the bulk of its activities. VCenter and vSphere hosts keep application-specific log files in different locations.
Forwarding event logs through a Syslog server
For the ESXi side of a vCenter installation, a Syslog collection may be the best option for centralized logging. As a network service, the Syslog server provides a standards-based approach. Running over TCP port 514, Syslog can send information from hosts to a centralized logging server, such as a Syslog server.
Several Syslog server products available, and Solarwinds Software's Kiwi Syslog Server is a popular option. In most situations, Kiwi Syslog Server's basic offering suffices, and the paid version includes additional features, such as Windows event-log collection and database logging, which you can apply to vCenter Server for a full vSphere centralized-logging solution.
Setting up Kiwi Syslog Server
The Kiwi Syslog Server runs on Windows platforms and utilizes the .NET framework. It can operate as an application or as a service. For most situations, the best option is a service-based installation; the data collection continues once a system reboots, so it's easier to manage.
Once you have installed Kiwi Syslog Server, configure an ESXi server to pass Syslog data to a designated server. Kiwi Syslog Server can receive messages from ESX or ESXi, regardless of whether it's managed by vCenter.
In the vSphere host, the Syslog service is configured in the Advanced Settings (software).
This example has a TCP/IP address, but it can also be a fully qualified DNS name if the ESXi host can resolve the name. Next, the host immediately sends Syslog messages to the Syslog server. No service or system restart is required.
As soon as the action completes, the Kiwi Syslog Server receives messages from the ESXi host.
The ESXi Syslog option can be applied to a vSphere host profile. It's a good way to ensure that you employ a consistent host configuration with centralized logging and other features.
Once additional systems report Syslog data to the Kiwi Syslog Server, each reporting host flows into the scrolling log on the application console. The onscreen data becomes a lot to manage, and it's usually unrealistic to review every message. So automation is the next logical step to interpret the centralized results.
The free version of Kiwi Syslog Server has basic email functionality for the summary data, and it's configured in the Kiwi Syslog Server setup window.
The licensed version can execute actions based on certain criteria, such as a specific string of ESXi log text.. It can send email or pager alerts, run a program or script, log a message in a database or play a notification sound.
The free edition provides a daily summary of event logs, but the summary data shouldn't be a replacement for vCenter Alarms. Instead, be on the alert for items that aren't visible in vCenter Alarms or the vSphere Client.
Over time, you should be able to establish a normal behavior pattern for hosts. If there's an even distribution of event messages to the Syslog server, for example, you may have a problem if one host produces 90% of the Syslog Server messages. In this situation, there may be an issue with the system or how it communicates with the other components.
Ultimately, centralized logging is critical for determining what happened to a system. Depending on several factors, the default retention period for log events may be too short. Using Syslog, system events are retained for a longer period of time. In many situations, this is limited only by storage space.
About the expert
Rick Vanover (firstname.lastname@example.org) has the vExpert, VCP, MCITP, MCTS and MCSA certifications. He is an IT Infrastructure Manager for Alliance Data in Columbus, Ohio and is an IT veteran that specializes in virtualization, server hardware, operating system support and technology management. Follow Rick on Twitter @RickVanover.
This was first published in November 2010