Sending VirtualCenter logs to Local7

Learn how to deploy software that will monitor VirtualCenter and log all VirtualCenter events and tasks to the Windows event log and a Linux Syslog Server with this tip.

There are seven aspects to monitoring VMware Infrastructure 3 (VI3). Monitoring:

  1. ESX server hardware
  2. ESX server software Virtual machines (VMs)
  3. Guest operating systems (OSes)
  4. VirtualCenter software
  5. VirtualCenter operating system
  6. VirtualCenter hardware (possibly a VM as well)

With the exception of the VirtualCenter software, every other item on the above list can be monitored using existing monitoring software such as the Windows Event Log, a Linux Syslog Server, or a vendor-provided hardware monitoring solution. Wouldn't it be nice if there was some way for VirtualCenter to send its logs to a Windows Event Log or a Linux Syslog Server? Of course it would be, and that is exactly what I am going to show you how to do using the VI3 SDK.

VirtualCenter monitoring processes

If you watch the VirtualCenter log at the bottom of any virtual infrastructure (VI) client you will see two types of items scroll by: tasks and events. Tasks are operations performed by VirtualCenter or ESX that can be initiated by an alarm, a schedule, a user, or the system. For example, if a user were to rename a virtual machine (VM), the Renamed_VM task would be initiated. An event, on the other hand, is a record of an operation that was performed. For example, if the distributed resource scheduler (DRS) was to initiate a VMotion operation, a task would be created for this operation, and in addition an event would be logged that a VMotion operation was initiated. When the operation completes, successfully or not, another event is recorded signaling that a VMotion operation was completed.

So what is the difference between tasks and events? A task is used to monitor an operation, whereas events are created during an operation's lifetime to describe the status of the operation. For example, to cancel a VMotion operation a user could right-click on the running task and select "cancel." This action would then generate an event describing the cancellation. It is okay if the reasoning behind separating tasks and events into two separate entities is still a bit of a puzzle. All will become clear when we take a look at how these two items are leveraged using the SDK.

Monitoring tasks and events

Hear Andrew Kutz speak at one of our virtualization seminars

Now that we know about tasks and events, how do we monitor them? If you've read my previous VI3 SDK articles, you know all about the data object ServiceContent. This data object contains two properties that are managed object references (morefs) which prove useful in this situation: taskManager and eventManager. These morefs provide references to the server's TaskManager and EventManager managed objects (respectively). Both of these objects contain properties that we can monitor to listen or changes. The TaskManager object has a property called recentTask. This property is a reference to an array of morefs for Task objects. Each time a new task is initiated on the server the recentTask property of the server's TaskManager object will be updated with a reference to its Task object. The EventManager object has a property called latestEvent which points to an Event data object.

I should point out that monitoring events is far easier than monitoring tasks, primarily because there are not as many moref lookups involved. However, if you should ever want your monitoring program to take an action on a current operation, such as canceling it, then it is necessary to monitor tasks as well.

To recap, we now know that we can monitor the server's TaskManager->recentTask and EventManager.latestEvent properties in order to determine if any new tasks or events have been initiated / recorded. However, it is not very efficient to constantly poll these properties for updates. What we need is method of waiting for the SDK to notify us of updates, instead of the other way around. Luckily VMware provides a method of doing just that. The WaitForUpdates method will block the current thread until there are new updates to be retrieved.

Once there are new updates we just have to retrieve them using the normal methods and then send them to a syslog server and/or write them to a Windows Event Log. The following code excerpts are from an application I wrote called vmmon. Vmmon is a .NET 2.0 command-line application written in C# that monitors a VC or ESX server for new tasks and events. Output is directed to stdout and optionally to a Windows Event Log called VirtualCenter and/or a syslog server. A couple of quick notes about vmmon:

  • Instead of linking to the VI3 SDK library I included the WSDL-generated source file in the project directly in order to avoid needing any other file than the vmmon binary to execute vmmon.

  • The project was created with Visual Studio .NET 2008 (9). You may not be able to open it with older versions of Visual Studio, but you can always create a new C# command line application and just copy the files in.

You can read more about vmmon, as well as download it. Existing readers of my VI3 SDK series will recognize a lot of the code snippets, but may also be a bit thrown off. I have reorganized some of my standard command-line application code because vmmon is multi-threaded and required me to move some function level variables to module-level fields to provide global, synchronized, access to them for multiple threads.

Because I will not be using this article to describe the code line-by-line, I will take a brief moment to outline the program structure. The program starts like any C# command-line-application, by entering "static void Main( string args[] )". As with my previous SDK command-line applications the next action is to parse the command-line arguments. Note that the variable "cmd_line_opts" has been moved to a module-level field called "m_cmd_line_opts" (Hint: All of my module-level fields begin with the "m_" prefix). It at this point where vmmon's structure diverges from my previous programs.

I use a thread from the applications thread pool and queue the method "StayLoggedIn" without passing any state. The StayLoggedIn method is necessary in order to retain a constant (not persistent) connection to the VI3 web service. Because the SDK is exposed via a Web service connections time out get interrupted, or severed for other reasons. This results in an annoying error, especially if someone were to turn my program into a Windows service. The solution is to trap any timeout exceptions inside the thread that is waiting for updates and then notify the thread for "StayLoggedIn" to log the monitor back in. The method "StayLoggedIn" continually loops, only logging into the web service if the module-level field "m_logged_in" is set to false.

Once the monitor has successfully logged in a new thread is queued for the method "WaitForUpdates". The "WaitForUpdates" method is what readers should pay attention to in this program. In teaches developers a few VI3 SDK advanced topics:

  • How to create and destroy PropertyFilters
  • How to check the server for updated Tasks and Events
  • How to check for only updated values

The first bit of code shows what properties we need to initially look at:

// Build a property specification object to get the 
// description property of the service content's TaskManager. 
// The TaskManager's description property is a pointer to a 
// data object that let's us look up verbose descriptions of tasks.
PropertySpec pspec_tskmgr = new PropertySpec();
pspec_tskmgr.all = false;
pspec_tskmgr.type = "TaskManager";
pspec_tskmgr.pathSet = new string[] { "description" };

ObjectSpec ospec_tskmgr = new ObjectSpec();
ospec_tskmgr.obj = m_vim_svc_content.taskManager;

PropertyFilterSpec pfspec = new PropertyFilterSpec();
pfspec.objectSet = new ObjectSpec[] { ospec_tskmgr };
pfspec.propSet = new PropertySpec[] { pspec_tskmgr };

PropertyFilterSpec[] pfspec_arr = new PropertyFilterSpec[] { pfspec };

// Get a reference to the TaskDescription object.
ObjectContent[] oc_coll = m_vim_svc.RetrieveProperties
     ( m_vim_svc_content.propertyCollector, pfspec_arr );
TaskDescription td = ( TaskDescription )oc_coll[ 0 ].propSet[ 0 ].val;

As you can tell from the code above, before we begin monitoring for updates we first need to grab a reference to the TaskManager's description property. It allows us to look up verbose descriptions of tasks using the task's "descriptionId" property. Next we create a property filter that will be used to listen for updates to tasks and events:

// Change the TaskManager property we are looking at to 'recentTask'.
pspec_tskmgr.pathSet[ 0 ] = "recentTask";

// Create a prop spec used to look at the EventManager's latestEvent 
// property.
PropertySpec pspec_evtmgr = new PropertySpec();
pspec_evtmgr.all = false;
pspec_evtmgr.type = "EventManager";
pspec_evtmgr.pathSet = new string[] { "latestEvent" };

ObjectSpec ospec_evtmgr = new ObjectSpec();
ospec_evtmgr.obj = m_vim_svc_content.eventManager;

The code above changes the TaskManager property we were looking at from "description" to "recentTask" and then creates a property specification object to look at the "latestEvent" property of the EventManager. The next bit of code looks different in the program's source file because here I have removed a switch statement that determines what type of property filter to create. We are going to assume that you want to look at both new tasks and events for the purposes of this article:

pfspec.objectSet = new ObjectSpec[] 
     { ospec_evtmgr, ospec_tskmgr };
pfspec.propSet = new PropertySpec[] 
    { pspec_evtmgr, pspec_tskmgr };

// Create a property specification filter using the property filter 
// specification we just created. These are global to the
// vim_svc session. When monitoring updates using
// the CheckForUpdates or WaitForUpdates method, a union 
// of all property specification filters is taken in order to figure
// out what properties to monitor.
ManagedObjectReference moref_pf = m_vim_svc.CreateFilter
     ( m_vim_svc_content.propertyCollector, pfspec, true );

As you can see above, we created a property filter specification that included looking at properties from both the TaskManager and EventManager. The next bit 'o code creates a property filter. The inline comments describe what happens pretty thoroughly. A global property filter (scoped to the current session) is created such that any method that checks for updates is checking on properties that are being filtered on by any property filter that has been created during said session. Thus, as of this moment if the "CheckForUpdates" or "WaitForUpdates" methods of the SDK are invoked, they will be listening for updates for the TaskManager->recentTask and EventManager.latestEvent properties.

*Editor's Note: The code in this section is available for viewing at the author's Web site.

Hopefully the provided in-line comments are enough for you to digest what went on. In case they are not, here are some tips. When we finally get an update it comes in the form of an UpdateSet object. Luckily we do not have to look up the updated property values: the UpdateSet object includes a property called "objectSet" that includes all the updated values. There are three possible updates occurring: an update on a recentTask, an event, and a taskInfo object. The first two updates are obvious, we created filters to monitor those updates, but what is going on with the taskInfo object?

Well, when we notice that a new task has appeared in the recentTask array we check to see if its progress property is set to 0 (0-100 are possible values). If the value of the progress property is set to 0 then we know that this task was just initiated. We want to know when the task completes so we can create a new property filter on the Task's "info" property and store a reference to that property filter in a module-level dictionary list (access to this list is synchronized with the lock statement to prevent reading and writing to the list at the same time). That way when there is an update to the Task we will get notified. Once the task is completed (we check this by seeing if its "completeTime" property has been set) we destroy that particular tasks property filter so we no longer get updates on it. At the very end of the loop we update the variable "update_version" with the version of this update set. That way we will not get the same updates twice.

Now, if you looked at the source files already you will notice that the entire "WaitForUpdates" method is encased in a try...catch block. This is to catch any exceptions that occur when waiting for updates. The following code is what is in the catch block:

catch
{
 if ( m_water_marker >= 3 && 
     ( DateTime.Now - m_start_of_flood < TimeSpan.FromMinutes( 1 ) ) )
 {
  Console.WriteLine( "Flood detected..." );
  throw;
 }
 
 if ( m_water_marker < 3 && 
     ( DateTime.Now - m_start_of_flood > TimeSpan.FromMinutes( 1 ) ) )
 {
  m_water_marker = 0;
  m_start_of_flood = DateTime.Now;
 }

 // Signal to the StayLoggedIn thread that we have been logged out.
 Console.WriteLine( "Connection timed out..." );
 m_logged_in = false;
 ++m_water_marker;
}
>

Okay, so here is what is going on. Because we cannot possibly handle all the exceptions that are occurring, we just assume that if an exception does occur that it is because the program was disconnected from the web service. Thus, the program should try to reconnect. However, we cannot make this assumption over and over again or we would be put into an endless loop. Therefore I have implemented flood detection. Here are the if-then rules for flood detection:

If there are more than three water markers (exceptions) in less than one minute since the last flood check, then throw an exception and quit

If there are less than three water markers in more than one minute since the last flood check, then reset the water marker count to 0 and reset the time of the flood check to DateTime.Now

Finally: alert all threads that the connection has been lost and increment the water marker count.

This bit of code allows four floods in one minute, after which the entire dam breaks and the water destroys the town. That is what is called bad. Implement your own error checking here. I recommend it!

And there you have it. That is how to monitor VI3 for tasks and events. Pretty slick. But how do you send them to the Windows Event Log or a Syslog Server? Well, did you notice the LogMessage invocations in the code way back there? That method is what does the trick.

Sending logs to the Windows Event Log and a Syslog Server

First up, the Windows Event Log:

if ( Regex.IsMatch( m_cmd_line_opts.WindowsEventLogServer,
 string.Format( "(localhost)|(127.0.0.1)|{0}", 
     Dns.GetHostName() ), RegexOptions.IgnoreCase ) )
{
 m_cmd_line_opts.WindowsEventLogServer = ".";
}

if ( !EventLog.SourceExists( "vmmon", 
     m_cmd_line_opts.WindowsEventLogServer ) )
{
 EventLog.CreateEventSource( "vmmon", "VirtualCenter", 
     m_cmd_line_opts.WindowsEventLogServer );
}
EventLog.WriteEntry( "vmmon", msg, EventLogEntryType.Information );
You will notice that I am replacing any instance of a reference for the localhost with a ".". This is because the .NET method that creates and writes to the event log only understands a "." when referencing the localhost. Anyone know why? The answer rhymes with "why" in fact: WMI. A "." is a WMI convention for localhost. Anyway, this bit 'o code creates an event log called "VirtualCenter" on the specified host if it does not exist and registers an event source called "vmmon". Once created (or not), the log message gets written to the "vmmon" (VirtualCenter) event source. That's it for the Windows event log. Syslogging is a little trickier, but not hard by stretch of the imagination. The code for syslogging is in a separate file called, not surprisingly, Syslog.cs. I am not going to discuss all of that code in this article as it is slightly out of scope, but I will tell you the core of what you need to know:

1) A syslog server listens on the UDP port 514 by default.

2) A syslog message has two properties: a type and facility.

3) The 8 types are:
Emergency = 0,
Alert = 1,
Critical = 2,
Error = 3,
Warning = 4,
Notice = 5,
Information = 6,
Debug = 7,

4) Facilities are system defined, but the usual suspects include "Auth", "System", "User", "Kernel", etc...

Once you know those things it is not hard to get here:

if (!SyslogHelper.IsActive)
{
 SyslogHelper.Connect(m_host_ip, m_port);
}
if (SyslogHelper.IsActive)
{
 string msg = String.Format(
  "<{0}>{1:MMM d HH:mm:ss} {2}",
  message.Facility * 8 + message.Level,
  DateTime.Now,
  message.Text);

 byte[] bytes = System.Text.Encoding.ASCII.GetBytes(msg);
 SyslogHelper.Send(bytes, bytes.Length);
}

The above code checks to see if there is an open connection to a syslog server. If not it opens a connection using the given server name and port (UDP 514 by default). Once the connection has been established a message is formatted in such a way standard for syslogging:

MMM d HH:mm:ss MESSAGE

The message is encoding as a buffer and then sent to the syslog server's port. That's all there is to it!

Conclusion

I hope you've found this article educational. It sure was fun to write vmmon. Be sure to email me if you have any questions via editor@searchvmware.com.

ABOUT THE AUTHOR: Andrew Kutz is deeply embedded in the dark, dangerous world of virtualization. He is a Microsoft Certified Solutions Developer (MCSD), a SANS/GIAC Certified Windows Security Administrator (GCWN) GOLD and a VMware Certified Professional (VCP) in Virtual Infrastructure 3 (VI3). Andrew is the author of the popular open source project, Sudo for Windows. He was employed by the University of Texas at Austin for nearly a decade as a systems administrator and developer and more recently as an analyst for Burton Group.

This was first published in January 2008

Dig deeper on Scripting administrative tasks

0 comments

Oldest 

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to:

-ADS BY GOOGLE

SearchServerVirtualization

SearchVirtualDesktop

SearchDataCenter

SearchCloudComputing

Close