About a week or so go I did a webex with vWire’s Mike Connors – in preparation to evaluating their product. This has been on my desk for months, and I finally got round to the product this week.
Two things really triggered my evaluation – firstly, the product has been recently released and is now compatible with vSphere4. I’ve been using vSphere4 since the beta programme, and all my lab kit is dedicated to run it, with no spare capacity for run (ahem) “legacy” versions of ESX and vCenter. Secondly, over the last couple of days/weeks/month I’ve been immersing myself in PowerShell – and vWire has a PowerShell component to it which is used to re-mediate problems it discovers.
So anyway, what is vWire? On the front side its a monitoring/health-check system that doesn’t just look at your vCenter DB, but also uses the log files of ESX and the SDK to build up a picture of your environment. So its very “operational” focused in managing a pre-configured environment. This might be its main selling point – the data it collects and reports with is denser than what the core vSphere4 product does for you. In this respect its simplifying the process of collating and acting on all that log info which is held by VMware in many different places. Additionally it has an alarm/alert function – together with a remediation engine – which allows PowerShell scripts to go out and fix the problem.
What’s interesting about that – is while it's all well and good to fix problems with PowerShell – what’s missing is a engine to trigger the remediation process. Whilst vCenter does now ship with lots of conditions and alarms – an ability to trigger scripts. Alarm/Alert system in vCenter is not extensible by the average admin unless you fancy writing your own application & plug-ins for vCenter.
As a tease to evaluation the vWire product there are some free tools on the site including OpsCheck and ConfigCheck. OpsCheck confirms if VMotion is correctly configured in your environment, whereas ConfigCheck confirms the security status/configuration of your ESX Host.
Additionally, vWire has “Community Content Library” where vWire uses can upload and share their solutions which they have created within vWire. These can be download by other users to extend the core functionality of vWire. It’s perhaps fair to say that vWire has some work to build and extend this community – as most of the contributions come from vWire staff such as now legendary, Steve Beaver.
Well, that’s the overview of the product – now for my evaluation…
I began by checking out the install requirements (Windows in a domain, .NET2.0, MS PowerShell and PowerCLI4.0 from VMware). I decide to use the built-in DB that comes with vWire, as this was just an evaluation and didn’t want to have the hassle of DSN configuration and SQL Administration. After installing all the PowerShell components, I cranked up the PowerCLI4.0 from VMware and made sure my vWire box could communicate to the vCenter server. I wanted to make sure that PowerShell was hunky-dory before even thinking about the install of vWire – so I made sure the “Set-ExecutionPolicy RemoteSigned” policy was all setup and good to go. Incidentally, I ran vWire in VM on the very system that it was monitoring. A little bit circular but I’m using to running AD, SQL and vCenter as VMs with ease, and want to see how vWire would handle this configuration.
For a more detailed list of requirements see here:
I clearly didn’t read the requirements too closely, as this VM I created had a 1GB of memory which the installer checks for – anyway, I ran through the installation anyway, and increased the memory to 2GB which is the recommended minimum.
Initially I was working from the instructions with the online help, until I switched to the PDF that is part of the vWire evaluation .zip file. After completing the install (and correcting my memory problem) my first task was to crank-up my web-browser to https://vwire.vi4book.com, log in (using the local administrator account to begin with) and license the product with my 30-day key, which as you can see is merely browsing for license file.
Directly after doing that, the main vWire window opened up and I was prompted to supply my vCenter details like so:
(Yes, that’s right “vmware” is my default password for most of my test systems. Not the most rigorous of passwords I’m sure you will agree – but hey, easy to remember! Incidentally, if you do use the SQL Express edition you password will have to be a LOT more rigorous for the DB!). AS you can see in the dialog box above vWire extends the functionality of vSphere Client with it own plug-in. This allows you to see the vWire server and its data in the vCenter windows as well:
After completing this information and clearing the obligatory tour/welcome screen this is what vWire looked liked. I was immediately struck by the “Open Alarms” indicating one of my ESX host had a “LUN Path Failure”. Early that day I’d been to my location where my hardware lives, and added two new NetApp FSAs. I’d been checking out the fibre-channel switch configuration on the Brocade Silkworm I have – and first concern was that I had balls up my switch configuration for one of my ESX hosts:
As you can see vWire has the classic “tab” view with a dashboard that summarizes the latest info. It reminded me a lot of my blogging software which has similar dashboard feel. I decided to check out that LUN Path Failure by clicking at the link labelled “1 Open Alerts affecting 1 hosts”.
After that I click the LUN Path failure button to investigate in a bit more detail:
And this in turn suggested the problem with a path failure to an iSCSI system.
Phew. The iSCSI system is just a VM which I’ve been using for demos in my video series. It’s not my regular fibre-channel storage (EMC Clarrion) or my new NetApp storage. So I went of to the vSphere Client. My theory was that the iSCSI VM is a bit fragile – and perhaps I’d powered it off at some point breaking the connection to the ESX2 host. That said I would have expected that to happen to all my ESX hosts in the vCenter. Sure enough vWire was right, the vSphere Client indicated the path was dead to the iSCSI LUN.
The interesting thing about this first initial experience was this. Firstly, I had no clue this problem existed until vWire told me. The vSphere client didn’t give me any alerts/alarms – in fact to the casual view the storage was fine as the VMFS volume was listed there. Dropping out of the vSphere Client, I did a rescan (as I knew the target was up and running) and the problem was resolved! To my great relief I hadn’t some how screwed up my fibre-channel switch in the morning!
Well. With all that resolved. I decided to make a cup of tea (very British of me, I know) and work my way through the PDF guide. I still hadn’t confirmed all my ESX host had been discovered, and setup the SMTP settings. After all, if that storage problem happened again, I would really like an alert on it that came to my inbox...
[After a brief pause, and cup of tea...]
Ahhhh. That’s better. Following the PDF guide I decide to check my ESX hosts had been discovered correctly. Under Administration, and Hosts – I could see all my hosts. vWire correctly figured out that ESX1/3 were running the “classic” edition of ESX with a “Service Console”, whereas ESX2/4 were running ESXi without a conventional console front-end.
Using vWire Server Settings, I set up the SMTP/Email requirements:
After clicking Apply and Send test e-mail message, one came through. (Although I didn’t use the settings in the screen grab as I don’t want to disclose my genuine SMTP settings in a post)…
Finally, I had to do some user admin. By default vWire allows you login as the local administrator of the vWire box. I wanted to use my domain account instead. I did try using a group to achieve this, but couldn’t get the group to be recognised, so I just used a single account. Adding a user in the “User Preferences” tab, and the Add button like so:
I was able to logout as the local administrator and in as the mikel account. So my initial setup work was done. It was time to investigate the other alert that had come up in vWire concerning “Port Group Network Security Policy”
By default vWire checks your security settings on vSwitches which cover the options on the security tab of a vSwitch or Portgroup. The default settings being Accept/Reject/Reject.
As a general rule I don’t change these settings. And you could regard that as a security weakness in my configuration. Incidentally, changing the two MAC address policies (MAC Address Changes, Forged Transmits) could cause issues in clustering software such as MSCS/MS-NLB). As a rule, Promiscuous Mode should be left as reject – unless your running some kind of intrusion detection system inside a VM or you doing something wacky like running ESX inside a VM. I wouldn’t hurt for my ESX host to be hardened up by having all these policies set to be reject.
Earlier in the day I was browsing the Content Library and noticed that there was a solution (in the form of a PowerShell script) which could be downloaded and imported to vWire. So I navigated to the vWire Content Library and searched there, locating the solution at:
Under the Workbench tab in vWire, I was able to find the Actions tab. The actions tab shows all the remediation tasks you can carry out – together with a handy import button that can be used to upload community solutions to vWire. Unfortunately, it appears the solution was written in a format compatible with vWire 1.0, and wasn’t in the right format for vWire 1.1. It returned an error that stated as much.
Anyway, I’m good pals with Steve Beaver from my forums days, so fortunately I was able to get back to Steve and he sent me a new version of the solution – which I was able to import – and by the time this is posted will be on the vWire Content Library.
This solution then got added to the list of possible remediation actions like so:
I was then able to go back to the alert, and ask vWire to fix this problem – by using the solution I added in the workbench – and use the Run option to allow the fix to take place…
Well, that more or less fixed me up on the alert/action front. All I need to do is close off the alerts to mark them as dealt with. At this point, I had a bit of senior moment as I couldn't tell right away how to do that. Of course, it was staring me in the mug all along.
The only thing I would say about the “Action” process – is that the PDF admin guide doesn’t give you much information on how to create actions of your own. That said Steve Beaver’s vWire blog does have some tutorials on how to do this here:
So I decided to check out the other tabs. You know what it's like with new bit of software – you sometimes dispense with the manual (RTFM!), click about a bit with the software and see what you can get it to do from the get go. Kind of like kicking the tires of a new car!
So I went to the search tab, to see what kind of searching you can do. From the pull-down list there is a list of categories you can search on. Clearly, this has had major overhaul in the latest release to cover the new object types in vSphere4 such as vApps and DvSwitches. Each “Search For:” category comes with pre-built search routines.
So for example, I selected “Host” from the pull-down list, and used the built-in search called “High Security Not Enabled on Host Firewall” – and discovered one of my ESX hosts was not configured for high-security. I’m figuring the reason being is that I’d enabled the SSH Client on this ESX host (which allows me to use scp/ssh from one ESX host to another…
Another neat feature of vWire which I liked was the ability in the search window to compare two vCenter object and compare them. For example, I could search for all my VMs, and then select two of them, and then using the comparison tab – set it to show how they were different from a configuration point of view.
The only thing I would say about this – it seemed to take vWire a while to pick on changes to my environment. For example I powered down both VMs, increased the allocation of CPU and Memory to one of the CTX boxes, and then ran the query again. Unfortunately, vWire seemed to think the amount of resources I’d allocated was the same.
In the screen grab below, I’d give CTX03 an additional CPU and 2GB of RAM, but vWire still reported the old configuration. I guess you wouldn’t want vWire constantly polling the vCenter/ESX hosts every nano-second.
Anyway, coming back a short time later – I checked again, and vWire did indeed update the changes I had made.
So this led me to look at the administrative settings to see whether there was some kind of polling interval that could be configured. I needed to be able to set my expectations correctly so I would know how long it would take to pick up my changes in vCenter. After a short while I found this configuration dialog box under the Administration/vCenter Tab…
The most frequent polling on this pull-down list is once every 15mins, and the least frequent is once every 24hrs. Down in the bottom right hand corner of the page is synch now force vWire to update all its information.
Finally, the Workbench tab is where customization can take place. Under the Workbench tab you can see the built-in Alerts, Searches, Actions, Comparison Views, and Property Sets. From here some heavy customization can be done. For example rather than manually running the Action to fix the port security problem I showed earlier – I could, if I wish, customize the alert that told me of the problem, and set the default action to the solution I downloaded and imported into vWire.
So below I select in the Alerts tab the alert called “Port Group Network Security Policy”, and then click the Edit button in the bottom right-hand corner. In the “Configuration Alert rule” dialog box under the “Criteria” tab, I could click and edit the “Actions to run” column.
Once I was done, under the “Script Credentials” tab I was able to set what user account would be used to carry out the task. As I was here, I decided to the same action with the LUN Path failure that vWire alerted me to before:
You can also make custom searches. These are constructed using the GUI and the save function
or you can take an existing search, duplicate and modify it
This new “saved” search then appeared in the Basic Search view, like so – flagging up my vmfs volume called vi4book was low on space:
In a very similar way it is possible to add your own actions (in PowerShell) under the actions tab. Not any old PowerShell script will run that you have found on the interwebs, as vWire has its own set of variables which must be referenced in the PowerShell such as the $input variable. If your familiar with PowerShell its just a question of supplanting your existing variables in your PowerShell scripts, with the vWire ones which isn’t the end of the world.
Once your action has been saved, and tested - it can then be shared with the wider vWire Community…
Well, I only spent a day or two playing with vWire of the back of 30/45 minute webex session. So my verdict must be tempered by that fact. But a couple of observations come to light even in that brief time. Firstly, to get the most out of the search/alarm facility, some knowledge of SQL Transact would be handy. These searches go to the vWire’s DB by the way not the vCenter DB. Secondly, its one thing to alert/alarm – what you will want to do is try to automate the remedial action – to do that you will need some good PowerShell skills. Now if these two skills (especially the former) are not in your skill set then you will have some work to do before you can really leverage what vWire is offering.
You see the built-in objects in vWire seem to be there to give you something to start from – the intention is for you the administrator to customize vWire to make it do what you want it to do. Different people have different views about this. Lazy administrators (like me) will be looking for something that is more off-the-shelf-just-do-it-all-for-me. Good administrators prefer systems that they can extend and build up the way they want it to be. Good administration comes with time and skills (SQL Transact/PowerShell) which you can’t just buy off the shelf. vWire doesn’t come with hundreds and hundreds of pre-built alerts, searches or actions. It comes with about 15-30 individual ones. So I don’t see vWire as one-size fits all, shrink-wrapped solution – its more extensible and configurable than that. And it needs to be. All I’m saying (and you could say this about a lot of products to the point that the next statement can become meaningless!) is that its going to take some investment of time and skill to leverage the power of the system to fullest.
Clearly, vWire is hoping that the Community will address the lazy administrator like me. With pages and pages of free searches, alerts and actions. However, right now this community is quite small – and number of available downloads in the Content Library is by no means exhaustive. In my webex with Mike Conner we discussed how vWire could grow this community more rapidly. Clearly one method is to reach out to people such as myself. I hope readers of this blog, haven’t thought this has been a infomercial for vWire. I am wondering if a smaller scale version of vWire could made free to download. Then, enthusiasts (the kind of people who like getting guru, god, etc status on forum boards) would be drawn into the product, and start contributing to the vWire community.