Problem solve Get help with specific problems with your technologies, process and projects.

VMware patches: Another quality assurance snafu

VMware patches have historically wrought havoc and broken functionality in virtual environments. The latest debacle with the vSphere 4 Update 2 hasn't helped the company's rep, either.

VMware's track record on patches has been spotty, to say the least. VMware was slow to release VMware Update Manager (VUM), a patch management system that rivals Microsoft's Windows Update or SUSE Linux's Yellow Dog Updater Modified. Prior to VUM's release in 2008, customers were forced to write scripts to patch ESX hosts --- often using text manipulation to interrogate .XML files to ensure that VMware patches were applied in the appropriate order.

ESXi and VUM were designed to reduce the frequency and complexity of VMware patches. While VUM excels at downloading and installing patches, it's dependent on VMware's spotty quality assurance (QA). Several VMware patches and updates have created problems for t hypervisor -- and, most recently, the VMware View virtual desktop product.

For more on the VMware patching process, check out the following resources:  
Two ways to patch VMware ESXi Installable  

VMware Infrastructure Client 2.5 harnesses third-party plug-in power

The update that disabled VMware View
In June 2010, a VMware representative who promotes new Knowledge Base (KB) articles posted an urgent note on Twitter: VSphere 4's Update 2 (or U2) breaks VMware View's PC-over-IP (PCoIP) functionality.

When I saw the tweet, I had already completed half the U2 installation in my lab. Fortunately, I hadn't upgraded VMware Tools, which was the component that disabled View's PCoIP.

Initially, VMware issued a new Knowledge Base article that told VMware View 4 users to avoid the U2 upgrade. Once the problem had been resolved for users that installed U2, the article was updated.

To fix U2's patch, VMware removed the troublesome VMware Super Video Graphics Array driver and installed an older driver that does not affect PCoIP. The fix had to be applied to every virtual desktop, template and parent virtual machine (VM) used by VMware Composer's Linked Clone feature.

Another VMware patch debacle
Sadly, a similar situation has happened before. When VMware Update 1 (U1) was released, I was one of the first to download and upgrade my VMware environment.

But the U1 upgrade had compatibility issues with the Hewlett-Packard (HP) Management Agents in HP servers. As a result, U1 caused an extremely rare Purple Screen of Death (PSOD) critical error on HP servers.

PSODs are usually caused by failures outside of the hypervisor's control, such as bad memory. (Being a sad, little geek, I collect PSODs on my RTFM blog. I don't get out much, and I have to do something to pass the time.)

VMware patches and quality assurance: Moving forward
What are we to make of these VUM shenanigans? The common response is that it's foolish to immediately install VMware patches. Wise and experienced administrators sit tight before deploying any major update from any vendor.

But that provides little comfort to customers who have downloaded the product in the last couple of weeks. Additionally, when you raise a support request with a software vendor, the first thing they usually ask is if you've installed the latest build. If not, they promptly bounce you out of the support process until you do

I installed U2 because of a new VMware View 4.5 beta. It's a bitter irony that U2 doesn't affect the PCoIP with VMware View 4.5 Beta 2.

That's right: The beta product is compatible, but the production product is not.

I've worried about VMware's QA for some time. I first raised these concerns at the Charlotte, N.C. User Summit in 2007. When VI3 was still an immature platform, I noticed an "Object reference not set to an instance of an object" error message that appeared on an hourly basis.

I worried that VMware was growing at such a rapid pace, in addition to unveiling a dizzying array of new features, that its quality control was slipping.
Currently, VMware enjoys an almost Apple-like reputation for products that "just work." I would hate to see that perception undermined by poor patch management QA. It only gives VMware's competitors a stick to beat them with.

It's worth noting that vSphere 4 is vastly different from VI2, the first widely adopted VMware virtualization platform. In the early days, VMware strictly controlled its hypervisor, and there was a healthy release of "maintenance" editions. VMware ensured stability and reliability with a very narrow Hardware Compatibility List that restricted ESX to a discreet list of server makes and models. There has been a paradigm shift with the introduction of vSphere4 that I think many in the VMware Community have missed. Now ESX is more vendor friendly, and many companies can add functionality to the hypervisor with their own products.

That said, few companies actually write vendor-specific code for ESX, with the exceptions of EMC PowerPath, Xsigo and Cisco Nexus 1000V VEM.

Eventually, VMware will discontinue ESX and only offer ESXi. But VMware will need to change its internal QA process and how the VUM works. I wouldn't be surprised if VMware adopts a process similar to the Microsoft Windows Update, in which third-party extensions are downloaded from VMware's website. In this model, third-party engineers would offer compatibility testing instead of VMware personnel. This approach might have avoided the HP and Cisco issues that customers faced with the previous VMware patches.

I also wouldn't be surprised to see a bigger commitment to the patching and QA of existing products, with VMware adjusting its roadmaps so that its engineering teams can find and fix bugs without the pressure of simultaneously developing new versions. If VMware can quickly refine this patching issue, it will certainly be a beautiful day.

Mike Laverick (VCP) has been involved with the VMware community since 2003. Laverick is a VMware forum moderator and member of the London VMware User Group Steering Committee. Laverick is the owner and author of the virtualization website and blog RTFM Education, where he publishes free guides and utilities aimed at VMware ESX/VirtualCenter users, and has recently joined as an Editor at Large. In 2009, Laverick received the VMware vExpert award and helped found the Irish and Scottish VMware user groups. Laverick has had books published on VMware Virtual Infrastructure 3, VMware vSphere4 and VMware Site Recovery Manager.

Dig Deeper on Troubleshooting VMware products

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.