Yesterday, I embarked on an upgrade from vSphere 4.0 U2 to vSphere 4.1. There are a couple of caveats here and some gotchas. Firstly, I backed up my vCenter DB, and also shutdown my SQL2K8 and vCenter VMs, and took a snapshot of them before beginning.
In terms of the order – you MUST get started with the upgrade of vCenter – it’s impossible to upgrade ESX 4.0-U2 to ESX 4.1 without doing this before hand. The upgrade of vCenter itself went relatively smoothly. My previous installation used a 32-bit DSN for both vCenter and VMware Update Manager. So I knew I would be safe from this upgrade bug:
VUM4.1 ONLY installs to a 64-bit OS. Apparently, VUM 4.1 although requiring a 64-bit OS, is actually a 32-bit application, but as such requires a 32-bit DSN. Go figure that? It’s actually quite difficult to know if your DSN is 32 or 64 bit. But I noticed when I upgraded VUM, that Database Information page showed a path to the SysWOW64 directory. This is the location where you can run the DSN Administrator for 32-bit on a 64-bit copy of Windows 2008 R2.
By digging around in the Registry I was able to see that VUM had a DSN entry held in HKEY_LOCAL_MACHINE \ SOFTWARE \ WOW6432Node \ ODBC \ ODBC.INI
The upgrade of VUM does mention the possibility of a reboot of the server it's running on at the end of the installation – I found I didn’t need to reboot.
The other bug to watch out for is the fact the VMware Composer 4.0 is incompatible with vCenter. I’ve been running the beta version (View 4.5) on my vCenter for a while. So I decided to take the risk to see if it would still work after the upgrade had completed. This View Composer incompatibility again seems to be a 32/64-bit issue. The KB article promised that View Composer with 4.5 will be compatible with vSphere 4.1 at the time of an unspecified GA. Right now, customers with View Composer and vCenter 4.0 ask not to upgrade.
“VMware View 4.0.x customers who use View Composer should not upgrade to vSphere vCenter Server 4.1 at this time. Our upcoming VMware View 4.5 will be supported on VMware vSphere 4.1.”
I got strange “Cannot execute SQL script warning” in the first attempt of upgrading vCenter, which didn’t appear during the second attempt – so I’m assuming I fat-figured an entry during the installation. During the installation there are two new questions which previously didn’t exist in previous installations and upgrades of vCenter. The first concerns how you want to handle the update of the vCenter Agent on your ESX hosts.
I chose to select the automatic upgrade, as I didn’t much like the idea of them being “disconnected” during any upgrade. During the upgrade, using the “Automatic” option didn’t seem to cause any issues…
Additionally, there is a new option to allocate how much RAM you wish to give the vCenter Web-Service that runs alongside the core vCenter service
I occasionally run the vSphere Client on the vCenter Server, which I guess is a little naughty of me. But as I was there, I decided to upgrade that also. You should know that from vSphere4.1 neither the vCenter nor ESX host webpages have the “client download” link anymore. However, after doing the upgrade I found that the ability to download the client from vCenter WAS STILL there, but it's been removed from the ESX host. I resolved to read release notes more closely in the future!
Anyway, one thing that appears to happen is that vCenter 4.1 upgrades the type of DSN you're using. Previously, if you want to run vCenter on 64-bit you needed to create a 32-bit DSN using the WOW utilities. After the upgrade it appears that VMware migrates your 32-bit DSN into a 64-bit one. The VUM DSN which is 32-bit remains 32-bit. So if you want to modify your DSN for vCenter and VUM you will need to run to separate utilities.
I was surprised to see that the vSphere Client didn’t offer me the chance to install or update the “Host Update Utility”. This allows you to update/upgrade ESXi (only) without the need for VUM, and I occasionally use it when VUM won’t play ball… If you plug an older version of the vSphere Client into the new vCenter – you get a standard message to upgrade the client. I’ve never really liked this method of upgrading – after once accidentally canceling an upgrade – which then broke the older Vi client. My advice is once you have embarked on the upgrade, don’t cancel it!
Once both vCenter and VUM have been upgraded, the VUM plug-in must be (re)installed before you think about upgrading your ESX hosts.
Sadly, after installing the new plug-in I got an error message saying that the VUM service was not available. So to cover my back I decided to reboot the vCenter/VUM server, and see if that resolved the problem. Unfortunately, the old Windows Sysadmin universal fix didn’t pan out as I hoped, so it was time to re-examine the service logon settings. I rather naughtily run VUM using Windows Authentication with previous versions of VUM you had to manually handle the “right to logon as service” privilege.
There now appears to be two VUM services (where previously there was just one) – the core VUM service and the VUM UFA service that is used specifically when VUM updates offline VMs. The core VUM service was marked as “Started” and the VUM UFA service was not, admittedly its default start-up settings were “Manual”. It was unclear if the service would or would not start properly when needed. So I decided to set about making sure that both VUM services had the correct logon details. After doing this the plug-in loaded correctly. PHEW! Without VUM working I wasn’t looking forward to the ESX host upgrade to ESX 4.1.
Before starting the upgrade to ESX 4.1 process – I ran a scan against my existing ESX hosts to confirm they were patched to the hilt for the ESX 4.0 U2 edition. When you use VUM to upgrade ESX 4.0 to Update1 or Update 2 – VMware just downloads the patches for you, and then you can re-mediate. VUM maintains a clear distinction between “updating” an existing release (ESX 4 U1/U2 and so on) and “upgrading” to a new release (ESX 4.0 to 4.1). So in my case I had to download the following binaries
- To upgrade ESXi 4.0 to 4.1 download the “pre-upgrade-from-ESX4.0-to-4.1.0-0.0.260247-release.zip” and the “upgrade-from-ESXi4.0-to-4.1.0-0.0.260247-release.zip”
- Note: That’s right, if you are using ESX 4.0 “Classic” – you first have to apply a pre-upgrade bundle to the ESX 4.0 servers, and then once completed, apply the second bundle
- To upgrade ESX 4.0 to 4.1 download the “upgrade-from-ESX4.0-to-4.1.0-0.0.260247-release.zip”
- Note: In contrast the ESXi 4.1 upgrade is a single bundle. But hey, I guess that proves upgrading, updating and patching ESXi is easier with ESXi rather than ESX Classic. Right?
These can then be “imported” using the “Host Upgrade Releases” and “Import Upgrade Release” link. There are NO default “Baselines” for these bundles, so these need to be created and then attached to the relevant hosts. Sadly, the official “Upgrade Guide” does NOT explain this process. Instead it directs you to read the “vSphere Update Manager Guide” which I struggled to locate amongst the standard docs page. Sadly, this guide doesn’t really explain the process of this Host Upgrade Releases process either.
I was able to get the upgrade-from-ESXi4.0-to-4.1.0-0.0.260247-release.zip and the upgrade-from-ESX4.0-to-4.1.0-0.0.260247-release.zip to import into VUM, but the pre-upgrade-from-ESX4.0-to-4.1.0-0.0.260247-release.zip package stubbornly refused to import. It kept on failing with a “Cannot verify checksum for imported file. This might be caused by corrupted metadata or binaries in the file. You can try to import and new copy of the upgrade file.”.
I got my files from the VMware Instructors download page, and double-checked the MD5SUM values to confirm I didn’t have a bum download. The MD5SUMs matched perfectly. Occasionally, there’s been a bum file on the site, so I thought I better draw their attention to it. I did try cranking up a new evaluation of vSphere4.1 using a “Mailinator” account to get past having a valid e-mail address. I’m forced to do this because, despite being vExpert, I’m not a partner nor customer and therefore my normal logins to vmware.com do not allow me to download software at will. It’s been an issue that has been a problem since 2003… Anyway, sadly I found that an evaluation does NOT allow access to the upgrade packages, so that turned out to be a dead end. So I thought I was stumped. I’d be able to use VUM to upgrade ESXi, but not ESX “Classic”. Then I noticed something on the web page for instructors. It said to “Deploy this [PRE] package before you upgrade from ESX 4.0 to ESX 4.1 using esxupdate or vihostupdate”. I found it interesting that there was NO mention of using VUM to apply the pre-upgrade package, maybe you HAVE to use esxupdate or vihostupdate first…
So I used the vimsh command to get the ESX hosts into maintanace mode, and this issued the esxupdate command:
vimsh -n -e /hostsvc/maintenance_mode_enter
esxupdate update –bundle=pre-upgrade-from-ESX4.0-to-4.1.0-0.0.260247-release.zip
This was successful using the esxupdate command, on the SAME package that wouldn’t import into VUM. So I’m pretty sure that a.) The file was good. b.) that VUM doesn’t know how to verify that the pre-upgrade ZIP file is good. It seems a little odd that in the end I had to resort to command-line tools to PRE-upgrade ESX “Classic” before I could upgrade to ESX 4.1…. It left me wondering what the point of VUM was if it couldn’t handle the whole process – I might as well have stuck with esxupdate/vihostupdate for the whole deal. Anyway, after ALL the effort of upgrading vCenter/VUM I thought I would persist with using VUM to complete the upgrade process rather than having to write scripts to achieve it.
Anyway, once the pre-upgrade package had been installed I rebooted the ESX Classic host, and exited maintenance mode. In the meantime the import of the other two upgrade bundles had completed. Although it’s TWO separate imports of the bundles – you are left with just ONE entry in the Import Upgrade Releases with TWO ticks, one next to ESX 4.0x and one for ESXi 4.x – allowing you to upgrade both to ESX 4.1 of either version
The reference to the Status being “Partial” is because the ESX 3.x to ESX 4.1 bundles had not been imported, as I didn’t need them. I think this is the reason why the “Upgrade Release Name” is in red… I try to not let red text worry me… but worry me it does.
Next I created my baseline which I called “Upgrade to ESX 4.1″ and then attached it to the first ESX hosts – and carried out a scan. Unsurprisingly that gave me red “non-compliant” marks because I’d neither staged nor re-mediated the host yet. I like to do this so I get a nice “before and after” feel once the whole process has been completed. I was surprised to find, that whilst you can “stage” (pre-download to the ESX Host) patches, it doesn’t appear to support the staging of upgrade packages. So I was forced to go straight for the remediate button – selecting my Upgrade Baseline along the way. At the time I only had two ESX hosts in the cluster (not my usual four), and one of them was in maintenance mode. So to get the upgrade process to go through I had to enabled the option to switch off HA Strict Admission Control.
This first upgrade of ESX Classic went through without an error, and the ESX host successfully exited maintenance mode, and rejoined the cluster.
Next up was the upgrade of ESXi 4.0 U2 to ESXi 4.1. Here I deliberately made my life harder. On the ESXi host was my domain controller, SQL and vCenter/VUM server. In the past I would have had to manually move vCenter/VUM host to another host, prior to engaging VUM remediation. I am very pleased to say this is no longer the case! The functionality that was first introduced in Vi3.5 has been restored, and now DRS successfully moves the vCenter/VUM server to another host.
Sadly, midway thru the remediation process something went wrong with my storage connection from the ESX host that was running my DC/SQL/VC/VUM. That caused vCenter to crash, and therefore I wasn’t able to monitor the remediation process from vCenter. [I guess some would say this is a good reason not to run virtual infrastructure services on a virtualization layer!!!] Anyway, I was able to connect to the ESXi box and confirm that it had indeed been upgraded to ESXi 4.1, and it was still in maintenance mode. Exiting maintenance mode brought the ESX4i box back into the cluster.
The last thing I did was check my HA Cluster settings. I found that strict admission control was still disabled. I wasn’t sure if this was expected behavior or because vCenter/VUM had crashed mid-way through the remediation process whether VUM hadn’t had an opportunity to re-enable it. The only thing I wasn’t able to test was upgrading to SRM 4.1 – I don’t have an SRM environment up and running at the moment so it wasn’t possible.
I do have another 2 ESX host to handle. They were running the BETA version of vSphere4.1. So I plan to wipe them this afternoon, and cleanly re-install them and add them back into vCenter. I still have VMware Tools updates to do. But I won’t be doing them on my virtual desktops until I can prove to myself that they won’t break VMware View PCoIP functionality…
After that I will be looking over my View 4.5 beta environment and see if it's still in a good state. I’ve lost my way a little on this View 4.5 guide I have been writing recently. So much work to do for TechTarget and the vSphere4.1 release has kept me away from writing the guide. So I want to get back to that pronto.
The other thing I want to do is play with vCenter HB service (AKA NeverFail) and Double-Take to compare and contrast the solutions. I’m also thinking of protecting vCenter with MSCS as well. My plan is to write up the configuration process and experiences…