Real-world suite implementations add to upgrade complexity
Upgrading ESX Server involved upgrading the related applications in the VMware Infrastructure Suite at the same time. Single VMware ESX Server upgrades minus VMware Infrastructure Suite applications require users to upgrade the ESX Server OS to 3.5, followed by a VMware Infrastructure client to version 2.5 upgrade.
However, most of us don't have such a simple configuration. Real VMware ESX Server production environments may consist of any number of ESX Servers, a VMware VirtualCenter Server, a VirtualCenter database, VMware License Server, VMware Consolidated Backup (VCB), VMware Tools in the Guest VMs, and the VI Clients.
Though upgrading a single ESX Server from 3.x to 3.5 is relatively straightforward, having so many different pieces across the entire VMware Infrastructure Suite installation adds complexity to upgrades. For that reason, the pitfalls I list out below have to do with not only upgrading VMware ESX Server but also the entire VMware Infrastructure Suite. A particular problem is that some settings in the local Linux Service Console may be unexpectedly modified, so it's a good idea to test systems before upgrading
Servers that run VMware ESX are usually powerful and expensive. So developing a group of test systems comparable to the "suite" of systems working together in your VMware Infrastructure can be difficult. An excellent way of doing this is to run VMware ESX Server on a small server that marginally meets the hardware requirements while running VMware VirtualCenter in a virtual guest OS on one of your existing servers. This configuration allows you to perform a number of tests before doing the real upgrade on the production network.
In what order does the upgrade need to be performed?
You should upgrade the VirtualCenter server to version 2.5 of VirtualCenter before upgrading your VMware ESX Servers to version 3.5. The upgrade path to go to ESX 3.5 and the latest VMware Infrastructure Suite is listed below:
- Upgrade VirtualCenter from 2.x to 2.5, license server, and VirtualCenter database, if needed.
- Upgrade ESX Server Hosts and Datastores, if needed.
- Upgrade Virtual Guest OS VMware Tools.
- Upgrade VMware Infrastructure Client on each management PC.
Many upgrades need to be done in this process. For example, 10 ESX Servers with 100 Virtual Guest operating systems, 1 VC Server with database, and 5 VI Client management stations comes to a total of 116 upgrades of various packages.
Sure, the VMware Tools can be set to automatically upgrade. But even if you remove those, the upgrade can take a lot of time. If you don't have the luxury of being down during the upgrade, you will have to be very, very careful and plan this upgrade very well. As always, the biggest sources of headache are the unexpected pitfalls that you may run into along the way.
HA, DRS and database issues
VM High Availability (HA) and Distributed Resource Scheduler (DRS) are tools to dynamically move guest operating systems around when systems go down. Make sure that any ESX Server host systems being upgraded are isolated from the HA and DRS Pools. A good way to do this is to remove any of those rules or pools that the current host systems are a part of, put the host system in maintenance mode, and move it to a newly created test/upgrade VMware virtual data center.
Reports have surfaced concerning database errors and upgrades, as well as the database that VirtualCenter uses to store its data. This is especially true if the database is not on the same server as VirtualCenter and you have a failed VirtualCenter upgrade. In this situation, the database may have been modified but not VirtualCenter.
The best way around this is to run an MSDE database on the VirtualCenter server and use it for your VMware VirtualCenter data. And run your VirtualCenter Server inside a VMware ESX Server Guest OS. In doing this, you can take a snapshot of your VirtualCenter Server, including the database. Additionally, if you have a failed upgrade of a VirtualCenter upgrade, you can simple revert to the previous snapshot and recover the server and the database.
Please note that prior to starting the VMware ESX Server 3.5 upgrade process, you should thoroughly read the VMware ESX Server 3.5 Release Notes and the VMware ESX Server 3.5 Upgrade Guide. The Upgrade Guide has a number of upgrade checklists that will help you in your planning and would prevent pitfalls in the upgrade process.
Is the upgrade to ESX 3.5 and VI2.5 worth the hassle?
VMware ESX Server 3.5 and the updates in the VMware Infrastructure 2.5 suite have a lot of new features that virtualization administrators must have. However, the VMware Infrastructure is a complex suite of applications and upgrading it -- especially on production systems -- will require careful planning. For more information, check out RTMF Education and their VMware ESX Server 3.5 Upgrade Guide, which offers a lot of real-world advice.
About the author: David Davis has served as an IT Manager for over 15 years. He has a number of certifications including CCIE #9369, MCSE, CISSP and VCP. David has authored over one hundred articles and a number of video training courses on VMware. David publishes Cisco Networking how-to articles and videos at his website, HappyRouter.com.
This was first published in February 2008