VMware's VMotion technology is the process by which a running Virtual Machine is migrated from one ESX host to another without incurring downtime. This how-to will explore how to obtain VMotion, how to configure it, what its appropriate usage scenarios are, and finally, how VMotion technology actually works.
How to obtain VMotion
VMotion is a component of VMware's Virtual Infrastructure 3 (VI3) enterprise virtualization solution. VMware offers three VI3 editions -- Starter, Standard, and Enterprise. VMotion is available a la carte for the Starter and Standard editions and comes with the Enterprise edition. For more information on pricing, call a VMware Sales Representative.
VMware licenses its products per every two processors. Notice the use of the word "processors," not "cores." For the moment, VMware is still licensing their software per processor, as long as the processor
Regardless of the avenue by which the software is obtained, it is necessary to create a VMware account. This process begins here at the VMware Website. This account will be linked with all purchased VMware licenses. It may be a good idea to create a group or user agnostic email address to create this account with in order to eliminate a single point of failure. Once the account is created and the VI3 Enterprise edition has been purchased, it is possible to obtain the license files needed by ESX and most importantly with regards to this article, VMotion.
Unfortunately for the customer, VMware's actual implementation of its licensing does not break down as nicely as its three VI3 editions. Once VI3 has been purchased, VMware will send the buyer several emails with licensing information. These e-mails will include the License Activation Code(s) that can be used to register the product(s) with VMware. Visit this VMware page to register the product(s). Once the product(s) is/are registered it is possible to activate the software license(s) at the license management page. There is no nice way to explain how to create the license file. The user interface has changed twice now since this author last activated VI3 licenses. VMware is aware of how cumbersome the licensing process is and provides online VI3 Licensing Specialists that will assist a user via a live chat session.
Once the license file has been created and obtained by downloading via e-mail, it is time to install it the file into the licensing server. VI3 provides its own licensing server, but it is also possible to install the license file into an existing FlexLM environment. It would be pointless to repeat here how to install the license file when VMware has already explained it in great detail and quite nicely on page 43 of the VI3 Installation Guide. When installing the license file, please be cognizant of whether or not all the VI3 licenses were combined into one license file or there were several license files received. Although VI3's components, VMware Distributed Resource Scheduler (DRS), High Availability (HA), VMotion, VirtualCenter, and ESX all exist under the aegis of VI3, each of their licenses can be obtained as single files. This used to be the way it was handled, but now VMware allows the combination of the separate licenses into a single file on their web site. If the licenses are not combined on the web site, it will be necessary to combine them into a single file before installing them into the license server.
Once the license file is installed, including the VMotion license, VMotion must be configured.
Once all the licensing has been worked out, it is time to "turn on" VMotion. This assumes that ESX and VirtualCenter have already been installed and properly configured. Launch the VI3 Client and click on one of the ESX hosts that should have VMotion enabled. Once the host is selected click on the "Configuration" tab. Click on the link "Licensed Features" inside of the box labeled "Software." There is a section called "Add-Ons." Click on the link "Edit..." that is grouped with the "Add-Ons" section. If VI3 licensing has been properly configured then it should be possible to select the checkbox next to "VMotion". Click "OK" to enable VMotion for this host. Enabling VMotion on one host by itself is actually rather useless. The above steps should be repeated for each host that will have VMs migrated to or from it.
VMotion requires a VMotion-enabled network to function. Although it is not required, it is *strongly* recommended that VMotion have a dedicated NIC. In fact, if a VMotion is not functioning correctly and by all appearances should be, the problem will most likely be network related. VMotion sends a lot of information over the wire and needs plenty of room and speed to do so. Regardless of whether or not a NIC can be dedicated for VMotion, a port must be enabled for VMotion. Launch the VI3 Client and click on one of the ESX hosts that should have VMotion enabled. Once the host is selected click on the "Configuration" tab. Click on the link "Networking" inside of the box labeled "Hardware." Click on on the "Add Networking..." link in the upper right-hand corner of the VI3 client. Select "VMKernel" for the connection type and click "Next." If there is a dedicated NIC available to VMotion then select "Create a virtual switch" otherwise select an existing vswitch and click "Next". On the next screen label the port group "VMotion" and click on the checkbox that says "Use this port group for VMotion." Do *not* fill out the VLAN ID field. Fill out the IP Address, Subnet Mask fields. Do not fill out the VMKernel Default Gateway field. Click "Next" to continue and then click "Finish" to create the port group. The above steps should be repeated for each host that will have VMs migrated to or from it.
Now that both licensing and the VMotion networks are configured, it is possible to migrate VMs from one host to another.
VMotion usage scenarios
Now that VMotion is enabled on two or more hosts, when should it be used? There are two primary reasons to use VMotion: to balance the load on the physical ESX servers and eliminate the need to take a service offline in order to perform maintenance on the server.
VI3 balances its load by using a new feature called DRS. DRS is included in the VI3 Enterprise edition along with VMotion. This is because DRS uses VMotion to balance the load of an ESX cluster in real time between all of the server involved in the cluster. For information on how to configure DRS see page 95 of the VMware VI3 Resource Management Guide. Once DRS is properly configured it will constantly be evaluating how best to distribute the load of running VMs amongst all of the host servers involved in the DRS-enabled cluster. If DRS decides that a particular VM would be better suited to run on a different host then it will utilize VMotion to seamlessly migrate the VM over to the other host.
While DRS migrates VMs here and there with VMotion, it is also possible to migrate all of the VMs off of one host server (resources permitting) and onto another. This is accomplished by putting a server into "maintenance mode." When a server is put into maintenance mode, VMotion will be used to migrate all of the running VMs off it onto another server. This way it is possible to bring the first server offline to perform physical maintenance on it without impacting the services that it provides.
How VMotion works
As stated above, VMotion is the process that VMware has invented to migrate, or move, a virtual machine that is powered on from one host server to another host server without the VM incurring downtime. This is known as a "hot-migration." How does this hot-migration technology that VMware has dubbed VMotion work? Well, as with everything, in a series of steps:
- A request has been made that VM-A should be migrated (VMotioned) from ESX-A to ESX-B
- VM-A's memory is pre-copied from ESX-A to ESX-B while ongoing changes are written to a memory
bitmap on ESX-A.
- VM-A is quiesced on ESX-A and VM-A's memory bitmap is copied to ESX-B.
- VM-A is started on ESX-B and all access to VM-A is now directed to the copy running on ESX-B.
- The rest of VM-A's memory is copied from ESX-A all the while memory is being read and written
from VM-A on ESX-A when applications attempt to access that memory on VM-A on ESX-B.
- If the migration is successful VM-A is unregistered on ESX-A.
For a VMotion event to be successful the following must be true:
*Editor's Note: Special thanks to Colin Stamp of IBM United Kingdom Ltd. for rewriting the following list.
- The VM cannot be connected to an internal vswitch.
- The VM cannot be connected to a CD-ROM or floppy drive that is using an ISO or floppy image
stored on a drive that is local to the host server.
- The VM's affinity must not be set, i.e., binding it to physical CPU(s).
- The VM must not be clustered with another VM (using a cluster service like the Microsoft
Cluster Service (MSCS)).
- The two ESX servers involved must both be using (the same!) shared storage.
- The two ESX servers involved must be connected via Gigabit Ethernet (or better).
- The two ESX servers involved must have access to the same physical networks.
- The two ESX servers involved must not have virtual switch port groups that are labeled the
- The two ESX servers involved must have compatible CPUs. (See support on Intel and AMD).
If any of the above conditions are not met, VMotion is not supported and will not start. The simplest way to test these conditions is to attempt a manual VMotion event. This is accomplished by right-clicking on VM in the VI3 client and clicking on "Migrate..." The VI3 client will ask to which host this VM should be migrated. When a host is selected, several validation checks are performed. If any of the above conditions are true then the VI3 client will halt the VMotion operation with an error.
The intent of this article was to provide readers with a solid grasp of what VMotion is and how it can benefit them. If you have any outstanding questions with regards to VMotion or any VMware technology please do not hesitate to send them to me via ask the experts.
About the author: Andrew Kutz is deeply embedded in the dark, dangerous world of
virtualization. He is a Microsoft Certified Solutions Developer (MCSD), a SANS/GIAC Certified
Windows Security Administrator (GCWN), and a VMware Certified Professional (VCP) in VI3. Andrew
also recently submitted a security practical entitled "Sudo for Windows (sudowin)" to the SANS
organization where it was accepted, thereby granting Andrew SANS GOLD status.
This was first published in October 2007