Part of an IT professional's job sometimes involves being creative, as was the case with our contributor and his...
task: converting a remote data center's physical servers to virtual machines -- without the ability to have anyone on-site. Through a variety of creative measures and much trial and error, he and his team successfully converted the data center's physical machines.
This article recaps his experience and shares his greatest takeaways involving networking, the ins and outs of VMware Converter, estimating conversion times, cleaning up snapshots post-conversion, the importance of testing, training a crew that had little VMware experience, and using VMware support.
After many, many months and much harrassment about how long the project had been open, I finally finished my last physical-to-virtual (P2V) conversion at our off-site disaster recovery location. What better way to finally relax than having a big fudge cupcake and vanilla ice cream while watching "The Biggest Loser" followed by a guilt-inspired brisk walk with my wife in the cool night air?
While enjoying the bit of exercise and the soothing smell of blossoming lilacs, it occurred to me that I could share some of what I have learned during this project to hopefully benefit others. Hey, I can't be the only one that tries most of the wrong paths before finding the right one!
This project involved converting all the physical servers in a remote location to virtual servers in a VMware ESX High Availability (HA) cluster, without being able to get on a plane and do the work on-site. To accomplish this, I needed to use our local employees (excellent people, albeit lacking VMware experience) to assist with setting up and configuring ESX hosts and the storage area network (SAN), and then use the various VMware Converter tools for the actual P2V migrations. So, here is a list of some of the lessons I learned the hard way during the process of this remote physical-to-virtual migration.
A (cabling) picture is worth 1,000 words
Thankfully, the hardware configuration in our remote office is very similar to what I have available locally. To aid in the set up and configuration of the SAN and ESX hosts, I took photographs of my local set up, imported them into Gimp to overlay the components and connection ports with labels, including which network interface cards (NICs) should be assigned to which network segments (for failover, redundancy) and how to cable the host bus adapters (HBAs). The pictures made the cabling much easier.
The devil is in the (VMware Converter) details
Even when you think you have outlined a bullet-proof process, there are steps that can be missed when someone else is trying to follow it. I found this out when using VMware Converter. Failing to specify the need to select the appropriate virtual switch for the network segment can cause problems when doing cold cloning from a bootable CD-ROM. Once you have the instructions, trying to follow them as if you had no background in the process is a good way to see if the instructions will actually be sufficient for your on-site resources.
Estimating conversion times: Are we there yet?
Estimating conversion times is part science, part guesswork. We were able to use some benchmarks of local network traffic from previous conversions, but they were only marginally useful. Each physical server was unique: different processors, memory, hard drive speed and disk configurations, and we also had to factor in whether or not the drives were being resized during the conversion. Furthermore, there were time differences based on whether the conversion was done at the block or file level. Towards the end, my time estimates became: "It will take as long as it takes, times two."
Don't forget to clean up snapshots
Taking a snapshot before removing physical hardware, specific drivers and support applications is a great thing to do in case something goes wrong. However, the ease of creating snapshots prior to each change means that it is easy for them to eat up a lot of space on your SAN. Therefore, part of our final P2V verification procedure involves removing all remaining snapshots once the conversion has been tested and verified.
Testing is key
The beauty of VMware is the flexibility it provides in using your resources. As part of our P2V verification, we tested the VMotion capabilities of each VM prior to deeming the conversion a success. This test can be revealing: Do you really have virtual switch redundancy on all the ESX hosts? Do the port group names match? How well does the new VM play with other VMs on the same ESX host? Is the CD-ROM configured correctly so that Distributed Resource Scheduler (DRS) can occur as configured? Only by testing these functions can you know if your newly-converted virtual machines are truly providing all of the benefits of a virtual environment.
I'd also suggest setting up a small test environment to get comfortable with what you are trying to do. If you have done it yourself it will be much easier to explain how to do it to others (particularly if there isn't any kind of remote access port such as HP's Integrated Lights Out and you're relying on your documentation, screenshots and imperfect memory to walk through the process.) Explain to others on the project that the project is not an exact science and that they should to try to be flexible.
Be creative to solve problems
In the rush of trying to get things done, it can be easy to make assumptions about what your on-site resources know and understand, which inevitably leads to confusion and frustration. I found it far better to explain not only what needed to be done, but how and why. For example, we discovered that the buttons and menu names differ between versions of VMware Converter; knowing why a person needed to make a certain selection helps gloss over these inconsistencies. Also, it is important that your resources are able to think on their feet. For example, when we started one of the cold conversions, we ran into difficulties because of insufficient RAM. The physical server was a Windows 2000 Server with 256 MB RAM -- not exactly a supercomputer (my iPod is more powerful than that), but the server could still function for many years. After hitting this snag, our on-site engineer was able to scrounge up a 128 MB chip out of a similar server from our junk pile. He added it in to our source server, which brought us up to 384 MB -- enough for VMware Converter 3.0.3 to run successfully.
Using VMware forums and support
Have an up-to-date VMware support agreement. Nobody can remember it all. Forums help, but they can only take you so far. For example, we had just converted our last VM, but it wouldn't boot. I suspected that it may be due to a hard disk conversion sequence problem. The VMware forums provided a solution that said to switch around the hard disks in the ESX console. It didn't explain, however, how to switch them.
A 15 minute support call to VMware and a WebEx session had me up and running. The alternative was to spend hours digging around going through my VMware Certified Professional Administration docs, but the clock was ticking.
Sometimes it is necessary to ask for help to keep the momentum going. Having a current support contract with VMware is on the top of my list of requirements prior to starting any big VMware project.
Keep learning and write it down
Each project or task will teach you something new. On my first conversion, I managed to confuse the functionality of virtual switches with open systems interconnection (OSI) level 3 functionality and couldn't understand why sometimes the VM worked and other times it didn't. It turned out that powering on the VM was causing something of a round robin on the switch that was connected to multiple segments, and it was randomly connecting to the correct port group. This situation is nearly impossible to troubleshoot, especially when you are really new to the game. Performing a cranial archeological dig and remembering the telecom layer functionality of switches and routers finally made it click. By my final P2V conversion, I thought I had it down cold. But as you know, that's the conversion I called VMware support for help with.
Take the time to write down the lessons you learn. Since starting this project, I have put together a document of over 100 pages of screenshots, how-tos and tricks of the trade typically learned by trying all the wrong sequences first, then hitting on the one good permutation. If your job is like mine, you probably have several big projects going simultaneously (the ones in all caps on your whiteboard) that are interleaved with countless daily tasks that really aren't that important, but someone who is not a bottom-feeder believes there might be a catastrophic hole in the universe if they aren't completed by "End of Day."
I hope you can benefit from some of the lessons I have learned during this project. Now that I've completed my conversions, I get to experience the satisfaction of checking off one of the all-capitalized items on the whiteboard, catch up on exercising, make my Espresso-Whey-Oatmeal-Banana protein bombs, enjoy the lilacs and contemplate my next VMware challenge.
Disclaimer: The information, views, and opinions expressed in this article are solely those of the author and do not represent the research, views or opinions of NYCE Payments Network, LLC. NYCE Payments Network, LLC is not responsible and cannot be held accountable for any information, views, or opinions expressed in this article. The author takes full responsibility for the information, views, and opinions presented.
Mak King has been in the IT industry for 14 years, progressing from his blissfully green days of DOS and sneakernet to VMware and storage area networks. He has certifications from Netware (CNE), Microsoft (MCP), CompTIA (iNet) and VMware (VCP Virtual Infrastructure 3). He is the virtualization and directory services subject matter expert for NYCE Payments Network, LLC (a Metavante company), where he has been employed for over 10 years.