How many VMware ESX Server guest operating systems can your hardware handle? How can you compare the performance of a dual-core server running at 3.2 GHz with a quad-core server running at 2 GHz? When the storage components of your guest OSes migrate from the old Fibre Channel (FC) storage system, to the new iSCSI storage system, how does the move affect performance? To answer these questions, virtualization administrators can run benchmarks on sets of hardware, compare the results and discover which aspects of overall performance change.
Benchmarks help compare the performance of different hardware running a single application. But traditional performance and benchmarking applications don't understand the concept of virtualization and the effect of virtual guests on server performance. A virtual environment --which often entails server consolidation -- rarely has a single application on one server. So benchmarking virtualization-tasked hardware requires an application that can create a unique and consistent load on a physical server that can then measure the performance of the underlying hardware as it changes. Hardware changes could simply involve swapping out a server with a different brand, changing the storage system or tweaking the hardware of the existing server.
Enter VMware's VMmark, a free benchmarking application for VMware ESX Server and VMware Infrastructure. I'll explain VMmark's benefits, requirements and availability in this tip.
How do I obtain and use VMmark?
You can download VMmark for free at the download VMmark website. The only requirement is that you must have a VMware login (also free). Once you get to the download site, there are a number of options for download. The core package is the VMmark Kit (circled below). There are also templates for several different types of servers.
Click image to enlarge
After downloading the 30MB file and unzipping it, I had a number of folders. Luckily though, I had a Readme file as well in that group of folders.
The Readme file contains the docs folder, which is where you should start. The vclient folder is what will be installed on all VMmark client systems. The vserver directory contains the applications that will need to be installed on the virtual machines (VMs) that will be benchmarked. Lastly, the VMmark folder contains the applications needed to be installed on the "prime client".
VMmark components and benefits
VMmark supports open standard and platform neutrality. It is a non-commercial product and is agnostic toward the underlying hardware that VMware ESX runs on. Furthermore, VMware is working with the standard performance evaluation corporation (SPEC) to develop a standard for virtualization benchmarking.
VMware calls the benchmarking done by VMmark "application-centric benchmarking." This is because the results provided are based on a certain number of real-world applications. VMmark is also a "tile-based implementation." A tile is a "unit of work in a virtualized environment, consisting of a collection of virtual machines running diverse workloads." The graphic below illustrates tiles on a hypervisor layer:
Graphic provided from VMware.com
VMmark employs what VMware calls "Precision Scoring Methodology." According to VMware, results from VMmark's benchmarking are very precise, the result of multi-hour testing with frequent samples. All of that data, from every virtual guest being measured, is summarized and the benchmarking scores are then calculated.
One of the potential downsides to VMmark are the requirements to put it in place. Implementing VMmark is no quick task as you need a fair amount of hardware and software installed. You need at least a 2 CPU server with 6GB of RAM and 15GB of free disk space. You will also need a private network that connects that server to PC clients – one PC for every "tile". There is a long list of VMmark software requirements that you will need. You can see the full list of requirements at the VMmark requirements website .
Obviously, installing VMmark and performing full server benchmarking is no small task and certainly is beyond the scope of this article. Stay tuned for more "VMmark how-to" articles from me in the future. You can find a very impressive white paper demonstrating how VMware's VMmark can be used for enterprise-grade benchmarking at VMmark: A Scaleable Benchmark for Virtualized Systems.
In Summary
While VMmark probably isn't for new VMware system admins, it is ideal for hardware vendors, consultants, and virtualization admins who need to do hardware comparisons and "what-if" scenario benchmarking. It's great that VMware offers VMmark for free and I look forward to using it more myself. If you have links, documents or virtual appliances that you can recommend to make VMmark easier to implement for readers of SearchVMware.com, please email them to the editor.
About the author: David Davis (CCIE #9369, VCP, CWNA, MCSE, CISSP, Linux+, CEH) is the Director of Infrastructure at Train Signal, Inc. He has written hundreds of articles and six video training courses – including the Train Signal VMware ESX Server video training series. His websites are Happy Router.com and VMwareVideos.com.
