Determining how to store virtual machines is one of the most common vSphere decisions. Should you use shared or local storage? Block or network-attached storage? What cluster file system works best: Virtual Machine File System (VMFS) or Raw Device Mapping (RDM)?
We'll focus here on the differences
How Virtual Machine File System works
A VMFS data store is the default way for the VMkernel to handle disks; the disk is partitioned and formatted by the VMkernel and nothing but the VMkernel can read the disk, now called a data store. The advantage of VMFS is that a single disk -- logical unit number (LUN) in storage-area network (SAN) terms -- can hold multiple virtual machines.
How many virtual machines (VMs) to assign per LUN is an age-old debate, but an average number would be a dozen VMs sharing one data store. Essentially, a data store can hold multiple VMs and can hold all of the files that make up each VM. These files include the VMX file that lists the VM hardware configuration, the VMDK files that are the VM's hard disks and the other sundry files that make up the VM.
How Raw Device Mapping works
With RDM, the VMkernel doesn't format the LUN; instead, the VM guest OS formats the LUN. Each RDM is a single VM hard disk and is usually attached to a single VM. An RDM takes the place of a VMDK file for a VM. This is where the VM's disk contents are stored. But this is not where the files that make up the VM are stored: These files need to be stored on a data store separate from the RDM.
RDMs are sometimes deployed based on the belief that they offer better performance since there is less file system overhead than VMDK files on VMFS. But, in some uses, RDMs are a little slower than VMFS. If a VM needs top disk performance, then dedicate a data store to the VMDK file.
How to choose between VMFS and RDM
There are a few things that require RDMs in vSphere:
- Microsoft Failover Cluster Services. MSCS uses shared disks to build a cluster out of VMs on different ESXi hosts. The shared disks cannot be VMDK files; RDMs are required if your storage is Fibre Channel. Check VMware's guidance on MSCS in VMs since it can be tricky to configure. Also, be sure you really need to use MSCS when vSphere HA isn't enough.
- Storage-area network Quality of Service. For the SAN fabric to apply QoS to traffic from one VM -- not the ESXi server -- the VM must use a unique Fibre Channel ID using a feature called N_Port Identity Virtualization (NPIV). NPIV only applies when the VM disk is an RDM.
- Managing some Fibre Channel storage from a VM. Some storage arrays are controlled using LUNs over the Fibre Channel network. To run the configuration software inside a VM, these control LUNs must be presented to the VM as RDMs. (This is not common; I've seen it only on high-end EMC storage.)
- Big VM disks. The largest VMDK file you can create is 2TB, but a single RDM can be up to 64TB. You need to decide if a VM with a huge disk is a good choice when you factor the backup size and how long it would take to do a restore.
The biggest limitation with RDMs is that the one LUN is only one VM disk. With a data store, the LUN could hold 20 VM disks in VMDK files. RDM can be very limiting, since an ESXi server can only handle 255 LUNs and the whole DRS and HA cluster should see the same LUNs.
Using all RDMs means there is only room for 254 RDM VM disks, plus one data store for the VM files. With VMFS data stores, the 255 LUNs could hold thousands of VM disks.
The option to use an RDM may be necessary in some situations, but your default choice when possible should be to use VMFS and store VM disks in VMDK files.
This was first published in August 2013