Once you've made the decision to deploy VMware vSAN as part of your next vSphere cluster, you must choose which...
storage devices to put in your ESXi hosts. Typically, your storage vendor chooses the balance of parts in a storage array, but with vSAN, it's up to you to pick the right components.
The right selection of vSAN storage media will deliver the capacity and performance that your VMs require at a significant cost savings compared to a separate storage area network, as well as equal or better reliability.
VMware doesn't publish a compatibility guide for individual vSAN storage media, like solid-state drives (SSD) and hard drives, the way it does for storage controllers. The guidance VMware does provide focuses on the endurance and performance of SSDs. Endurance is the volume of data writes the device can perform without failing. SSDs wear out as a result of writes, not reads. Performance is usually transactional and is measured in the number of IOPS rather than as throughput. Higher endurance and performance typically come together and translate to higher cost per GB of capacity.
Are we talking tiers or caches?
While we often talk about tiers of drives in vSAN, we aren't referring to storage tiering. VMware vSAN uses two tiers of drive: a performance drive tier and a capacity drive tier. However, the performance drives do not count towards the capacity of your vSAN data store and are not the persistent location for your data -- so, the performance drives are a cache, not a tier. VMware vSAN data resides permanently only in the capacity drives and is copied rather than moved to the performance drives to improve performance. From here on out, I will talk about layers rather than tiers: the performance layer and the capacity layer.
Performance and capacity layers
The performance layer acts as a write buffer and a read cache; the aim is that most VM I/O will happen on the performance drives. The performance layer needs to be fast and must consist of SSDs. The performance layer only needs to be large enough to hold the active set of disk blocks in use. The rule of thumb is to have a performance layer that is 10% the size of your capacity layer.
The capacity layer is the permanent storage location for your data. This layer needs enough capacity for all of your data, plus the redundant copies necessary for availability. A hybrid vSAN configuration uses hard disks for capacity, while an all-flash configuration uses larger SSDs for capacity. With all-flash and the VMware vSAN Advanced license, you can use deduplication, compression and erasure coding to reduce the required physical capacity. In many cases, the extra cost of the Advanced license is more than offset by the reduced cost of solid-state capacity for the layer. The capacity layer also needs adequate performance. The capacity layer needs to be able to drain the writes out of the performance layer before the performance layer runs out of capacity. As with all write caches, it smoothes performance peaks. The persistent storage must be able to keep up with the total I/O load.
How to choose SSDs
Capacity, performance and endurance all drive the cost of an SSD. You need enough of each for your workload, but there's no point in paying for more than you'll use. The performance layer needs high performance and endurance, but only a fraction of your total capacity. The capacity layer needs capacity, but probably far less performance and endurance. Fast Peripheral Component Interconnect Express or nonvolatile memory express flash suits the performance layer, while larger and cheaper serial-attached SCSI or Serial Advanced Technology Attachment flash are a better fit for the capacity layer. Keep in mind that your storage only needs to be fast enough for your workload; if your workload only needs 20,000 IOPS and you bought 200,000 IOPS, then you probably spent too much money. One feature that you need in both layers is power loss protection. This feature ensures data protection in an unexpected power loss and avoids any possibility of data corruption.
How to choose hard drives
Hard drives have fallen out of favor as a place to store VMs but offer the most capacity for your dollar. The downside to hard drives is that they don't provide the same performance as flash storage. To drain the writes out of the performance layer, you need a larger number of hard disks than you do SSDs. Beware of large hard disks that use shingled magnetic recording. These drives can be very slow to overwrite existing data and, as such, have variable write latency.
When you design a VMware vSAN deployment, you take responsibility for the performance and capacity of your vSAN storage media. Using drives with the right mix of performance, endurance and capacity for your workload will provide the best value for the money you spend. As always, the more you know about your applications' demands, the easier it is to choose the right vSAN storage media.
Data storage growth stays a step ahead of capacity
How to survive data storage capacity growth
Tips to improve private cloud storage performance