Problem solve Get help with specific problems with your technologies, process and projects.

Preventing VMware driver-generated errors and VMware Backdoor misuse

The VMware hypervisor interacts with virtual machines via three avenues: paravirtualized drivers, regular drivers and VMware Tools. Regulating interactions with certain configurations can prevent hypervisor errors and misuse of the connection VMware Tools has with VMware Backdoor.

There is ample information available about securing operating systems running on VMware ESX and ESXi systems but not as much about possible security vulnerabilities that can occur due to the way the VMware hypervisor and the guest OSes interact. These interactions occur via three channels: paravirtualized drivers, regular drivers and VMware Tools. As such, there are two areas system administrators need to consider when composing a secure VMware environment.

The first component is the interaction between the hypervisor and the virtual machine (VM). This layer extends into the guest OS via paravirtualized drivers that make up VMware Tools, which may be news for concern to those who didn't previously know about it. The second component is the guest OS itself. Each guest OS has security hardening scripts, guides and benchmarks that should be followed, yet none of these guides, scripts or benchmarks truly take the way the virtualization layer and the guest OS interact into account, hence why the first component affects the second. There are certain configurations you can use, however, that will help to create more secure interactions between the VMware hypervisor and its virtual machines.

Hardening the guest OS
Hardening the guest OS will not be discussed within this tip, but here are some reference links that will be useful if you need to harden the guest OS:

Apply one or all of these benchmarks or guides to your virtual machines. Remember that the vSwitch does not contain a built-in firewall so once the guest OS is connected to a network it must protect itself.

Interaction between the hypervisor and the virtual machine
The interaction between the hypervisor and the VM occurs through three components: paravirtualized drivers, normal drivers and the VMware Backdoor.

Paravirtualized drivers are drivers that know they are running within a virtual machine and either use out of band communication with devices (perhaps through the VMware Backdoor) or take advantage of this to use code specific to the virtualization host in use. For example, within a VMware Guest, the vmxnet driver is paravirtualized. This driver gains some performance advantages due to this knowledge. The Virtual Machine Interface (VMI) proposed by VMware will make writing paravirtualized drivers for Linux much easier and nearly transparent. Poorly written paravirtualized drivers can cause crashes that could be an attempt to escape the VM into the hypervisor. While this is not currently possible it is best to vet all paravirtualized drivers before putting them in use. In essence only use paravirtualized drivers from known sources i.e. VMware.

Normal drivers do not know they are running upon a hypervisor and often require translation by the hypervisor to the underlying hardware in use. These drivers interact with the guest OS kernel only and the guest OS kernel interacts with the hypervisor through normal means. In some cases, it is possible that the hypervisor does not know about the command issued by the driver (or to the driver) and will show errors within the per-VM vmware.log to this effect. Functionality suffers, but this may not be noticeable to the VM. In other cases it could cause the VM to crash. For example, the vmkernel, VMware's hypervisor, did not implement every SCSI instruction or command available. Some of the esoteric commands would cause errors to appear within the vmware.log file. In odd cases, the VM would panic.

VMware Backdoor
The VMware Backdoor is the one that confuses, and most people think gets abused. Some simple settings within the VM, however, will protect the VMware Backdoor. The backdoor gives an alternative path to the hypervisor and is used to provide out of band communication between the VM and the hypervisor. The VMware Backdoor is mainly used by VMware Tools.

VMware Tools can run as any user, and therefore any user can issue many of the VMware Tools commands that run through the VMware Backdoor. The average user does not need access to VMware Tools to execute commands, so VMware Tools access should be restricted to administrators when on VMware ESX. Unfortunately, the VMware Backdoor is available to anyone and cannot be closed off from within the guest OS at this time.

Securing the VMware Backdoor
The main way to secure the VMware Backdoor is to disable functionality using advanced VMware configuration settings. Each of the current security standards suggest subsets of all the different isolation options to set and none of them agree on the minimal set required which is actually all the ones listed. Here are some settings that should be set within the Advanced Options under each VM's settings (or added directly into each VM's .vmx file.)


  • Disable copy from remote console of a VM to the workstation: => false

  • Disable paste from workstation into remote console of VM: => false

  • Disable changing screen resolution and depth: => false

CISecurity ESX Benchmark

  • Disable copy from remote console of a VM to the workstation: => false

  • Disable paste from workstation into remote console of VM: => false

  • Disable changing screen resolution and depth: => false

  • Disable the ability for the VMware Tools to make some configuration changes => true

VMware VI3.5 hardening guidelines

  • Disable some aspects of logging during the life of the VM to vmware.log. This greatly reduces logging but does not entirely remove it. This could help with disk I/O issues. => true

  • Rotate the vmware.log every specified bytes, else the vmware.log file can grow very large
    log.rotatesize => 100000

  • Keep only the specified number of historical vmware.log files, else an infinite number will be kept, which can take up quite a bit of disk space. On a VMFS it can quickly reach the 32 K file limit.
    log.keepold => 10

  • Limit how much data can be sent to the VMware Back door.
    tools.setinfo.sizeLimit => 1048576

  • Disable the ability to set some of the information from within the VM about the VM through the VMware Backdoor => true

  • Disable the ability for the VM to set the connection state through the VMware Backdoor of those aspects of the virtual hardware that can be connected and disconnected (floppy, CD-ROM, network, etc.) => true => true

  • Disable the ability of the VM to call diskwiper routines through the VMware Backdoor. => true
Depending on security requirements, all of these options should be set as they will secure interactions between guests and VMware remote console hosts as well as between the virtual machine, the hypervisor and the file system. These settings can prevent some very interesting (and sometimes confusing) hypervisor errors caused by lack of space.

Protecting the VMware Backdoor is very important. I must reiterate that it is best to limit access to the tools, but not the drivers, as appropriate. Also, the implementation of mandatory access controls such as Window's User Access Control (UAC) and SELinux within the guest OS can limit who and what can access the VMware Backdoor.

Security of the VM is mostly within the guest OS, but virtual hardware settings can make a large difference as well. Make a practice of reading the current hardening guidelines, benchmarks and checklists to aid in properly securing your virtual machines.

ABOUT THE AUTHOR: Edward L. Haletky is the author of VMware ESX Server in the Enterprise: Planning and Securing Virtualization Servers. He recently left Hewlett-Packard, where he worked on the virtualization, Linux and High-Performance Computing teams. Haletky owns AstroArch Consulting, Inc. and is a champion and moderator for the VMware Communities Forums.

Dig Deeper on VMware and networking

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.