Remember the email server or payroll system that you virtualized? Someone with administrator access to your virtual...
environment could easily swipe it and all the data without anybody knowing. Stealing a physical server out of a data center is very difficult and is sure to be noticed, stealing a virtual machine (VM), however, can be done from anywhere on your network, and someone could easily walk out with it on a flash drive in their pocket.
Virtualization offers many benefits over physical servers, but there are some pitfalls you should be aware of and protect against to avoid losing sensitive data. Because a virtual machine is encapsulated into a single virtual disk file that resides on a virtual host server it is not all that difficult for someone with the appropriate access to make a copy of that disk file and access any of the data on it. This is a fairly simple thing to do, and we will show you how to do it here so you can protect your environment against it.
There are basically two ways one could access the virtual disk (.vmdk) file of a virtual machine. The first would be using the ESX Service Console. If someone knew the root password or had a user account on the host, they could gain access to the VMFS volumes that contain the virtual machine files and use copy tools like Secure Copy, or SCP, to copy files from it. The second is using the vSphere/VMware Infrastructure Client which contains a built-in datastore browser; this is the method we will cover here.
Step 1: Snapshot the VM
The first thing you need to do is take a snapshot of the VM. This makes the VM's virtual disk read-only and creates a new delta file for any new writes to the disk. You must do this step because the VM is powered on and using its disk file -- if you do not do it you will not be able to copy the VMDK file because it is locked. This is the same method that many virtual backup applications use when backing up virtual machines. If the VM is powered off or suspended then there is no need for this step.
Step 2: Copy the virtual disk
The next thing someone would do is make a copy of the virtual disk file. You have the option to copy the configuration file (.vmx) for the VM to make it easier to import into Player or Workstation. The virtual disk file (.vmdk) actually consists of two files, a larger one that contains the blocks of the disk and a smaller one that is a text file that is the disk descriptor file. If you use the vSphere client built-in datastore browser these files will appear as one and will both be copied, if you use another copy program like WinSCP you will need to copy both of them.
Browse the datastore that the VM is on; go to the VM's home directory, select the files and then select Download. You can then select a folder on your PC that you are running the vSphere client on to download the files to. Depending on the size of the virtual disk file and your network speed the copy can take either minutes or hours. Once the file downloads have completed you're ready for the final step. You can now delete (commit) the VM snapshot since you have finished copying the disk file.
Step 3: Access the virtual disk file
Now that you have the virtual disk file you can access the data on it. You can either copy it to a flash drive and take it home with you or access it on the PC you downloaded it to. There are two methods you can use to access the data:
- Mount the virtual disk file on your PC using the vmware-mount command from VMware's Virtual Disk Development Kit (VDDK). The virtual disk will appear as a drive letter on your PC which you can browse. This method just allows you to access the files on the VM and doesn't allow you to run any applications on it.
- Import the VM into VMware Player, Workstation or Server and power it on. One roadblock to this method can be if you do not have any credentials to log in to the operating system. To get around that you can boot from a Live CD and either crack/change the administrator password or access the file system directly.
Let's do a step-by-step of each method:
First step for both methods
Because we created a snapshot of the VM, we need to change the configuration (.vmx) file of the VM to make it forget about it. To do this, simply edit it in a text editor. Locate the line that begins with "scsi0:0.fileName" and remove the -000001 from the filename and save the file.
Method 1: Mounting the disk
- First you need to download the VDDK from VMware's website. It's available as either a 33 MB Windows download or a 49 MB Linux download.
- Next, install the VDDK. Typically this will be in C:Program FilesVMwareVMware Virtual Disk Development Kit.
- The vmware-mount.exe command is located in the bin sub-directory; open a CMD prompt and switch to that directory. You can type vmware-mount /? to see the options for the command. To mount a disk file use this format: vmware-mount.exe
- Once it mounts, you can use the vmware-mount.exe /L command to see the mount, or simply go to the new drive letter and do a directory listing.
- Now that the disk is mounted you can access anything you want on it just as if it was a local drive on your PC.
Method 2: Import the VM
- For this method you need to import the VM into a product that can read it. The easiest way is to use the free VMware Player, which is available for both Windows and Linux. You could also use VMware Server or Workstation instead. Download and install one of these products if you do not already have it installed.
- For this example we'll use VMware Player. Once you start the application select Open a Virtual Machine and browse to your VM's .vmx file on your PC.
- Now just click the Play Virtual Machine link and the VM will power on. You will most likely get a message stating that the VM has been moved or copied because some of its file are missing and some info in the configuration file doesn't match its new home. Just select "I copied it" and click OK, the missing files will be created automatically and the configuration file updated. You may also get a VMware Tools message even if it was previously installed; the reason for this is the ESX version is slightly different from the hosted products like Player. You can either choose to download it or not, it won't affect the VM all that much.
- Once the VM is up you can log in to it if you have credentials to it, if you wish to disable the network for it just click Settings in player, select the virtual network interface card (NIC) and uncheck Connected, otherwise you'll need to reconfigure the networking inside the OS so it's configured similarly (IP address/gateway, etc.) to the PC you are running it on.
- If you don't know the administrator's password you can attempt to crack or reset it using some of the many tools available on Live CD's such as PC Login Now, Ophcrack or Offline NT Password and Registry Editor, once you have downloaded them and extract the ISO file you can simply mount it on the VM's CD-ROM drive so it can boot from it instead of the hard disk. Then when you power on the VM, you need to quickly click inside the VM windows and press ESC to get the boot menu so you can select to boot from the CD-ROM. Instead of using a utility like this you could also use a Live CD like the Ultimate Boot CD or Knoppix that supports browsing the file system of the VM, you can get a full list of them here.
So there you have it, it's really not all that difficult to do once you know how to do it. It is fairly easy for anyone with the appropriate access to steal sensitive data on a VM which is why you should protect the raw .vmdk files of your virtual machines at all costs.
Currently with vSphere there is no native encryption of .vmdk files, but .vmdk file encryption was introduced in the Workstation 7 release so it may eventually make it to vSphere. Not all administrators will have the urge to snoop on sensitive data but ensuring that you protect against it is good insurance. Here are some ways that you can protect against this threat:
- Protect sensitive data at the operating system or application layer if possible; this is typically done using encryption. This isn't always an option though and typically consumes extra host resources because of the encryption overhead. That way even if someone gets a hold of the virtual disk it will be extremely difficult for them to read the contents of it.
- Restrict access inside vCenter Server to privileges that allow file operations of the host datastores. You don't have to take away the Browse Datastore privilege from a user role; you can allow that, but take away the Low Level File Operations which prevents downloading, copying and renaming of files. Not everyone that has access to vCenter Server requires this type of access so be sure and only grant it to those that absolutely need it. Resist granting full access inside vCenter Server to users; utilize custom roles to tailor the access to exactly what is required.
- Unfortunately there is no auditing or logging built in to vCenter Server or ESX that will alert you to file operations using the vSphere client. When you use the Datastore Browser the underlying application that is used is Secure File Transfer Protocol (SFTP), there is some output in /var/log/sercure file inside the ESX Service Console, but nothing that will tell you any detail about file downloads.
You might consider implementing a single point of entry application like the HyTrust Appliance that can provide very granular access control and a centralized logging facility for all your hosts. The HyTrust application can provide the missing logging of SCP/SFTP transfers and advanced programming interface (API) calls made from using the Datastore Browser as well as lock down access to the ESX Service Console and vCenter Server.
- Limit access to the ESX service console to people that absolutely need it, you can use sudo to further limit what someone can access and execute inside the service console. For most normal operations there is no need to access the service console as everything can be done using the vSphere Client instead.
The bottom line is there are multiple layers you need to protect to ensure your data is safe. Protect the data, the application, the operating system and the physical server, and make sure you also protect the virtualization layer. Don't focus your security efforts in all those other areas and forget one that can compromise them all. Not understanding and addressing the security challenges that are unique to virtual environments can be a costly mistake that you don't want to make.
Eric Siebert is a 25-year IT veteran with experience in programming, networking, telecom and systems administration. He is a guru-status moderator on the VMware community VMTN forums and maintains VMware-land.com, a VI3 information site.