The components of VMware Infrastructure 3 (VI3), ESX 3 and Virtual Center 2 (VC2) all run a Tomcat Web server that provides access to the VI3 SDK. This Web server restricts access to the SDK via SSL and uses self-signed certificates from VMware. In this article, I will describe the different approaches that can circumvent some of the problems associated with this configuration.
Imagine a scenario in which you run your company's VI3 installation, and a developer in another department is writing some custom reporting tools for it using the VI3 SDK. She has asked you to install some SSL certificates so that her custom program won't bomb out. You tell her that you are pretty sure that VI3 comes with SSL certificates installed, but she says that her code fails because it cannot negotiate a secure connection with VI3. Are you wrong about VI3 shipping with SSL capabilities? And if so, how do you install an SSL certificate so that the developer can connect successfully to the server?
Actually, you are correct that VI3 ships with SSL enabled, but you should also believe your developer when she says that she cannot connect to VI3 successfully.
It is important to remember that VI3 is actually comprised of two primary components, the ESX 3 servers and the VC 2 server that manages them. Both ESX 3 and VC 2 run an instance of the Tomcat Web server that provides the access endpoint for the VI3 SDK. Your developer could actually connect to either the Web server running on the ESX 3 servers or the Web server running on the VC 2 server with the same code. That is, if she could connect at all. The Tomcat Web server is configured for SSL out of the box, but it uses a VMware self-signed certificate whose root CA will very likely not be listed in any common trusted CA store. Because the client -- in this case, the software attempting to access the SDK -- does not trust the certificate used by the Web service that exposes the SDK, the client will fail to connect to the Web service with a SSL handshake error.
There are three solutions for this problem: 1) turn off the SSL requirement for the VI3 SDK, 2) explicitly trust the VMware self-signed certificate or 3) install certificates on the ESX 3 and VC 2 servers that are signed by a trusted CA.
The first solution, to turn off the SSL requirement for the VI3 SDK, is a security hole. I do not recommend it, but I present it regardless because it is a solution to the problem and should not be ignored because it is unfavorable. To disable the SSL requirement on a VirtualCenter server, you need to edit the file C:\Documents and Settings\All Users\Application Data\VMware\VMwareVirtualCenter\vpxd.cfg. There is a section that starts with <proxyDatabase>. Uncomment this section of text in that section:
<!-- <server id="1"> --> <!-- <namespace> /sdk </namespace> --> <!-- <host> localhost </host> --> <!-- <port> -2 </port> --> <!-- </server> -->
Once uncommented, save the file and restart the VC Management Service. The VI3 SDK will now respond over with non-SSL over port 80 on the VC server.
To enable this on the ESX 3 servers, you need to edit the file /etc/vmware/hostd/config.xml and remove this bit of text:
<redirect id="2"> /sdk </redirect>
Save the file, and then restart the vmware-hostd daemon. If you invoke the command 'lsof | grep LISTEN', you will see that the process listening on port 443 is vmware-hostd, but there is no init file in /etc/init.d for vmware-hostd. If you grepped through /etc/init.d, you would find that the init file /etc/init.d/mgmt-vmware is actually used to start the vmware-hostd daemon, so that is the init file we want to use. Invoke /etc/init.d/mgmt-vmware restart to restart vmware-hostd.
The problem with this approach is that all the information attained via the SDK is traveling in the clear (unless your company implements a Layer-2 security measure like IPSec). To prevent this security risk, you should use solutions two or three.
The second solution is for people who need to access the VI3 SDK on ESX 3 servers and VC 2 to download the certificates from those servers and configure them as explicitly trusted certificates. Keep in mind, however, that software developed using this practice will fail to connect to the VI3 SDK if the security principal that runs that software does not trust the certificates on the ESX 3 servers and VirtualCenter2 server. This problem is rectified using the third solution.
The third solution is to install new SSL certificates on the ESX 3 servers and the VC 2 server. If your company has its own CA, you can just generate SSL certificates with that or you can always purchase them from Verisign. The key is to use certificates that are signed by a CA that is already trusted by all the security principals that will be used to run the developer's software. Once you have the certificates and they are in a base-64 encoded PEM format, we can proceed.
The first step to installing a new certificate on a VC 2 server is to back up the old certificate. The old certificate is located in C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\SSL. The private key is named rui.key and the public certificate is named rui.crt. Rename the existing files as rui.key.old and rui.crt.old. Now upload the new key file and certificate and move them into /etc/vmware/ssl. Rename the new key file to rui.key and rename the new certificate to rui.crt.
There is a security flaw out-of-the-box with the original key file. It is not set to read-only by SYSTEM and Administrators. Set the permissions on rui.key.old and rui.key so that only the SYSTEM user and the Administrators group have read access to them. Restart the Tomcat Web server by restarting the VC Management Service to load the new certificate.
With ESX 3, as with VC 2, we should first back up the existing certificate. The SSL certificate and its key file for the Tomcat server on ESX 3 are stored in /etc/vmware/ssl. The private key is named rui.key and the public certificate is named rui.crt. Rename the existing files as rui.key.old and rui.crt.old. Now upload the new key file and certificate and move them into /etc/vmware/ssl. Rename the new key file to rui.key and rename the new certificate to rui.crt. Be sure that both the certificate and the key file are owned by user and group root and that the permissions on the key file 400 (owner read) and the permissions on the certificate are 644 (owner read/write, group read, others read). Next, restart the Tomcat Web server. Invoke /etc/init.d/mgmt-vmware restart, and this will cause the Tomcat Web server to use the new certificate we just put into place.
Repeat this process for the rest of your ESX 3 servers.
In conclusion, I hope that I have shown you the various methods that can be used to get around a real-life problem scenario caused by the VI3 self-signed certificates.
Andrew Kutz is deeply embedded in the dark, dangerous world of virtualization. Andrew is an avid fan of .NET, open source, Terminal Services, coding and comics. He is a Microsoft Certified Solutions Developer (MCSD) and a SANS/GIAC Certified Windows Security Administrator (GCWN). Andrew graduated from the University of Texas at Austin with a B.A. in Ancient History and Classical Civilization and currently lives in Austin, Tex., with his wife Mandy and their two puppies, Lucy and CJ.