As VMware continues to grow, the software giant is shipping more appliances than ever. Currently, when you choose...
the Windows version of a VMware offering, there's a fair chance that you'll only be able to choose an appliance with a future version. Therefore, it's a good idea to be prepared for troubleshooting these virtual machines and to know how to configure features that are not available through the Web management interface.
Instead of troubleshooting a VMware appliance, you could choose to delete it and deploy a new one, but that's not always practical, nor is it the best choice if your data is stored inside the appliance itself. The best option is to have a backup so you can restore yesterday's working version of the appliance but sometimes troubleshooting is worth the effort. Read these practical tips to figure out where to get started.
The anatomy of a VMware appliance
Most of the recent appliances shipped by VMware are based on SUSE Linux Enterprise Server (SLES) 11. You can easily verify which version of Linux your appliance uses. One indicator is the information in the vSphere Web Client VM summary, which shows which OS is installed. However, this isn't always completely accurate. SLES allows you to install another OS or version without any complaint from VMware. To access more accurate information, it's best to log in to the command line and look into the file /etc/SUSE-release with the following command:
This will report that you are using SLES 11 with a given patch level. This information is useful if you would want to troubleshoot an appliance that no longer functions and want to start it with a rescue system.
Not all troubleshooting needs to start with booting from a rescue disc. There are a few scenarios where doing so could be beneficial such has having to reset the root user's password. Let's take a look at how to boot with a rescue disc.
Booting with a rescue disk
To boot with a rescue disk, you must first download a CD or DVD for the same version of Linux running inside your VMware appliance. This is why it's useful to find out which distribution your appliance is running. You should find out this information right after deploying the appliance and have a rescue disc available so you don't have to stress out when there's an actual problem.
It's best to download the same version of SLES -- currently 11 -- but if that is no longer available your best bet is to download a newer one.
Place the ISO file of your download on a data store or on your local hard disk and attach it to the VM. You can connect the ISO file -- don't forget to enable the checkbox to connect at power on -- but the VM will keep booting from its hard disk. Therefore, use one of the two options shown in Figure A to boot from a CD or DVD. Increasing the boot time to, say, 10,000 milliseconds -- 10 seconds -- allows you to press "Escape" in the boot screen or enable the checkbox to have the VM boot into the BIOS at next boot. This setting allows you to modify the BIOS permanently to be able to boot from an ISO file. Be careful with this setting because if you forget to eject the ISO file, it will keep booting from the removable media.
In the SUSE install disk, choose to start the Rescue System. We'll discuss how to change the root user's password later.
When the rescue system has loaded, you can log in as root user without a password.
Accessing the command prompt
As I explained, you don't need a rescue disc to work on your system's command line. In most appliances, you can access the command line on the console of the VM. When accessing the console, the screen might have a menu without a command prompt, but if you press Alt+F1, a new Shell window opens that allows you to log in. Figure D shows what the vCenter Server Appliance (vCSA) login process looks like.
As you can see, this vCenter appliance requires you to enter the commands shell.set –enabled True and shell to access the true Linux Bash Shell.
Once logged in to the appliance, you might want to investigate if the services needed for the appliance are running. For a vCenter appliance, this can be accomplished with the service command and the name of the service:
service vmware-vpxd status
You can use commands to stop, start and restart with the name of the service in much the same way. But what if you don't know which service you're trying to check the status of or restart? The easiest way to find out is to look at the /etc/init.d folder because it lists the startup scripts for all services installed in the appliance. Use the following command to look at the directory listing:
You could also enter ls -l /etc/init.d to see more details.
You should also investigate the log files. The location of the log files varies from appliance to appliance, but a good starting point is the /var/log folder or, specifically, the /var/log/vmware folder. You can use the cat command to view the folder's entire contents, or tail to access just the last ten lines of the file. You can use of the most useful commands – less – to use a browser to look up and down the file. You can search for specific keywords with the following command:
While in the browser screen, press the H key for help, which will tell you to that using the slash allows you to search for keywords.
You can always access the command line on the console, but sometimes it's more practical to access it though Secure Shell (SSH) with a tool such as Putty or another SSH client. For some appliances, you can enable SSH during the deployment, but if you choose not to or choose to enable it when it's not supported, you can still go back and enable it afterward. Be sure to check if permanently enabling this service complies with your organization's security policy and otherwise disable the service after using it. One reason to enable SSH is to use session control protocol (SCP) -- or WinSCP -- to copy files to a VMware appliance.
Use this command to enable the service, also called daemon:
service sshd start (replace "start" with "stop" in order to stop)
To permanently enable it, use this command:
chkconfig sshd on
These commands will change when VMware makes the shift to SLES 12, which uses systemd.
Changing the password for user root
When you no longer know the root user's password, you can use the rescue disc procedure to access the local Linux-system to set a new password. After booting the Rescue System, execute the following commands to show disk assignments and device names:
The root partition is most likely at /dev/sda2 or /dev/sda3, so that is the partition you need to mount. You can do so with this command:
mount /dev/sda2 /mnt
Next, mount the so-called pseudo file systems that are loaded into RAM:
mount -o bind /dev /mnt/dev
mount -o bind /proc /mnt/proc
mount -o bind /sys /mnt/sys
Finally, change the running root file system to the appliance's file system by loading it in a chroot jail:
From now on, all commands you execute will be relative to the installed file system, so you can execute the passwd root command to set a new password for user root. Likewise, you can issue other commands, as they are all relative to the active chroot file system. After making changes, reboot the appliance and access it with the new password.
Working with disks and partitions
You can also use commands to manipulate the file system when logging in. But first, a warning: When making changes to the system, it's a good idea to first create a snapshot of your VMware appliance. VMware offers information on how to do this for a vCenter 6 appliance. This procedure also works for other appliances that use Logical Volume Manager. Otherwise, if you have traditional partitions, you'll need to take a Linux-based approach to decrease the VM's disk size and to increase the partition size with the GParted tool. You could also use a live CD to boot and resize partitions on the hard disk of the machine.
Troubleshooting VMware vCloud Automation Center
The basics of troubleshooting VMware performance
Troubleshooting VMware vSphere Replication