Even though David Williams authored the chapter on programming with VMware APIs in the popular book "VMware Power Scripting Tools," he doesn't claim to be a developer by trade. "My background is on the system administration side. I always saw a need to do things better and smarter. Things that took hours to do I wanted to do in minutes." That mentality led him to try his hand at scripting, Perl, and eventually programming in Java and .NET. Those skills came in handy when he started working with VMware ESX 2.0, back in 2003. "The [VMware] management tools are great, but they only take you to a certain extent. When you're trying to manage large environment, you need tools to automate tedious tasks, usually through scripts using the VI client or directly on the host," Smith said. As principal of the data center optimization consulting practice Williams & Garcia, Williams and his consultants have lots of opportunities to work with the different scripting and automation options available to VMware ESX. And he's also observed how that interface has changed over the years. In this Q&A, Williams describes the common tasks VMware administrators can accomplish with scripting, the different approaches, some best practices, and also how he sees things changing with ESX 3i, the new embedded version.
SearchVMware.com: Why do VMware administrators need to learn scripting or development? Why can't they just manage using VirtualCenter?
David Williams: The biggest thing I see users doing is handling group administration, making mass changes, and also doing change management and configuration management to adhere to IT governance principles.
The thing about VirtualCenter is that even today, most operations are performed against a single object – against a host, against a VM – and that's where the tediousness comes in. There's no concept of group objects. But let's say you have to make a change to 30 VMs – that's an error-prone task, because it's a repetitive task performed by a human.
Can you give some specific examples?
Williams: Sure. For example, let's say you have an environment that contains a lot of VMs, for example PeopleSoft, and you're about to do a new release. Before that, though, you want to take a snap of that environment before making the change. Today, you'd have to dedicate 30 minutes to take snaps of all those VMs, or… you could create a script.
On the group administration side is where a lot of benefit [from automation] can be gained. With it, you can do things like ensure consistent configurations, like the number of VMs for a given project. For example, for every customer, an ASP might want to deploy a web server, a database server, etc. Rather than set up the environment for each customer, you could script it, and just pass the program a couple of parameters.
Then there's the whole paradigm shift to IT governance. [Before], infrastructure was not usually tested for IT governance. But nowadays, when you want to go in and make a change to a host, you're going to have to bring it up in front of change control board, and they're going to say, 'Have you tested this change?' Having a way to script [that change] is usually a good way to satisfy change management requirements.
Scripting is also a way to provide functionality that the management tools don't even have – like creating a workflow model for someone to request a VM, get it approved, created and launched… That's really popular, because today, that process is a really tedious, with a lot of embedded workflow logic.
What are some of the scripting facilities VMware supports today?
Williams: Well, first off, there are shell scripts, which are really easy and that everyone can do. There's also the suite of esxcfg-* tools, which you can chain together and execute on an ESX server. But that's not really scripting; it's pretty rudimentary and basic.
There is also of course the shell – the service console – and while scripting directly on top of ESX is OK, it's not ideal. You can't do everything from shell that you could do from VirtualCenter. Sure, you could do something like have your HBA scan for new storage, but if it's more complex, you'll eventually run in to things where there is no shell script to perform that action. You just don't have all functionality exposed to shell commands. Also, if you have multiple ESX hosts, shell scripting won't give you access to the whole environment in the same way that if you were scripting against VirtualCenter.
For both ESX as well as VirtualCenter, there's also the VMperl and VMcom API, which you can script against. But if you really want the full power of automation, you're going to have to rely on the VMware Web service, i.e., the VMware VI3 SDK. It does require more knowledge, but you get more functionality. From there, you get a larger list of operations you can perform, and you can write the code with anything that can consume a web service: Java, .NET, C#, VB, Perl... The most popular is Java because it allows you to take advantage of all the open source libraries that are available. In the book, I give examples in Perl, Java, and .NET.
Some people have called the VMware APIs "cumbersome." Do you agree?
Williams: Well, everything is relative to your level of programming. But yes, I've worked with some senior programmers that think that the SDK is a cumbersome thing to work with. It's interesting to see the learning curve, the ramp up. They're pretty used to Web services, but then they start working with the VMware APIs and they have to change gears and their way of thinking. The way that it's done, it's different. [The VMware developers] got really creative and think it's a cool product -- and it is a cool project -- but they didn't think about a programmer and the common ways of doing things; instead they thought about the best way of doing things.
But it's evolved. [ESX] 2.5 was very cumbersome to work with. You could do it, but it was very different from the way everyone else was doing things in the web services community. VMware got the feedback and with VI3, totally revised the web service to the point where old code will not work on the new web service. Is it improved? Yes. Is it easier? Yes. Is it perfect? No.
What about doing development against ESX 3.5?
Williams: It's hard to say because 3.5 is still in beta. But I think that the same thing they did between 2.5 to 3.0, they're doing with 3.5. VMware does listen to their customers and their community, so it promises to be easier. 3.5 is not quite a major release, so it will keep compatibility with 3.0.2 SDK, but it will expose some new functionality and do things a little differently. From what I've seen and heard, they did overcome some of the things that were frequent complaints.
Williams: Like the way you have to traverse through the SDK to get down to the objects that you wanted to perform an operation against. Or the amount of exchange between program and the SDK. As it stands, there's a lot of cross talk, back and forth, to finally get down to what you really want. [Fixing] that should improve the performance and flexibility of the application.
My bid is that in 3.5 things become more standard. My gold standard would be a combination of an SDK and your development environment, like Eclipse or Visual Studio. That way, it becomes easier -- more click and drop -- rather than having to bash out a lot of code to have to do a basic operation. Today, it's pretty much development.
What do you think about ESX 3i?
Williams: I think 3i will become popular quickly, because what does it take to get up with 3i? Just a flash device that you pop in to the server, and boom, you're up and running. Compare that with today: you have to go and install [ESX], run all the patches. With 3i, you take out a lot of the complexity and overhead of deploying an ESX server.
But 3i does pull out the rather bloated service console. Since it's rather limited anyway, [most developers] will go to VI. Even if they choose to stick with the [command line interface], they'll still have to run it off the desktop, not off ESX. It's going to be a big change.
But programming on the service console should really only happen if you have no other choice. Doing systems management on the thing you're managing isn't a good idea. It's that whole idea of a dependency injection, where no one object is responsible for all the other objects. You want to create a higher level entity that has the responsibility of managing and linking together the different pieces underneath it. If I was writing systems management scripts, I wouldn't want to run them on ESX, but on a higher entity. You're going to see a lot of that.
Let us know what you think about the story; email: Alex Barrett, News Director.