A few days ago, I was talking about spinning up a new VM to take on some random task, and a fellow Redspin geek jokingly asked if I had ever heard of virtualization sprawl.  I took a second to think about the population of Debian VM’s I had built in the past year;  I had more than doubled the headcount in our server block.  The geek in me says “Spin em up! Disk space is cheap! Cacti loves to make graphs!”, while the security engineer in me says “How the heck are you going to keep all these boxes secure?”

Virtualization is huge, and its here to stay.  It just makes the trigger so easy to pull.  It takes 20 minutes to build a new Debian VM from scratch, or only 3 minutes to copy an image and give it a new IP.  No more hunting for hardware or trying to salvage some old server.  You get an entire server, with a entire stack of services and a fresh operating system at the click of a mouse.  One of the last customer sites I visited had grown their VM farm as well, from 30ish virtual machines in 2008, to 200+ virtual machines in 2009.  While virtualization is fantastic, it comes with a little baggage:

• Machine Management. You’ve gone from 4 VM’s to 400, and you’re a little lost on how to manage them.  Your old policies on managing hardware-based servers don’t apply, and you have everyone in your company begging you to let them spin up a new virtual server.  How can you keep track of machines if you don’t even know they exist in the first place?  How do you limit or control the creation of new virtual machines, without squashing all the sweet benefits that virtualization offers?
• Security. Just because they are virtual doesn’t make them unhackable.  In fact, virtual machines have a higher attack surface than their hardware-based brethren since attacks can be focused either at the hypervisor, or at the VM itself.  They require the same security TLC as a regular server (patching, hardening, GPO policies, auditing, etc) but the speed at which these machines arrive makes it hard to manage.  Are you sure all those windows servers made it into your WSUS schedule?   Mix that in with 12 different operating systems across 3 hypervisors, and you’ve got a playground for conficker.
• Support. Here comes the hard part.  You’re not sure who built the VM, when it got spun up, or who takes care of it – but you know its down because your phone is ringing off the hook.  You don’t know the operating system, let alone admin credentials to even begin troubleshooting.  The VM is a ghost, but obviously a critical ghost.  How can you expect to fire up the engine again when you can’t even pop the hood?
• Cost: One of the main keys of virtualization is to save money.  Less hardware = more in the IT fund…right?  Where hardware costs go down, the cost of licensing can skyrocket.  Since VM’s are so easy to create – sometime they get built out of ease and convenience instead of business needs (or the VM’s built outrun your licensing pool).  Why bog down a domain controller when you can build a new VM to serve up NTP?

So whats an admin to do?  You cant fight the charm of virtualization, but you want to do it right.  As the coming months and years roll by, virtualization is going to get bigger, more affordable, and more mainstream.  If setting up some sort of virtual infrastructure is in your plan, now is the time to start some prep work.  If you already have a ton of VM’s floating around – take a second look at your deployment and management policies and see if they could use a tuneup.  The benefits it brings to your organization can drastically outweigh the cons by considering the following:

• Policies: Like we said before – your old hardware-based policies don’t apply.  Create a guideline to enforce deployment standards.  Specify who can build images, who is responsible for importing them into hypervisors, and who will manage them.  Create a detailed list of all current VM’s and their use (see below), and be sure to audit this list on a regular basis to find VM’s that are no longer used and can be decommissioned or consolidated.  If the rules are laid out beforehand, anyone who wants to play will have to abide by them.
• Standard Images: Creating some standard images to build off of will help streamline the process.  This creates a baseline for all VM’s to be built from that can include basic hardening, auditing, installation of company-wide applications (antivirus, logging agents, etc).  This gives you tighter control over the deployment process by specifying the operating systems and configuration baselines that get pushed out.
• VM Lifecycle Management: A very important issue with VM’s is lifecycle management.  It basically boils down to keeping track of VM’s and their use.  VM’s, unlike physical servers, easily travel from hypervisor to hypervisor, making them somewhat difficult to keep track of.  VM’s also have lifespan issues – where a forgotten VM that is no longer used, is no longer on a patching schedule, but still has a live IP address can cause lots of problems.  Microsoft and Novell (along with others) have released software applications to help with VM lifecycle management.  These work by tracking new virtual machines are they are spun-up, moved into production, bounced around hypervisors, and then retired.

Its up to you to decide how fancy you get.  Your tools to combat VM Sprawl can be as simple as Nmap and a spreadsheet, or as complex as enterprise management applications that can store entire network state.   Whatever path you find yourself on, take an hour or two to go through your existing VM’s to document them, and get some sort of plan in place for adding more.  You’ll thank me later.