polysulfide

This morning my IT Director and the Senior Application Architect were discussing upgrading some production hardware. The system requirements for the software platform in question had already been delivered to me so I knew what they needed. The IT Director was asking about cost and timeline and asked me what these servers would cost.

I told him they were good virtualization candidates which made him happy (We're still justifying VI to the board). The Application Architect didn't respond but I saw the screwed up look on his face and the pain / concern for his platform raging in his head. I pulled up a whiteboard and went over the architecture with him and explained the resource scalability. By the end of our quick meeting I had buy off. The problem is that so many people still think of virtual servers as some sort of VMware workstation or MS Virtual PC product, MAYBE they're able to think about in terms of MS Virtual Server or VMware Server which still inspires dread for the Enterprise Production environment. Seems most people don't have a clue about ESX, HA, DRS, or anything like that. I need to find a good multi-audience slide show.

http://www.youtube.com/watch?v=qWf_WiaFedc

This one is great for geek circles but doesn't really scale up to directors and executive well. My IT Director bought off on VI a long time ago but he was amazed to watch this video and learn how some of these things work. We need more tools to spread the word. They need to be informative but also technical. Its the people with some depth of technical knowledge who have a hard time with virtualization. They understand the design challenges and assume they haven't been overcome. Joe Schmo, just says "Cool, that's a great idea"

All right, here's the fun part. Now I know what I'm doing to a strong enough degree that the only way I'm going to get better is to buy a bunch of blades with a SAN and set them up in a basement someplace or to work for somebody who has VI or is open to implementing VI.

SO AHEM, I don't personally know anyone who's going to go to the trouble of wiring blades into a stove outlet to teach themselves VI, so here's the options:

1) Beg your current employer to let you help administer their VI environment.
2) Convince your current employer that you need VMware and then also convince them that you're the man for the job
3) Look for a new job where you can leverage the skillz you have to get the skillz you need.

On a side note, unless you work for a small shop who's willing to give you a lot of leeway in terms of time to implementation, training curve, etc. You should know that setting up a top-notch VI environment is no small undertaking. Here's the basic checklist for a consolidation project:

1) Determine the annual cost for each of your physical servers, power, cooling, maintenance.
2) Determine the number of man hours that it takes to admin over the life of the server. Racking, firmware updates, HW events, decommission time
3) Use this information to determine the TCO for your existing servers and new physical servers
4) Use a product like capacity planner to determine your virtualization candidates
5) Determine how many VMs you can run on your target HW platform
6) Get quotes for hardware and software (Acceleration kit and 3 hosts is a great entry level proof of concept)
7) Be smart with your storage investment. Use an existing SAN, or a SAN that will scale up according to your eventual need.
8) Consider products like SANiQ from Lefthand networks, create a SAN out of the unused diskspace on your ESX servers.
9) Determine your hardware and licensing costs including any value-add software
10) Use this information to determine what the average TCO of a virtual server is.

Armed with this information you should be able to justify Virtualization over physical server refresh, its all about the bottom line. Be conservative about capacity and ROI, set achievable metrics for success. If its a tough sell, try to get your first implementation to use the funds that would be dedicated to server refresh and server purchases for upcoming initiatives.

11) Talk about VMware, anyone without your level of investment in virtualization is going to be doubtful. You'll probably need buy-off from other departments to help sell the project to executives. Dispel the myths, alleviate the doubts. Take every opportunity to say "If we had VMware this would be cheaper and easier because......."
12) Carefully design your infrastructure, switches, NICs, VLANS, trunks, remote management, SAN connectivity, redundant HBAs, etc. MAX OUT RAM
13) Design your hardware validation plan. Do you have a test lab you can vitalize and hammer your ESX servers with? Are you going to use VMmark and simulate loads, or are you going to just plug it all in and start migrating machines? If that's your plan, get some 3rd party reporting tools budgeted to scrutinize your environment for you and be prepared to tune pre-production
14) Plan you Virtual Center Server, which database will you use, what is it's availability, who is going to provide access, back it up, etc.
15) If you're planning more than 3 or 4 hosts for inital setup plan your DRS clusters.
16) Get buy-off and approval, generate POs order equipment
17) Rack hardware, install components, configure BIOS, remote management, install firmware updates, initial disk setup
18) Prepare your VC and UM databases, install a physical Virtual Center server (use economy hardware for this, not the same class as your ESX boxes)
19) If using AD, create your VC access role AD groups
20) Setup user groups in VC, map VC roles to AD groups for ease of administration and audit trails.
21) DOCUMENT YOUR CONFIGURATION AND RECOVERY PLAN TO THIS POINT

Use the VMware CPU info ISO to validate your ESX hosts for 64-bit compat.

22) Install ESX on your Host servers
23) Enable remote access to your Service Console
24) SECURE YOUR SERVICE CONSOLES
25) Join your ESX hosts to VC
26) Configure your ESX network components, firewall, NTP, AD integration, etc
27) Configure your switch with appropriate VLANS, Trunks, etc.
28) Configure your vKernels, service consoles, VLAN switches, NIC teams, etc
29) Use local storage to create templates of your server OSs
30) Install sysprep files in Virtual Center, create customizations
31) Provision LUNs on your SAN
32) Connect ESX Servers to Storage, Detect LUNs, format VMFS filesystems
33) Create DRS/HA cluster, Resource Pools, Basic folder structure
34) Create an ISO folder on your VMFS volumes and upload all key ISO files
35) Update the SSL certs on your ESX servers and Virtual Center
36) CONFIGURE VCB or your preferred backup method DO IT DO IT DO IT DO IT NOW
37) Deploy some VMs into your cluster and common storage
38) Play with them, test them, experiment with HA and DRS, unplug servers, disconnect storage, do it a lot, while it doesn't matter
39) DOCUMENT YOUR ESX SERVER CONFIG
40) DOCUMENT AND TEST YOUR BACKUP VM level BACKUP STRATEGIES
41) Validate your Hardware configuration, load it with VMs and run the heck out of them, if you can't generate a capacity load, use a tool like VMmark
42) Identify the low-hanging fruit for virtualization. The old, the slow, the weak. Pick some that will be easy and low impact
43) Develop, document, and test a p2v strategy on non-production equipment. (document roll-back plan in case of failure)
44) Present the plan to the business owners of the low-hanging fruit systenms
45) Virtualize the easy guys
46) Celebrate your success, build confidence with business unit or server owners.
47) Monitor your utilization, tune, improve your performance, eliminate bottlenecks, test your backup and recovery
48) Plan for more p2v, deploy some new services from templates
49) Develop a plan to prevent server sprawl, virtual server isn't free server, don't get overloaded with junk systems.
50) Consider chargeback reports and other types of reporting to promote awareness.

There are pieces missing and a few things out of order. My main goal here was to say "Look, this isn't easy, know what you're getting into if you plan to do it on your own" There are a lot of consulting companies out there that can help do the hard stuff. If you have doubts, contact one of them, get them interacting with your Business leaders, make sure that training you is on the agenda if implentation actually happens.

I write these when I'm too tired to do my job anymore so I apologize if they lack coherence. I think I'm almost caught up so I can start focusing on specific issues and details to provide more relevant experience.