Skip navigation


This morning my IT Director and the Senior Application Architect were discussing upgrading some production hardware.  The system requirements for the software platform in question had already been delivered to me so I knew what they needed.  The IT Director was asking about cost and timeline and asked me what these servers would cost.



I told him they were good virtualization candidates which made him happy (We're still justifying VI to the board).  The Application Architect didn't respond but I saw the screwed up look on his face and the pain / concern for his platform raging in his head.  I pulled up a whiteboard and went over the architecture with him and explained the resource scalability.  By the end of our quick meeting I had buy off.  The problem is that so many people still think of virtual servers as some sort of VMware workstation or MS Virtual PC product, MAYBE they're able to think about in terms of MS Virtual Server or VMware Server which still inspires dread for the Enterprise Production environment.  Seems most people don't have a clue about ESX, HA, DRS, or anything like that.  I need to find a good multi-audience slide show.



This one is great for geek circles but doesn't really scale up to directors and executive well.  My IT Director bought off on VI a long time ago but he was amazed to watch this video and learn how some of these things work.  We need more tools to spread the word.  They need to be informative but also technical.  Its the people with some depth of technical knowledge who have a hard time with virtualization.  They understand the design challenges and assume they haven't been overcome.  Joe Schmo, just says "Cool, that's a great idea"







We use commvault and I'm a decent scripter so luckily I can use VCB and not have to worry about selecting a backup product.



Now having attended some user groups and seminars, I have every software company under the sun calling me to buy their software.  Many of these packages are similarly priced to VI itself and when you add them all up then yes, they do cost more.



Some of the software that intrigues me is charge back.  We don't have any chargeback in place at the moment, IT is a cost center and we budget to provide for the whole enterprise.  Adding some use metrics and reporting by service or department can really help with cost viability.  I think it will be helpful in the future to identify where the cost is going and who's consuming the investments.  Cost visibility rather than charge back but it uses the same tools and methodology.



Optimization products like Virtugo Optimize are intriguing as well.  This product dynamically adjusts reservations based on a lot of metrics gathered by their agents on both the host and guest systems.  They claim to provide up to a 30% increase in the amount of workload a host can handle.  I'm still working on measuring this.  Internally they really only do a CPU based stress test.  I want to see it under real load and that is difficult because I don't have a functional prod environment yet.  If I did, I might not run their software in it for testing either.



What I'm doing now is configuring VMmark.  Once that is in place and I'm able to add enough tiles to make my test environment puke, I'll turn on optimization and see if my benchmarks improve.  VMmark takes about a week to configure from scratch.  Once the templates and config files exist, an environment could probably be whipped out in a day.  BIG HURDLE!!!  VMmark is mostly open source.  It uses tools that are freely available for download.  Software licenses for guest products like Windows and Exchange are generally available in most shops.  There are 2 required products however which need to be purchased.  SPECweb2005 and SPECjbb2005.  SPEC requires payment in full before they will snail mail you a CD.  They will not extend discounts so I can expense it and they will not offer terms so I can receive it sooner.  I have to go though the PO generation process, give them a PO, wait for an invoice, wait for approval on the invoice and then wait for a monthly check run.  then I have to wait for snail mail both directions.  This really puts a hamper on my VMmarking.  I have no qualms about acquiring the software illegitimately AFTER I have the purchase approved NOT before but its not available though any of the standard illegitimate routes.  So I wait.







All right, here's the fun part.  Now I know what I'm doing to a strong enough degree that the only way I'm going to get better is to buy a bunch of blades with a SAN and set them up in a basement someplace or to work for somebody who has VI or is open to implementing VI.



SO AHEM, I don't personally know anyone who's going to go to the trouble of wiring blades into a stove outlet to teach themselves VI, so here's the options:


1) Beg your current employer to let you help administer their VI environment.

2) Convince your current employer that you need VMware and then also convince them that you're the man for the job

3) Look for a new job where you can leverage the skillz you have to get the skillz you need.


On a side note, unless you work for a small shop who's willing to give you a lot of leeway in terms of time to implementation, training curve, etc.  You should know that setting up a top-notch VI environment is no small undertaking.  Here's the basic checklist for a consolidation project:


1) Determine the annual cost for each of your physical servers, power, cooling, maintenance.

2) Determine the number of man hours that it takes to admin over the life of the server.  Racking, firmware updates, HW events, decommission time

3) Use this information to determine the TCO for your existing servers and new physical servers

4) Use a product like capacity planner to determine your virtualization candidates

5) Determine how many VMs you can run on your target HW platform

6) Get quotes for hardware and software (Acceleration kit and 3 hosts is a great entry level proof of concept)

7) Be smart with your storage investment.  Use an existing SAN, or a SAN that will scale up according to your eventual need.

8) Consider products like SANiQ from Lefthand networks, create a SAN out of the unused diskspace on your ESX servers.

9) Determine your hardware and licensing costs including any value-add software

10) Use this information to determine what the average TCO of a virtual server is.



Armed with this information you should be able to justify Virtualization over physical server refresh, its all about the bottom line.  Be conservative about capacity and ROI, set achievable metrics for success.  If its a tough sell, try to get your first implementation to use the funds that would be dedicated to server refresh and server purchases for upcoming initiatives.



11) Talk about VMware, anyone without your level of investment in virtualization is going to be doubtful.  You'll probably need buy-off from other departments to help sell the project to executives.  Dispel the myths, alleviate the doubts.  Take every opportunity to say "If we had VMware this would be cheaper and easier because......."

12) Carefully design your infrastructure, switches, NICs, VLANS, trunks, remote management, SAN connectivity, redundant HBAs, etc. MAX OUT RAM

13) Design your hardware validation plan.  Do you have a test lab you can vitalize and hammer your ESX servers with?  Are you going to use VMmark and simulate loads, or are you going to just plug it all in and start migrating machines?  If that's your plan, get some 3rd party reporting tools budgeted to scrutinize your environment for you and be prepared to tune pre-production

14) Plan you Virtual Center Server, which database will you use, what is it's availability, who is going to provide access, back it up, etc.

15) If you're planning more than 3 or 4 hosts for inital setup plan your DRS clusters.

16) Get buy-off and approval, generate POs order equipment

17) Rack hardware, install components, configure BIOS, remote management, install firmware updates, initial disk setup

18) Prepare your VC and UM databases, install a physical Virtual Center server (use economy hardware for this, not the same class as your ESX boxes)

19) If using AD, create your VC access role AD groups

20) Setup user groups in VC, map VC roles to AD groups for ease of administration and audit trails.




Use the VMware CPU info ISO to validate your ESX hosts for 64-bit compat. 



22) Install ESX on your Host servers

23) Enable remote access to your Service Console


25) Join your ESX hosts to VC

26) Configure your ESX network components, firewall, NTP, AD integration, etc

27) Configure your switch with appropriate VLANS, Trunks, etc.

28) Configure your vKernels, service consoles, VLAN switches, NIC teams, etc

29) Use local storage to create templates of your server OSs

30) Install sysprep files in Virtual Center, create customizations

31) Provision LUNs on your SAN

32) Connect ESX Servers to Storage, Detect LUNs, format VMFS filesystems

33) Create DRS/HA cluster, Resource Pools, Basic folder structure

34) Create an ISO folder on your VMFS volumes and upload all key ISO files

35) Update the SSL certs on your ESX servers and Virtual Center

36) CONFIGURE VCB or your preferred backup method DO IT DO IT DO IT DO IT NOW

37) Deploy some VMs into your cluster and common storage

38) Play with them, test them, experiment with HA and DRS, unplug servers, disconnect storage, do it a lot, while it doesn't matter



41) Validate your Hardware configuration, load it with VMs and run the heck out of them, if you can't generate a capacity load, use a tool like VMmark

42) Identify the low-hanging fruit for virtualization.  The old, the slow, the weak.  Pick some that will be easy and low impact

43) Develop, document, and test a p2v strategy on non-production equipment. (document roll-back plan in case of failure)

44) Present the plan to the business owners of the low-hanging fruit systenms

45) Virtualize the easy guys

46) Celebrate your success, build confidence with business unit or server owners.

47) Monitor your utilization, tune, improve your performance, eliminate bottlenecks, test your backup and recovery

48) Plan for more p2v, deploy some new services from templates

49) Develop a plan to prevent server sprawl, virtual server isn't free server, don't get overloaded with junk systems.

50) Consider chargeback reports and other types of reporting to promote awareness.


There are pieces missing and a few things out of order. My main goal here was to say "Look, this isn't easy, know what you're getting into if you plan to do it on your own"  There are a lot of consulting companies out there that can help do the hard stuff.  If you have doubts, contact one of them, get them interacting with your Business leaders, make sure that training you is on the agenda if implentation actually happens.



I write these when I'm too tired to do my job anymore so I apologize if they lack coherence.  I think I'm almost caught up so I can start focusing on specific issues and details to provide more relevant experience.



Okay, here's where it gets fun. I have access to VI evaluation. But I'm missing features DAMNIT, back to the whole realistic eval, ethical issues, educational license thing..... But I shall refrain from ranting because this also has been addressed in 3.5. The eval is full-featured.


So ESX hosts, configured, VC configured, migration of loads of VMware server guests to ESX. This brings up the first big best practice. Volume Licensing. My Volume License guests transfer to ESX just fine. My OEM and Action Pack boxen, not so much. They want activation. Technically you need to be using VLM licenses to do vMotion and not OEM so if you need activation, get the right bits. On another note: Once on ESX if you don't need to maintain portability between other products just activate before making your template. Activate persists through a sysprep, I think? I could be wrong. If not, it DOES persist through a ghostwalk, just use a dos boot disk image.



Okay the next piece was figuring out WTF the difference is between a Service Console, a vkernel, a vNIC, a vSwitch, and a Guest network so the spice could flow. There are a lot of best practices and service dependencies to be sorted out here. My number one piece of advice for your first run on ESX is do these things first: Allow NFS client, Software iSCSI client, SSHD (Inbound), and NTP client in through the firewall.



Now before you do anything else, configure NTP. If you know linux and want to do things like ssh and scp create user accounts and give them wheel access/ configure sudo or just allow root login to SSH. Edit /etc/ssh/sshd_config and change the line regarding root login. You'll have to restart sshd afterward. If you don't know how, its probably irrelevant. If you don't know how then here is where you branch off on your education to learn Linux. I wont encourage this or give detail because once again, this has been addressed by 3i and Linux expertise will soon cease to be a requirement.



Now create a vKernel switch and mark it for HA.



This covers most of the irritating prerequisites for configuring other services while you play around and learn.



Join the VMTN network. Start looking a posts every day or two, see what the common issues are. Start to contribute where you can so you have some visibility and credibility by the time you have real questions. Sign up for the local user group. Sign up for webinars. Read / Listen / Watch as many of the VMworld sessions as you can stand (I actually have a spreadsheet and a set of take-away documents so I can keep track of what I've watched and what I've learned) Tell your friends and industry peers about VMware. Get them excited, get them on board and collaborate.



Implement all the features on your test lab, HA, DRS, various storage. Opensource iSCSI works well even with junk in a test lab. Get familiar with what you can do and how. Then start playing with p2v, start using VCB if you have the right storage requirements. Educate yourself on the items you'll need to really support a VI and not just what you need to implement one.



Once you "Know what you're doing," read some books. ISBN: 0-9711510-6-7 by Ron Oglesby and Scott Herold, Looking forward to the next book. Once you've read some books, implemented all the features, figured out all the bugs and know what you're doing, go to a VMware user group, ask some questions, answer some questions. You'll find if you get this far you know more than most of the enthusiasts out there.



If you're interested in certification, this is the point you should get to before you consider taking the classes and preparing for the test. I haven't taken it yet, I have a bootcamp in my near future but its my understanding that knowledge is prerequisite to be successful in the classes.




Okay so here I am now (Then) thinking that it would be nice to have an ESX environment to educate myself in.  Well part of mastery isn't just learning one technology.  There are supporting technologies, competing technologies, and conceptual items like best practices to learn.



While I'm trying to piece together ESX capable hardware (I would rant about this for hours too but SAS drives now work without hacking, still need to check SATA-II RAID but kudos none the less) and figure out how I'm going to master ESX, its time to learn some of the INs and OUTs of virtualization.  Time to download the free versions.



Zen, Linux Virtual Machine, Microsoft Virtual PC, Microsoft Virtual Server, VMware Server, Evaluation versions of Workstation 6 and even Viridian recently.  If my target technology is a pain in the ass to educate myself on then I'll check out all the other free and relatively compatible options. (Another big selling point for educational license of VI)



An interesting yet equally frustrating development:  With VT technology and undocumented switches, I can run ESX 3.0 inside of VMware Workstation 6!  It even works relatively well.  HCL issues resolved!!! Still constrained by evaluation and ethics!!!!



Time to focus on best practice and living inside of virtual machines (Technology independent)



A professional peer of mine and I (who happen to have a personal WAN, domain, and TONS of junky lab hardware) standardized on the following Virtualization test environment.



Linux host servers

VMware Server

LVM Volumes formatted XFS (one per Guest)  *Later re-vamped to ext3 for simpler volume extends


With a farm of VMware servers we were able to gain a lot of insight into working with virtual machines.  The Linux host and Dynamic/snappable file systems gave a somewhat representative experience to ESX hosts. Although the Virtual Server Client isn't VC, this model really gave us something to work with that was free, educational, and gave us a test bed for some of the required learning.  No DRS, no HA, no fancy features without the use of scripts but its still good and still in use today.



We're both fairly comfortable staging lab environments in this configuration at this point.  We've got adequate backup and template capabilities scripted and he has actually implemented this config at his job.  Each engineer is given one test/dev box.  With this configuration, a single test/dev box and adequate storage can provide exponentially more value.



HEY!  Turns out we both work for a large enterprise who's made a significant investment in VI.  Great!  Now good luck getting access to it.  I can't get access to ours because I wasn't on the team that got trained and implemented it.  The joys of being in a remote site!  My cohort is in a similar boat.  New set of griping!!  What if my employer has VI but wont let me play with it?  shrug never figured that one out, I moved on to a new company where I am the top System Engineer.  Now I have the opportunity, how do I get these guys to implement?



Meanwhile on various attempts, I've gotten ESX capable hardware (HP blades and EMC SAN out of my own pocket!)  I told you I was serious and dedicated (Love Craig's list and eBay).  So now with another eval license and some VMware experience behind me I'm actually able to get some value out of an evaluation.



More to come.



Okay, so I've realized the inevitable future that is virtualization. I've accepted that expertise in virtualization will be necessary for senior infrastructure professionals in the very near future. I've accepted that there will be significant educational investment. My interest is peaked, my passion is high, I am fully dedicated to the task at hand. NOW WHAT!?


The bread and butter right? Dive strait into ESX and VC. This is the epitome of virtualization and the core technology of the market leader. Time to learn this technology. Okay so I'm an IT professional, it's simple enough to download an evaluation license. I should be able to throw this baby on some of my lab junk and come up to speed fairly quickly.


No such luck. There wasn't an HCL crammed down my throat prior to registering for 30 days. Along side my other priorities, gathering ESX capable hardware didn't happen for at least 30 days. After this time period I'm no longer permitted to re register for another evaluation. This can be circumvented once or twice right? But still barely ethical and somewhat of a pain.



I happen to be of a somewhat common creed of IT professional. I don't want to steal your software. I don't even want to borrow your software. I don't have an advanced skill set in your new technology so a 30 day evaluation doesn't give me adequate time to test a real implementation. I want to LEARN your software. I want to have it available to me in a fully functional state for my own educational purposes. not becuase I want to evaluate it in my near production environment indefinitely but because I want to support, design, implement, and guide people to purchase your software. For that very point I beleive that an educational license is necesary. Think Microsoft Action Pack. The AP is an acknowledgement. "We appreciate the fact that you can steal our software if you want to. We appreciate that you need copies to become proficient, that you need to use it to master it before you can sell it or direct it to be purchased. SO here, we'll sell it to you for a reasonable price to be used within special constraints." VMware needs something like this for those of us who have moral conflicts between our industry ethics and our need to gain mastery of expensive technologies.



I would rant about this for much longer except for the fact that this has been remedied somewhat satisfactorily. With 3.5 I can now run fully functional for 60 days (Not constrained by original date) provided I wipe and rebuild my infrastructure after each cycle. This is still a pain but I can appreciate why they wouldn't want my educational environment to have too much longevity. I approve. Kudos. (I'd still prefer an educational version.... Make it an acceleration kit that isn't expandable, do something. Give us access to learn so we can design, implement, and sell. I would not have been able to sell VMware to my executives if I hadn't had as much experience with VI as I have. I've gotten my experience in a variety of places but it would have been simpler if I could have purchased an educational version that would be full featured without interruption. If I had been less persistent in my desire to learn I may not be staring at the prospect of potentially virtualizing hundreds of machines.



I'm more than a year into my journey from "WOW this is the future of the Data Center" epiphany to realizing a production ESX infrastructure in an Enterprise Data Center environment.


I'm going to try to play catch up by summarizing my experiences to date and hopefully catch up to real time where I can post my trials and tribulations as I encounter them.


My largest goals are to come up with distilled useful information, best practices, and some basic guides to help those in a similar situation.



I'm a highly senior IT generalist with decades of experience. I have bountiful hardware, software, OS, networking, coding, analysis, and integration experience so I am able to conceptually appreciate Virtualization technology and its place in the world of IT infrastructure. Virtualization has caught my interest to the degree that I am prepared to leave the generalist mentality behind and focus on becoming an expert in the realm of virtualization and supporting technologies such as enterprise storage.



I am encountering issues in terms of educational labs, skepticism, availability of a local community, etc. This is all to be expected of an emerging technology and I embrace it. This will be my journal of overcoming those obstacles and contributing to the main stream success of virtualization