I consider there to be a gap in SMB confidence in starting a consolidation exercise (in whatever guise) to a VMware virtualised Infrastructure.
For this proven practice I will impart my experience and opinions from two high pressure and successful migrations (a company merger and a Disaster Recovery / Business Continuity gap) to a VMware ESX 2.5 Farm. The ESX host count at the time was just 6 but adequately catered for a migration of 78 physical servers plus Business As Usual server requirements.
The present day sees the adoption of VMware's Infrastructure to the extent all server provisions are answered with a "right click deploy".
I would just to add the point that I continue to use the approach I adopted for my VMware ESX 2.5 implementation for my VMware Infrastructure 3 environment, after all, it just works.
VMware Certified Professionals architecting VMware Infrastructures for consolidation. Technology departments looking to reduce physical server sprawl, manage legacy services and reduce utility expenditure.
The topics covered in this paper are written for consumption by Technical personnel and focus on migrating Microsoft Windows Operating Systems. As no one perfect solution exists for consolidation & migration exercises this document will remind you the infrastructure you're working on underpins a business. You must consider all aspects of the building blocks in your infrastructure in order to make your transition as smooth as possible. Take this document, read it and understand some of the impacts discussed from the expressed real world experience. Remember, due diligence in your upfront investigation will increase your awareness of the environments and dictate your considerations and approach. This guide could possibly save your career and maybe your life.
Business process impact
Supporting the VI with business unit buy-in
DR & BC
The estate review
Hardware - outside looking in
Hardware - inside looking out
Operating System , Service Pack & Hot Fixes
Face lifting the Virtual Machine
What's in a name?
Consider your host
Setting them apart
During a migration
Monitoring and backups
P2V Procedures & tools
With over 10 years experience architecting application service models I continue to build on a great wealth of knowledge and experience designing, building and supporting the Compaq & HP Proliant server range. From hardware to operating systems I have created, deployed and supported many of the offerings from Microsoft ensuring reliability and compatibility are key success factors.
My current technical product portfolio centres on and around the VMware Virtualisation offerings extracting the benefits of sustainable solutions through server consolidation, repurposing freed up equipment, full utilisation of hardware and providing ‘services' not ‘servers' to the customer. A core deliverable has always been to address Disaster Recovery and Business Continuity and is an area I have specialised in ensuring business survival activities are addressed.
The content of this document is purely reported and implied from personal experience of the author and not the opinion of Nationwide Building Society or any other of its employees.
Intellectual property rights for the creation and content of this paper belong the author.
Copyright (c) Darren Woollard 2008, Nationwide Building Society.
A physical to virtual (P2V) project should not be simply viewed as a lift and drop exercise , a like for like server migration or a quick win solution to address server sprawl in a shrinking Datacenter. Transferral of servers into a Virtual Infrastructure requires dedicated thinking and commitment. Ask yourself the question, "What does success look like?". Now is the time to streamline the physical clutter, analyse and move it to a new utopian world.
Remember, a Virtual Infrastructure is flexible. You can add or adjust your hardware components. Memory, disk, network bandwidth and, importantly, processor (the raw horsepower) are now at your fingertips. Forget the mentality of server provision but now consider service provision and allocate resources as they're required.
Is there a wrong candidate for a VMware Infrastructure? A bad candidate is only bad if it drains your estate of resources. Consider this, you have an aging physical server with high processor and memory demands. The hardware is failing and your options for spares are exhausted. The application is no longer supported and it's business critical. P2V it. You now have a portable machine for any VMware environment. You can constrain it's resource demands and give it more if necessary. You could dedicate and ESX to this one machine, at least it's up and running and on current hardware. I appreciate there's a license cost involved but what would the cost be if the physical expired?
The start of your migration journey will doubtless be traumatic at times but not due to technical challenges. A key element of the migration exercise will require reassuring the business and battling with prejudice towards virtualisation and change. People fear change especially naive people.
Throughout a migration exercise keep in mind this thought, "Less touch less blame". No matter how innocent your actions and work may be the reality is if you're near pieces of business critical infrastructure and something fails you'll more than likely pick up the blame. Communicate your involvement and interaction with the servers up front.
Get yourself a local administrative account created on the server(s) targeted for migration - it's quite likely you'll be refused access to the real administrator account name and password and in the long run you'll save yourself time by removing the need to change, test and document new passwords. But why have an administrative account? Initially the migrated server won't have the necessary drivers to use the virtual hardware devices and without drivers for the devices you won't be on the network - you'll need a local logon to install them.
During the estate review ensure you have an account with the relevant level of access to review the hardware from the Operating System perspective. Avoid using other users' administrative equivalent accounts or Service Accounts. Ask for an account to be created specifically for your work and remind your peers system level auditing can be invoked to trace your activities should there be any concerns.
I cannot stress to you how much your approach to communication plays an important role for this type of project. Always keep your communication, whether verbal or written, positive. By all means tell someone their poor implementation is preventing some migrations from succeeding but have a solid plan or tested idea of how you'll be working around the problem.
1. Business considerations
Within an IT infrastructure two types of boundaries are typically found. The formal, usually derived from a departmental name and / or geographical location and the ownership, unknowingly formed from a system spanning many department and / or job roles. To elaborate, an ownership boundary may be a live system with a development team providing support whilst the support department handover completes. Of course there will be political tangents on this ownership boundary but I'm sure I don't have to remind you of this.
Understanding and identifying the boundaries during your migration planning will help to reduce animosity and prevent information from being withheld - another element to a successful migration.
1.2 Business process impact
Considering business regulations and process isn't typically the thought in the front of your mind when planning how you'll move an estate of legacy equipment into a VMware Infrastructure but it's a topic that must not be forgotten. Remember, you're just about to potentially break the tried and tested business processes and for Business Continuity and Disaster Recovery.
Use the Proof of Concept environment to test your new procedures, document and demonstrate to the business unit(s) how these new steps will replace the existing documented survival activities. I've used the word existing here and try to keep this in mind. Your remit as part of this project is to change the existing procedures and, if you can, avoid picking up the missing ones.
Locating the owners of the existing tried and tested documentation will be easier than asking them to do the necessary re-working of their documents. Don't despair though, some documentation may become obsolete once systems have transitioned into the new VMware Infrastructure. As an example, transitioning legacy servers with locally attached tape backup mechanisms may become redundant in favour of disk / site replication technologies or simply just a single tape library for all virtual machines.
1.3 Supporting the VI with business unit buy-in
Continuing on from the Boundary theme discussed above it's worthwhile understanding the ownership of the VMware Infrastructure components.
It would be fair to say this topic is a ‘hot potato' for many organisations and one of the stumbling blocks when introducing and adopting a virtualised infrastructure. Whilst it's not part of your remit to sit and draw up how things should be you may want to think about who will be responsible the aspects of a VMware Infrastructure, just a little thought preparation should a conversation ever arise.
Consider, who is responsible for :
2. Business requirements
You're ready to start your project but do you know exactly what the business is expecting from the deliverable and does it tally with your understanding of the project remit? In some way it's a misconception a consolidation exercise will fix your Business Continuity and Disaster Recovery woes. It will if the design caters for it. Double check, use your Proof of Concept environment.
Talking about consolidation work in the business world with peers imminently embarking on a P2V invariably draws out my favourite question, "How many physical servers can I squeeze on to an ESX host?". This is echoed throughout organisations and one that has no definitive answer. Once again use your Proof of Concept environment to generate your rule of thumb, this way you can communicate your expectations based upon your specific business application testing.
Recouping physical space in your Datacenter and saving power are really by-products of a consolidation exercise and shouldn't be seen to be the main drive behind a project but just one of the key benefits. Promote these ideas as part of a collective.
If the project is a cost saving exercise be mindful although you may have squeezed 20 physical servers onto 1 large physical VMware ESX host how do the consumption figures compare to their physical originals? Remember, these ESX hosts can be run at a higher processor utilisation and draw more power. Consider the purchase cost of new large ‘enterprise' hosts, the ownership in terms of hardware support for the asset life. Reviewing just these few suggestions it's fair to say you wouldn't expect to see your costs drop to 1/20th of their original combined total.
3. The estate review
3.1 Hardware - outside looking in
It goes without saying up front investigation of your server hardware is needed but what questions need answering? Let's breakdown the key elements of a server provision then drill into specifics.
Picture yourself standing in front of the sever, what do can you see?
Walk to the rear of the server, now what can you see?
Is it apparent where all the (tangled) cables go? Investigate what's connected.
3.2 Hardware - inside looking out
Remember server (or) service owners are invariably reluctant to allow inquisitive minds reviewing their equipment so poking your head inside a server, assuming you're granted the downtime, can prove tricky so don't ask unless you really need to take the lid off.
There are tools at hand to reveal configurations of hardware and the operating system but hold off from installing expensive diagnostic or freeware tools, try the pre-supplied first.
Use these commands and piping the output to a text file for collection and storage :
Use these for reviewing and taking screen grabs :
I suggest creating a network share and moving the files you've created into it remembering to pre-fix the output filenames with the server name, asset tag or something meaningful when you come to review them later.
Working examples :
You've started gathering the basics now let's drill down into some important items.
Not all servers are straight forward in their configuration. What might seem quite obvious from a quick glance can reveal past upgrades, misleading naming conventions and startling partitioning techniques.
Q. After a migration where will the virtual memory file exist?
A. See 4.5
This may seem to be a simple counting exercise but later I'll explain why the quantity and perceived speed play an important part in what's presented to the business after migration.
Q. After a migration does the CPU count need to be the same?
A. See 4.6
Virtualisation and peripheral support is long way off but don't rule out servers with devices attached there are many vendor solutions in the market.
Consider (externally) :
Consider (internally) :
Q. If the server has security dongle can it be migrated in to a Virtual Infrastructure?
3.2.4 Operating System & Service Pack
The revision of Operating Systems becomes important during a migration and not after. Later in this paper the section entitled Helper Servers explains why.
Did you know a quick and effective way to find your Operating System revision?
Whatever Windows Operating System you're looking into it's quicker than navigating through Control Panel or running the Computer Management MMC.
An entire inventory isn't mandatory but reviewing the list of installed applications can provoke a thought in to where the server fits into your jigsaw puzzle of infrastructure.
Gathering the trends of servers is a time consuming process. What metrics are important? How much and what frequency should a capture be?
There is no definitive answer but I'd approach the gathering exercise (if you have time) towards your Target Environment boundaries.
Consider gathering :
By now your spreadsheet, or database, will be brimming full of interesting technical information about your server estate. I suspect you will have learned why some of your ex-colleagues never lasted the distance after reviewing the server builds and the standard non-standards.
But what about one item you can't learn from the examination? What's the key to preventing a migration from happening at all? The downtime window.
The downtime window has the potential to be an immense challenge with its associated mental anguish and I strongly recommend keeping a trail (as you should be doing) of commitments and information given to you.
4. Target environment
4.1 Face lifting the Virtual Machine
4.1.1 What's in a name?
Server naming conventions in a Small / Medium Business or Corporate environment invariably are constructed in all manner of forms. Perhaps a role, a location or a famous animated television series but whatever is active it should remain in place until after a migration. If you're looking to change server names and become one step closer to the utopian world - hold off. Too much change is dangerous. Take easy steps. After the dust has settled and the virtual machines have been accepted and stabilised consider renaming them.
Are you changing the network architecture? Perhaps the new VMware Infrastructure is housed in new racks with cable managed networking to different networking infrastructure.
An Operating System installation for a server isn't complete without a CD or two of extra drivers, management agents and utilities to make "Your life easier". With that last quote firmly in your mind and undoubtedly hitting a nerve surely isn't it time to rip out the hardware monitoring agents, network card configuration utility and handful of other utilities forced on the server?
Consider the :
4.2 Consider your host
The ESX host has a busy life dealing with all the virtual machine resource demands and it would only seem sensible that reducing these pool demands will increase the available amount. A less stressed host is a happier host and subsequently can balance and efficiently run virtual machines. A few simple tweaks can be made to reduce the resource demands.
4.3 Setting them apart
The virtual machine estate will expand as the adoption of the VMware Infrastructure becomes accepted. No doubt the Resource Pools and Folders have been planned but do you know where the migrated machines will reside? Will legacy servers require delegation rights of a different level (lower or higher) for ongoing support and maintenance?
There's no hard and fast rule just remember the administrative boundaries and resist the urge to create a complicated hierarchy of virtual machine Folders.
4.4 Helper servers
The VMware conversion tool is key to the success of a migration from a physical to virtual world, without this application the virtual hardware drivers and registry settings would need manually piecing together. The role of the Helper Server is simply to provide a host operating system for the VMware conversion tool.
You would expect one Helper server would be enough but a Helper server running the Windows 2003 Server Operating System will not convert a Windows NT 4.0 Server. Why? It's down to the NTFS version and compatibility between Operating System editions, disks can be read but in order for writes to be made the disk would need to be converted. Not a good idea.
Q. "How many Helper Servers do I need?"
A. The estate review will provide the answer.
Experience has shown a Microsoft Windows 2003 Server Standard Edition will convert nearly all Windows 2000 & 2003 servers. A Windows XP conversion will need a Window XP Helper and a Windows NT 4 Server will be sufficient for both Server & Workstation. In one real world migration a Windows 2000 Server running with Service Pack 2 refused to successfully convert even with a Helper Server at the same level Operating System and Service Pack. The solution was to build the Helper Server from an Operating System disk with the Service Pack already slip-streamed. It worked first time - easy when you know how.
4.5 Disk space
You can never have enough disk space and yet I'm sure the estate review has revealed a few surprises and even raised an eyebrow. Now is a good time to resize and play Robin Hood by taking from the (disk) rich and give to the (disk) poor.
Earlier the question "After a migration where will the virtual memory file exist?" was posed. The intended audience for this question is mainly SAN focused but it may provoke a thought or two for your environment. SAN disk is expensive so why host PAGEFILE.SYS on your spindles and then back it up? It's not only the cost of hosting the file but think about the retention storage. A virtual memory file can be re-created if it becomes lost or corrupted without loss of data. Create a new virtual hard disk on ‘cheaper' disk spindles.
Moving a physical environment to a VMware Infrastructure not only reduces the worry of hardware failure but also removes its constraints. The majority of client / server implementations run on at least dual CPU servers, this may be due to buying deals, reseller offers or vendor recommendations. Whatever the reason it's now time to change this thought process.
Q. "Should I place all the migrated physical to virtual machines into one large VMware Infrastructure?"
A. As your VI grows the resources will be under greater strain, read 4.6.1 below.
A high percentage, if not all, of the migrations can be given a single vCPU.
In a real world example a Quad Pentium Pro 200mhz server was migrated to a single vCPU Virtual Machine. Prior to the migration the server performed intensive data analysis every night taking upwards of 10 hours to complete. After the migration to a Quad Xeon 2.8ghz VMware ESX 2.5 Server the overnight process now completes in 1½ hours. It's not about the quantity of the vCPUs assigned but about the physical CPU cycles behind the scenes serving the requests.
4.6.1 Granting processor power - a scenario
So, it's Day 1 and your VMware Infrastructure is ready to take on the work load of the failing physical estate. The migration completes with great success and the business subsequently adopts this new infrastructure as the way forward for all new client /server service provisions. The months tick by with more and more virtual machines being added. One day you're told to re-visit and health check the VMware Infrastructure as it's been reported to be running slowly. Upon your investigation you can't pin point any one bottleneck or virtual machine choking. You approach the business and learn their applications aren't running as fast as they were when they first entered the virtual world.
The problem? Setting expectations
Imagine your ESX cluster. You have a pool of 80ghz of CPU and 64gb RAM ready to flex it's virtual muscle and migrate 20 virtual machines into it. This pool of power is shared amongst 20 virtual machines. Six months later you may have 80 machines but they're still all sharing the same resources.
Think of resource allocation as a volume control on your amplifier. You can just turn up the power as the business demands.
Internal and external modems, licensing and security dongles need not stop you migrating a physical server into a VMware Infrastructure. Third Party solutions exist in providing connectivity over an IP network for USB and Serial devices and for a lot less money than you'd initially think. Typically a small client application will be required to be installed on the server but this is a small price to pay if you're able to dispose or be able to re-purpose a physical server.
5. During a migration
5.1 Controlling changes
Every organisation has a different approach and approval mechanism for controlling changes to their environment. Whether it's a formal electronically driven or paper shuffling process you must not deviate or try to beat the system to save time. I'd be the first to admit I'm not a fan of paperwork and it takes a toll on elapsed time but this is the most politically correct approach of saying "Don't blame me, you signed it off." when blame may unnecessarily be thrown your way.
Consider communicating the impact of change :
5.2 Elapsed time
Factors often forgotten for the time it takes to perform a migration aren't typically what you'd expect. Here's a few thought provoking ideas.
The server estate review is the basis for your timetable and it's your technical awareness of the environment, the political slants and downtime windows that enable you to be in the best position to draw up and publish a timetable for the migrations.
Consider including :
Publishing the timetable and reminding your communication channels of this is key to keeping everyone informed and in fact everyone, not literally of course, should be able to review it. There may be a chance that someone is aware of a little task, database or program running on a machine your review might have missed. Remember, this one person may have not been party to the original ‘ownership' discussions when planning for downtime but nevertheless it's a positive way of introducing a ‘catch all'.
5.4 Migration template
Creating a one page template with the details of the server you're migrating and where it's going to will force you in to a routine of checking and not forgetting. You can add as much or as little information as you see fit.
Here's a sample of items I used in my template.
Consider (source server details) :
Consider (migration details) :
Consider (target virtual machine details) :
5.6 Monitoring and backups
Of the many hurdles and expectation management exercises you're juggling at once during your migration you must not forget to keep the support and operations management teams aware. Out of hours support personnel will not be impressed being called out only to find the server is down due to your migration.
5.7 Migration framework
As I discussed at the beginning of this paper your first migration will present you with a myriad of challenges and steep learning curves, but once completed you'll have achieved a mammoth task only a small number of IT individuals have ever undertaken.
Of course it's not only about completing a migration exercise but also formulating a winning method for your next migration project - oh yes, there will be others.
My ‘Winning Formula or, framework, looks like this.
1. Complete the estate review.
2. Pull together a draft timetable.
3. Use a Test, Development or Proof of Concept environment for performance testing.
4. Review and update your timetable based on results found in (3).
5. Formalise and publish timetable.
6. Raise change control(s).
7. Start (or continue) with the P2Vs.
8. Communicate your results as soon as possible.
9. Communicate the next round of migrations.
10. GOTO 7 until the migrations are complete.
6. P2V Procedures & Tools
6.1 Wise Words
An empty environment awaits your migration candidates but do you have the tools in place to capture and configure the servers before bringing them to life? Are you confident your toolkit of imaging tools will be able to detect the hardware? Did you answer "Yes"? You'd be surprised.
I strongly suggest the approach to the imaging process avoids the installation of software on to the server you're planning to migrate as it's important to be able to return it back into service in the state you found it. Remember some migrations can take a few attempts before they complete successfully and if something has changed, however insignificant, the blame may come your way.
Cloning takes time and once a few server migrations have completed you'll have an appreciation for an average speed of transfer based upon server model, disk technologies and network connectivity. Factor these into your future migration plans.
So, the cloning process has finished and you want to get started on the re-configuration but wait a moment. The server should be left in state so it can't be returned in to service (even by accident) and shouldn't be powered down. Why? The server may have been running continuously for a couple of years and once it's cooled down it may not start back up. It happens. Removing the network cable(s) and perhaps halting the server POST at the boot loader should be enough.
A myriad of tools are available and quite honestly you can make the migration as complicated as you want but why make your task harder than necessary?
Vendors offer tempting features in their P2V product suites. The ability to review prospective candidate servers and based upon a pre-defined rule set determine their suitability in a VMware Infrastructure. Full hardware detection, scheduling and reporting. This is all very well but they come with a cost , their own hardware requirements and some require software agent installs.
I strongly recommend trying a few migrations manually, understand what issues crop up and how to fix them. If you then decide to use a vendor product for the migrations at least you'll have an appreciation for the ‘behind the scenes' process in a P2V migration.
Consider creating your own boot CD :
both with :
VMware Converter :
Tape backup :
In summary you need to spend time investigating your infrastructure, perform test migrations, liaise with business members and communicate positively. I won't lie by telling you the migration will run smoothly but if you've gleaned anything from this paper that's made you think for at least moment I hope it'll help iron out the creases.
If you've read this far then you must be committed to ensuring your migration will be as near as perfect as you can. It remains for me to leave you with a final nugget of wisdom, something that has made it worthwhile for you just getting to this paragraph.
So here it is... "There isn't a perfect script for P2V migrations otherwise I'd written it by now."