Needing to Validate an upgrade path I'm wanting to do

[benny_hauk]

My VC 1 server is terribly old and has always ran on an Access DB which meant I got to start over with a brand new VC 2 server (yeah!). That also means my upgrade path looks a \*little* different that the documented upgrade paths (not much talk about migrating VMs or hosts between a VC1 server and a VC2 server). I'm mostly doing very manual migrations using swing boxes and swing VMFS3 partitions and it's working great so far.

Now I want to upgrade 3 VMs with particularly huge vmdk files (I don't have the swing space to manually migrate this one) all on the same 550GB LUN to the VI3 environment. Here is the path I want to take (currently running on ESX 2.5.x):

1) remove the ESX2 host with the 3 VMs on it from VC1.3's inventory (make it standalone)

2) Add that same ESX2 host to the VC2 inventory

3) Power down all VMs on that host

4) Migrate all those VMs to an ESX3 host already in the VC2 inventory (VMDK files stay on the same, shared datastore)

5) Upgrade that shared datastore's partition from VMFS2 to VMFS3

6) Upgrade Virt HW on all 3 VMs

7) Turn on all 3 VMs and Upgrade VMWare Tools on them all

And then at my leisure I can blow away the now unused ESX2 host and rebuild it from scratch with ESX3.

Any technical road-blocks in this path? Potential problem areas I see include #2 (is that even possible?) and maybe #4 and/or #5 except for the fact I took those steps straight out of the manual.

While I've backup up all 3 VMs and am prepared for the worst, I don't have test resources available to test this path prior to doing it. I've migrated 35 VMs in a very manual fashion (cp vmdk from VMFS2 part. to VMFS3 part. and cp vmx, nvram, etc and manually modify vmx file to point to new vmdk location) so I feel like I have a pretty good understanding of the environment, just not sure about a couple steps that look right but that I've never verified actually work.

Thanks!

[bigvee · Accepted solution]

Ive done the exact same scenario with a few of my migrations. I had no issues at all.

I had quiet a few 2.5.3 hosts in VC2 while I was migrating VMs to the new environment.

Id say you'll be fine.

[ANSA]

You're on the right track. Should be fine.

[benny_hauk]

Excellent!

After some thought I came up with a way to test it with some old VMs I'm ready to destroy. I'll probably have time to test it later this afternoon once they finish migrating to an empty VMFS2 partition. If this test upgrade works without a hitch, I'll give you guys credit for the quick reply and do my production upgrade this weekend.

Thanks again.

If this works well enough, I may just use this scenerio going forward and upgrade an entire LUN at a time. I couldn't consider this option at first because so many of my VMs have their VMDK files spread across multiple LUNs. Now that I've already manually migrated most of those and that's not the case anymore, I'll take it under advisement to finish my migration using this method. It sure would be much faster with less downtime.

[benny_hauk]

Having problems with step 4 (see orginal post).

All my ESX3 hosts are in a cluster in VC2. Does the destination ESX3 host have to NOT be in a Cluster? When I try to migrate my powered down VMs it doesn't let me keep them on the VMFS2 partition where they are now - the destination datastores are only my VMFS3 partitions I've already created.

Was I supposed to upgrade the VMFS2 partition first? I'm pretty sure I'm not.

I'm taking a break for today and will look at it again tomorrow. I guess I'll try taking an ESX3 host out of the cluster and see if I can migrate from the ESX2 host to it and let the VMDKs stay on the VMFS2 partition. If that doesn't work either... anyone know what I'm doing wrong?

[benny_hauk]

I'm still in the midst of disaster recovery, but so far this scenario hasn't panned out well at all. I tried just testing the procedure using to VMs I cared nothing about and planned on destroying anyway. 2 things happened, one my fault, one is still sort of a mystery. First of all when creating an empty VMFS2 partition to put my test VMs on I removed an active VMFS3 partition, completely destroying 2 VMs. In a showing of divine mercy only 2 VMs were affected and they were both test machines and weren't hard to get recreated (cloned from their prod counterparts then full file restore from tape - LONG night).

Currently, 2 other VMs that are running and responding on the network simply lost their datastore (the only 2 on that datastore). It's the freakiest thing, but the datastore really doesn't appear to be present anywhere but the VMs are obviously up and running. The best guess I have is that the ESX2 host I added knew something about a logical LUN 20 and the VI3 hosts already there knew something about an entirely different logical LUN 20 and one of them won out and the VMFS3 partition's vmhba:1:0:20:? got forgotten about in lieu of the VMFS2 vmhba:0:0:20:?. Either way it appears the problem is that the ESX3 host's partition table doesn't know anything about the LUN that disappeared even though the LUN is still very much active and online (I've been told rebooting the VM or the host might be fatal at this stage). Support is working me through a procedure to try and restore the wiped out partition table on the host without harming the integrity of the VMs supposedly still on that partition so last night I created clones of those 2 VMs in case the worst happens.

Summary: From here on out I find a way to continue on the long and winding, manual road. It's never come back to bite me and trying to take shortcuts (al beit seemingly supported shortcuts) has bit me on 2 separate occasions now. I admit a lot of this is my own fault/carelessness so I'm not bashing the product or even the process necessarily. But if I'm less likely to screw up again by taking the long road then the long road will be the easiest and quickest in the end.

[benny_hauk]

I'm marking the first response as correct because even though I didn't quite get it to work myself, I think it was due to mistakes I made - not the procedure itself. Specifically, I think I wiped out a working VMFS3 partition from a 2.5 MUI when I thought I was wiping out an empty VMFS2 partition in order to create a new VMFS3 partition from the VC2 client. I'm not positive how I did something so stupid, but I suppose I was due for making a stupid mistake. I would have just upgraded the VMFS2 partition to VMFS3 from vc2 client, but in order to do that you have to put an ESX3 host in Maintenance Mode for some strange reason and I didn't feel like VMotioning all the VMs off of a host them VMotioning them back on.

Anyway, I followed enough of the procedure above to realize that it probably works fine. I still wouldn't manipulate any VMFS partition with production VMDK files on it without backing up the entire VMDK files first - I don't care how many times the procedure has been tested. As I can now atest to, you can't fully test for human error!