ESXi 5.1 to 5.5 post upgrade invalid/unknown VM

[srwsol]

Hi folks:

I performed an upgrade of ESXi 5.1 to 5.5 and after the upgrade found that one of my VMs showed as invalid/unknown in the inventory.  I tried removing and re-adding it to inventory with no success, and then I went through the procedure described in the kb about renaming vmInventory.xml in the /etc/hostd directory of the host and then manually re-adding the VMs.  That didn't work either, as soon as I re-added the problem VM it showed as invalid/unknown again in the inventory.  I ended up restoring the original copy of the vmInventory.xml file and then rolling back to version 5.1 via the recovery mode option during boot.  When 5.1 came back up the VM appeared perfectly in the inventory and started up with no problem.  I'm at a loss as to how to proceed now.  Could there be something in the VMX file that 5.1 thinks is OK, but 5.5 doesn't?  Also, can a bad VMX file result in this behavior rather than simply a failure of the VM to boot when started, which is my usual experience with a bad VMX file?

Suggestions welcome.

[nkrishnan]

Can you please post the VMX file details. Also make sure you are on the latest VM Hardware and tool version.

Do you have any pass through devices connected to VM? and 3rd party driver installed on ESXi

[srwsol]

I've posted the VMX file.   Regarding your questions:

1).  The hardware version is 7, and there are other version 7 VMs that didn't have the problem.

2).  I think the VMTools version is older, and I was planning to upgrade it after going to 5.5, but since the VM never started I can't see that it mattered here.

3).  The host is a Dell 2970 (AMD processors) so there is no vt-d.

4).  I do have some Dell openmanage vibs installed that I kept by using "update" instead of "install" on the 5.5 package.

From looking at the VMX I do see a few things that I'm not sure what they are, specifically the linux debugging lines, but they don't seem to be a problem under 5.1.   This VM is a Windows SBS 2008 instance and was originally created under ESXi 4.   

[nkrishnan]

Take a snapshot and upgrade VM hardware to version 9 (latest) and upgrade to ESXi5.5. I also didn't find anything suspicious in VMX file.

[zXi_Gamer]

extendedConfigFile = "SBS2008.vmxf"

In the datastore, can you confirm, if this file is available and post the contents of this file.

fileSearchPath = "/vmfs/volumes/RAID_5_Datastore/SBS2008/snapshots;."

IIRC, the fileSearchPath has to start with "."

Note: Parent_Path is the path to the parent virtual machine of the linked clone. The filePath always starts with a period character, indicating the current directory. The .lck files are  created in the current location.

VMware KB: A virtual machine fails to power on and requests the location of .lck files

[a_p_]

This configuration file looks like it was edited manually at some point in time!? What you may try is to go to VM's the settings, select "CPUID Mask" in the "Options" tab and click the "Reset All to Defaults" button in both tabs of the "Advanced..." settings. Alternatively you could delete

cpuid.80000001.edx = ""

cpuid.80000001.edx.amd = ""

from the current .vmx file, register the VM in ESXi 5.5 and then reset the CPU settings to default.

André

[srwsol]

Here are the contents of the two directories you mentioned.  Note that I have some of the disk files for this VM on a different datastore than where the VMX is as there is limited space on the high performance datastore, so I only keep the operating system files and the VMX and related files there.  The big data files for this VM are on another datastore that's Raid 5 instead of Raid 1, but has more space.  Also, this directory listing is taken back under 5.1 while the VM is running.  I'll add one more thing I thought of that was different about this VM when I did the upgrade.  While I was waiting for the other VMs on the box to shutdown so that I could do the upgrade and this one was already shutdown, I took a snapshot of this VM because I knew I was going to have to upgrade vmware tools and upgrade the VM version level after the upgrade.  I never rebooted the VM under 5.1 after taking the snapshot, and I'm wondering if there is some aspect of the snapshot process, if you do it while the VM is down, that takes place during the next VM startup, which in this case would have occurred under 5.5.  That's the only thing I can think of that I did different regarding this VM versus the others. 

/vmfs/volumes/4b2037b1-db63cd61-7b6b-00221966cf48/SBS2008 # ls -l

-rw-------    1 root     root         32441 Apr 11  2011 SBS2008-Snapshot42.vmsn

-rw-r--r--    1 root     root            13 Jan  7  2013 SBS2008-aux.xml

-rwx------    1 root     root          5686 Apr 14  2011 SBS2008.bak

-rw-------    1 root     root          8684 Oct  8 03:09 SBS2008.nvram

-rw-------    1 root     root          1092 Oct  8 01:57 SBS2008.vmsd

-rw-------    1 root     root          6775 Oct  8 06:14 SBS2008.vmx

-rw-------    1 root     root             0 Oct  8 03:09 SBS2008.vmx.lck

-rw-------    1 root     root           365 Oct  8 06:13 SBS2008.vmxf

-rw-------    1 root     root          6776 Oct  8 06:14 SBS2008.vmx~

-rw-------    1 root     root      16801792 Oct  8 06:13 SBS2008_3-000001-delta.vmdk

-rw-------    1 root     root           347 Oct  8 06:14 SBS2008_3-000001.vmdk

-rw-------    1 root     root     10737418240 Oct  8 01:56 SBS2008_3-flat.vmdk

-rw-------    1 root     root           545 Oct  7 21:32 SBS2008_3.vmdk

-rw-------    1 root     root     1845518336 Oct 10 01:47 SBS2008_4-000001-delta.vmdk

-rw-------    1 root     root           347 Oct  8 06:14 SBS2008_4-000001.vmdk

-rw-------    1 root     root     10737418240 Oct  8 01:56 SBS2008_4-flat.vmdk

-rw-------    1 root     root           519 Oct  7 21:32 SBS2008_4.vmdk

-rw-------    1 root     root     7919095808 Oct 10 02:21 SBS2008_5-000001-delta.vmdk

-rw-------    1 root     root           348 Oct  8 06:14 SBS2008_5-000001.vmdk

-rw-------    1 root     root     128849018880 Oct  8 01:56 SBS2008_5-flat.vmdk

-rw-------    1 root     root           521 Oct  7 21:31 SBS2008_5.vmdk

-rw-r--r--    1 root     root        263278 Oct  5 04:42 vmware-213.log

-rw-r--r--    1 root     root        256376 Oct  6 00:57 vmware-214.log

-rw-r--r--    1 root     root        257706 Oct  7 00:51 vmware-215.log

-rw-r--r--    1 root     root        242046 Oct  7 02:09 vmware-216.log

-rw-r--r--    1 root     root        240728 Oct  7 19:15 vmware-217.log

-rw-r--r--    1 root     root        240211 Oct  8 01:56 vmware-218.log

-rw-r--r--    1 root     root        228755 Oct  9 15:35 vmware.log

-rw-------    1 root     root     149946368 Oct  8 03:09 vmx-SBS2008-1880149363-1.vswp

/vmfs/volumes/4b2037b1-db63cd61-7b6b-00221966cf48/SBS2008 #

/vmfs/volumes/4b20543d-aa14d6ba-2359-00221966cf48/SBS2008/snapshots # ls -l

-rw-r--r--    1 root     root             1 Jul 22  2011 SBS2008-7010cd73.hlog

-rw-------    1 root     root     8589934592 Oct  8 03:09 SBS2008-7010cd73.vswp

-rw-------    1 root     root         35051 Oct  8 01:57 SBS2008-Snapshot106.vmsn

/vmfs/volumes/4b20543d-aa14d6ba-2359-00221966cf48/SBS2008/snapshots #

[srwsol]

a.p. wrote:

This configuration file looks like it was edited manually at some point in time!? What you may try is to go to VM's the settings, select "CPUID Mask" in the "Options" tab and click the "Reset All to Defaults" button in both tabs of the "Advanced..." settings. Alternatively you could delete

cpuid.80000001.edx = ""

cpuid.80000001.edx.amd = ""

from the current .vmx file, register the VM in ESXi 5.5 and then reset the CPU settings to default.

André

I'm not sure that I ever edited the file by hand, but I have played around with the CPU affinity with this VM in the past via the viclient and/or webclient.   I'll try, either later tonight or tomorrow night, removing that stuff and doing the upgrade again.  I've already upgraded vmware tools and I'll update the VM version to the latest possible under 5.1 prior to retrying the upgrade as well.  Seems like if they are going to change the syntax editing in the VMX files they should provide a way to check it ahead of time, or at the very least put something in the log file about what lines they didn't like. 

[srwsol]

After doing a number of things I got the upgrade to work.  I'm uncertain which thing I did fixed it; unfortunately I didn't have enough time to do this scientifically and change just one thing at a time and retry the upgrade over and over until it worked.  Here are all the things that I did:

• Upgraded the VM version to version 9 prior to the upgrade
• Installed the latest version of VMWare tools prior to the upgrade and then again post upgrade
• Reset the CPU ID mask to defaults (still saw those lines in the VMX file though)
• Removed a host USB device (a modem) from the configuration prior to the upgrade and put it back afterward (had read that there were some issues in 5.5 concerning USB devices)
• Booted the VM under the old version after taking snapshot so that an empty snapshot wasn't there post upgrade
• Removed the VM from the inventory prior to the upgrade and then re-added it after the upgrade so that it wasn't present during the upgrade itself.

Thanks to everyone who provided suggestions.  Hopefully if this happens to someone else they can try some of these things and maybe narrow down which item(s) are actually necessary.