VMware Cloud Community
agriesser
Contributor
Contributor

Unable to hot migrate some VMs after upgrade to ESX 3.5 (VC2.5)

Hi!

Today I upgraded from VC2.0.1/ESX 3.0.1 to VC2.5/ESX 3.5.

I'm running three ESX hosts and was able to upgrade two of them to 3.5 by now (Virtual Center has already been upgraded to 2.5). Currently I'm trying to get the last ESX 3.0.1 server free by migrating all virtual

machines hosted on this server to the other two servers.

What's really strange, is that I could migrate some of the VMs hosted on ESX1 (currently ESX 3.0.1) without problems to ESX2 (v3.5) and ESX3 (v2.5), but some virtual machines can't be migrated.

The migration meter is stalled at 10% and after quite some time I get an error message "Operation timed out".

I tried to migrate an already migrated machine back to ESX1, that worked. Afterwards I tried to migrate this VM again to the host it was prior to migrating to ESX1 and that worked too, so I'm sure that there's

no problem with the ESX Server, the VMotion network or any other thing that would prevent all virtual machines from doing migrations using VMotion.

The problem seems to be related to only some specific VMs.

In the logfiles I get the following lines after starting the migration:

VMotionLastStatusCb: Failed with error 3: Timed out waiting for migration start request.

VMotionResolveCheck: Operation in progress

VMotionStatusCb: Completed

Retrieved current power state from foundry 1

VMotionResolveCheck: Firing ResolveCb

ResolveCb: VMX reports gone = false

ResolveCb: Failed with fault: (vim.fault.Timedout) {

  • msg = ""*

}

Any idea what I could try to get these machines migrated?

TIA,

Alex

Reply
0 Kudos
13 Replies
Texiwill
Leadership
Leadership

Hello,

Did you do the migrates simultaneously or one at a time? In general I would start looking for SCSI Reservation Conflicts, they tend to be the leading cause of failed migrations. How many SCSI Operations were happening on the LUN at the same time?


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
Reply
0 Kudos
agriesser
Contributor
Contributor

I tried it simultaneously at first and afterwards I tried it one at a time.

I don't think that SCSI reservation conflicts are the cause for this problem. Yesterday I found out that there were also some messages from the license server in the logfiles and I'm wondering if applying the latest patches to ESX3.5 will solve this issue, so I'll give it a try and report back afterwards.

Any other ideas while I'm trying to update my machines?

Update: I now installed the patches and the problems are still there. I think it has todo with my "old" ESX Server not being perfectly up-to-date (I didn't install the latest patches and also didn't upgrade to 3.0.2 before attempting to upgrade to 3.5). I'll have a look at the ESX3.0.1 patches and will check if there's possibly a patch that can be installed without rebooting the ESX host to take effect. If this is not the case, I'll need a downtime anyway, so then I'll directly upgrade to 3.5.

Reply
0 Kudos
agriesser
Contributor
Contributor

Today I had the opportunity to play a bit with my VI.

The first thing I tried was to shut down one of the guests on the ESX 301 host and tried to cold migrate it to one of my new ESX35 hosts. That didn't work either, BUT I finally got useful error messages in the /var/log/vmware/hostd.log file on the ESX35 host. It complained about invalid configuration options for my virtual IDE and virtual floppy images. It seems as if ESX35 handles things a bit different and only allows image files being used from specific locations. Needless to say, that the place where my image files resided wasn't listed, so I gave it a quick shot and added this directory to /etc/vmware/configrules.

That did fix the issue immediately, afterwards I was even able to hot migrate all other virtual machines which were still running on the old ESX301 host to the newer ESX35 hosts without downtime.

So long story short:

If SOME of your VMs won't let themselves migrate to new ESX 3.5 hosts, check the VM configuration file and especially watch out for paths to image files, etc. Check against /etc/vmware/configrules if they're still allowed on your updated ESX hosts.

Reply
0 Kudos
opbz
Hot Shot
Hot Shot

intersting solution...

got similar case on some servers I will try your fix

Actually I do not hve a /etc/vmware/configrules on my ESX 3.01 server... I do see it on ESX 3.5

thanks

Reply
0 Kudos
DigitalVoodoo
Enthusiast
Enthusiast

That happened with my upgrades as well, and was quite maddening to troubleshoot until the right error message presented itself. I wish VMware would put up some KBs or other information on this and the other new "features" they've added to the product.

Reply
0 Kudos
azn2kew
Champion
Champion

Can you upload both contents of your config file and .vmx file I would like to compare my to see what specific settings added.

libdir = "/usr/lib/vmware"

authd.fullpath = "/usr/sbin/vmware-authd"

authd.client.port = "902"

vmware.fullpath = "/usr/bin/vmware"

control.fullpath = "/usr/bin/vmware-cmd"

serverd.fullpath = "/usr/sbin/vmware-serverd"

serverd.init.fullpath = "/usr/lib/vmware/serverd/init.pl"

authd.proxy.vim = "vmware-hostd:hostd-vmdb"

authd.proxy.nfc = "vmware-hostd:ha-nfc"

Thanks in advance!

Regards,

Stefan Nguyen

"The Power of Knowledge"

If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!! Regards, Stefan Nguyen VMware vExpert 2009 iGeek Systems Inc. VMware vExpert, VCP 3 & 4, VSP, VTSP, CCA, CCEA, CCNA, MCSA, EMCSE, EMCISA
Reply
0 Kudos
agriesser
Contributor
Contributor

Well, part of my /etc/vmware/configrules file looks like this (the whole config file has more than 200 lines):

---

  1. Virtual IDE devices can point to VMFS volume, raw device, or virtual

  2. tools media.

rule "Virtual IDE Devices"

{

vm regex ".*"

  1. IDE device backend. 2 controllers, 2 devices each

key regex "^ide[0-1]:[0-1]\.fileName$"

  1. Allow CDROM devices

accept regex_case "^/dev/cdrom[0-9]*$"

accept regex_case "^/dev/hd[a-z]$"

accept regex_case "^/dev/scd[0-9]+$"

  1. Only allow paths under /vmfs/, /vmimages, and relative paths

accept prefix_case "/vmfs/"

accept prefix_case "/vmimages/"

accept prefix_case "/virtual-machines/"

accept prefix_case "/usr/lib/vmware/isoimages/"

accept !prefix "/"

  1. Virtual Center sets dummy values

accept match "/null.iso"

}

---

My ISO images resided in /virtual-machines/ so I added it to all appropriate places in /etc/vmware/configurules to ease the migration. New images now reside on a shared VMFS volume to avoid such problems in the future.

My .vmx file contained the path to the image like this:

---

vimsap23/vimsap23.vmx:ide0:0.fileName = "/virtual-machines/iso-images/en_2000server_SP4.iso"

---

Nothing special as long as you know where to change the default behaviour if you're not using the default paths.

Reply
0 Kudos
peterjakobs
Contributor
Contributor

What guest OS are you running inside those machines?

I had this issue with Virtual machines configured as Red hat 32bit, but the OS was 64bit.

It ran fine on 3.0.2, but i could not move the machines to the 3.5 server.

After changing the OS setting to red Hat 64 bit the virtual machine migrated just fine.

Reply
0 Kudos
Frank_Poelert
Contributor
Contributor

Hi Alex,

I have been playing around with the migration of VM's from 3.0.2. to 3.5.0 and found out the following:

The first time you migrate a VM to 3.5.0 some CPU masks need to be set. VC can only set these if the VM is down. Once the settings are meade you can mve the VM around between 3.0.2 and 3.5.0 hosts.

I have copied the following settings from the .vmx of an already migrated VM into the .vmx of a VM that I wanted to move for the first time. And then simply migrated the thing to the 3.5.0. host with Vmotion.

cpuid.1.eax = "xxxx--


xx--


"

cpuid.1.ecx = "--


RRR----


"

cpuid.1.edx = "--


T--"

cpuid.80000001.eax.amd = "xxxx--


xx--


"

cpuid.80000001.ecx.amd = "--


0-"

cpuid.80000001.edx = "--


H--


"

cpuid.80000001.edx.amd = "---R


H--T--"

virtualHW.productCompatibility = "hosted"

tools.upgrade.policy = "manual"

I guess it will be better to check a vmx file of your own instead of using these settings directly.

I am running this on Proliant DL580G5's and DL380G5's (Intel 5130, E5345 and X7350 CPU's).

Might be different on other CPU's.

.Kind regards,

Frank Poelert

Reply
0 Kudos
jpoling
Enthusiast
Enthusiast

I just ran into this issue as well. ..the vmx file is indeed different between a 3.0.x and 3.5 VM. . .has anyone gotten a resolution from VMware on this? Or do I need to:

a. modify all the vmx files?

b. do cold migrations?

Thanks,

Jeff

Reply
0 Kudos
jpoling
Enthusiast
Enthusiast

My issue turned out to be a silly admin mistake - mismatched network settings on vmotion vswitch for a couple hosts. . .once I corrected the network settings vmotion sailed.

Jeff

Reply
0 Kudos
Remko_N
Contributor
Contributor

After migrating 1 VM offline, all the other VM's can migrate online, no manual reconfiguration done. The .vmx file is updated automatically. I don't know why this is, but at least I can migrate my 3.0.2 VM's to the 3.5 hosts online.

Reply
0 Kudos
Funtoosh
Enthusiast
Enthusiast

I also came across similar issue when I was doing Storage VMotion on 3.5 host. I found that OS is the constrain. In my case my VM's were running XP

Thanks,

Funtoosh

Reply
0 Kudos