Same hardware, can't vmotion. Stops at 13% with A general system error occurred: Invalid argument.

[jakirby]

I have four IBM (now Lenovo) x3550 M4 (7914AC1) hosts that were purchased in groups of two several months apart.  The ones purchased at the same time can vmotion between each other but they can't vmotion with the ones purchased later OR with any of our older infrastructure (Nehalem level IBM's -- x3650 M3, x3550 M3, etc.).  The newer M4's can talk to the old hosts fine.  I can migrate powered-down guests.

VMware support hasn't been able to give any aid on this and have instead wasted my time (including not contacting me for weeks and also going on leave without passing it to another engineer; manager has failed on it, too). 

I've pulled the hardware apart -- as far as I can tell it's all the same hardware (looks the same, has the same part numbers and FRU numbers).  The machines were configured the same.

Firmware is not exactly the same on the hardware but is VERY close.  I've updated one of each generation with the same IBM Xpresspacks and so except for the pre-boot diagnostic image, I think they are the same.  I'll be checking this yet again next.

VMware drivers are the same.  They weren't.  They are now.  I've pulled latest 5.5 drivers for everything I can find a driver for (including the FC cards -- using the native Qlogic driver from vmware; the NICs (igb); the 10Gb nics (not in use ixgbe but did it anyway); the CIM provider, MegaRAID controller.  I used VUM to update them to latest 5.5 (standard license) with all patches.  vCenter is a vCSA 5.5 latest update.  I've gone through the BIOS settings side-by-side and made sure everything is the same.  I've done the same with everything I could think of in vSphere (found about 5 differences but nothing fixed it).

The Management network is the vMotion network.  It is on the same VLAN as one of the ones the guests are on (guests are split over 5 different VLANs).  VMware support set up a different vSwitch for vMotion with no luck.

Symptoms are the same regardless of direction (old to new, new to old) and don't care about the VM type (Windows 2008R2, CentOS 5.5, RHEL 6.7) or the hardware level (VM 10 or VM 7).  It also doesn't care if it's standard or high priority or if it's with the web client or the Windows client. 

The place with the most characteristic symptom that doesn't seem to show anywhere in googleland except in pastebin is in the host's /scratch/log/hostd.log file.  The host that is sending the guest throws a vmsvc.vm error VMotionCmdCb [1461010590046389] 'to': error -6: Invalid argument.

If anyone else has seen this (obviously someone has as it's on pastebin) and figured out, please enlighten me.

Thanks!

jeff

These are the two hostd.log files (newer, originating host first, then older, receiving host).

-----------------------

Originating server hostd.log:

-----------------------

2016-04-18T20:16:29.927Z [37501B70 verbose 'Default' opID=F2241F42-000018D1-64-1b user=vpxuser] AdapterServer: target='vim.VirtualMachine:2', method='refreshStorageInfo'

2016-04-18T20:16:29.956Z [37501B70 verbose 'Hbrsvc' opID=F2241F42-000018D1-64-1b user=vpxuser] Replicator: VmFileProviderCallback VM (id=2)

2016-04-18T20:16:29.972Z [37501B70 verbose 'Default' opID=cb7f34c4 user=vpxuser] AdapterServer: target='vim.PerformanceManager:ha-perfmgr', method='GetPerfCounter'

2016-04-18T20:16:29.978Z [37280B70 verbose 'Locale' opID=cb7f34c4 user=vpxuser] Default resource used for 'counter.vsanDomObj.writeThroughput.summary' expected in module 'perf'.

2016-04-18T20:16:29.992Z [37280B70 verbose 'Default' opID=cb7f34c4 user=vpxuser] AdapterServer: target='vim.HostSystem:ha-host', method='retrieveInternalCapability'

2016-04-18T20:16:29.993Z [37280B70 verbose 'Default' opID=cb7f34c4 user=vpxuser] AdapterServer: target='vim.PerformanceManager:ha-perfmgr', method='queryPerfCounterInt'

2016-04-18T20:16:30.002Z [37280B70 verbose 'Default' opID=cb7f34c4 user=vpxuser] AdapterServer: target='vim.LicenseManager:ha-license-manager', method='GetLicenses'

2016-04-18T20:16:30.002Z [37280B70 verbose 'Vimsvc.ha-license-manager' opID=cb7f34c4 user=vpxuser] Load: Loading existing file: /etc/vmware/license.cfg

2016-04-18T20:16:30.014Z [37280B70 verbose 'Default' opID=cb7f34c4 user=vpxuser] ha-license-manager:Validate -> Valid license detected for "VMware ESX Server 5.0" (lastError=0, desc.IsValid:Yes)

2016-04-18T20:16:30.172Z [35981B70 verbose 'Default' opID=F2241F42-000018D1-64-a2-71 user=vpxuser] AdapterServer: target='vim.host.VMotionManager:ha-vmotionmgr', method='prepareSourceEx'

2016-04-18T20:16:30.172Z [37280B70 info 'Vcsvc.VMotion' opID=F2241F42-000018D1-64-a2-71 user=vpxuser] PrepareSourceEx [1461010590046389], VM = '2'

2016-04-18T20:16:30.172Z [37280B70 info 'Vcsvc.VMotionSrc (1461010590046389)' opID=F2241F42-000018D1-64-a2-71 user=vpxuser] VMotionEntry: migrateType = 1

2016-04-18T20:16:30.172Z [37280B70 verbose 'Vcsvc.VMotionSrc (1461010590046389)' opID=F2241F42-000018D1-64-a2-71 user=vpxuser] JrnlCreate: Created 1461010590.14

2016-04-18T20:16:30.172Z [37280B70 info 'Vmsvc.vm:/vmfs/volumes/526aca09-9e531e90-b07b-e61f135547a7/guestvm.mydomain.edu/guestvm.mydomain.edu.vmx' opID=F2241F42-000018D1-64-a2-71 user=vpxuser] State Transition (VM_STATE_ON -> VM_STATE_EMIGRATING)

2016-04-18T20:16:30.172Z [37280B70 info 'Vmsvc.vm:/vmfs/volumes/526aca09-9e531e90-b07b-e61f135547a7/guestvm.mydomain.edu/guestvm.mydomain.eduvmx' opID=F2241F42-000018D1-64-a2-71 user=vpxuser] VMotionPrepare (1461010590046389): Sending 'to' srcIp=10.10.6.108 dstIp=10.10.6.73, type=1, encrypted=false, remoteThumbprint=28:5C:C3:9C:3E:BA:79:EA:B5:19:CC:A9:36:34:C1:10:48:46:F0:0E

2016-04-18T20:16:30.172Z [37280B70 info 'Vmsvc.vm:/vmfs/volumes/526aca09-9e531e90-b07b-e61f135547a7/guestvm.mydomain.edu/guestvm.mydomain.edu.vmx' opID=F2241F42-000018D1-64-a2-71 user=vpxuser] VMotionPrepare: srcLoggingIp=

2016-04-18T20:16:30.172Z [37280B70 info 'Vmsvc.vm:/vmfs/volumes/526aca09-9e531e90-b07b-e61f135547a7/guestvm.mydomain.edu/guestvm.mydomain.edu.vmx' opID=F2241F42-000018D1-64-a2-71 user=vpxuser] VMotionPrepare: dstLoggingIp=

2016-04-18T20:16:30.172Z [37280B70 info 'Vmsvc.vm:/vmfs/volumes/526aca09-9e531e90-b07b-e61f135547a7/guestvm.mydomain.edu/guestvm.mydomain.edu.vmx' opID=F2241F42-000018D1-64-a2-71 user=vpxuser] VMotionPrepare: srcMgmtIp=10.10.6.108

2016-04-18T20:16:30.172Z [37280B70 info 'Vmsvc.vm:/vmfs/volumes/526aca09-9e531e90-b07b-e61f135547a7/guestvm.mydomain.edu/guestvm.mydomain.edu.vmx' opID=F2241F42-000018D1-64-a2-71 user=vpxuser] VMotionPrepare: dstMgmtIp=2620:91:0:96::73

2016-04-18T20:16:30.172Z [37280B70 info 'Vmsvc.vm:/vmfs/volumes/526aca09-9e531e90-b07b-e61f135547a7/guestvm.mydomain.edu/guestvm.mydomain.edu.vmx' opID=F2241F42-000018D1-64-a2-71 user=vpxuser] VMotionPrepare: dstVmDirPath=/vmfs/volumes/526aca09-9e531e90-b07b-e61f135547a7/guestvm.mydomain.edu/

2016-04-18T20:16:30.172Z [37280B70 info 'Vmsvc.vm:/vmfs/volumes/526aca09-9e531e90-b07b-e61f135547a7/guestvm.mydomain.edu/guestvm.mydomain.edu.vmx' opID=F2241F42-000018D1-64-a2-71 user=vpxuser] VMotionPrepare: dstVmFileName=guestvm.mydomain.edu.vmx

2016-04-18T20:16:30.173Z [37280B70 info 'Vmsvc.vm:/vmfs/volumes/526aca09-9e531e90-b07b-e61f135547a7/guestvm.mydomain.edu/guestvm.mydomain.edu.vmx' opID=F2241F42-000018D1-64-a2-71 user=vpxuser] SetVmdbVirtualDeviceChange: deviceChange # 0

SIOC: SIOC is notified not to start injector

2016-04-18T20:16:30.173Z [37280B70 verbose 'Vcsvc.VMotionSrc (1461010590046389)' opID=F2241F42-000018D1-64-a2-71 user=vpxuser] Migration changed state from NONE to BEGIN

2016-04-18T20:16:30.173Z [37280B70 info 'Vcsvc.VMotion' opID=F2241F42-000018D1-64-a2-71 user=vpxuser] Completed scheduling PrepareSourceEx [1461010590046389].

2016-04-18T20:16:30.173Z [36591B70 info 'vm:/vmfs/volumes/526aca09-9e531e90-b07b-e61f135547a7/guestvm.mydomain.edu/guestvm.mydomain.edu.vmx']  Disconnect check in progress.

2016-04-18T20:16:30.173Z [FFA4FB70 verbose 'Default' opID=F2241F42-000018D1-64-a2-71 user=vpxuser] AdapterServer: target='vmodl.query.PropertyCollector:ha-property-collector', method='createFilter'

2016-04-18T20:16:30.174Z [372C1B70 info 'Vmsvc.vm:/vmfs/volumes/526aca09-9e531e90-b07b-e61f135547a7/guestvm.mydomain.edu/guestvm.mydomain.edu.vmx'] VMotionCmdCb [1461010590046389] 'to': error -6: Invalid argument

2016-04-18T20:16:30.174Z [372C1B70 info 'Vmsvc.vm:/vmfs/volumes/526aca09-9e531e90-b07b-e61f135547a7/guestvm.mydomain.edu/guestvm.mydomain.edu.vmx'] VMotionCmdCb [1461010590046389] : Prepare task completed with error vmodl.fault.SystemError

2016-04-18T20:16:30.174Z [372C1B70 verbose 'Vmsvc.vm:/vmfs/volumes/526aca09-9e531e90-b07b-e61f135547a7/guestvm.mydomain.edu/guestvm.mydomain.edu.vmx'] VMotionCmdCb: Firing ResolveCb

2016-04-18T20:16:30.174Z [372C1B70 info 'Vcsvc.VMotionSrc (1461010590046389)'] ResolveCb: VMX reports needsUnregister = false for migrateType MIGRATE_TYPE_VMOTION

2016-04-18T20:16:30.174Z [372C1B70 info 'Vcsvc.VMotionSrc (1461010590046389)'] ResolveCb: Failed with fault: (vmodl.fault.SystemError) {

-->    dynamicType = <unset>,

-->    faultCause = (vmodl.MethodFault) null,

-->    reason = "Invalid argument",

-->    msg = "",

--> }

2016-04-18T20:16:30.174Z [372C1B70 verbose 'Vcsvc.VMotionSrc (1461010590046389)'] Migration changed state from BEGIN to DONE

2016-04-18T20:16:30.174Z [372C1B70 verbose 'Vcsvc.VMotionSrc (1461010590046389)'] Finish called

2016-04-18T20:16:30.174Z [372C1B70 info 'Vmsvc.vm:/vmfs/volumes/526aca09-9e531e90-b07b-e61f135547a7/guestvm.mydomain.edu/guestvm.mydomain.edu.vmx'] State Transition (VM_STATE_EMIGRATING -> VM_STATE_ON)

2016-04-18T20:16:30.174Z [37280B70 verbose 'Hostsvc'] Received state change for VM '2'

2016-04-18T20:16:30.174Z [37280B70 info 'Guestsvc.GuestFileTransferImpl'] Entered VmPowerStateListener

2016-04-18T20:16:30.174Z [37280B70 info 'Guestsvc.GuestFileTransferImpl'] VmPowerStateListener succeeded

2016-04-18T20:16:30.174Z [37280B70 info 'Hbrsvc'] Replicator: powerstate change VM: 2 Old: 1 New: 1

2016-04-18T20:16:30.174Z [372C1B70 info 'Hostsvc'] Decremented SIOC Injector Flag2

2016-04-18T20:16:30.174Z [372C1B70 info 'Vmsvc.vm:/vmfs/volumes/526aca09-9e531e90-b07b-e61f135547a7/guestvm.mydomain.edu/guestvm.mydomain.edu.vmx'] VMotion cleanup completed

2016-04-18T20:16:30.174Z [372C1B70 verbose 'Vcsvc.VMotionSrc (1461010590046389)'] JrnlEnd: Wrote active=0

2016-04-18T20:16:30.174Z [372C1B70 verbose 'Vcsvc.VMotionSrc (1461010590046389)'] JrnlEnd: Finished journal

2016-04-18T20:16:30.175Z [372C1B70 verbose 'Vcsvc.VMotionSrc (1461010590046389)'] [JrnlFilePersistenceProvider::DeleteJournal] Deleting /var/lib/vmware/hostd/journal/1461010590.14.

2016-04-18T20:16:30.175Z [372C1B70 verbose 'Vcsvc.VMotionSrc (1461010590046389)'] JrnlDelete: Deleted journal

2016-04-18T20:16:30.175Z [37280B70 verbose 'Hbrsvc'] Replicator: Remove group no matching entry for VM (id=2)

2016-04-18T20:16:30.180Z [37501B70 verbose 'Default' opID=F2241F42-000018D1-64-a2-71 user=vpxuser] AdapterServer: target='vmodl.query.PropertyCollector.Filter:session[574b3c38-6a6b-214e-4bf3-428b10abded9]52760c3e-4722-159c-5c3a-df6eb23ddbbc', method='destroy'

2016-04-18T20:16:30.201Z [372C1B70 verbose 'Default' opID=44d5f9ee user=vpxuser] AdapterServer: target='vim.PerformanceManager:ha-perfmgr', method='GetPerfCounter'

2016-04-18T20:16:30.207Z [372C1B70 verbose 'Locale' opID=44d5f9ee user=vpxuser] Default resource used for 'counter.vsanDomObj.writeThroughput.summary' expected in module 'perf'.

2016-04-18T20:16:30.223Z [37501B70 verbose 'Default' opID=44d5f9ee user=vpxuser] AdapterServer: target='vim.HostSystem:ha-host', method='retrieveInternalCapability'

2016-04-18T20:16:30.224Z [372C1B70 verbose 'Default' opID=44d5f9ee user=vpxuser] AdapterServer: target='vim.PerformanceManager:ha-perfmgr', method='queryPerfCounterInt'

2016-04-18T20:16:31.162Z [35981B70 verbose 'SoapAdapter'] Responded to service state request

2016-04-18T20:16:31.314Z [35981B70 verbose 'Default' opID=F2241F42-000018D1-64-a2-b7 user=vpxuser] AdapterServer: target='vim.host.VMotionManager:ha-vmotionmgr', method='completeSource'

2016-04-18T20:16:31.314Z [37501B70 info 'Vcsvc.VMotion' opID=F2241F42-000018D1-64-a2-b7 user=vpxuser] CompleteSource [1461010590046389]

2016-04-18T20:16:31.320Z [35981B70 verbose 'Default' opID=8590ebdf user=vpxuser] AdapterServer: target='vim.PerformanceManager:ha-perfmgr', method='GetPerfCounter'

2016-04-18T20:16:31.326Z [35981B70 verbose 'Locale' opID=8590ebdf user=vpxuser] Default resource used for 'counter.vsanDomObj.writeThroughput.summary' expected in module 'perf'.

2016-04-18T20:16:31.340Z [35981B70 verbose 'Default' opID=8590ebdf user=vpxuser] AdapterServer: target='vim.HostSystem:ha-host', method='retrieveInternalCapability'

2016-04-18T20:16:31.342Z [37501B70 verbose 'Default' opID=8590ebdf user=vpxuser] AdapterServer: target='vim.PerformanceManager:ha-perfmgr', method='queryPerfCounterInt'

2016-04-18T20:16:31.356Z [35981B70 verbose 'Default' opID=8590ebdf user=vpxuser] AdapterServer: target='vim.LicenseManager:ha-license-manager', method='GetLicenses'

2016-04-18T20:16:31.356Z [35981B70 verbose 'Vimsvc.ha-license-manager' opID=8590ebdf user=vpxuser] Load: Loading existing file: /etc/vmware/license.cfg

2016-04-18T20:16:31.371Z [35981B70 verbose 'Default' opID=8590ebdf user=vpxuser] ha-license-manager:Validate -> Valid license detected for "VMware ESX Server 5.0" (lastError=0, desc.IsValid:Yes)

-----------------------

Receiving host hostd.log:

-----------------------

2016-04-18T20:16:29.798Z [FFD35B70 verbose 'Hostsvc.DvsManager'] PersistAllDvsInfo called

2016-04-18T20:16:30.245Z [29940B70 verbose 'Default' opID=ac976ee5 user=vpxuser] AdapterServer: target='vim.PerformanceManager:ha-perfmgr', method='GetPerfCounter'

2016-04-18T20:16:30.251Z [29940B70 verbose 'Locale' opID=ac976ee5 user=vpxuser] Default resource used for 'counter.vsanDomObj.writeThroughput.summary' expected in module 'perf'.

2016-04-18T20:16:30.265Z [29940B70 verbose 'Default' opID=ac976ee5 user=vpxuser] AdapterServer: target='vim.HostSystem:ha-host', method='retrieveInternalCapability'

2016-04-18T20:16:30.267Z [29940B70 verbose 'Default' opID=ac976ee5 user=vpxuser] AdapterServer: target='vim.PerformanceManager:ha-perfmgr', method='queryPerfCounterInt'

2016-04-18T20:16:31.412Z [2BAC2B70 verbose 'Default' opID=F2241F42-000018D1-64-a2-e5 user=vpxuser] AdapterServer: target='vim.host.VMotionManager:ha-vmotionmgr', method='completeDestination'

2016-04-18T20:16:31.412Z [2BAC2B70 info 'Vcsvc.VMotion' opID=F2241F42-000018D1-64-a2-e5 user=vpxuser] CompleteDestination [1461010590046389]

2016-04-18T20:16:31.412Z [2BAC2B70 warning 'Vcsvc.VMotion' opID=F2241F42-000018D1-64-a2-e5 user=vpxuser] CompleteDestination: Map entry not found

2016-04-18T20:16:31.417Z [29940B70 verbose 'Default' opID=fa72f2f7 user=vpxuser] AdapterServer: target='vim.PerformanceManager:ha-perfmgr', method='GetPerfCounter'

2016-04-18T20:16:31.424Z [29940B70 verbose 'Locale' opID=fa72f2f7 user=vpxuser] Default resource used for 'counter.vsanDomObj.writeThroughput.summary' expected in module 'perf'.

2016-04-18T20:16:31.439Z [FFD35B70 verbose 'Default' opID=fa72f2f7 user=vpxuser] AdapterServer: target='vim.HostSystem:ha-host', method='retrieveInternalCapability'

2016-04-18T20:16:31.441Z [29940B70 verbose 'Default' opID=fa72f2f7 user=vpxuser] AdapterServer: target='vim.PerformanceManager:ha-perfmgr', method='queryPerfCounterInt'

[rcporto · Accepted solution]

Are you using IPv6 on the new hosts ? If yes, try disabling:

2016-04-18T20:16:30.172Z [37280B70 info 'Vmsvc.vm:/vmfs/volumes/526aca09-9e531e90-b07b-e61f135547a7/guestvm.mydomain.edu/guestvm.mydomain.eduvmx' opID=F2241F42-000018D1-64-a2-71 user=vpxuser] VMotionPrepare (1461010590046389): Sending 'to' srcIp=10.10.6.108 dstIp=10.10.6.73, type=1, encrypted=false, remoteThumbprint=28:5C:C3:9C:3E:BA:79:EA:B5:19:CC:A9:36:34:C1:10:48:46:F0:0E

2016-04-18T20:16:30.172Z [37280B70 info 'Vmsvc.vm:/vmfs/volumes/526aca09-9e531e90-b07b-e61f135547a7/guestvm.mydomain.edu/guestvm.mydomain.edu.vmx' opID=F2241F42-000018D1-64-a2-71 user=vpxuser] VMotionPrepare: srcLoggingIp=

2016-04-18T20:16:30.172Z [37280B70 info 'Vmsvc.vm:/vmfs/volumes/526aca09-9e531e90-b07b-e61f135547a7/guestvm.mydomain.edu/guestvm.mydomain.edu.vmx' opID=F2241F42-000018D1-64-a2-71 user=vpxuser] VMotionPrepare: dstLoggingIp=

2016-04-18T20:16:30.172Z [37280B70 info 'Vmsvc.vm:/vmfs/volumes/526aca09-9e531e90-b07b-e61f135547a7/guestvm.mydomain.edu/guestvm.mydomain.edu.vmx' opID=F2241F42-000018D1-64-a2-71 user=vpxuser] VMotionPrepare: srcMgmtIp=10.10.6.108

2016-04-18T20:16:30.172Z [37280B70 info 'Vmsvc.vm:/vmfs/volumes/526aca09-9e531e90-b07b-e61f135547a7/guestvm.mydomain.edu/guestvm.mydomain.edu.vmx' opID=F2241F42-000018D1-64-a2-71 user=vpxuser] VMotionPrepare: dstMgmtIp=2620:91:0:96::73

[jakirby]

Thank you, Richardson.  You were spot on.

It's more substantial than just what I reported before.  It appears that the situation is that when our networking team put the problematic pair into DNS, they entered both IPv4 and IPv6 addresses.  The non-problematic pair only was entered in DNS with IPv4 addresses (though they have IPv6 enabled and are reachable that way). 

I turned off IPv6 on the problematic test machine and rebooted it.  After that, vCenter could not find it.  I removed it from the inventory and added it back in to the same cluster it was in before.  It came back (though it took some time to find) and brought its guests back with it.  I was able to vMotion the test machine onto and off of it again.

However, with IPv6 addressing showing in the DNS entry and with no IPv6 enabled on the machine, it does not report most of the host's configuration.  It shows everything in the summary page, but in the configuration page it only shows processors, datastores, and power management.  It is unaware of network adapters, networking, storage and storage adapters, DNS and routing, Security profile, etc.  It's all blank.

In reality, two NICs are trunked in a port group together and are assigned the same IPv4 and IPv6 address.  Load Balancing is by IP Hash.

I'm having networking drop IPv6 on the hosts and I'm getting on with life. 

Thanks again!

jeff