Migration fails but completes???

[sjjustin]

I'm getting an error in an ESXi 5.5 cluster when trying to live migrate VMs from hosts and datastores with the web client.  The VMs are located on local storage attached to the servers so I can't just do a host migration like with the VMs on our SAN.
The thing is, despite reporting as a failure (And taking a bit longer than usual to migrate) it actually does migrate and shows as complete in task details despite showing as a failure in tasks.

Here's a screenshot of what I mean, it might make it easier for people to understand.

[corvettefisher]

This most likely means that it could not communicate to the management agents properly.

You may want to check /var/log/hostd.log for errors or put the host in maintenance mode and restart the management agents on the affected hosts

[mathanmailid]

Restart the vpxa service on the target host. Refer - http://kb.vmware.com/kb/1003490

[sjjustin]

That log is huge!  After filtering out the info and verbose entries and looking at 1 migration event I get this:

2014-08-20T00:31:26.253Z [FFFD9B70 warning 'Vmsvc.vm___________.vmx'] Failed to find activation record, event user unknown.

2014-08-20T00:31:26.253Z [26C01B70 warning 'Vmsvc' opID=hostd-2adb] Vmsvc: Filtering VMs: ignored unexpected exception N5Vmomi5Fault21ManagedObjectNotFound9ExceptionE(vmodl.fault.ManagedObjectNotFound)

2014-08-20T00:40:20.619Z [26A42B80 warning 'Vmsvc.vm:___________.vmx' opID=76b516e4-2b-10 user=vpxuser] PopulateCache failed: _diskAccess : false, _storageAccessible : true

2014-08-20T00:40:20.622Z [26A42B80 warning 'Vmsvc.vm:___________.vmx' opID=76b516e4-2b-10 user=vpxuser] CannotRetrieveCorefiles: VM disk access is turned off

2014-08-20T00:40:21.957Z [26A42B80 warning 'Vmsvc.vm:___________.vmx'] PopulateCache failed: _diskAccess : false, _storageAccessible : true

2014-08-20T00:40:21.960Z [26A42B80 warning 'Vmsvc.vm:___________.vmx'] CannotRetrieveCorefiles: VM disk access is turned off

2014-08-20T00:40:21.965Z [26A42B80 warning 'Vmsvc.vm:___________.vmx'] Could not read host log /vmfs/volumes/50c9a07-c9348521-5685-001ec4ce13f6/[servername]/

2014-08-20T00:40:21.965Z [26A42B80 warning 'Default'] Complete Task: task haTask-ha-folder-vm-vim.Folder.registerVm-278144968 not registered as started

SIOC: SIOC is notified not to start injector

2014-08-20T00:40:34.742Z [25FA4B80 warning 'Vmsvc.vm:___________.vmx'] PopulateCache failed: _diskAccess : false, _storageAccessible : true

2014-08-20T00:40:34.752Z [25FA4B80 warning 'Vmsvc.vm:___________.vmx'] CannotRetrieveCorefiles: VM disk access is turned off

2014-08-20T00:40:34.768Z [25FA4B80 warning 'Vmsvc.vm:___________.vmx'] FetchUpdatedLayout: No cached layout files available. Doing a full fetch

I'm not sure what it means though, how can disk access be off?  And if there's an error why is it still completing with an error saying it failed?  I'm not sure how critical this is but I'd rather not leave the systems in limbo with migrations in case it causes further issues down the track.

As for the KB article I don't think it applies in this situation:

•You cannot connect ESXi/ESX to VirtualCenter / vCenter Server.

All hosts are connected to the vCenter Server

•You cannot connect directly to the ESXi/ESX host from the VMware Infrastructure Client / vSphere Client.

I can connect via vCenter or directly to each of the hosts

•You cannot stop or start a virtual machine.

I can start or stop a VM

•A virtual machine is shown as running in vCenter Server when it is not.

It is showing as running and is actually running, even after the error about migration

•vCenter Server displays the error: Virtual machine creation may fail because agent is unable to retrieve VM creation options from the host

That error does not appear.