vMotion stalls at 83% then times out

[ShawnPD]

Two strange occurrences on a Sunday evening...I am never doing maintenance on a Sunday evening again...

We have a cluster of 4 - HP BL 680c G7 blades. 

I am attempting to put hosts in maintenance mode to remediate them to the current patch level.

We are using esxi 5.1.0 trying to move all hosts to build 1063671

On each host the vms will evacuate, and the enter maintenance mode operation will get to 83% and stall.  After sometime the operation will time out in vCenter.   If I attempt to place the host into maintenance mode again, the host very quickly with complete the operation and enter maintenance mode.

This error messages shows up in the hostd.log of the affected host...not sure what they mean, but looks interesting

(this error repeated multiple times) 2013-05-06T02:59:58.200Z [5454DB90 info 'Vmomi'] Activation [N5Vmomi10ActivationE:0xffdbc1b8] : Invoke done [waitForUpdates] on [vmodl.query.PropertyCollector:ha-property-collector]

2013-05-06T02:59:58.200Z [5454DB90 verbose 'Vmomi'] Arg version:

--> "583"

2013-05-06T02:59:58.200Z [5454DB90 info 'Vmomi'] Throw vmodl.fault.RequestCanceled

2013-05-06T02:59:58.200Z [5454DB90 info 'Vmomi'] Result:

--> (vmodl.fault.RequestCanceled) {

-->    dynamicType = <unset>,

-->    faultCause = (vmodl.MethodFault) null,

-->    msg = "",

--> }

2013-05-06T02:59:58.201Z [54DD8B90 error 'SoapAdapter.HTTPService'] HTTP Transaction failed on stream TCP(error:Transport endpoint is not connected) with error N7Vmacore15SystemExceptionE(Connection reset by peer)

2013-05-06T03:05:02.700Z [54DD8B90 verbose 'Default'] CloseSession called for session id=79256627-c45a-0d32-bcfc-5c8acd7fa76f

2013-05-06T03:05:02.702Z [54DD8B90 info 'ha-eventmgr'] Event 448 : User vpxuser@127.0.0.1 logged out (login time: Thursday, 01 January, 1970 17:09:00, number of API invocations: 0, user agent: VMware-client/5.1.0)

2013-05-06T03:05:10.597Z [FF906D20 verbose 'Locale' opID=f3b563dc-a6] Default resource used for 'counter.sys.resourceMemConsumed.summary' expected in module 'perf'.

2013-05-06T03:05:10.601Z [544CBB90 verbose 'Locale' opID=f3b563dc-b3] Default resource used for 'counter.sys.resourceMemConsumed.summary' expected in module 'perf'.

2013-05-06T03:05:19.246Z [5446AB90 verbose 'SoapAdapter'] Responded to service state request

2013-05-06T03:09:04.093Z [54D80B90 error 'EnvironmentManager'] Unable to obtain the default HW version

2013-05-06T03:09:04.114Z [54922B90 verbose 'DvsTracker'] FetchSwitches: added 1 items

2013-05-06T03:09:04.114Z [54922B90 verbose 'DvsTracker'] FetchDVPortgroups: added 21 items

2013-05-06T03:09:04.114Z [54922B90 info 'DvsTracker'] FetchUplinkDVPortgroups: Appended uplinkPg for dvs bf d4 29 50 a8 e0 28 cb-aa 05 76 62 56 c9 9a 2a portgroup dvportgroup-31

2013-05-06T03:09:04.114Z [54922B90 info 'DvsTracker'] FetchUplinkDVPortgroups: added 21 items

Nothing useful shows up in the vCenter Task & Events Tab.  Is there a better place to look for errors or issues?