Solved: Re: Hung VM's - Frequently Recurring Issue

FinFreeTX · ‎11-08-2010

This is a repost from the original thread here: http://communities.vmware.com/message/1641669

Just as the original poster stated, the issue is that with minimal load on the host machine all of my VM's get 'hung' about once a week - usually around the same time (12.45am), but on different days. I've gone through all of the automated processes on the VM's and my network and none of them correspond to the dates and times of the issue - so I'm not really sure what is causing this. It also happens occassionally at other times throughout the day/night. Here are the symptoms when the VM's 'hang':

All VM's do-not respond to ping requests or any other method of remote connection
CAN connect to host with vSphere Client, but no indication of issue under Performance or Events - all looks 'normal' (no spikes)
Attempting to click 'Console' tab, or launch console with the button causes vSphere Client to 'lock up' - cannot do anything else until vSphere Client is restarted
Attempting to reboot host via vSphere Client starts Maintenance Mode, begins attempting to shut down guest OS'es, then kicks you off - but never comes back online
After attempting to reboot via vSphere Clilent, Host console is 'locked' - no response when pushing F12 or any other keys
Host machine must be manually restarted by physically powering off and back on

I've enabled Tech Support Mode, so I am collecting logs of the issue - but I am unable to make heads or tails of them - and even worse the time stamps don't seem to be accurrate which makes it even harder. If there is anything specific I should be looking for, please advise and I will post.

Here are the details of my environment:

Host:

ESXi 4.1.0 26027
Dell PowerEdge 2900
Dual Xeon Quad-Core's
16GB RAM
Local SATA RAID 5 array
1x 1GB NIC - no VLANs

Guests:

1x Windows Server 2008 x64 - 2 vCPU - 4088MB
1x Windows Server 2008 x32 - 1vCPU - 2048MB
5x Windows XP 32bit (only 2 used - others are always OFF unless
needed) - 1 vCPU each - 256MB each

Host BIOS and firmwares on latest versions, no hardware issues detected

with Dell self diagnostics.

Please help??? Any suggestions would be greatly appreciated!

jamesbowling · ‎11-08-2010

How many ports is not really a concern right this second because you don't have anywhere near 56 ports being used. I would only increase this if needed anyway. I would hope that the RAID isn't a RAID0. Are the VMs that are running on this host critical? If so, then you may want to look at getting enough drives in place for a RAID5. We would want to look at the hostd.log file to see if there are any errors in it regarding these weird issues. Can you look through the hostd.log and see if there is anything odd in them and if so, post that portion of the log here?

If you found this at all helpful please award points by using the correct or helpful buttons! Thanks!

James B. | Blog: http://www.vSential.com | Twitter: @vSential --- If you found this helpful then please awards helpful or correct points accordingly. Thanks!

View solution in original post

golddiggie · ‎11-08-2010

Before I'd do anything else, I'd add more pNIC's to that box... A single NIC inside a host is horrible... Add either a dual, pair of dual, or quad port Intel (Gb) server class NIC (make sure it's listed on the VMware HCL) and set it up for VM traffic. I do hope that your physical switches are Gb...

How many drives are you using to make up the RAID 5 array? What RAID controller are you using? How is it configured (stripe size)??

VMware VCP4

Consider awarding points for "helpful" and/or "correct" answers.

jamesbowling · ‎11-08-2010

I would definitely make sure that you add more pNICs to this host. It is recommended best practice to have multiple pNICs on your hosts for redundancy.I am assuming that you have management network and everything running on that 1 pNIC through a single vSwitch?

If you found this at all helpful please award points by using the correct or helpful buttons! Thanks!

James B. | Blog: http://www.vSential.com | Twitter: @vSential --- If you found this helpful then please awards helpful or correct points accordingly. Thanks!

FinFreeTX · ‎11-08-2010

I went ahead and enabled the 2nd port on the onboard NIC (Broadcom NeXtreme II BCM5708), and added it as an active adaptor in the vSwitch. Any suggestions on how many ports to assign it? It was set to 56, so I bumped it up to 120. I'll look into installing a secondary NIC as-well. I am using Dell PowerConnect 24xx 1Gb switches, but they are not managed

I didn't actually setup the RAID array, so I'd have to reboot to check the settings on the console - but it is 2x 1TB SATA drives, so I assume it is a standard RAID 0. It's a SAS6iR SAS RAID Controller.

jamesbowling · ‎11-08-2010

How many ports is not really a concern right this second because you don't have anywhere near 56 ports being used. I would only increase this if needed anyway. I would hope that the RAID isn't a RAID0. Are the VMs that are running on this host critical? If so, then you may want to look at getting enough drives in place for a RAID5. We would want to look at the hostd.log file to see if there are any errors in it regarding these weird issues. Can you look through the hostd.log and see if there is anything odd in them and if so, post that portion of the log here?

If you found this at all helpful please award points by using the correct or helpful buttons! Thanks!

James B. | Blog: http://www.vSential.com | Twitter: @vSential --- If you found this helpful then please awards helpful or correct points accordingly. Thanks!

FinFreeTX · ‎11-08-2010

Checked with our old Sys Admin and he says it's a RAID 1 array.

Not sure exactly what I am looking for in the hostd.log file, but the only error/failure I am seeing frequently is:

Failed to read header on stream TCP(local=127.0.0.1:58689, peer=127.0.0.1:0): N7Vmacore15SystemExceptionE(Connection reset by peer)

Ticket issued for CIMOM version 1.0, user root

Some of these do match up with the times of the outages - is this what I am looking for???

jamesbowling · ‎11-08-2010

You definitely want to focus on times on or right before you see this oddity happen. We would want to look for any errors, warnings, any other logged information that may show something attempting to happen or what not.

If you found this at all helpful please award points by using the correct or helpful buttons! Thanks!

James B. | Blog: http://www.vSential.com | Twitter: @vSential --- If you found this helpful then please awards helpful or correct points accordingly. Thanks!

FinFreeTX · ‎11-08-2010

OK, tomorrow I'll go through that log and lookup the last couple of times the issue occurred and see what I find. I assume the error I posted earlier doesn't tell you anything?

jamesbowling · ‎11-08-2010

Not really. Let us know what you find.

James Bowling

James B. | Blog: http://www.vSential.com | Twitter: @vSential --- If you found this helpful then please awards helpful or correct points accordingly. Thanks!

FinFreeTX · ‎11-10-2010

Sorry, I'm just getting the chance to go through these logs and unfortunately they do-not go back far enough to view the last few times this happened. In-fact, all of the log entries are from today. Are these archived anywhere? If-not, I'll have to wait for the issue to happen again before I will be able to get a sample from the logs...

jamesbowling · ‎11-10-2010

Logs are rotated and live in /var/log/vmware. You will see them there with a .gz extension where you can use zless hostd-1.log.gz to view the logs from the cli.

If you found this at all helpful please award points by using the correct or helpful buttons! Thanks!

James B. | Blog: http://www.vSential.com | Twitter: @vSential --- If you found this helpful then please awards helpful or correct points accordingly. Thanks!

FinFreeTX · ‎11-10-2010

I've checked through all of the archived logs and the farthest they are going back is 11/05 which is not far-enough...

jamesbowling · ‎11-10-2010

So the logs have rotated out since then. You can check out my post about changing the amount of hostd logs kept here: http://vsential.com/archives/adjusting-the-amount-of-hostd-log-files/

If you found this at all helpful please award points by using the correct or helpful buttons! Thanks!

James B. | Blog: http://www.vSential.com | Twitter: @vSential --- If you found this helpful then please awards helpful or correct points accordingly. Thanks!

FinFreeTX · ‎11-12-2010

The issue happened again this morning - 2nd time this week (it's getting more frequent). Here are the hostd logs from the 10 minute period around the time the issue occurred. I didn't see anything that stood out, but I don't really know what I'm looking for. Please review and let me know if you see anything helpful? I'm also attaching the full log file if you need to see a wider time-frame.

THANKS for all your help with this issue - it's killing me!

New proxy client TCP(local=127.0.0.1:61002, peer=127.0.0.1:80)

Activation : Invoke done on

Arg version:

"42"

Throw vmodl.fault.RequestCanceled

Result:

(vmodl.fault.RequestCanceled) {

dynamicType = <unset>,

faultCause = (vmodl.MethodFault) null,

msg = "",

}

Failed to read header on stream TCP(local=127.0.0.1:60369, peer=127.0.0.1:0): N7Vmacore15SystemExceptionE(Connection reset by peer)

Ticket issued for CIMOM version 1.0, user root

Actual VM overhead: 112205824 bytes

RefreshVms updated overhead for 1 VM

Ticket issued for CIMOM version 1.0, user root

New proxy client TCP(local=192.168.1.143:4789, peer=192.168.1.63:80)

Ticket issued for CIMOM version 1.0, user root

New proxy client TCP(local=192.168.1.143:4790, peer=192.168.1.63:80)

PersistAllDvsInfo called

Ticket issued for CIMOM version 1.0, user root

New proxy client TCP(local=127.0.0.1:55350, peer=127.0.0.1:80)

Activation : Invoke done on

Arg version:

"42"

Throw vmodl.fault.RequestCanceled

Result:

(vmodl.fault.RequestCanceled) {

dynamicType = <unset>,

faultCause = (vmodl.MethodFault) null,

msg = "",

}

Failed to read header on stream TCP(local=127.0.0.1:50148, peer=127.0.0.1:0): N7Vmacore15SystemExceptionE(Connection reset by peer)

New proxy client SSL(TCP(local=192.168.1.143:4791, peer=192.168.1.63:443))

New proxy client SSL(TCP(local=192.168.1.143:4792, peer=192.168.1.63:443))

Activation : Invoke done on [vmodl.query.PropertyCollector:session[529ac840-c0d8-74a5-7c8d-133887163e8e]5226072c-58ee-bbff-d378-769979f48a41]

Arg version:

"1"

Throw vmodl.fault.RequestCanceled

Result:

(vmodl.fault.RequestCanceled) {

dynamicType = <unset>,

faultCause = (vmodl.MethodFault) null,

msg = "",

}

Activation : Invoke done on [vmodl.query.PropertyCollector:session[529ac840-c0d8-74a5-7c8d-133887163e8e]5226072c-58ee-bbff-d378-769979f48a41]

Arg version:

"1"

Throw vmodl.fault.RequestCanceled

Result:

(vmodl.fault.RequestCanceled) {

dynamicType = <unset>,

faultCause = (vmodl.MethodFault) null,

msg = "",

}

Failed to read header on stream TCP(local=127.0.0.1:56942, peer=127.0.0.1:0): N7Vmacore15SystemExceptionE(Connection reset by peer)

Ticket issued for CIMOM version 1.0, user root

New proxy client TCP(local=192.168.1.143:4793, peer=192.168.1.63:80)

New proxy client TCP(local=192.168.1.143:4794, peer=192.168.1.63:80)

Ticket issued for CIMOM version 1.0, user root

PersistAllDvsInfo called

New proxy client TCP(local=192.168.1.143:4807, peer=192.168.1.63:80)

New proxy client TCP(local=127.0.0.1:63397, peer=127.0.0.1:80)

Activation : Invoke done on

Arg version:

"42"

Throw vmodl.fault.RequestCanceled

Result:

(vmodl.fault.RequestCanceled) {

dynamicType = <unset>,

faultCause = (vmodl.MethodFault) null,

msg = "",

}

Failed to read header on stream TCP(local=127.0.0.1:52944, peer=127.0.0.1:0): N7Vmacore15SystemExceptionE(Connection reset by peer)

Ticket issued for CIMOM version 1.0, user root

Actual VM overhead: 248696832 bytes

RefreshVms updated overhead for 1 VM

Ticket issued for CIMOM version 1.0, user root

New proxy client TCP(local=192.168.1.143:4813, peer=192.168.1.63:80)

Ticket issued for CIMOM version 1.0, user root

PersistAllDvsInfo called

New proxy client TCP(local=192.168.1.143:4814, peer=192.168.1.63:80)

New proxy client TCP(local=127.0.0.1:63153, peer=127.0.0.1:80)

Activation : Invoke done on

Arg version:

"42"

Throw vmodl.fault.RequestCanceled

Result:

(vmodl.fault.RequestCanceled) {

dynamicType = <unset>,

faultCause = (vmodl.MethodFault) null,

msg = "",

}

Failed to read header on stream TCP(local=127.0.0.1:62034, peer=127.0.0.1:0): N7Vmacore15SystemExceptionE(Connection reset by peer)

Ticket issued for CIMOM version 1.0, user root

Actual VM overhead: 113254400 bytes

RefreshVms updated overhead for 1 VM

FinFreeTX · ‎11-12-2010

Also, per the suggestion of our former sys admin who actually setup this environment, we are running the Kiwi Syslog Daemon tool to catch all of the log info from the host, and here are the logs it caught from the 10 minute period prior to the outage:

2010-11-12 00:32:04 Local6.Warning 192.168.1.63 Nov 12 06:32:04 vmkernel: 6:17:02:34.587 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:05 Local6.Warning 192.168.1.63 Nov 12 06:32:05 vmkernel: 6:17:02:34.932 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:06 Local6.Warning 192.168.1.63 Nov 12 06:32:06 vmkernel: 6:17:02:36.282 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:06 Local6.Warning 192.168.1.63 Nov 12 06:32:06 vmkernel: 6:17:02:36.285 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:06 Local6.Warning 192.168.1.63 Nov 12 06:32:06 vmkernel: 6:17:02:36.290 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:06 Local6.Warning 192.168.1.63 Nov 12 06:32:06 vmkernel: 6:17:02:36.294 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:06 Local6.Warning 192.168.1.63 Nov 12 06:32:06 vmkernel: 6:17:02:36.297 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:06 Local6.Warning 192.168.1.63 Nov 12 06:32:06 vmkernel: 6:17:02:36.300 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:06 Local6.Warning 192.168.1.63 Nov 12 06:32:06 vmkernel: 6:17:02:36.303 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:07 Local6.Warning 192.168.1.63 Nov 12 06:32:07 vmkernel: 6:17:02:37.307 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:07 Local6.Warning 192.168.1.63 Nov 12 06:32:07 vmkernel: 6:17:02:37.310 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:07 Local6.Warning 192.168.1.63 Nov 12 06:32:07 vmkernel: 6:17:02:37.314 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:09 Local6.Warning 192.168.1.63 Nov 12 06:32:09 vmkernel: 6:17:02:39.376 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:09 Local6.Warning 192.168.1.63 Nov 12 06:32:09 vmkernel: 6:17:02:39.380 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:09 Local6.Warning 192.168.1.63 Nov 12 06:32:09 vmkernel: 6:17:02:39.383 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:09 Local6.Warning 192.168.1.63 Nov 12 06:32:09 vmkernel: 6:17:02:39.386 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:09 Local6.Warning 192.168.1.63 Nov 12 06:32:09 vmkernel: 6:17:02:39.389 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:09 Local6.Warning 192.168.1.63 Nov 12 06:32:09 vmkernel: 6:17:02:39.392 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:09 Local6.Warning 192.168.1.63 Nov 12 06:32:09 vmkernel: 6:17:02:39.395 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:09 Local6.Warning 192.168.1.63 Nov 12 06:32:09 vmkernel: 6:17:02:39.397 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:10 Local6.Warning 192.168.1.63 Nov 12 06:32:10 vmkernel: 6:17:02:40.457 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:11 Local6.Warning 192.168.1.63 Nov 12 06:32:11 vmkernel: 6:17:02:41.461 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:11 Local6.Warning 192.168.1.63 Nov 12 06:32:11 vmkernel: 6:17:02:41.464 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:11 Local6.Warning 192.168.1.63 Nov 12 06:32:11 vmkernel: 6:17:02:41.467 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:11 Local6.Warning 192.168.1.63 Nov 12 06:32:11 vmkernel: 6:17:02:41.470 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:11 Local6.Warning 192.168.1.63 Nov 12 06:32:11 vmkernel: 6:17:02:41.473 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:11 Local6.Warning 192.168.1.63 Nov 12 06:32:11 vmkernel: 6:17:02:41.476 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:11 Local6.Warning 192.168.1.63 Nov 12 06:32:11 vmkernel: 6:17:02:41.478 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:12 Local6.Warning 192.168.1.63 Nov 12 06:32:12 vmkernel: 6:17:02:42.699 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:12 Local6.Warning 192.168.1.63 Nov 12 06:32:12 vmkernel: 6:17:02:42.703 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:12 Local6.Warning 192.168.1.63 Nov 12 06:32:12 vmkernel: 6:17:02:42.706 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:12 Local6.Warning 192.168.1.63 Nov 12 06:32:12 vmkernel: 6:17:02:42.709 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:12 Local6.Warning 192.168.1.63 Nov 12 06:32:12 vmkernel: 6:17:02:42.712 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:12 Local6.Warning 192.168.1.63 Nov 12 06:32:12 vmkernel: 6:17:02:42.715 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:12 Local6.Warning 192.168.1.63 Nov 12 06:32:12 vmkernel: 6:17:02:42.718 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:12 Local6.Warning 192.168.1.63 Nov 12 06:32:12 vmkernel: 6:17:02:42.720 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:13 Local6.Warning 192.168.1.63 Nov 12 06:32:14 vmkernel: 6:17:02:43.775 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:13 Local6.Warning 192.168.1.63 Nov 12 06:32:14 vmkernel: 6:17:02:43.779 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:13 Local6.Warning 192.168.1.63 Nov 12 06:32:14 vmkernel: 6:17:02:43.782 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:13 Local6.Warning 192.168.1.63 Nov 12 06:32:14 vmkernel: 6:17:02:43.785 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:14 Local4.Info 192.168.1.63 Nov 12 06:32:15 Hostd: Ticket issued for CIMOM version 1.0, user root

2010-11-12 00:32:56 Local6.Warning 192.168.1.63 Nov 12 06:32:56 vmkernel: 6:17:03:26.292 cpu2:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:56 Local6.Warning 192.168.1.63 Nov 12 06:32:56 vmkernel: 6:17:03:26.298 cpu3:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:56 Local6.Warning 192.168.1.63 Nov 12 06:32:56 vmkernel: 6:17:03:26.299 cpu2:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.030 cpu2:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f4ec940) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x