FinFreeTX
Contributor
Contributor

Hung VM's - Frequently Recurring Issue

Jump to solution

This is a repost from the original thread here: http://communities.vmware.com/message/1641669

Just as the original poster stated, the issue is that with minimal load on the host machine all of my VM's get 'hung' about once a week - usually around the same time (12.45am), but on different days. I've gone through all of the automated processes on the VM's and my network and none of them correspond to the dates and times of the issue - so I'm not really sure what is causing this. It also happens occassionally at other times throughout the day/night. Here are the symptoms when the VM's 'hang':

  • All VM's do-not respond to ping requests or any other method of remote connection

  • CAN connect to host with vSphere Client, but no indication of issue under Performance or Events - all looks 'normal' (no spikes)

  • Attempting to click 'Console' tab, or launch console with the button causes vSphere Client to 'lock up' - cannot do anything else until vSphere Client is restarted

  • Attempting to reboot host via vSphere Client starts Maintenance Mode, begins attempting to shut down guest OS'es, then kicks you off - but never comes back online

  • After attempting to reboot via vSphere Clilent, Host console is 'locked' - no response when pushing F12 or any other keys

  • Host machine must be manually restarted by physically powering off and back on

I've enabled Tech Support Mode, so I am collecting logs of the issue - but I am unable to make heads or tails of them - and even worse the time stamps don't seem to be accurrate which makes it even harder. If there is anything specific I should be looking for, please advise and I will post.

Here are the details of my environment:

Host:

  • ESXi 4.1.0 26027

  • Dell PowerEdge 2900

  • Dual Xeon Quad-Core's

  • 16GB RAM

  • Local SATA RAID 5 array

  • 1x 1GB NIC - no VLANs

Guests:

  • 1x Windows Server 2008 x64 - 2 vCPU - 4088MB

  • 1x Windows Server 2008 x32 - 1vCPU - 2048MB

  • 5x Windows XP 32bit (only 2 used - others are always OFF unless
    needed) - 1 vCPU each - 256MB each

Host BIOS and firmwares on latest versions, no hardware issues detected

with Dell self diagnostics.

Please help??? Any suggestions would be greatly appreciated!

0 Kudos
1 Solution

Accepted Solutions
jamesbowling
VMware Employee
VMware Employee

How many ports is not really a concern right this second because you don't have anywhere near 56 ports being used. I would only increase this if needed anyway. I would hope that the RAID isn't a RAID0. Are the VMs that are running on this host critical? If so, then you may want to look at getting enough drives in place for a RAID5. We would want to look at the hostd.log file to see if there are any errors in it regarding these weird issues. Can you look through the hostd.log and see if there is anything odd in them and if so, post that portion of the log here?

If you found this at all helpful please award points by using the correct or helpful buttons! Thanks!

James B. | Blog: http://www.vSential.com | Twitter: @vSential --- If you found this helpful then please awards helpful or correct points accordingly. Thanks!

View solution in original post

0 Kudos
37 Replies
golddiggie
Champion
Champion

Before I'd do anything else, I'd add more pNIC's to that box... A single NIC inside a host is horrible... Add either a dual, pair of dual, or quad port Intel (Gb) server class NIC (make sure it's listed on the VMware HCL) and set it up for VM traffic. I do hope that your physical switches are Gb...

How many drives are you using to make up the RAID 5 array? What RAID controller are you using? How is it configured (stripe size)??

VMware VCP4

Consider awarding points for "helpful" and/or "correct" answers.

Hosted Systems Engineer IV (VMware environment)
Brewing beer again!
jamesbowling
VMware Employee
VMware Employee

I would definitely make sure that you add more pNICs to this host. It is recommended best practice to have multiple pNICs on your hosts for redundancy.I am assuming that you have management network and everything running on that 1 pNIC through a single vSwitch?

If you found this at all helpful please award points by using the correct or helpful buttons! Thanks!

James B. | Blog: http://www.vSential.com | Twitter: @vSential --- If you found this helpful then please awards helpful or correct points accordingly. Thanks!
FinFreeTX
Contributor
Contributor

I went ahead and enabled the 2nd port on the onboard NIC (Broadcom NeXtreme II BCM5708), and added it as an active adaptor in the vSwitch. Any suggestions on how many ports to assign it? It was set to 56, so I bumped it up to 120. I'll look into installing a secondary NIC as-well. I am using Dell PowerConnect 24xx 1Gb switches, but they are not managed Smiley Sad

I didn't actually setup the RAID array, so I'd have to reboot to check the settings on the console - but it is 2x 1TB SATA drives, so I assume it is a standard RAID 0. It's a SAS6iR SAS RAID Controller.

0 Kudos
jamesbowling
VMware Employee
VMware Employee

How many ports is not really a concern right this second because you don't have anywhere near 56 ports being used. I would only increase this if needed anyway. I would hope that the RAID isn't a RAID0. Are the VMs that are running on this host critical? If so, then you may want to look at getting enough drives in place for a RAID5. We would want to look at the hostd.log file to see if there are any errors in it regarding these weird issues. Can you look through the hostd.log and see if there is anything odd in them and if so, post that portion of the log here?

If you found this at all helpful please award points by using the correct or helpful buttons! Thanks!

James B. | Blog: http://www.vSential.com | Twitter: @vSential --- If you found this helpful then please awards helpful or correct points accordingly. Thanks!

View solution in original post

0 Kudos
FinFreeTX
Contributor
Contributor

Checked with our old Sys Admin and he says it's a RAID 1 array.

Not sure exactly what I am looking for in the hostd.log file, but the only error/failure I am seeing frequently is:

Failed to read header on stream TCP(local=127.0.0.1:58689, peer=127.0.0.1:0): N7Vmacore15SystemExceptionE(Connection reset by peer)

Ticket issued for CIMOM version 1.0, user root

Some of these do match up with the times of the outages - is this what I am looking for???

0 Kudos
jamesbowling
VMware Employee
VMware Employee

You definitely want to focus on times on or right before you see this oddity happen. We would want to look for any errors, warnings, any other logged information that may show something attempting to happen or what not.

If you found this at all helpful please award points by using the correct or helpful buttons! Thanks!

James B. | Blog: http://www.vSential.com | Twitter: @vSential --- If you found this helpful then please awards helpful or correct points accordingly. Thanks!
0 Kudos
FinFreeTX
Contributor
Contributor

OK, tomorrow I'll go through that log and lookup the last couple of times the issue occurred and see what I find. I assume the error I posted earlier doesn't tell you anything?

0 Kudos
jamesbowling
VMware Employee
VMware Employee

Not really. Let us know what you find.

James Bowling

James B. | Blog: http://www.vSential.com | Twitter: @vSential --- If you found this helpful then please awards helpful or correct points accordingly. Thanks!
0 Kudos
FinFreeTX
Contributor
Contributor

Sorry, I'm just getting the chance to go through these logs and unfortunately they do-not go back far enough to view the last few times this happened. In-fact, all of the log entries are from today. Are these archived anywhere? If-not, I'll have to wait for the issue to happen again before I will be able to get a sample from the logs...

0 Kudos
jamesbowling
VMware Employee
VMware Employee

Logs are rotated and live in /var/log/vmware. You will see them there with a .gz extension where you can use zless hostd-1.log.gz to view the logs from the cli.

If you found this at all helpful please award points by using the correct or helpful buttons! Thanks!

James B. | Blog: http://www.vSential.com | Twitter: @vSential --- If you found this helpful then please awards helpful or correct points accordingly. Thanks!
0 Kudos
FinFreeTX
Contributor
Contributor

I've checked through all of the archived logs and the farthest they are going back is 11/05 which is not far-enough...

0 Kudos
jamesbowling
VMware Employee
VMware Employee

So the logs have rotated out since then. You can check out my post about changing the amount of hostd logs kept here: http://vsential.com/archives/adjusting-the-amount-of-hostd-log-files/

If you found this at all helpful please award points by using the correct or helpful buttons! Thanks!

James B. | Blog: http://www.vSential.com | Twitter: @vSential --- If you found this helpful then please awards helpful or correct points accordingly. Thanks!
0 Kudos
FinFreeTX
Contributor
Contributor

The issue happened again this morning - 2nd time this week (it's getting more frequent). Here are the hostd logs from the 10 minute period around the time the issue occurred. I didn't see anything that stood out, but I don't really know what I'm looking for. Please review and let me know if you see anything helpful? I'm also attaching the full log file if you need to see a wider time-frame.

THANKS for all your help with this issue - it's killing me!

New proxy client TCP(local=127.0.0.1:61002, peer=127.0.0.1:80)

Activation : Invoke done on

Arg version:

"42"

Throw vmodl.fault.RequestCanceled

Result:

(vmodl.fault.RequestCanceled) {

dynamicType = <unset>,

faultCause = (vmodl.MethodFault) null,

msg = "",

}

Failed to read header on stream TCP(local=127.0.0.1:60369, peer=127.0.0.1:0): N7Vmacore15SystemExceptionE(Connection reset by peer)

Ticket issued for CIMOM version 1.0, user root

Actual VM overhead: 112205824 bytes

RefreshVms updated overhead for 1 VM

Ticket issued for CIMOM version 1.0, user root

New proxy client TCP(local=192.168.1.143:4789, peer=192.168.1.63:80)

Ticket issued for CIMOM version 1.0, user root

New proxy client TCP(local=192.168.1.143:4790, peer=192.168.1.63:80)

PersistAllDvsInfo called

Ticket issued for CIMOM version 1.0, user root

New proxy client TCP(local=127.0.0.1:55350, peer=127.0.0.1:80)

Activation : Invoke done on

Arg version:

"42"

Throw vmodl.fault.RequestCanceled

Result:

(vmodl.fault.RequestCanceled) {

dynamicType = <unset>,

faultCause = (vmodl.MethodFault) null,

msg = "",

}

Failed to read header on stream TCP(local=127.0.0.1:50148, peer=127.0.0.1:0): N7Vmacore15SystemExceptionE(Connection reset by peer)

New proxy client SSL(TCP(local=192.168.1.143:4791, peer=192.168.1.63:443))

New proxy client SSL(TCP(local=192.168.1.143:4792, peer=192.168.1.63:443))

Activation : Invoke done on [vmodl.query.PropertyCollector:session[529ac840-c0d8-74a5-7c8d-133887163e8e]5226072c-58ee-bbff-d378-769979f48a41]

Arg version:

"1"

Throw vmodl.fault.RequestCanceled

Result:

(vmodl.fault.RequestCanceled) {

dynamicType = <unset>,

faultCause = (vmodl.MethodFault) null,

msg = "",

}

Activation : Invoke done on [vmodl.query.PropertyCollector:session[529ac840-c0d8-74a5-7c8d-133887163e8e]5226072c-58ee-bbff-d378-769979f48a41]

Arg version:

"1"

Throw vmodl.fault.RequestCanceled

Result:

(vmodl.fault.RequestCanceled) {

dynamicType = <unset>,

faultCause = (vmodl.MethodFault) null,

msg = "",

}

Failed to read header on stream TCP(local=127.0.0.1:56942, peer=127.0.0.1:0): N7Vmacore15SystemExceptionE(Connection reset by peer)

Ticket issued for CIMOM version 1.0, user root

New proxy client TCP(local=192.168.1.143:4793, peer=192.168.1.63:80)

New proxy client TCP(local=192.168.1.143:4794, peer=192.168.1.63:80)

Ticket issued for CIMOM version 1.0, user root

Ticket issued for CIMOM version 1.0, user root

PersistAllDvsInfo called

New proxy client TCP(local=192.168.1.143:4807, peer=192.168.1.63:80)

New proxy client TCP(local=127.0.0.1:63397, peer=127.0.0.1:80)

Activation : Invoke done on

Arg version:

"42"

Throw vmodl.fault.RequestCanceled

Result:

(vmodl.fault.RequestCanceled) {

dynamicType = <unset>,

faultCause = (vmodl.MethodFault) null,

msg = "",

}

Failed to read header on stream TCP(local=127.0.0.1:52944, peer=127.0.0.1:0): N7Vmacore15SystemExceptionE(Connection reset by peer)

Ticket issued for CIMOM version 1.0, user root

Actual VM overhead: 248696832 bytes

RefreshVms updated overhead for 1 VM

Ticket issued for CIMOM version 1.0, user root

New proxy client TCP(local=192.168.1.143:4813, peer=192.168.1.63:80)

Ticket issued for CIMOM version 1.0, user root

PersistAllDvsInfo called

New proxy client TCP(local=192.168.1.143:4814, peer=192.168.1.63:80)

New proxy client TCP(local=127.0.0.1:63153, peer=127.0.0.1:80)

Activation : Invoke done on

Arg version:

"42"

Throw vmodl.fault.RequestCanceled

Result:

(vmodl.fault.RequestCanceled) {

dynamicType = <unset>,

faultCause = (vmodl.MethodFault) null,

msg = "",

}

Failed to read header on stream TCP(local=127.0.0.1:62034, peer=127.0.0.1:0): N7Vmacore15SystemExceptionE(Connection reset by peer)

Ticket issued for CIMOM version 1.0, user root

Actual VM overhead: 113254400 bytes

RefreshVms updated overhead for 1 VM

0 Kudos
FinFreeTX
Contributor
Contributor

Also, per the suggestion of our former sys admin who actually setup this environment, we are running the Kiwi Syslog Daemon tool to catch all of the log info from the host, and here are the logs it caught from the 10 minute period prior to the outage:

2010-11-12 00:32:04 Local6.Warning 192.168.1.63 Nov 12 06:32:04 vmkernel: 6:17:02:34.587 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:05 Local6.Warning 192.168.1.63 Nov 12 06:32:05 vmkernel: 6:17:02:34.932 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:06 Local6.Warning 192.168.1.63 Nov 12 06:32:06 vmkernel: 6:17:02:36.282 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:06 Local6.Warning 192.168.1.63 Nov 12 06:32:06 vmkernel: 6:17:02:36.285 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:06 Local6.Warning 192.168.1.63 Nov 12 06:32:06 vmkernel: 6:17:02:36.290 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:06 Local6.Warning 192.168.1.63 Nov 12 06:32:06 vmkernel: 6:17:02:36.294 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:06 Local6.Warning 192.168.1.63 Nov 12 06:32:06 vmkernel: 6:17:02:36.297 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:06 Local6.Warning 192.168.1.63 Nov 12 06:32:06 vmkernel: 6:17:02:36.300 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:06 Local6.Warning 192.168.1.63 Nov 12 06:32:06 vmkernel: 6:17:02:36.303 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:07 Local6.Warning 192.168.1.63 Nov 12 06:32:07 vmkernel: 6:17:02:37.307 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:07 Local6.Warning 192.168.1.63 Nov 12 06:32:07 vmkernel: 6:17:02:37.310 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:07 Local6.Warning 192.168.1.63 Nov 12 06:32:07 vmkernel: 6:17:02:37.314 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:09 Local6.Warning 192.168.1.63 Nov 12 06:32:09 vmkernel: 6:17:02:39.376 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:09 Local6.Warning 192.168.1.63 Nov 12 06:32:09 vmkernel: 6:17:02:39.380 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:09 Local6.Warning 192.168.1.63 Nov 12 06:32:09 vmkernel: 6:17:02:39.383 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:09 Local6.Warning 192.168.1.63 Nov 12 06:32:09 vmkernel: 6:17:02:39.386 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:09 Local6.Warning 192.168.1.63 Nov 12 06:32:09 vmkernel: 6:17:02:39.389 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:09 Local6.Warning 192.168.1.63 Nov 12 06:32:09 vmkernel: 6:17:02:39.392 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:09 Local6.Warning 192.168.1.63 Nov 12 06:32:09 vmkernel: 6:17:02:39.395 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:09 Local6.Warning 192.168.1.63 Nov 12 06:32:09 vmkernel: 6:17:02:39.397 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:10 Local6.Warning 192.168.1.63 Nov 12 06:32:10 vmkernel: 6:17:02:40.457 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:11 Local6.Warning 192.168.1.63 Nov 12 06:32:11 vmkernel: 6:17:02:41.461 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:11 Local6.Warning 192.168.1.63 Nov 12 06:32:11 vmkernel: 6:17:02:41.464 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:11 Local6.Warning 192.168.1.63 Nov 12 06:32:11 vmkernel: 6:17:02:41.467 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:11 Local6.Warning 192.168.1.63 Nov 12 06:32:11 vmkernel: 6:17:02:41.470 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:11 Local6.Warning 192.168.1.63 Nov 12 06:32:11 vmkernel: 6:17:02:41.473 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:11 Local6.Warning 192.168.1.63 Nov 12 06:32:11 vmkernel: 6:17:02:41.476 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:11 Local6.Warning 192.168.1.63 Nov 12 06:32:11 vmkernel: 6:17:02:41.478 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:12 Local6.Warning 192.168.1.63 Nov 12 06:32:12 vmkernel: 6:17:02:42.699 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:12 Local6.Warning 192.168.1.63 Nov 12 06:32:12 vmkernel: 6:17:02:42.703 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:12 Local6.Warning 192.168.1.63 Nov 12 06:32:12 vmkernel: 6:17:02:42.706 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:12 Local6.Warning 192.168.1.63 Nov 12 06:32:12 vmkernel: 6:17:02:42.709 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:12 Local6.Warning 192.168.1.63 Nov 12 06:32:12 vmkernel: 6:17:02:42.712 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:12 Local6.Warning 192.168.1.63 Nov 12 06:32:12 vmkernel: 6:17:02:42.715 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:12 Local6.Warning 192.168.1.63 Nov 12 06:32:12 vmkernel: 6:17:02:42.718 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:12 Local6.Warning 192.168.1.63 Nov 12 06:32:12 vmkernel: 6:17:02:42.720 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:13 Local6.Warning 192.168.1.63 Nov 12 06:32:14 vmkernel: 6:17:02:43.775 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:13 Local6.Warning 192.168.1.63 Nov 12 06:32:14 vmkernel: 6:17:02:43.779 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:13 Local6.Warning 192.168.1.63 Nov 12 06:32:14 vmkernel: 6:17:02:43.782 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:13 Local6.Warning 192.168.1.63 Nov 12 06:32:14 vmkernel: 6:17:02:43.785 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:14 Local4.Info 192.168.1.63 Nov 12 06:32:15 Hostd: Ticket issued for CIMOM version 1.0, user root

2010-11-12 00:32:56 Local6.Warning 192.168.1.63 Nov 12 06:32:56 vmkernel: 6:17:03:26.292 cpu2:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:56 Local6.Warning 192.168.1.63 Nov 12 06:32:56 vmkernel: 6:17:03:26.298 cpu3:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:32:56 Local6.Warning 192.168.1.63 Nov 12 06:32:56 vmkernel: 6:17:03:26.299 cpu2:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.030 cpu2:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f4ec940) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x

2010-11-12 00:33:11 Local7.Debug 192.168.1.63 0 0x0 0x0.

2010-11-12 00:33:11 Local6.Warning 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.030 cpu2:5901)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe: NMP device "naa.600508e00000000059cd61e3918dc00b" state in doubt; requested fast path state update...

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.030 cpu2:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.031 cpu2:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f3e8140) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x

2010-11-12 00:33:11 Local7.Debug 192.168.1.63 0 0x0 0x0.

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.031 cpu2:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.031 cpu2:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f37ca40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x

2010-11-12 00:33:11 Local7.Debug 192.168.1.63 0 0x0 0x0.

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.031 cpu2:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.032 cpu2:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f380540) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x

2010-11-12 00:33:11 Local7.Debug 192.168.1.63 0 0x0 0x0.

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.032 cpu2:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.033 cpu2:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f3e0a40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x

2010-11-12 00:33:11 Local7.Debug 192.168.1.63 0 0x0 0x0.

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.033 cpu2:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.033 cpu2:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f480140) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x

2010-11-12 00:33:11 Local7.Debug 192.168.1.63 0 0x0 0x0.

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.033 cpu2:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.034 cpu2:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f4b0740) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x

2010-11-12 00:33:11 Local7.Debug 192.168.1.63 0 0x0 0x0.

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.034 cpu2:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.034 cpu2:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f43e240) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x

2010-11-12 00:33:11 Local7.Debug 192.168.1.63 0 0x0 0x0.

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.034 cpu2:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.035 cpu2:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f43cd40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x

2010-11-12 00:33:11 Local7.Debug 192.168.1.63 0 0x0 0x0.

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.035 cpu2:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.035 cpu2:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f37a040) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x

2010-11-12 00:33:11 Local7.Debug 192.168.1.63 0 0x0 0x0.

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.035 cpu2:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.036 cpu2:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f488540) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x

2010-11-12 00:33:11 Local7.Debug 192.168.1.63 0 0x0 0x0.

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.036 cpu2:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.036 cpu2:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f3a6340) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x

2010-11-12 00:33:11 Local7.Debug 192.168.1.63 0 0x0 0x0.

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.036 cpu2:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2010-11-12 00:33:11 Local6.Warning 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.041 cpu2:5901)WARNING: LinScsi: SCSILinuxQueueCommand: queuecommand failed with status = 0x1055 Host Busy vmhba1:1:1:0 (driver name: MPT SAS Host) - Message repeated 2 times

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.041 cpu2:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f3e8140) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x0 D:0x8 P:0x0 Possible sense data: 0x

2010-11-12 00:33:11 Local7.Debug 192.168.1.63 0 0x0 0x0.

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.041 cpu2:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0.

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.042 cpu2:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f380540) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x0 D:0x8 P:0x0 Possible sense data: 0x

2010-11-12 00:33:11 Local7.Debug 192.168.1.63 0 0x0 0x0.

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.042 cpu2:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0.

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.043 cpu2:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f3e0a40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x0 D:0x8 P:0x0 Possible sense data: 0x

2010-11-12 00:33:11 Local7.Debug 192.168.1.63 0 0x0 0x0.

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.043 cpu2:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0.

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.043 cpu2:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f480140) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x0 D:0x8 P:0x0 Possible sense data: 0x

2010-11-12 00:33:11 Local7.Debug 192.168.1.63 0 0x0 0x0.

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.043 cpu2:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0.

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.044 cpu2:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f4b0740) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x0 D:0x8 P:0x0 Possible sense data: 0x

2010-11-12 00:33:11 Local7.Debug 192.168.1.63 0 0x0 0x0.

2010-11-12 00:33:11 Local6.Notice 192.168.1.63 Nov 12 06:33:11 vmkernel: 6:17:03:41.044 cpu2:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0.

2010-11-12 00:33:46 Local4.Info 192.168.1.63 Nov 12 06:33:46 Hostd: Ticket issued for CIMOM version 1.0, user root

2010-11-12 00:33:56 Local6.Warning 192.168.1.63 Nov 12 06:33:56 vmkernel: 6:17:04:26.300 cpu2:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:33:56 Local6.Warning 192.168.1.63 Nov 12 06:33:56 vmkernel: 6:17:04:26.306 cpu3:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:33:56 Local6.Warning 192.168.1.63 Nov 12 06:33:56 vmkernel: 6:17:04:26.307 cpu3:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:33:58 Local4.Info 192.168.1.63 Nov 12 06:33:58 Hostd: Actual VM overhead: 144269312 bytes

2010-11-12 00:33:58 Local4.Info 192.168.1.63 Nov 12 06:33:58 Hostd: RefreshVms updated overhead for 1 VM

2010-11-12 00:34:53 Local4.Info 192.168.1.63 Nov 12 06:34:53 Hostd: PersistAllDvsInfo called

2010-11-12 00:34:56 Local6.Warning 192.168.1.63 Nov 12 06:34:56 vmkernel: 6:17:05:26.310 cpu3:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:34:56 Local6.Warning 192.168.1.63 Nov 12 06:34:56 vmkernel: 6:17:05:26.315 cpu3:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:34:56 Local6.Warning 192.168.1.63 Nov 12 06:34:56 vmkernel: 6:17:05:26.317 cpu3:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:35:17 Local4.Info 192.168.1.63 Nov 12 06:35:17 Hostd: Ticket issued for CIMOM version 1.0, user root

2010-11-12 00:35:50 Local6.Notice 192.168.1.63 Nov 12 06:35:51 vmkernel: 6:17:06:20.775 cpu3:4099)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f40ac40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x

2010-11-12 00:35:50 Local7.Debug 192.168.1.63 0 0x0 0x0.

2010-11-12 00:35:50 Local6.Warning 192.168.1.63 Nov 12 06:35:51 vmkernel: 6:17:06:20.775 cpu3:4099)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe: NMP device "naa.600508e00000000059cd61e3918dc00b" state in doubt; requested fast path state update...

2010-11-12 00:35:50 Local6.Notice 192.168.1.63 Nov 12 06:35:51 vmkernel: 6:17:06:20.775 cpu3:4099)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2010-11-12 00:35:50 Local6.Notice 192.168.1.63 Nov 12 06:35:51 vmkernel: 6:17:06:20.777 cpu3:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f42a140) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x

2010-11-12 00:35:50 Local7.Debug 192.168.1.63 0 0x0 0x0.

2010-11-12 00:35:50 Local6.Notice 192.168.1.63 Nov 12 06:35:51 vmkernel: 6:17:06:20.777 cpu3:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2010-11-12 00:35:50 Local6.Notice 192.168.1.63 Nov 12 06:35:51 vmkernel: 6:17:06:20.777 cpu3:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f4a2b40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x

2010-11-12 00:35:50 Local7.Debug 192.168.1.63 0 0x0 0x0.

2010-11-12 00:35:50 Local6.Notice 192.168.1.63 Nov 12 06:35:51 vmkernel: 6:17:06:20.777 cpu3:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2010-11-12 00:35:56 Local6.Warning 192.168.1.63 Nov 12 06:35:56 vmkernel: 6:17:06:26.319 cpu2:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:35:56 Local6.Warning 192.168.1.63 Nov 12 06:35:56 vmkernel: 6:17:06:26.324 cpu3:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:35:56 Local6.Warning 192.168.1.63 Nov 12 06:35:56 vmkernel: 6:17:06:26.326 cpu3:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:35:56 Local4.Info 192.168.1.63 Nov 12 06:35:56 Hostd: New proxy client TCP(local=127.0.0.1:59933, peer=127.0.0.1:80)

2010-11-12 00:35:56 Local4.Info 192.168.1.63 Nov 12 06:35:56 Hostd: Activation : Invoke done on

2010-11-12 00:35:56 Local4.Info 192.168.1.63 Nov 12 06:35:56 Hostd: Arg version:

2010-11-12 00:35:56 Local4.Info 192.168.1.63 Nov 12 06:35:56 Hostd: "42"

2010-11-12 00:35:56 Local4.Info 192.168.1.63 Nov 12 06:35:56 Hostd: Throw vmodl.fault.RequestCanceled

2010-11-12 00:35:56 Local4.Info 192.168.1.63 Nov 12 06:35:56 Hostd: Result:

2010-11-12 00:35:56 Local4.Info 192.168.1.63 Nov 12 06:35:56 Hostd: (vmodl.fault.RequestCanceled) {

2010-11-12 00:35:56 Local4.Info 192.168.1.63 Nov 12 06:35:56 Hostd: dynamicType = <unset>,

2010-11-12 00:35:56 Local4.Info 192.168.1.63 Nov 12 06:35:56 Hostd: faultCause = (vmodl.MethodFault) null,

2010-11-12 00:35:56 Local4.Info 192.168.1.63 Nov 12 06:35:56 Hostd: msg = "",

2010-11-12 00:35:56 Local4.Info 192.168.1.63 Nov 12 06:35:56 Hostd: }

2010-11-12 00:35:56 Local4.Info 192.168.1.63 Nov 12 06:35:56 Hostd: Failed to read header on stream TCP(local=127.0.0.1:61295, peer=127.0.0.1:0): N7Vmacore15SystemExceptionE(Connection reset by peer)

2010-11-12 00:36:48 Local4.Info 192.168.1.63 Nov 12 06:36:48 Hostd: Ticket issued for CIMOM version 1.0, user root

2010-11-12 00:36:56 Local6.Warning 192.168.1.63 Nov 12 06:36:56 vmkernel: 6:17:07:26.327 cpu2:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:36:56 Local6.Warning 192.168.1.63 Nov 12 06:36:56 vmkernel: 6:17:07:26.332 cpu2:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:36:56 Local6.Warning 192.168.1.63 Nov 12 06:36:56 vmkernel: 6:17:07:26.334 cpu3:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:36:58 Local4.Info 192.168.1.63 Nov 12 06:36:58 Hostd: Actual VM overhead: 143212544 bytes

2010-11-12 00:36:58 Local4.Info 192.168.1.63 Nov 12 06:36:58 Hostd: RefreshVms updated overhead for 1 VM

2010-11-12 00:37:56 Local6.Warning 192.168.1.63 Nov 12 06:37:56 vmkernel: 6:17:08:26.336 cpu2:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:37:56 Local6.Warning 192.168.1.63 Nov 12 06:37:56 vmkernel: 6:17:08:26.342 cpu2:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:37:56 Local6.Warning 192.168.1.63 Nov 12 06:37:56 vmkernel: 6:17:08:26.344 cpu2:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:38:19 Local4.Info 192.168.1.63 Nov 12 06:38:19 Hostd: Ticket issued for CIMOM version 1.0, user root

2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.770 cpu4:4100)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900) 2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.770 cpu4:4100)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f3ae540) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:38:24 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:38:24 Local6.Warning 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.770 cpu4:4100)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe: NMP device "naa.600508e00000000059cd61e3918dc00b" state in doubt; requested fast path state update... 2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.770 cpu4:4100)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.770 cpu4:4100)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900) 2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.770 cpu4:4100)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f4a2b40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:38:24 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.770 cpu4:4100)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.771 cpu4:4100)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900)

2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.771 cpu4:4100)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f498740) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x

2010-11-12 00:38:24 Local7.Debug 192.168.1.63 0 0x0 0x0.

2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.771 cpu4:4100)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.771 cpu4:4100)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900) 2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.771 cpu4:4100)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f44a240) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:38:24 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.771 cpu4:4100)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.771 cpu4:4100)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900) 2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.771 cpu4:4100)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f47e040) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:38:24 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.771 cpu4:4100)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.772 cpu4:4100)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900)

2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.772 cpu4:4100)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f4c4840) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x

2010-11-12 00:38:24 Local7.Debug 192.168.1.63 0 0x0 0x0.

2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.772 cpu4:4100)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.772 cpu4:4100)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900) 2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.773 cpu4:4100)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900) 2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.773 cpu4:4100)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f4c0c40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:38:24 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.773 cpu4:4100)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.773 cpu4:4100)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900)

2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.773 cpu4:4100)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f484840) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x

2010-11-12 00:38:24 Local7.Debug 192.168.1.63 0 0x0 0x0.

2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.773 cpu4:4100)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.774 cpu4:4100)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900) 2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.774 cpu4:4100)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f3ead40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:38:24 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.774 cpu4:4100)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.774 cpu4:4100)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900) 2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.774 cpu4:4100)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f4a4640) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:38:24 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.774 cpu4:4100)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.774 cpu4:4100)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900)

2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.775 cpu4:4100)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f480540) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x

2010-11-12 00:38:24 Local7.Debug 192.168.1.63 0 0x0 0x0.

2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.775 cpu4:4100)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.775 cpu4:4100)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900) 2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.775 cpu4:4100)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f3f4d40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:38:24 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.775 cpu4:4100)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.775 cpu4:4100)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900) 2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.775 cpu4:4100)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f42ea40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:38:24 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:38:24 Local6.Notice 192.168.1.63 Nov 12 06:38:25 vmkernel: 6:17:08:54.775 cpu4:4100)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:38:42 Daemon.Notice 192.168.1.63 Nov 12 06:38:42 ntpd[565974]: kernel time sync status change 6001 2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.019 cpu1:4097)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900)

2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.019 cpu1:4097)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f482a40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x

2010-11-12 00:38:50 Local7.Debug 192.168.1.63 0 0x0 0x0.

2010-11-12 00:38:50 Local6.Warning 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.019 cpu1:4097)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe: NMP device "naa.600508e00000000059cd61e3918dc00b" state in doubt; requested fast path state update...

2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.019 cpu1:4097)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.019 cpu1:4097)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900) 2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.019 cpu1:4097)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f4b8340) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:38:50 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.019 cpu1:4097)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.020 cpu1:4097)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900) 2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.020 cpu1:4097)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f438b40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:38:50 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.020 cpu1:4097)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.020 cpu1:4097)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900)

2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.020 cpu1:4097)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f43ca40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x

2010-11-12 00:38:50 Local7.Debug 192.168.1.63 0 0x0 0x0.

2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.020 cpu1:4097)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.021 cpu1:4097)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900) 2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.021 cpu1:4097)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f4be840) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:38:50 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.021 cpu1:4097)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.021 cpu1:4097)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900) 2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.021 cpu1:4097)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f416a40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:38:50 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.021 cpu1:4097)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.021 cpu1:4097)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900)

2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.021 cpu1:4097)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f41c840) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x

2010-11-12 00:38:50 Local7.Debug 192.168.1.63 0 0x0 0x0.

2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.022 cpu1:4097)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.022 cpu1:4097)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900) 2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.022 cpu1:4097)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f3d2640) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:38:50 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.022 cpu1:4097)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.023 cpu1:4097)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900) 2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.023 cpu1:4097)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f4ce540) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:38:50 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.023 cpu1:4097)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.023 cpu1:4097)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900)

2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.023 cpu1:4097)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f3ea140) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x

2010-11-12 00:38:50 Local7.Debug 192.168.1.63 0 0x0 0x0.

2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.023 cpu1:4097)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.024 cpu1:4097)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900) 2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.024 cpu1:4097)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f4e0e40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:38:50 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.024 cpu1:4097)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.024 cpu1:4097)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900) 2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.024 cpu1:4097)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f46e640) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:38:50 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.024 cpu1:4097)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.025 cpu1:4097)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900)

2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.025 cpu1:4097)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f3c8b40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x

2010-11-12 00:38:50 Local7.Debug 192.168.1.63 0 0x0 0x0.

2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.025 cpu1:4097)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2010-11-12 00:38:50 Local6.Notice 192.168.1.63 Nov 12 06:38:50 vmkernel: 6:17:09:20.025 cpu1:4097)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900) 2010-11-12 00:38:56 Local6.Warning 192.168.1.63 Nov 12 06:38:56 vmkernel: 6:17:09:26.347 cpu2:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block 2010-11-12 00:38:56 Local6.Warning 192.168.1.63 Nov 12 06:38:56 vmkernel: 6:17:09:26.352 cpu2:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block 2010-11-12 00:38:57 Local6.Warning 192.168.1.63 Nov 12 06:38:57 vmkernel: 6:17:09:27.354 cpu2:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block 2010-11-12 00:39:50 Local4.Info 192.168.1.63 Nov 12 06:39:50 Hostd: Ticket issued for CIMOM version 1.0, user root 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.767 cpu0:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f472b40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:39:50 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:39:50 Local6.Warning 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.767 cpu0:5901)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe: NMP device "naa.600508e00000000059cd61e3918dc00b" state in doubt; requested fast path state update... 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.767 cpu0:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.768 cpu0:4104)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f416e40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:39:50 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.768 cpu0:4104)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.768 cpu0:4104)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f39e340) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:39:50 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.768 cpu0:4104)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.769 cpu0:4104)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f416040) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:39:50 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.769 cpu0:4104)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.769 cpu0:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f3ee740) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:39:50 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.769 cpu0:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.770 cpu0:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f3ae740) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:39:50 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.770 cpu0:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.770 cpu0:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f43e040) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:39:50 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.770 cpu0:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.771 cpu0:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f49c140) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:39:50 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.771 cpu0:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.771 cpu0:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f476640) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:39:50 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.771 cpu0:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.772 cpu0:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f4d2e40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:39:50 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.772 cpu0:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.772 cpu0:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f372d40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:39:50 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.772 cpu0:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.773 cpu0:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f3a8c40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:39:50 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.773 cpu0:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.773 cpu0:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f482140) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:39:50 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.773 cpu0:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.779 cpu0:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f416e40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x0 D:0x8 P:0x0 Possible sense data: 0x 2010-11-12 00:39:50 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.779 cpu0:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.779 cpu0:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f39e340) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x0 D:0x8 P:0x0 Possible sense data: 0x 2010-11-12 00:39:50 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.779 cpu0:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.780 cpu0:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f416040) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x0 D:0x8 P:0x0 Possible sense data: 0x 2010-11-12 00:39:50 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.780 cpu0:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.780 cpu0:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f3ee740) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x0 D:0x8 P:0x0 Possible sense data: 0x 2010-11-12 00:39:50 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.780 cpu0:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:39:50 Local6.Notice 192.168.1.63 Nov 12 06:39:51 vmkernel: 6:17:10:20.781 cpu0:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f3ae740) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x0 D:0x8 P:0x0 Possible sense data: 0x 2010-11-12 00:39:50 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:39:53 Local4.Info 192.168.1.63 Nov 12 06:39:53 Hostd: PersistAllDvsInfo called 2010-11-12 00:39:53 Local4.Info 192.168.1.63 Nov 12 06:39:53 Hostd: Root pool capacity changed from 18446MHz/13222MB to 18446MHz/13221MB 2010-11-12 00:39:57 Local6.Warning 192.168.1.63 Nov 12 06:39:57 vmkernel: 6:17:10:27.357 cpu3:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block 2010-11-12 00:39:57 Local6.Warning 192.168.1.63 Nov 12 06:39:57 vmkernel: 6:17:10:27.362 cpu3:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block 2010-11-12 00:39:57 Local6.Warning 192.168.1.63 Nov 12 06:39:57 vmkernel: 6:17:10:27.364 cpu3:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block 2010-11-12 00:39:58 Local4.Info 192.168.1.63 Nov 12 06:39:58 Hostd: Actual VM overhead: 142155776 bytes 2010-11-12 00:39:58 Local4.Info 192.168.1.63 Nov 12 06:39:58 Hostd: RefreshVms updated overhead for 1 VM 2010-11-12 00:40:53 Local4.Info 192.168.1.63 Nov 12 06:40:53 Hostd: Root pool capacity changed from 18446MHz/13221MB to 18446MHz/13222MB 2010-11-12 00:40:57 Local4.Info 192.168.1.63 Nov 12 06:40:57 Hostd: New proxy client TCP(local=127.0.0.1:59132, peer=127.0.0.1:80) 2010-11-12 00:40:57 Local4.Info 192.168.1.63 Nov 12 06:40:57 Hostd: Activation : Invoke done on 2010-11-12 00:40:57 Local4.Info 192.168.1.63 Nov 12 06:40:57 Hostd: Arg version: 2010-11-12 00:40:57 Local4.Info 192.168.1.63 Nov 12 06:40:57 Hostd: "42" 2010-11-12 00:40:57 Local4.Info 192.168.1.63 Nov 12 06:40:57 Hostd: Throw vmodl.fault.RequestCanceled 2010-11-12 00:40:57 Local4.Info 192.168.1.63 Nov 12 06:40:57 Hostd: Result: 2010-11-12 00:40:57 Local4.Info 192.168.1.63 Nov 12 06:40:57 Hostd: (vmodl.fault.RequestCanceled) { 2010-11-12 00:40:57 Local4.Info 192.168.1.63 Nov 12 06:40:57 Hostd: dynamicType = <unset>, 2010-11-12 00:40:57 Local4.Info 192.168.1.63 Nov 12 06:40:57 Hostd: faultCause = (vmodl.MethodFault) null, 2010-11-12 00:40:57 Local4.Info 192.168.1.63 Nov 12 06:40:57 Hostd: msg = "", 2010-11-12 00:40:57 Local4.Info 192.168.1.63 Nov 12 06:40:57 Hostd: } 2010-11-12 00:40:57 Local4.Info 192.168.1.63 Nov 12 06:40:57 Hostd: Failed to read header on stream TCP(local=127.0.0.1:51090, peer=127.0.0.1:0): N7Vmacore15SystemExceptionE(Connection reset by peer) 2010-11-12 00:40:57 Local6.Warning 192.168.1.63 Nov 12 06:40:57 vmkernel: 6:17:11:27.366 cpu3:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block 2010-11-12 00:40:57 Local6.Warning 192.168.1.63 Nov 12 06:40:57 vmkernel: 6:17:11:27.370 cpu3:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block 2010-11-12 00:40:57 Local6.Warning 192.168.1.63 Nov 12 06:40:57 vmkernel: 6:17:11:27.372 cpu3:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block 2010-11-12 00:41:22 Local4.Info 192.168.1.63 Nov 12 06:41:22 Hostd: Ticket issued for CIMOM version 1.0, user root 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.013 cpu7:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f430e40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:41:27 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:41:27 Local6.Warning 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.013 cpu7:5901)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe: NMP device "naa.600508e00000000059cd61e3918dc00b" state in doubt; requested fast path state update... 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.013 cpu7:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.014 cpu7:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f42a540) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:41:27 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.014 cpu7:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.014 cpu7:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f3b0740) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:41:27 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.014 cpu7:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.015 cpu7:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f374d40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:41:27 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.015 cpu7:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.015 cpu7:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f3e6240) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:41:27 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.015 cpu7:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.016 cpu7:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f4b8b40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:41:27 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.016 cpu7:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.019 cpu7:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f496d40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:41:27 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.019 cpu7:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.024 cpu7:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f430e40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x0 D:0x8 P:0x0 Possible sense data: 0x 2010-11-12 00:41:27 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.024 cpu7:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.025 cpu7:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f42a540) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x0 D:0x8 P:0x0 Possible sense data: 0x 2010-11-12 00:41:27 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.025 cpu7:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.025 cpu7:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f3b0740) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x0 D:0x8 P:0x0 Possible sense data: 0x 2010-11-12 00:41:27 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.025 cpu7:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.025 cpu7:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f374d40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x0 D:0x8 P:0x0 Possible sense data: 0x 2010-11-12 00:41:27 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.025 cpu7:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.025 cpu2:4414)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x41027f46ab40) to NMP device "naa.5000c500151b3264" failed on physical path "vmhba1:C0:T0:L0" H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.025 cpu2:4414)ScsiDeviceIO: 1672: Command 0x28 to device "naa.5000c500151b3264" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.026 cpu7:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f3e6240) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x0 D:0x8 P:0x0 Possible sense data: 0x 2010-11-12 00:41:27 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.026 cpu7:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.026 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x41027f3bc040) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x0 D:0x8 P:0x0 Possible sense data: 0x 2010-11-12 00:41:27 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.026 cpu6:4102)ScsiDeviceIO: 1672: Command 0x28 to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.026 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x41027f42a940) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x0 D:0x8 P:0x0 Possible sense data: 0x 2010-11-12 00:41:27 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.026 cpu6:4102)ScsiDeviceIO: 1672: Command 0x28 to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.026 cpu7:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f4b8b40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x0 D:0x8 P:0x0 Possible sense data: 0x 2010-11-12 00:41:27 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.026 cpu7:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.026 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x41027f38a140) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x0 D:0x8 P:0x0 Possible sense data: 0x 2010-11-12 00:41:27 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.026 cpu6:4102)ScsiDeviceIO: 1672: Command 0x28 to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:41:27 Local6.Notice 192.168.1.63 Nov 12 06:41:27 vmkernel: 6:17:11:57.026 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x41027f4c0840) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x0 D:0x8 P:0x0 Possible sense data: 0x 2010-11-12 00:41:27 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:41:31 Local4.Info 192.168.1.63 Nov 12 06:41:31 Hostd: New proxy client TCP(local=192.168.1.143:5122, peer=192.168.1.63:80) 2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.263 cpu6:5901)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900) 2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.263 cpu6:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f40a540) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:41:51 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:41:51 Local6.Warning 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.263 cpu6:5901)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe: NMP device "naa.600508e00000000059cd61e3918dc00b" state in doubt; requested fast path state update... 2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.263 cpu6:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.263 cpu6:5901)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900)

2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.263 cpu6:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f420240) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x

2010-11-12 00:41:51 Local7.Debug 192.168.1.63 0 0x0 0x0.

2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.263 cpu6:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.264 cpu6:5901)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900) 2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.264 cpu6:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f488640) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:41:51 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.264 cpu6:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.264 cpu6:5901)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900) 2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.264 cpu6:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f486040) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:41:51 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.264 cpu6:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.264 cpu6:5901)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900)

2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.264 cpu6:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f40aa40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x

2010-11-12 00:41:51 Local7.Debug 192.168.1.63 0 0x0 0x0.

2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.264 cpu6:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.265 cpu6:5901)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900) 2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.265 cpu6:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f410a40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:41:51 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.265 cpu6:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.265 cpu6:5901)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900) 2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.265 cpu6:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f372040) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:41:51 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.265 cpu6:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.266 cpu6:5901)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900)

2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.266 cpu6:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f3cc340) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x

2010-11-12 00:41:51 Local7.Debug 192.168.1.63 0 0x0 0x0.

2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.266 cpu6:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.266 cpu6:5901)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900) 2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.266 cpu6:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f452640) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:41:51 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.266 cpu6:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.267 cpu6:5901)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900) 2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.267 cpu6:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f462d40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:41:51 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.267 cpu6:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.267 cpu6:5901)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900)

2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.267 cpu6:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f3ace40) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x

2010-11-12 00:41:51 Local7.Debug 192.168.1.63 0 0x0 0x0.

2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.267 cpu6:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.268 cpu6:5901)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900) 2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.268 cpu6:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f412140) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:41:51 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.268 cpu6:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.268 cpu6:5901)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900) 2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.268 cpu6:5901)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027f460540) to NMP device "naa.600508e00000000059cd61e3918dc00b" failed on physical path "vmhba1:C1:T1:L0" H:0x8 D:0x0 P:0x0 Possible sense data: 0x 2010-11-12 00:41:51 Local7.Debug 192.168.1.63 0 0x0 0x0. 2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.268 cpu6:5901)ScsiDeviceIO: 1672: Command 0x2a to device "naa.600508e00000000059cd61e3918dc00b" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. 2010-11-12 00:41:51 Local6.Notice 192.168.1.63 Nov 12 06:41:51 vmkernel: 6:17:12:21.269 cpu6:5901)<6>mptbase: ioc0: LogInfo(0x31110900): Originator=, Code=, SubCode(0x0900)

2010-11-12 00:41:57 Local6.Warning 192.168.1.63 Nov 12 06:41:57 vmkernel: 6:17:12:27.373 cpu2:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:41:57 Local6.Warning 192.168.1.63 Nov 12 06:41:57 vmkernel: 6:17:12:27.378 cpu2:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:41:57 Local6.Warning 192.168.1.63 Nov 12 06:41:57 vmkernel: 6:17:12:27.380 cpu3:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:13 Local6.Warning 192.168.1.63 Nov 12 06:42:14 vmkernel: 6:17:12:43.787 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:14 Local6.Warning 192.168.1.63 Nov 12 06:42:14 vmkernel: 6:17:12:44.133 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:14 Local6.Warning 192.168.1.63 Nov 12 06:42:14 vmkernel: 6:17:12:44.481 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:15 Local6.Warning 192.168.1.63 Nov 12 06:42:15 vmkernel: 6:17:12:45.485 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:15 Local6.Warning 192.168.1.63 Nov 12 06:42:15 vmkernel: 6:17:12:45.489 cpu7:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:15 Local6.Warning 192.168.1.63 Nov 12 06:42:15 vmkernel: 6:17:12:45.495 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:15 Local6.Warning 192.168.1.63 Nov 12 06:42:15 vmkernel: 6:17:12:45.497 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:15 Local6.Warning 192.168.1.63 Nov 12 06:42:15 vmkernel: 6:17:12:45.499 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:15 Local6.Warning 192.168.1.63 Nov 12 06:42:15 vmkernel: 6:17:12:45.502 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:15 Local6.Warning 192.168.1.63 Nov 12 06:42:15 vmkernel: 6:17:12:45.505 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:15 Local6.Warning 192.168.1.63 Nov 12 06:42:15 vmkernel: 6:17:12:45.508 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:15 Local6.Warning 192.168.1.63 Nov 12 06:42:15 vmkernel: 6:17:12:45.511 cpu6:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:16 Local6.Warning 192.168.1.63 Nov 12 06:42:16 vmkernel: 6:17:12:46.573 cpu5:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:16 Local6.Warning 192.168.1.63 Nov 12 06:42:16 vmkernel: 6:17:12:46.577 cpu4:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:16 Local6.Warning 192.168.1.63 Nov 12 06:42:16 vmkernel: 6:17:12:46.580 cpu4:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:17 Local6.Warning 192.168.1.63 Nov 12 06:42:17 vmkernel: 6:17:12:47.582 cpu5:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:17 Local6.Warning 192.168.1.63 Nov 12 06:42:17 vmkernel: 6:17:12:47.586 cpu4:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:17 Local6.Warning 192.168.1.63 Nov 12 06:42:17 vmkernel: 6:17:12:47.588 cpu4:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:17 Local6.Warning 192.168.1.63 Nov 12 06:42:17 vmkernel: 6:17:12:47.591 cpu4:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:17 Local6.Warning 192.168.1.63 Nov 12 06:42:17 vmkernel: 6:17:12:47.594 cpu4:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:18 Local6.Warning 192.168.1.63 Nov 12 06:42:18 vmkernel: 6:17:12:48.655 cpu4:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:18 Local6.Warning 192.168.1.63 Nov 12 06:42:18 vmkernel: 6:17:12:48.658 cpu5:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:18 Local6.Warning 192.168.1.63 Nov 12 06:42:18 vmkernel: 6:17:12:48.661 cpu4:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:18 Local6.Warning 192.168.1.63 Nov 12 06:42:18 vmkernel: 6:17:12:48.664 cpu4:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:18 Local6.Warning 192.168.1.63 Nov 12 06:42:18 vmkernel: 6:17:12:48.666 cpu5:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:18 Local6.Warning 192.168.1.63 Nov 12 06:42:18 vmkernel: 6:17:12:48.669 cpu5:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:18 Local6.Warning 192.168.1.63 Nov 12 06:42:18 vmkernel: 6:17:12:48.672 cpu5:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:18 Local6.Warning 192.168.1.63 Nov 12 06:42:18 vmkernel: 6:17:12:48.675 cpu5:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:20 Local6.Warning 192.168.1.63 Nov 12 06:42:20 vmkernel: 6:17:12:49.881 cpu5:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:20 Local6.Warning 192.168.1.63 Nov 12 06:42:20 vmkernel: 6:17:12:49.884 cpu4:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:20 Local6.Warning 192.168.1.63 Nov 12 06:42:20 vmkernel: 6:17:12:49.887 cpu4:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:20 Local6.Warning 192.168.1.63 Nov 12 06:42:20 vmkernel: 6:17:12:49.890 cpu4:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:20 Local6.Warning 192.168.1.63 Nov 12 06:42:20 vmkernel: 6:17:12:49.893 cpu4:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:20 Local6.Warning 192.168.1.63 Nov 12 06:42:20 vmkernel: 6:17:12:49.896 cpu5:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:20 Local6.Warning 192.168.1.63 Nov 12 06:42:20 vmkernel: 6:17:12:49.899 cpu4:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:20 Local6.Warning 192.168.1.63 Nov 12 06:42:20 vmkernel: 6:17:12:49.902 cpu4:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:21 Local6.Warning 192.168.1.63 Nov 12 06:42:21 vmkernel: 6:17:12:50.959 cpu4:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:21 Local6.Warning 192.168.1.63 Nov 12 06:42:21 vmkernel: 6:17:12:50.965 cpu5:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:21 Local6.Warning 192.168.1.63 Nov 12 06:42:21 vmkernel: 6:17:12:50.968 cpu5:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:21 Local6.Warning 192.168.1.63 Nov 12 06:42:21 vmkernel: 6:17:12:50.971 cpu4:566486)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:53 Local4.Info 192.168.1.63 Nov 12 06:42:53 Hostd: Ticket issued for CIMOM version 1.0, user root

2010-11-12 00:42:57 Local6.Warning 192.168.1.63 Nov 12 06:42:57 vmkernel: 6:17:13:27.382 cpu3:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:57 Local6.Warning 192.168.1.63 Nov 12 06:42:57 vmkernel: 6:17:13:27.388 cpu2:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:57 Local6.Warning 192.168.1.63 Nov 12 06:42:57 vmkernel: 6:17:13:27.390 cpu2:566487)WARNING: UserObj: 569: Failed to crossdup fd 1, fs: def5 oid: 17000000030000008 type CHAR: Would block

2010-11-12 00:42:58 Local4.Info 192.168.1.63 Nov 12 06:42:58 Hostd: Actual VM overhead: 141099008 bytes

2010-11-12 00:42:58 Local4.Info 192.168.1.63 Nov 12 06:42:58 Hostd: RefreshVms updated overhead for 1 VM

0 Kudos
jamesbowling
VMware Employee
VMware Employee

Do you have anything that is doing any monitoring?

If you found this at all helpful please award points by using the correct or helpful buttons! Thanks!

James B. | Blog: http://www.vSential.com | Twitter: @vSential --- If you found this helpful then please awards helpful or correct points accordingly. Thanks!
0 Kudos
jamesbowling
VMware Employee
VMware Employee

I would also check your SCSI devices/controllers. I see quite a few failed command being sent around 00:33:11. I would assume that this is when it is happening. It definitely looks as if this is related to your storage though.

Check vmhba1:1:1:0 (driver name: MPT SAS Host).

Also, are you doing any type of backups or anything during these times?

If you found this at all helpful please award points by using the correct or helpful buttons! Thanks!

James B. | Blog: http://www.vSential.com | Twitter: @vSential --- If you found this helpful then please awards helpful or correct points accordingly. Thanks!
0 Kudos
FinFreeTX
Contributor
Contributor

I don't have any monitoring other than the syslogd and native tech support mode. Is there anyting specific you would recommend?

Can you be more specific about checking the SCSI devices/controllers? I've run the Dell self diagnostics and it didn't find any problems - once again, is there anything specific you would recommend?

Thanks again!

0 Kudos
jamesbowling
VMware Employee
VMware Employee

Well, I am not sure what diags to run but all I see during these events are issues with read/writes to you storage. If this happens then of course the host will freeze up. I would recommend either opening a support case with VMware, if you can, and also one with Dell. Do you have a backup of all of the VMs on this host?

If you found this at all helpful please award points by using the correct or helpful buttons! Thanks!

James B. | Blog: http://www.vSential.com | Twitter: @vSential --- If you found this helpful then please awards helpful or correct points accordingly. Thanks!
0 Kudos
FinFreeTX
Contributor
Contributor

I'm trying to avoid opening a ticket with VMware as we don't currently have a service contract, but I've contacted Dell and I'm awaiting a response.

I will make backups of the VM's this weekend. What method would you recommend to accomplish this to ensure compatibility if we ever had to fire them up from another location?

Thanks again for all your help!!!

0 Kudos