VMware Cloud Community
JayDeah
Contributor
Contributor

extremely long boot time (crash?) on ESXi5 host, pause after "iscsi_vmk started"

ok ive got some Dell poweredge 1950 servers connected to an md3200i that have been happily running ESXi 4.1 U1 for some time and i have put ESXi5 on them using a number of scenarios all of which having the same problem. Any thoughts?

the boot process hangs near the end of the progress bar and the last entry on the screen is "iscsi_vmk started successfully"

if i alt+f12 i can see the server hasnt crashed.

i am using vcenter 5.0 with latest update manager and webclient.

i have used the following scenarios to install and configure:

1) host upgrade using updatemanager

2) clean instakll of ESXi5, apply old esxi4.1 host profile, update settings reboot

3) clean install of ESXi5, clean configuration, reboot

scenario 1 appeared to leave the host hanging , i gave up after 30minutes and reinstalled

scenario 2 didnt seem to like having the ESXi4 settings as i ended up with my swiscsi adapter on vmhba39, it did eventually start after about half an hour!

scenario 3 has been hanging for about 15minutes sofar, im hoping it finally boots!

Reply
0 Kudos
95 Replies
wdroush1
Hot Shot
Hot Shot

I've had this happen on my 3.5 box under my desk when the iSCSI server can't be contacted.

With a clean install though and hand-configured everything, I'm confused.

Reply
0 Kudos
bretti
Expert
Expert

It sounds like the host is trying to mount all the datastores on boot and is having trouble contacting the iSCSI target.  You may have to just wait until it comes up or times out and check the configuration to see if something changed.

I'm curious, can you do a vmkping to your target in the ALT-F2 session while it's booting and see if it can reach the target?

Reply
0 Kudos
JayDeah
Contributor
Contributor

i found a post from someone saying they had the same problem with the MD3200i under 4.1

to be honest i hadnt tried 4.1 with the MD3200i so dont know if it's a "new" issue with 5 or not.

also when the hosts FINALLY did boot, one only saw half the paths to the san, the other didnt show any VMFS datastores.

all fixed after a rescan.

as far as i can tell everything is configured properly!

SWiscsi with 2x 5709 NICs with 9000mtu connected to seperate switches on seperate subnets, 2 paths per vmk so 4 paths total on the host

Reply
0 Kudos
bretti
Expert
Expert

I think you're right, it does not sound like a "new" issue with 5.

It is curious that a rescan after the host booted corrected the problem.  I would think if there was a communication issue, that it would persist past boot up and a rescan would not correct it.

You might consider trying to change some of the settings on the vSwtich to see what helps.  What load balancing policy are you using?

Reply
0 Kudos
MauroBonder
VMware Employee
VMware Employee

already checked if all are compatible ? http://www.vmware.com/go/hcl

put here logs about it. if possible.

Good Luck

*Please, don't forget the awarding points for "helpful" and/or "correct" answers. *Por favor, não esqueça de atribuir os pontos se a resposta foi útil ou resolveu o problema.* Thank you/Obrigado
Reply
0 Kudos
JayDeah
Contributor
Contributor

Yes it's all on the HCL and it's using the right mpio drivers etc.

I'm not entirely sure how to get the logs off and which ones I'm looking for. Any quick answers before I start a google?

As for switches I'm using dvswitches and there is no loadbalancong each vmk has access to a single nic (well, dvuplink but you know what I mean) each vmk has a single path to each controller and they are in different null routed subnets

Sent from my iPhone

Reply
0 Kudos
JayDeah
Contributor
Contributor

extract from vmkernel.log below:

Please note one thing i failed to mention but from reading that log MAY be relevant, i also have 4x NFS shares mounted on the hosts. these are from my old openfiler stores. these are connected to via a seperate vmk on a seperave dvswitch/physical switch/subnet.

i'm currently in the process of svMotioning all of my vmdks from these openfilers to the new SAN LUNs and will be decomissioning the openfilers as soon as it's done.

EDIT: Sorry i attached the wrong section of the log, doh! here is the dodgy bit. lots of "msleep returned 4"

i also just built a new ESXi5 host and did nothing other than add the networking and the iscsi storage (no NFS/openfiler) it's currently sat booting doing the same thing (zzzZZZ)

2011-08-25T14:10:24.406Z cpu3:2610)Loading module iscsi_vmk ...
2011-08-25T14:10:24.409Z cpu3:2610)Elf: 1862: module iscsi_vmk has license VMware
2011-08-25T14:10:24.414Z cpu3:2610)DMA: 524: DMA Engine 'vmhba39' created.
2011-08-25T14:10:24.414Z cpu3:2610)iscsi_vmk: iscsivmk_RegisterSCSIAdapter:1569: iscsi_vmk module registered iSCSI adapter vmhba39 with the mid-layer
2011-08-25T14:10:24.414Z cpu3:2610)Mod: 4015: Initialization of iscsi_vmk succeeded with module ID 48.
2011-08-25T14:10:24.414Z cpu3:2610)iscsi_vmk loaded successfully.
2011-08-25T14:10:28.624Z cpu0:2079)NetPort: 1426: disabled port 0x2000003
2011-08-25T14:10:28.624Z cpu0:2079)NetPort: 2600: resuming traffic on DV port 262
2011-08-25T14:10:28.624Z cpu0:2079)Uplink: 5244: enabled port 0x2000003 with mac 00:10:18:ae:a8:7e
2011-08-25T14:10:28.636Z cpu0:2079)NetPort: 1426: disabled port 0x3000003
2011-08-25T14:10:28.636Z cpu0:2079)NetPort: 2600: resuming traffic on DV port 278
2011-08-25T14:10:28.636Z cpu0:2079)Uplink: 5244: enabled port 0x3000003 with mac 00:10:18:ae:a8:7c
2011-08-25T14:10:28.636Z cpu0:2079)NetPort: 1426: disabled port 0x3000002
2011-08-25T14:10:28.636Z cpu0:2079)NetPort: 2600: resuming traffic on DV port 277
2011-08-25T14:10:28.636Z cpu0:2079)Uplink: 5244: enabled port 0x3000002 with mac 00:1e:c9:b8:69:5e
2011-08-25T14:10:28.649Z cpu0:2079)NetPort: 1426: disabled port 0x2000002
2011-08-25T14:10:28.649Z cpu0:2079)NetPort: 2600: resuming traffic on DV port 261
2011-08-25T14:10:28.649Z cpu0:2079)Uplink: 5244: enabled port 0x2000002 with mac 00:10:18:ae:a8:78
2011-08-25T14:10:30.444Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:10:35.447Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:10:40.450Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:10:45.452Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:10:50.457Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:10:55.461Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:11:00.464Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:11:05.467Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:12:01.918Z cpu2:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:12:06.920Z cpu2:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:12:11.958Z cpu2:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:12:16.993Z cpu2:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:12:21.998Z cpu2:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:12:27.001Z cpu2:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:12:32.006Z cpu2:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:12:37.009Z cpu2:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:13:27.496Z cpu0:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:13:27.549Z cpu0:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:13:27.551Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:13:27.551Z cpu2:2610)VC: 1452: Filesystem probe time 53 msec (devices probed 6 of 6)
2011-08-25T14:13:27.615Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:13:27.615Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:13:27.624Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:13:27.624Z cpu2:2610)VC: 1452: Filesystem probe time 7 msec (devices probed 6 of 6)
2011-08-25T14:13:27.694Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:13:27.694Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:13:27.696Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:13:27.696Z cpu2:2610)VC: 1452: Filesystem probe time 7 msec (devices probed 6 of 6)
2011-08-25T14:13:27.758Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:13:27.758Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:13:27.768Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:13:27.768Z cpu2:2610)VC: 1452: Filesystem probe time 7 msec (devices probed 6 of 6)
2011-08-25T14:13:27.830Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:13:27.837Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:13:27.839Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:13:27.839Z cpu2:2610)VC: 1452: Filesystem probe time 7 msec (devices probed 6 of 6)
2011-08-25T14:13:42.859Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:13:47.861Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:13:52.866Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:13:57.869Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:14:02.873Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:14:07.877Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:14:12.881Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:14:17.883Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:15:13.320Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:15:18.323Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:15:23.328Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:15:28.331Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:15:33.336Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:15:38.338Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:15:43.342Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:15:48.345Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:16:39.831Z cpu3:2051)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x1a (0x4124007c4e00) to dev "t10.DP______BACKPLANE000000" on path "vmhba1:C0:T32:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.Act:NONE
2011-08-25T14:16:39.831Z cpu3:2051)ScsiDeviceIO: 2316: Cmd(0x4124007c4e00) 0x1a, CmdSN 0x229 to dev "t10.DP______BACKPLANE000000" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2011-08-25T14:16:39.834Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:16:39.844Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:16:39.845Z cpu2:2610)VC: 1449: Device rescan time 28 msec (total number of devices 6)
2011-08-25T14:16:39.845Z cpu2:2610)VC: 1452: Filesystem probe time 10 msec (devices probed 6 of 6)
2011-08-25T14:16:39.907Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:16:39.907Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:16:39.917Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:16:39.917Z cpu2:2610)VC: 1452: Filesystem probe time 8 msec (devices probed 6 of 6)
2011-08-25T14:16:39.978Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:16:39.986Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:16:39.987Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:16:39.987Z cpu2:2610)VC: 1452: Filesystem probe time 7 msec (devices probed 6 of 6)
2011-08-25T14:16:40.049Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:16:40.049Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:16:40.058Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:16:40.058Z cpu2:2610)VC: 1452: Filesystem probe time 7 msec (devices probed 6 of 6)
2011-08-25T14:16:40.127Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:16:40.127Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:16:40.128Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:16:40.128Z cpu2:2610)VC: 1452: Filesystem probe time 7 msec (devices probed 6 of 6)
2011-08-25T14:16:55.149Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:17:00.152Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:17:05.157Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:17:10.159Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:17:15.163Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:17:20.166Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:17:25.169Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:17:30.172Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:18:26.588Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:18:31.589Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:18:36.593Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:18:41.595Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:18:46.600Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:18:51.603Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:18:56.607Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:19:01.609Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:19:53.060Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:19:53.069Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:19:53.070Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:19:53.070Z cpu2:2610)VC: 1452: Filesystem probe time 9 msec (devices probed 6 of 6)
2011-08-25T14:19:53.132Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:19:53.132Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:19:53.141Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:19:53.141Z cpu2:2610)VC: 1452: Filesystem probe time 7 msec (devices probed 6 of 6)
2011-08-25T14:19:53.202Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:19:53.210Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:19:53.211Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:19:53.211Z cpu2:2610)VC: 1452: Filesystem probe time 8 msec (devices probed 6 of 6)
2011-08-25T14:19:53.273Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:19:53.281Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:19:53.282Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:19:53.282Z cpu2:2610)VC: 1452: Filesystem probe time 7 msec (devices probed 6 of 6)
2011-08-25T14:19:53.343Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:19:53.344Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:19:53.352Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:19:53.352Z cpu2:2610)VC: 1452: Filesystem probe time 7 msec (devices probed 6 of 6)
2011-08-25T14:20:08.372Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:20:13.375Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:20:18.380Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:20:23.382Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:20:28.387Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:20:33.391Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:20:38.395Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:20:43.399Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:21:38.836Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:21:43.839Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:21:48.844Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:21:53.847Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:21:58.852Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:22:03.855Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:22:08.860Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:22:13.862Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:23:05.362Z cpu3:2051)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x1a (0x4124007b1340) to dev "t10.DP______BACKPLANE000000" on path "vmhba1:C0:T32:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.Act:NONE
2011-08-25T14:23:05.362Z cpu3:2051)ScsiDeviceIO: 2316: Cmd(0x4124007b1340) 0x1a, CmdSN 0x5cd to dev "t10.DP______BACKPLANE000000" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2011-08-25T14:23:05.365Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:23:05.374Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:23:05.375Z cpu2:2610)VC: 1449: Device rescan time 28 msec (total number of devices 6)
2011-08-25T14:23:05.375Z cpu2:2610)VC: 1452: Filesystem probe time 9 msec (devices probed 6 of 6)
2011-08-25T14:23:05.437Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:23:05.437Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:23:05.446Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:23:05.446Z cpu2:2610)VC: 1452: Filesystem probe time 7 msec (devices probed 6 of 6)
2011-08-25T14:23:05.508Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:23:05.508Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:23:05.517Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:23:05.517Z cpu2:2610)VC: 1452: Filesystem probe time 7 msec (devices probed 6 of 6)
2011-08-25T14:23:05.578Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:23:05.586Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:23:05.587Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:23:05.587Z cpu2:2610)VC: 1452: Filesystem probe time 7 msec (devices probed 6 of 6)
2011-08-25T14:23:05.649Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:23:05.649Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:23:05.658Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:23:05.658Z cpu2:2610)VC: 1452: Filesystem probe time 7 msec (devices probed 6 of 6)
2011-08-25T14:23:20.679Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:23:25.682Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:23:30.687Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:23:35.689Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:23:40.695Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:23:45.698Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:23:50.702Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:23:55.705Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:24:51.150Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:24:56.152Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:25:01.157Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:25:06.160Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:25:11.165Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:25:16.168Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:25:21.172Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:25:26.175Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:26:17.639Z cpu3:2051)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x1a (0x4124007bd8c0) to dev "t10.DP______BACKPLANE000000" on path "vmhba1:C0:T32:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.Act:NONE
2011-08-25T14:26:17.639Z cpu3:2051)ScsiDeviceIO: 2316: Cmd(0x4124007bd8c0) 0x1a, CmdSN 0x7a0 to dev "t10.DP______BACKPLANE000000" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2011-08-25T14:26:17.642Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:26:17.652Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:26:17.653Z cpu2:2610)VC: 1449: Device rescan time 28 msec (total number of devices 6)
2011-08-25T14:26:17.653Z cpu2:2610)VC: 1452: Filesystem probe time 9 msec (devices probed 6 of 6)
2011-08-25T14:26:17.715Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:26:17.723Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:26:17.725Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:26:17.725Z cpu2:2610)VC: 1452: Filesystem probe time 8 msec (devices probed 6 of 6)
2011-08-25T14:26:17.794Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:26:17.794Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:26:17.795Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:26:17.795Z cpu2:2610)VC: 1452: Filesystem probe time 7 msec (devices probed 6 of 6)
2011-08-25T14:26:17.857Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:26:17.857Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:26:17.866Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:26:17.866Z cpu2:2610)VC: 1452: Filesystem probe time 7 msec (devices probed 6 of 6)
2011-08-25T14:26:17.927Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:26:17.927Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:26:17.936Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:26:17.936Z cpu2:2610)VC: 1452: Filesystem probe time 7 msec (devices probed 6 of 6)
2011-08-25T14:26:32.957Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:26:37.960Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:26:42.965Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:26:47.968Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:26:52.971Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:26:57.975Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:27:02.979Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:27:07.981Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:28:03.417Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:28:08.420Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:28:13.424Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:28:18.427Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:28:23.431Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:28:28.435Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:28:33.439Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:28:38.441Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:29:29.886Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:29:29.895Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:29:29.897Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:29:29.897Z cpu2:2610)VC: 1452: Filesystem probe time 9 msec (devices probed 6 of 6)
2011-08-25T14:29:29.958Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:29:29.966Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:29:29.967Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:29:29.967Z cpu2:2610)VC: 1452: Filesystem probe time 7 msec (devices probed 6 of 6)
2011-08-25T14:29:30.029Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:29:30.029Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:29:30.038Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:29:30.038Z cpu2:2610)VC: 1452: Filesystem probe time 7 msec (devices probed 6 of 6)
2011-08-25T14:29:30.099Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:29:30.099Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:29:30.108Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:29:30.108Z cpu2:2610)VC: 1452: Filesystem probe time 7 msec (devices probed 6 of 6)
2011-08-25T14:29:30.188Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:29:30.188Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:29:30.189Z cpu2:2610)VC: 1449: Device rescan time 16 msec (total number of devices 6)
2011-08-25T14:29:30.189Z cpu2:2610)VC: 1452: Filesystem probe time 8 msec (devices probed 6 of 6)
2011-08-25T14:29:45.209Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:29:50.211Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:29:55.216Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:30:00.219Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:30:05.224Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:30:10.227Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:30:15.231Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:30:20.234Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:31:15.687Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:31:20.689Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:31:25.695Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:31:30.697Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:31:35.701Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:31:40.704Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:31:45.707Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:31:50.710Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:32:42.179Z cpu2:2050)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x1a (0x4124007acbc0) to dev "t10.DP______BACKPLANE000000" on path "vmhba1:C0:T32:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.Act:NONE
2011-08-25T14:32:42.179Z cpu2:2050)ScsiDeviceIO: 2316: Cmd(0x4124007acbc0) 0x1a, CmdSN 0xb44 to dev "t10.DP______BACKPLANE000000" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2011-08-25T14:32:42.182Z cpu3:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:32:42.191Z cpu3:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:32:42.192Z cpu3:2610)VC: 1449: Device rescan time 28 msec (total number of devices 6)
2011-08-25T14:32:42.192Z cpu3:2610)VC: 1452: Filesystem probe time 9 msec (devices probed 6 of 6)
2011-08-25T14:32:42.254Z cpu3:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:32:42.262Z cpu3:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:32:42.263Z cpu3:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:32:42.263Z cpu3:2610)VC: 1452: Filesystem probe time 8 msec (devices probed 6 of 6)
2011-08-25T14:32:42.333Z cpu3:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:32:42.333Z cpu3:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:32:42.334Z cpu3:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:32:42.334Z cpu3:2610)VC: 1452: Filesystem probe time 8 msec (devices probed 6 of 6)
2011-08-25T14:32:42.404Z cpu3:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:32:42.404Z cpu3:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:32:42.405Z cpu3:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:32:42.405Z cpu3:2610)VC: 1452: Filesystem probe time 7 msec (devices probed 6 of 6)
2011-08-25T14:32:42.466Z cpu3:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:32:42.466Z cpu3:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:32:42.475Z cpu3:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:32:42.475Z cpu3:2610)VC: 1452: Filesystem probe time 7 msec (devices probed 6 of 6)
2011-08-25T14:32:57.496Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:33:02.499Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:33:07.502Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:33:12.505Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:33:17.510Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:33:22.514Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:33:27.518Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:33:32.520Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:34:27.963Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:34:32.965Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:34:37.970Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:34:42.972Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:34:47.978Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:34:52.980Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:34:57.984Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:35:02.988Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:35:54.493Z cpu2:2050)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x1a (0x4124007c87c0) to dev "t10.DP______BACKPLANE000000" on path "vmhba1:C0:T32:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.Act:NONE
2011-08-25T14:35:54.493Z cpu2:2050)ScsiDeviceIO: 2316: Cmd(0x4124007c87c0) 0x1a, CmdSN 0xd17 to dev "t10.DP______BACKPLANE000000" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2011-08-25T14:35:54.508Z cpu3:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:35:54.508Z cpu3:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:35:54.510Z cpu3:2610)VC: 1449: Device rescan time 30 msec (total number of devices 6)
2011-08-25T14:35:54.510Z cpu3:2610)VC: 1452: Filesystem probe time 9 msec (devices probed 6 of 6)
2011-08-25T14:35:54.573Z cpu3:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:35:54.581Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:35:54.583Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:35:54.583Z cpu2:2610)VC: 1452: Filesystem probe time 7 msec (devices probed 6 of 6)
2011-08-25T14:35:54.646Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:35:54.653Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:35:54.655Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:35:54.655Z cpu2:2610)VC: 1452: Filesystem probe time 7 msec (devices probed 6 of 6)
2011-08-25T14:35:54.716Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:35:54.724Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:35:54.725Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:35:54.725Z cpu2:2610)VC: 1452: Filesystem probe time 7 msec (devices probed 6 of 6)
2011-08-25T14:35:54.795Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:35:54.795Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:35:54.796Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:35:54.796Z cpu2:2610)VC: 1452: Filesystem probe time 7 msec (devices probed 6 of 6)
2011-08-25T14:36:09.817Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:36:14.818Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:36:19.822Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:36:24.825Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:36:29.828Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:36:34.830Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:36:39.834Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:36:44.837Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:37:41.274Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:37:46.277Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:37:51.282Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:37:56.285Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:38:01.290Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:38:06.293Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:38:11.298Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:38:16.300Z cpu1:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:39:07.745Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:39:07.745Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:39:07.747Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:39:07.747Z cpu2:2610)VC: 1452: Filesystem probe time 8 msec (devices probed 6 of 6)
2011-08-25T14:39:07.816Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:39:07.816Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:39:07.817Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:39:07.817Z cpu2:2610)VC: 1452: Filesystem probe time 7 msec (devices probed 6 of 6)
2011-08-25T14:39:07.879Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:39:07.886Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:39:07.887Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:39:07.887Z cpu2:2610)VC: 1452: Filesystem probe time 7 msec (devices probed 6 of 6)
2011-08-25T14:39:07.959Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:39:07.967Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:39:07.968Z cpu2:2610)VC: 1449: Device rescan time 16 msec (total number of devices 6)
2011-08-25T14:39:07.968Z cpu2:2610)VC: 1452: Filesystem probe time 8 msec (devices probed 6 of 6)
2011-08-25T14:39:08.048Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:39:08.048Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:39:08.049Z cpu2:2610)VC: 1449: Device rescan time 16 msec (total number of devices 6)
2011-08-25T14:39:08.049Z cpu2:2610)VC: 1452: Filesystem probe time 8 msec (devices probed 6 of 6)
2011-08-25T14:39:08.151Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:39:08.151Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:39:08.160Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:39:08.160Z cpu2:2610)VC: 1452: Filesystem probe time 7 msec (devices probed 6 of 6)
2011-08-25T14:39:08.215Z cpu2:2610)Loading module vmfs3 ...
2011-08-25T14:39:08.218Z cpu2:2610)Elf: 1862: module vmfs3 has license VMware
2011-08-25T14:39:08.225Z cpu2:2610)FSDisk: 501: Initialized SCSI-3 registration key: 1ec9b8695c
2011-08-25T14:39:08.225Z cpu2:2610)FSS: 798: Registered fs vmfs3, module 31, fsTypeNum 0xf530
2011-08-25T14:39:08.225Z cpu2:2610)Mod: 4015: Initialization of vmfs3 succeeded with module ID 49.
2011-08-25T14:39:08.225Z cpu2:2610)vmfs3 loaded successfully.
2011-08-25T14:39:08.236Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:39:08.249Z cpu2:2610)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:39:08.250Z cpu2:2610)VC: 1449: Device rescan time 6 msec (total number of devices 6)
2011-08-25T14:39:08.250Z cpu2:2610)VC: 1452: Filesystem probe time 13 msec (devices probed 6 of 6)
2011-08-25T14:39:08.297Z cpu2:2610)Loading module nfsclient ...
2011-08-25T14:39:08.300Z cpu2:2610)Elf: 1862: module nfsclient has license VMware
2011-08-25T14:39:08.303Z cpu2:2610)FSS: 798: Registered fs nfs, module 32, fsTypeNum 0xb00f
2011-08-25T14:39:08.303Z cpu2:2610)NFS: 83: Multiple outstanding NFS requests are supported
2011-08-25T14:39:08.303Z cpu2:2610)Mod: 4015: Initialization of nfsclient succeeded with module ID 50.
2011-08-25T14:39:08.303Z cpu2:2610)nfsclient loaded successfully.
2011-08-25T14:39:08.328Z cpu2:2610)Loading module dell ...
2011-08-25T14:39:08.331Z cpu2:2610)Elf: 1862: module dell has license VMware
2011-08-25T14:39:08.332Z cpu2:2610)Mod: 4015: Initialization of dell succeeded with module ID 51.
2011-08-25T14:39:08.332Z cpu2:2610)dell loaded successfully.
2011-08-25T14:39:08.373Z cpu1:2680)NFS: 157: Command: (remount) Server: (10.64.66.20) IP: (10.64.66.20) Path: (/mnt/volgroup1/volume1/VMFSNFS01/) Label: (VMFSOFL01) Options: (None)
2011-08-25T14:39:08.377Z cpu1:2680)WARNING: NFS: 1367: Server 10.64.66.20 supports read transfer of 524288 truncating to 65536
2011-08-25T14:39:08.377Z cpu1:2680)NFS: 292: Restored connection to the server 10.64.66.20 mount point /mnt/volgroup1/volume1/VMFSNFS01/, mounted as 38d0f4c3-ba9755a3-0000-000000000000 ("VMFSOFL01")
2011-08-25T14:39:08.377Z cpu1:2680)NFS: 217: NFS mount 10.64.66.20:/mnt/volgroup1/volume1/VMFSNFS01/ status: Success
2011-08-25T14:39:08.395Z cpu1:2680)NFS: 157: Command: (remount) Server: (10.64.66.21) IP: (10.64.66.21) Path: (/mnt/volgroup1/volume1/VMFSNFS02/) Label: (VMFSOFL02) Options: (None)
2011-08-25T14:39:08.400Z cpu1:2680)WARNING: NFS: 1367: Server 10.64.66.21 supports read transfer of 524288 truncating to 65536
2011-08-25T14:39:08.400Z cpu1:2680)NFS: 292: Restored connection to the server 10.64.66.21 mount point /mnt/volgroup1/volume1/VMFSNFS02/, mounted as 7b59043b-b8cc9b82-0000-000000000000 ("VMFSOFL02")
2011-08-25T14:39:08.400Z cpu1:2680)NFS: 217: NFS mount 10.64.66.21:/mnt/volgroup1/volume1/VMFSNFS02/ status: Success
2011-08-25T14:39:08.417Z cpu1:2680)NFS: 157: Command: (remount) Server: (10.64.66.22) IP: (10.64.66.22) Path: (/mnt/volgroup1/volume1/VMFSNFS03/) Label: (VMFSOFL03) Options: (None)
2011-08-25T14:39:08.421Z cpu1:2680)WARNING: NFS: 1367: Server 10.64.66.22 supports read transfer of 262144 truncating to 65536
2011-08-25T14:39:08.421Z cpu1:2680)NFS: 292: Restored connection to the server 10.64.66.22 mount point /mnt/volgroup1/volume1/VMFSNFS03/, mounted as a72c87b3-acbed301-0000-000000000000 ("VMFSOFL03")
2011-08-25T14:39:08.421Z cpu1:2680)NFS: 217: NFS mount 10.64.66.22:/mnt/volgroup1/volume1/VMFSNFS03/ status: Success
2011-08-25T14:39:08.439Z cpu1:2680)NFS: 157: Command: (remount) Server: (10.64.66.23) IP: (10.64.66.23) Path: (/mnt/volgroup1/volume1/VMFSNFS04/) Label: (VMFSOFL04) Options: (None)
2011-08-25T14:39:08.441Z cpu1:2680)WARNING: NFS: 1367: Server 10.64.66.23 supports read transfer of 524288 truncating to 65536
2011-08-25T14:39:08.441Z cpu1:2680)NFS: 292: Restored connection to the server 10.64.66.23 mount point /mnt/volgroup1/volume1/VMFSNFS04/, mounted as f4089d51-4f69ec7f-0000-000000000000 ("VMFSOFL04")
2011-08-25T14:39:08.441Z cpu1:2680)NFS: 217: NFS mount 10.64.66.23:/mnt/volgroup1/volume1/VMFSNFS04/ status: Success
2011-08-25T14:39:08.471Z cpu2:2610)Loading module ipmi_msghandler ...
2011-08-25T14:39:08.473Z cpu2:2610)Elf: 1862: module ipmi_msghandler has license GPL
2011-08-25T14:39:08.474Z cpu2:2610)module heap: Initial heap size: 16384, max heap size: 5562368
2011-08-25T14:39:08.474Z cpu2:2610)vmklnx_module_mempool_init: Mempool max 5562368 being used for module: 52

2011-08-25T14:39:08.474Z cpu2:2610)vmk_MemPoolCreate passed for 4 pages

2011-08-25T14:39:08.474Z cpu2:2610)module heap: using memType 0
2011-08-25T14:39:08.474Z cpu2:2610)module heap vmklnx_ipmi_msghandler: creation succeeded. id = 0x410019c68000
2011-08-25T14:39:08.474Z cpu2:2610)<6>ipmi message handler version 39.1-4vmw
2011-08-25T14:39:08.474Z cpu2:2610)Mod: 4015: Initialization of ipmi_msghandler succeeded with module ID 52.
2011-08-25T14:39:08.474Z cpu2:2610)ipmi_msghandler loaded successfully.
2011-08-25T14:39:08.498Z cpu2:2610)Loading module ipmi_si_drv ...
2011-08-25T14:39:08.500Z cpu2:2610)Elf: 1862: module ipmi_si_drv has license GPL
2011-08-25T14:39:08.501Z cpu2:2610)module heap: Initial heap size: 16384, max heap size: 5562368
2011-08-25T14:39:08.501Z cpu2:2610)vmklnx_module_mempool_init: Mempool max 5562368 being used for module: 53

2011-08-25T14:39:08.501Z cpu2:2610)vmk_MemPoolCreate passed for 4 pages

2011-08-25T14:39:08.501Z cpu2:2610)module heap: using memType 0
2011-08-25T14:39:08.501Z cpu2:2610)module heap vmklnx_ipmi_si_drv: creation succeeded. id = 0x410019c70000
2011-08-25T14:39:08.501Z cpu2:2610)<6>IPMI System Interface driver.
2011-08-25T14:39:08.501Z cpu2:2610)<6>ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca8, slave address 0x20, irq 0
2011-08-25T14:39:08.883Z cpu2:2610)<6>ipmi: Found new BMC (man_id: 0x   0002a2,  prod_id: 0x 0100, dev_id: 0x20)
2011-08-25T14:39:08.883Z cpu2:2610)<6>IPMI kcs interface initialized
2011-08-25T14:39:08.883Z cpu2:2610)PCI: driver ipmi_si is looking for devices
2011-08-25T14:39:08.883Z cpu2:2610)PCI: driver ipmi_si claimed 0 device
2011-08-25T14:39:08.883Z cpu2:2610)Mod: 4015: Initialization of ipmi_si_drv succeeded with module ID 53.
2011-08-25T14:39:08.883Z cpu2:2610)ipmi_si_drv loaded successfully.
2011-08-25T14:39:08.908Z cpu2:2610)Loading module ipmi_devintf ...
2011-08-25T14:39:08.911Z cpu2:2610)Elf: 1862: module ipmi_devintf has license GPL
2011-08-25T14:39:08.911Z cpu2:2610)module heap: Initial heap size: 16384, max heap size: 5562368
2011-08-25T14:39:08.911Z cpu2:2610)vmklnx_module_mempool_init: Mempool max 5562368 being used for module: 54

2011-08-25T14:39:08.911Z cpu2:2610)vmk_MemPoolCreate passed for 4 pages

2011-08-25T14:39:08.911Z cpu2:2610)module heap: using memType 0
2011-08-25T14:39:08.911Z cpu2:2610)module heap vmklnx_ipmi_devintf: creation succeeded. id = 0x410019c78000
2011-08-25T14:39:08.911Z cpu2:2610)<6>ipmi device interface
2011-08-25T14:39:08.911Z cpu2:2610)Mod: 4015: Initialization of ipmi_devintf succeeded with module ID 54.
2011-08-25T14:39:08.911Z cpu2:2610)ipmi_devintf loaded successfully.
2011-08-25T14:39:08.938Z cpu2:2610)Loading module vmkstatelogger ...
2011-08-25T14:39:08.941Z cpu2:2610)Elf: 1862: module vmkstatelogger has license VMware
2011-08-25T14:39:08.943Z cpu2:2610)VMKStateLogger: 98: Registering vmkStateLogger module version 1.0
2011-08-25T14:39:08.943Z cpu2:2610)Mod: 4015: Initialization of vmkstatelogger succeeded with module ID 55.
2011-08-25T14:39:08.943Z cpu2:2610)vmkstatelogger loaded successfully.
2011-08-25T14:39:08.980Z cpu2:2610)Loading module migrate ...
2011-08-25T14:39:08.983Z cpu2:2610)Elf: 1862: module migrate has license VMware
2011-08-25T14:39:08.989Z cpu2:2610)Migrate: 120: Registering migration module version 3.0
2011-08-25T14:39:08.989Z cpu2:2610)Migrate: 2822: Initializing migration module with module id 38.
2011-08-25T14:39:08.989Z cpu2:2610)Migrate: 1091: Migrate: Using a default heap size of 17206334 bytes, based on a 36853 MB VM.
2011-08-25T14:39:08.989Z cpu2:2610)Migrate: 2995: Registering module 'VMotion' version 3.0 for migrate type 1
2011-08-25T14:39:08.989Z cpu2:2610)Migrate: 2995: Registering module 'FT' version 3.0 for migrate type 3
2011-08-25T14:39:08.989Z cpu2:2610)Migrate: 2995: Registering module 'FSR' version 3.0 for migrate type 2
2011-08-25T14:39:08.989Z cpu2:2610)Migrate: 3091: Successfully enabled migration support.
2011-08-25T14:39:08.989Z cpu2:2610)Mod: 4015: Initialization of migrate succeeded with module ID 56.
2011-08-25T14:39:08.989Z cpu2:2610)migrate loaded successfully.
2011-08-25T14:39:09.011Z cpu2:2610)Loading module cbt ...
2011-08-25T14:39:09.014Z cpu2:2610)Elf: 1862: module cbt has license VMware
2011-08-25T14:39:09.015Z cpu2:2610)FDS: 386: cbt
2011-08-25T14:39:09.015Z cpu2:2610)Mod: 4015: Initialization of cbt succeeded with module ID 57.
2011-08-25T14:39:09.015Z cpu2:2610)cbt loaded successfully.
2011-08-25T14:39:09.037Z cpu2:2610)Loading module svmmirror ...
2011-08-25T14:39:09.040Z cpu2:2610)Elf: 1862: module svmmirror has license VMware
2011-08-25T14:39:09.041Z cpu2:2610)FDS: 386: svm
2011-08-25T14:39:09.041Z cpu2:2610)Mod: 4015: Initialization of svmmirror succeeded with module ID 58.
2011-08-25T14:39:09.041Z cpu2:2610)svmmirror loaded successfully.
2011-08-25T14:39:09.068Z cpu2:2610)Loading module hbr_filter ...
2011-08-25T14:39:09.071Z cpu2:2610)Elf: 1862: module hbr_filter has license VMware
2011-08-25T14:39:09.076Z cpu2:2610)Hbr: 772: Hbr initialize runtime heap: (20368553:315247846)
2011-08-25T14:39:09.076Z cpu2:2610)VSCSIFilter: 905: Registered VSCSI filter type hbr_filter
2011-08-25T14:39:09.076Z cpu2:2610)Hbr: 328: HBR crc32 instruction: disabled
2011-08-25T14:39:09.077Z cpu2:2610)Hbr: 159: HBR bitmap mempool initializing (4096:2147483646)
2011-08-25T14:39:09.077Z cpu2:2610)Mod: 4015: Initialization of hbr_filter succeeded with module ID 59.
2011-08-25T14:39:09.077Z cpu2:2610)hbr_filter loaded successfully.
2011-08-25T14:39:09.650Z cpu0:2738)Config: 346: "SIOControlFlag1" = 0, Old Value: 0, (Status: 0x0)
2011-08-25T14:39:13.969Z cpu1:2816)VisorFSRam: 577: hostdstats with (0,247,755)
2011-08-25T14:39:13.969Z cpu1:2816)FSS: 6334: Mounting fs visorfs (41000aa2c0d0) with -o 0,247,0755,hostdstats on file descriptor 41000a7537a0
2011-08-25T14:39:16.062Z cpu3:2828)Config: 346: "SIOControlFlag2" = 0, Old Value: 0, (Status: 0x0)
2011-08-25T14:39:20.473Z cpu3:3056)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:39:20.473Z cpu3:3056)Vol3: 647: Couldn't read volume header from control: Invalid handle
2011-08-25T14:39:20.473Z cpu3:3056)FSS: 4333: No FS driver claimed device 'control': Not supported
2011-08-25T14:39:20.490Z cpu3:3056)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:39:20.491Z cpu3:3056)VC: 1449: Device rescan time 8 msec (total number of devices 7)
2011-08-25T14:39:20.491Z cpu3:3056)VC: 1452: Filesystem probe time 17 msec (devices probed 6 of 7)
2011-08-25T14:39:20.523Z cpu1:3058)<6>usb passthrough enabled; all eligible devices will be unclaimed by kernel drivers
2011-08-25T14:39:20.523Z cpu0:2629)<6>usb-storage 1-1.2:1.1: unclaiming vmhba33
2011-08-25T14:39:20.524Z cpu3:2160)ScsiPath: 4963: DeletePath : adapter=vmhba33, channel=0, target=0, lun=0
2011-08-25T14:39:20.524Z cpu3:2160)WARNING: NMP: nmpUnclaimPath:1577:Physical path "vmhba33:C0:T0:L0" is the last path to NMP device "Unregistered Device". The device has been unregistered.
2011-08-25T14:39:20.524Z cpu3:2160)WARNING: ScsiPath: 5022: Remove path: vmhba33:C0:T0:L0
2011-08-25T14:39:20.524Z cpu0:2629)LinScsiLLD: scsi_remove_host:857: Removed Host Adapter vmhba33
2011-08-25T14:39:20.524Z cpu0:2629)<6>usb-storage 1-1.2:1.0: interface is claimed by usb-storage
2011-08-25T14:39:20.524Z cpu0:2629)<6>usb 1-1.2: device is not available for passthrough
2011-08-25T14:39:20.524Z cpu0:2629)ScsiAdapter: 2228: Unregistering adapter vmhba33
2011-08-25T14:39:20.524Z cpu0:2629)DMA: 569: DMA Engine 'vmhba33' destroyed.
2011-08-25T14:39:20.608Z cpu0:3060)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:39:20.611Z cpu1:3060)Vol3: 647: Couldn't read volume header from control: Invalid handle
2011-08-25T14:39:20.611Z cpu1:3060)FSS: 4333: No FS driver claimed device 'control': Not supported
2011-08-25T14:39:20.618Z cpu0:3060)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:39:20.618Z cpu0:3060)VC: 1449: Device rescan time 3 msec (total number of devices 7)
2011-08-25T14:39:20.618Z cpu0:3060)VC: 1452: Filesystem probe time 9 msec (devices probed 6 of 7)
2011-08-25T14:39:21.043Z cpu3:2828)Config: 346: "VMOverheadGrowthLimit" = -1, Old Value: 0, (Status: 0x0)
2011-08-25T14:39:25.717Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:39:30.719Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:39:35.725Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:39:40.727Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:39:45.733Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:39:50.737Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:39:55.742Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:40:00.745Z cpu0:2668)Tcpip: 2236: msleep returned 4
2011-08-25T14:40:49.586Z cpu0:2828)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x1a (0x4124007a5540) to dev "t10.DP______BACKPLANE000000" on path "vmhba1:C0:T32:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.Act:NONE
2011-08-25T14:40:49.586Z cpu0:2828)ScsiDeviceIO: 2316: Cmd(0x4124007a5540) 0x1a, CmdSN 0x1284 to dev "t10.DP______BACKPLANE000000" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2011-08-25T14:40:52.222Z cpu2:3061)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:40:52.222Z cpu2:3061)Vol3: 647: Couldn't read volume header from control: Invalid handle
2011-08-25T14:40:52.222Z cpu2:3061)FSS: 4333: No FS driver claimed device 'control': Not supported
2011-08-25T14:40:52.232Z cpu2:3061)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:40:52.233Z cpu2:3061)VC: 1449: Device rescan time 3 msec (total number of devices 7)
2011-08-25T14:40:52.233Z cpu2:3061)VC: 1452: Filesystem probe time 10 msec (devices probed 6 of 7)
2011-08-25T14:40:52.243Z cpu2:3061)ScsiScan: 1098: Path 'vmhba1:C0:T0:L0': Vendor: 'FUJITSU '  Model: 'MAY2036RC       '  Rev: 'D108'
2011-08-25T14:40:52.243Z cpu2:3061)ScsiScan: 1101: Path 'vmhba1:C0:T0:L0': Type: 0x0, ANSI rev: 3, TPGS: 0 (none)
2011-08-25T14:40:52.244Z cpu2:3061)megasas_slave_configure: do not export physical disk devices to upper layer.
2011-08-25T14:40:52.244Z cpu2:3061)WARNING: ScsiScan: 1485: Failed to add path vmhba1:C0:T0:L0 : Not found
2011-08-25T14:40:52.244Z cpu2:3061)ScsiScan: 1098: Path 'vmhba1:C0:T1:L0': Vendor: 'FUJITSU '  Model: 'MAY2036RC       '  Rev: 'D108'
2011-08-25T14:40:52.244Z cpu2:3061)ScsiScan: 1101: Path 'vmhba1:C0:T1:L0': Type: 0x0, ANSI rev: 3, TPGS: 0 (none)
2011-08-25T14:40:52.245Z cpu2:3061)megasas_slave_configure: do not export physical disk devices to upper layer.
2011-08-25T14:40:52.245Z cpu2:3061)WARNING: ScsiScan: 1485: Failed to add path vmhba1:C0:T1:L0 : Not found
2011-08-25T14:40:52.363Z cpu2:3061)Vol3: 647: Couldn't read volume header from control: Invalid handle
2011-08-25T14:40:52.363Z cpu2:3061)FSS: 4333: No FS driver claimed device 'control': Not supported
2011-08-25T14:40:52.366Z cpu2:3061)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:40:52.367Z cpu2:3061)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:40:52.368Z cpu2:3061)VC: 1449: Device rescan time 3 msec (total number of devices 7)
2011-08-25T14:40:52.368Z cpu2:3061)VC: 1452: Filesystem probe time 10 msec (devices probed 6 of 7)
2011-08-25T14:40:52.377Z cpu3:2651)usb-storage: detected SCSI revision number 0 on vmhba32
2011-08-25T14:40:52.377Z cpu3:2651)usb-storage: patching inquiry data to change SCSI revision number from 0 to 2 on vmhba32
2011-08-25T14:40:52.377Z cpu2:2157)ScsiUid: 273: Path 'vmhba32:C0:T0:L0' does not support VPD Device Id page.
2011-08-25T14:40:52.377Z cpu2:2157)VMWARE SCSI Id: Could not get disk id for vmhba32:C0:T0:L0
2011-08-25T14:40:52.382Z cpu2:3061)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:40:52.386Z cpu2:3061)Vol3: 647: Couldn't read volume header from control: Invalid handle
2011-08-25T14:40:52.386Z cpu2:3061)FSS: 4333: No FS driver claimed device 'control': Not supported
2011-08-25T14:40:52.392Z cpu3:3061)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:40:52.393Z cpu3:3061)VC: 1449: Device rescan time 3 msec (total number of devices 7)
2011-08-25T14:40:52.393Z cpu3:3061)VC: 1452: Filesystem probe time 10 msec (devices probed 6 of 7)
2011-08-25T14:40:52.444Z cpu3:3061)Vol3: 647: Couldn't read volume header from control: Invalid handle
2011-08-25T14:40:52.444Z cpu3:3061)FSS: 4333: No FS driver claimed device 'control': Not supported
2011-08-25T14:40:52.451Z cpu1:3061)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:40:52.451Z cpu1:3061)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:40:52.452Z cpu1:3061)VC: 1449: Device rescan time 3 msec (total number of devices 7)
2011-08-25T14:40:52.452Z cpu1:3061)VC: 1452: Filesystem probe time 10 msec (devices probed 6 of 7)
2011-08-25T14:40:52.511Z cpu0:3061)Vol3: 647: Couldn't read volume header from control: Invalid handle
2011-08-25T14:40:52.511Z cpu0:3061)FSS: 4333: No FS driver claimed device 'control': Not supported
2011-08-25T14:40:52.517Z cpu1:3061)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:40:52.518Z cpu1:3061)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:40:52.518Z cpu1:3061)VC: 1449: Device rescan time 3 msec (total number of devices 7)
2011-08-25T14:40:52.518Z cpu1:3061)VC: 1452: Filesystem probe time 9 msec (devices probed 6 of 7)
2011-08-25T14:40:52.579Z cpu1:3061)Vol3: 647: Couldn't read volume header from control: Invalid handle
2011-08-25T14:40:52.579Z cpu1:3061)FSS: 4333: No FS driver claimed device 'control': Not supported
2011-08-25T14:40:52.582Z cpu1:3061)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:40:52.583Z cpu1:3061)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:40:52.583Z cpu1:3061)VC: 1449: Device rescan time 3 msec (total number of devices 7)
2011-08-25T14:40:52.583Z cpu1:3061)VC: 1452: Filesystem probe time 9 msec (devices probed 6 of 7)
2011-08-25T14:40:52.645Z cpu1:3061)Vol3: 647: Couldn't read volume header from control: Invalid handle
2011-08-25T14:40:52.645Z cpu1:3061)FSS: 4333: No FS driver claimed device 'control': Not supported
2011-08-25T14:40:52.648Z cpu0:3061)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:40:52.648Z cpu0:3061)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:40:52.649Z cpu0:3061)VC: 1449: Device rescan time 3 msec (total number of devices 7)
2011-08-25T14:40:52.649Z cpu0:3061)VC: 1452: Filesystem probe time 9 msec (devices probed 6 of 7)
2011-08-25T14:40:52.662Z cpu0:3061)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-25T14:40:52.662Z cpu0:3061)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-08-25T14:40:52.665Z cpu0:3061)Vol3: 647: Couldn't read volume header from control: Invalid handle
2011-08-25T14:40:52.665Z cpu0:3061)FSS: 4333: No FS driver claimed device 'control': Not supported
2011-08-25T14:40:52.672Z cpu1:3061)VC: 1449: Device rescan time 3 msec (total number of devices 7)
2011-08-25T14:40:52.672Z cpu1:3061)VC: 1452: Filesystem probe time 9 msec (devices probed 6 of 7)
2011-08-25T14:40:55.305Z cpu0:3337)Boot Successful

Reply
0 Kudos
DSeaman
Enthusiast
Enthusiast

I had the same problem when I took a USB stick with 5.0 on it from one white box server and put it into another (slightly different hardware) and tried to boot. I gave it a good 10-15 minutes and eventually I got the DCUI, where I could go in and reconfigure my management network since the NICs were a bit different. 

Derek Seaman
Reply
0 Kudos
indico
Contributor
Contributor

I am also seeing this. Everything was fine until we started testing path failover and replugging. The paths stayed dead and also the hosts hangs after reboot.

If we reinstall the host it works until the next reboot. After a couple of reboots it also seems to lose the software iSCSI initiator and the vDS...

Our hosts are DL380 G5/G6's (booting from USB) and the SAN is HP P2000 G3 with latest firmware.

More testing:


After doing some more testing it seems to be a vmk issue. If the paths shows up as dead or the iSCSI software adapter fails to find both paths, try change the failover order, of your iSCSI vmk's, back and forth while doing vmkping testing. In my case I could only (if lucky) ping one of the SAN controller IP's, and after a bit of changing back and forth of the failover order the second SAN IP also started working. I've been able to replicate this on three different hosts but still havent found a way around the boot issue. It just seems to take forever to boot but the hosts do come up, can take between 30-120 minutes sometimes.

I should also add that all these hosts, switches and the SAN have been running ESXi 4.1 flawlessly prior to upgrade.

Reply
0 Kudos
ashleyw
Enthusiast
Enthusiast

On cisco UCS blade infrastructure with MSA2000 + MSA2012i + NexentaStor iSCSI SANs, I see the same delays on reboot. The boot process first shows "iscsi_vmk loaded successfully" for around 2 minutes, then "vmw_satp_alwa loaded sucessfully" on the console for around 10 minutes while booting. If it makes a difference, we had previously changed the default paths to round robin via the CLI with;

esxcli storage nmp satp set --default-psp="VMW_PSP_RR" --satp="VMW_SATP_ALUA"

Previous workaround on 4.1i for the same delays on start don't seem to be available under esx5i.

I have checked the paths to the storage and all seem to be OK.

Reply
0 Kudos
indico
Contributor
Contributor

Just curious, when the host has booted does your paths come up OK? Mine stays dead or the hosts reports no available datastores. After moving the NIC's failover order (it seems to be enough with just moving one vmk nic from active to unused and then back) I can vmkping the iSCSI IP's and after a rescan the paths comes up. I have to change the path selection manually after that though.

We are also using roundrobin and VMW_SATP_ALUA.

Reply
0 Kudos
JayDeah
Contributor
Contributor

I don't need to do anything other than a rescan post-boot to get it to show. Rescans happen pretty quick too.

I raised a support ticket just waiting for them to finish investigating.

I really did hope it'd be a misconfig as at least that will be easy to fix!

Sent from my iPhone

Reply
0 Kudos
Jae-Hoon_Choi
Enthusiast
Enthusiast

That's a critical vSphere 5.0's bug.

Tiny solution is below that I found.

1. Go to the vSphere 5.0 iSCSI Software Adapter property.

2. Go to network configuration tab

3. unbind all pNIC that bind to vSphere 5.0 iSCSI Software Adapter (Just click the "Remove" button.)

4. Register iSCSI Storage ip address using Dynamic discovery or Static discovery tab

5. Rescan using "Rescan All" link.

6. That's all...

then......

Host reboot speed will normal

P.S

I can't understand this unbelivable(or funny) BUG!!!

P.S.2

I don't permit deploy vSphere 5.0 in our production environment!!!!!

- Also development, test VM environment, too.

I don't permit purchase vSphere 5.0 in our company.

I'll go to Hyper-V 3.0 in Windows Server 2012 + System Center 2012 later....

Reply
0 Kudos
indico
Contributor
Contributor

Hi, some other findings point to a blank and empty RAID controller (vmhba1) which the ESXi seems to be trying to read from. Why?

2011-08-29T15:36:45.238Z cpu2:39236)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x28 (0x4124003b6580) to dev "mpx.vmhba1:C0:T0:L0" on path "vmhba1:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.Act:NONE
2011-08-29T15:36:45.238Z cpu2:39236)ScsiDeviceIO: 2316: Cmd(0x4124003b6580) 0x28, CmdSN 0xc46 to dev "mpx.vmhba1:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2011-08-29T15:36:45.238Z cpu2:39236)Partition: 484: Read of GPT header failed on "mpx.vmhba1:C0:T0:L0": I/O error
2011-08-29T15:36:45.238Z cpu2:39236)WARNING: LinBlock: LinuxBlockCommand:1371:READ past end of device on major 104 - 0 + 1 > 0
2011-08-29T15:36:45.238Z cpu2:39236)ScsiDeviceIO: 2316: Cmd(0x4124003b6580) 0x28, CmdSN 0xc47 to dev "mpx.vmhba1:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2011-08-29T15:36:45.238Z cpu2:39236)WARNING: Partition: 944: Partition table read from device mpx.vmhba1:C0:T0:L0 failed: I/O error
2011-08-29T15:36:45.238Z cpu2:39236)WARNING: LinBlock: LinuxBlockCommand:1371:READ past end of device on major 104 - 0 + 1 > 0


The controller is an empty HP SmartArray P400/P410 and has no configured disks. Anyone knows how to disable this controller with bootparameters?

Update:

This is getting silly. After disabling the P410 controller on one of the DL380G6's we now have another similar issue.

2011-08-29T21:10:21.175Z cpu4:3057)VC: 1449: Device rescan time 17 msec (total number of devices 7)
2011-08-29T21:10:21.175Z cpu4:3057)VC: 1452: Filesystem probe time 102 msec (devices probed 7 of 7)
2011-08-29T21:10:21.373Z cpu3:3058)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-29T21:10:21.403Z cpu3:3058)Vol3: 647: Couldn't read volume header from control: Invalid handle
2011-08-29T21:10:21.403Z cpu3:3058)FSS: 4333: No FS driver claimed device 'control': Not supported
2011-08-29T21:10:21.447Z cpu3:3058)VC: 1449: Device rescan time 17 msec (total number of devices 7)
2011-08-29T21:10:21.447Z cpu3:3058)VC: 1452: Filesystem probe time 101 msec (devices probed 7 of 7)
2011-08-29T21:10:21.596Z cpu5:3058)FSS: 4333: No FS driver claimed device 'mpx.vmhba0:C0:T0:L0': Not supported
2011-08-29T21:10:21.624Z cpu4:3058)Vol3: 647: Couldn't read volume header from control: Invalid handle
2011-08-29T21:10:21.624Z cpu4:3058)FSS: 4333: No FS driver claimed device 'control': Not supported
2011-08-29T21:10:21.641Z cpu4:3058)VC: 1449: Device rescan time 18 msec (total number of devices 7)
2011-08-29T21:10:21.641Z cpu4:3058)VC: 1452: Filesystem probe time 102 msec (devices probed 7 of 7)
2011-08-29T21:10:21.689Z cpu6:2175)ScsiUid: 273: Path 'vmhba0:C0:T0:L0' does not support VPD Device Id page.
2011-08-29T21:10:21.691Z cpu6:2175)VMWARE SCSI Id: Could not get disk id for vmhba0:C0:T0:L0

2011-08-29T21:10:20.953Z cpu6:3055)<6>usb passthrough enabled; all eligible devices will be unclaimed by kernel drivers except for ESXi boot device vmhba32

vmhba0 is the DVD in the server. vmhba32 is the USB-memory used for booting. Why these stupid probes and long timeouts? Any way to disable usb-passthrough?

Reply
0 Kudos
JayDeah
Contributor
Contributor

I can't just unbind pnocs I need the server to boot unaided. Was really looking at using power drs in 5.0 as I hoped the ha bug would be cured. As for vmhba1 it doesn't look as if it's pausing with that error a the timestamps are all the same so probably just a red herring?

Sent from my iPhone

Reply
0 Kudos
indico
Contributor
Contributor

I think unbinding helped in my case for the dead paths. It now seems to reconnect after reboot. I've also switched from one vSwitch to separate vSwitches for the iSCSI-NIC's. I havent had a failed vmkping after any recent reboot or path failover.

The boots are still very slow though, around 10-20 minutes.

Reply
0 Kudos
JayDeah
Contributor
Contributor

One thing I have just noticed and it may be relevant.

The MD3200i has 2 controllers with 4 ports per controller. I am connecting up only the first 2 ports on each controller. The other 2 ports on each controller are still enabled and have their default IP addresses assigned. When performing a discovery these IP addresses get discovered fro the array and appear in the "static discovery" tab.

due to a bug in the gui on the MD3200i software it will be difficult (but not impossible) to disable the extra 4 un-used ports.

[EDIT]

nope, doesnt look like it's that, i just disabled the unused ports and built a brand new host with the correct discovery options and it's been sat booting for a while still Smiley Sad

Reply
0 Kudos
HayamaMarcelo
Contributor
Contributor

Indico,

May be I´m a little naive to help you. Anyway, I was having a similar problem with USB and ESXi 5.

I think as I´ve read in some article I can´t recall, usbarbitrator is the responsible for usb passthrough. Then if you issue this command:

/etc/init.d usbarbitrator stop

the usbarbitrator service will stop, so usb passthrough won´t work until you issue a START command line (/etc/init.d/usbarbitrator start).

So, I hope it helps.

Best regards,

Marcelo Hayama

http://tiny.cc/HayCard

Reply
0 Kudos
ashleyw
Enthusiast
Enthusiast

Hi, after spending the last few hours going through the log files, I can 100% confirm that the issue with long boot times appears to be a bug in vsphere5 for some reason trying to scan the CD-ROM drive corresponding to vmhba32. On a nested ESX5i instance, this causes a delay of 5 minutes, on an HP DL360 G7, this causes a delay of 30 minutes, on a Cisco UCS blade this causes a delay of approximately 15 minutes.

Just ssh into the ESX5i host and go to the /var/log directory and do a "less vmkernel.log" and go to the end of the file and page up and you'll see repetitions of the lines below accounting for the majority of the boot time - as to why this happens during the iscsi section I guess only VMware can answer.

Is there anyone in VMware that can escalate this and get a hotfix out - it has a big impact for iscsi shops.

I have attached the segments from the log file from the nested instance.

<pre>

2011-09-14T04:50:43.471Z cpu0:2050)LVM: 13188: One or more LVM devices have been discovered.

2011-09-14T04:51:06.754Z cpu1:2604)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-09-14T04:51:06.775Z cpu1:2604)VC: 1449: Device rescan time 22 msec (total number of devices 5)
2011-09-14T04:51:06.775Z cpu1:2604)VC: 1452: Filesystem probe time 19 msec (devices probed 5 of 5)
2011-09-14T04:51:32.987Z cpu0:2604)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-09-14T04:51:33.009Z cpu0:2604)VC: 1449: Device rescan time 24 msec (total number of devices 5)
2011-09-14T04:51:33.009Z cpu0:2604)VC: 1452: Filesystem probe time 21 msec (devices probed 5 of 5)
2011-09-14T04:51:59.208Z cpu0:2604)FSS: 4333: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': Not supported
2011-09-14T04:51:59.231Z cpu0:2604)VC: 1449: Device rescan time 23 msec (total number of devices 5)
2011-09-14T04:51:59.231Z cpu0:2604)VC: 1452: Filesystem probe time 21 msec (devices probed 5 of 5)

</pre>

Reply
0 Kudos