VMware Cloud Community
Freaky201110141
Contributor
Contributor
‎02-29-2012 08:15 AM

ESXi 5.0, Xyratex SAN's, slow boot (cmd 0x12 failed)

Hi there,

we have a customer that's currently running 3.5u3. Needless to say, we want to get rid of it (they actually run 2008 R2 on it, which isn't supported until u4 or u5...).

Anyways, they have 3 Xyratex SAN's, 2 SAS and 1 SATA. These cause a lot of issues during boot. Apparently the LUN's don't understand SCSI Command 0x12, which is INQUIRY from what I understand and *should* be supported by any SCSI device. ESXi keeps resending these commands a lot of times, causing the boot to be really really slow. Of course, since it's LUN based, adding LUN's will make the boot time longer and removing them shorter.

3.5u3 boots in like 2-3 mins btw.

I know I can reconfigure some things so it will retry less etc. but was more hoping for a solution with the cmd Smiley Happy. Below is a piece of log on a full try (well... it'll try again later...) for 1 LUN. Do note however the data stores are there once it's booted and I can access them. Haven't done any speed tests yet nor tested for a prolonged time to see if these retries will choke storage for a while. Also please note we tested by putting ESXi 5.0 on USB stick and booting it on one of the current 3.5u3 hosts. It should see exactly the same as 3.5u3 thus since it uses fibre HBA's.

Any ideas?

TIA

2012-02-28T13:55:29.350Z cpu2:2630)vmw_psp_fixed: psp_fixedSelectPathToActivateInt:479: Changing active path from NONE to vmhba4:C0:T2:L3 for device "Unregistered Device".

2012-02-28T13:55:29.350Z cpu2:2630)VMWARE SCSI Id: Id for vmhba4:C0:T2:L3

0x60 0x05 0x0c 0xc0 0x00 0x20 0x27 0x6c 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x07 0x46 0x35 0x34 0x30 0x32 0x45

2012-02-28T13:55:29.350Z cpu3:2051)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x12 (0x4124007cbac0) to dev "naa.60050cc00020276c0000000000000007" on path "vmhba4:C0:T2:L3" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x25 0x0.Act:FAILOVER

2012-02-28T13:55:29.350Z cpu3:2051)WARNING: NMP: nmp_DeviceRetryCommand:133:Device "naa.60050cc00020276c0000000000000007": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.

2012-02-28T13:55:30.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:55:30.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:55:30.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:55:31.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:55:31.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:55:31.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:55:32.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:55:32.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:55:32.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:55:32.351Z cpu2:2050)LVM: 13188: One or more LVM devices have been discovered.

2012-02-28T13:55:33.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:55:33.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:55:33.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:55:34.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:55:34.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:55:34.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:55:35.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:55:35.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:55:35.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:55:36.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:55:36.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:55:36.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:55:37.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:55:37.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:55:37.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:55:38.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:55:38.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:55:38.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:55:39.347Z cpu6:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:55:39.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:55:39.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:55:40.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:55:40.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:55:40.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:55:41.347Z cpu7:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:55:41.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:55:41.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:55:42.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:55:42.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:55:42.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:55:43.347Z cpu7:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:55:43.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:55:43.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:55:44.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:55:44.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:55:44.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:55:45.347Z cpu4:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:55:45.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:55:45.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:55:46.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:55:46.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:55:46.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:55:47.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:55:47.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:55:47.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:55:48.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:55:48.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:55:48.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:55:49.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:55:49.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:55:49.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:55:50.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:55:50.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:55:50.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:55:51.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:55:51.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:55:51.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:55:52.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:55:52.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:55:52.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:55:53.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:55:53.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:55:53.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:55:54.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:55:54.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:55:54.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:55:55.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:55:55.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:55:55.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:55:56.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:55:56.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:55:56.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:55:57.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:55:57.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:55:57.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:55:58.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:55:58.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:55:58.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:55:59.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:55:59.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:55:59.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:00.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:00.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:00.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:01.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:01.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:01.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:02.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:02.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:02.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:03.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:03.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:03.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:04.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:04.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:04.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:05.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:05.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:05.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:06.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:06.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:06.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:07.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:07.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:07.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:08.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:08.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:08.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:09.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:09.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:09.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:09.351Z cpu4:2177)ScsiDeviceIO: 2316: Cmd(0x4124007cbac0) 0x12, CmdSN 0x68 to dev "naa.60050cc00020276c0000000000000007" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x5 0x25 0x0.

2012-02-28T13:56:09.351Z cpu4:2177)WARNING: NMP: nmp_DeviceStartLoop:721:NMP Device "naa.60050cc00020276c0000000000000007" is blocked. Not starting I/O from device.

2012-02-28T13:56:09.351Z cpu2:2630)WARNING: NMP: nmp_DeviceStartLoop:721:NMP Device "naa.60050cc00020276c0000000000000007" is blocked. Not starting I/O from device.

2012-02-28T13:56:10.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:562:Retry world restore device "naa.60050cc00020276c0000000000000007" - no more commands to retry

2012-02-28T13:56:10.347Z cpu2:2630)ScsiDeviceIO: 5837: QErr is correctly set to 0x0 for device naa.60050cc00020276c0000000000000007.

2012-02-28T13:56:10.348Z cpu2:2630)ScsiDeviceIO: 6333: Could not detect setting of sitpua for device naa.60050cc00020276c0000000000000007. Error Failure.

2012-02-28T13:56:10.349Z cpu3:2051)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x12 (0x4124007cbac0) to dev "naa.60050cc00020276c0000000000000007" on path "vmhba4:C0:T2:L3" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x25 0x0.Act:FAILOVER

2012-02-28T13:56:10.349Z cpu3:2051)WARNING: NMP: nmp_DeviceRetryCommand:133:Device "naa.60050cc00020276c0000000000000007": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.

2012-02-28T13:56:11.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:11.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:11.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:12.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:12.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:12.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:13.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:13.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:13.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:14.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:14.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:14.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:15.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:15.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:15.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:16.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:16.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:16.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:17.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:17.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:17.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:18.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:18.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:18.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:19.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:19.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:19.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:20.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:20.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:20.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:21.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:21.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:21.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:22.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:22.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:22.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:23.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:23.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:23.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:24.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:24.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:24.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:25.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:25.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:25.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:26.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:26.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:26.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:27.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:27.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:27.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:28.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:28.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:28.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:29.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:29.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:29.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:30.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:30.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:30.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:31.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:31.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:31.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:32.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:32.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:32.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:33.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:33.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:33.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:34.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:34.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:34.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:35.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:35.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:35.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:36.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:36.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:36.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:37.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:37.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:37.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:38.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:38.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:38.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:39.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:39.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:39.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:40.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:40.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:40.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:41.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:41.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:41.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:42.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:42.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:42.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:43.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:43.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:43.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:44.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:44.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:44.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:45.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:45.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:45.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:46.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:46.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:46.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:47.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:47.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:47.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:48.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:48.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:48.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:49.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:49.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:49.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:50.347Z cpu6:2670)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60050cc00020276c0000000000000007" - issuing command 0x4124007cbac0

2012-02-28T13:56:50.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:348:Retry command 0x12 (0x4124007cbac0) to NMP device "naa.60050cc00020276c0000000000000007" failed on physical path "vmhba4:C0:T2:L3" H:0x0 D:0x2 P:0x0 Valid sense data: 0x [0$

2012-02-28T13:56:50.347Z cpu3:2051)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.60050cc00020276c0000000000000007": awaiting fast path state update before retrying failed command again...

2012-02-28T13:56:50.350Z cpu4:2177)ScsiDeviceIO: 2316: Cmd(0x4124007cbac0) 0x12, CmdSN 0x6f to dev "naa.60050cc00020276c0000000000000007" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x5 0x25 0x0.

2012-02-28T13:56:50.350Z cpu4:2177)WARNING: NMP: nmp_DeviceStartLoop:721:NMP Device "naa.60050cc00020276c0000000000000007" is blocked. Not starting I/O from device.

2012-02-28T13:56:50.350Z cpu2:2630)WARNING: NMP: nmp_DeviceStartLoop:721:NMP Device "naa.60050cc00020276c0000000000000007" is blocked. Not starting I/O from device.

2012-02-28T13:56:51.347Z cpu5:2669)WARNING: NMP: nmpDeviceAttemptFailover:562:Retry world restore device "naa.60050cc00020276c0000000000000007" - no more commands to retry

2012-02-28T13:56:51.372Z cpu2:2630)ScsiDevice: 3121: Successfully registered device "naa.60050cc00020276c0000000000000007" from plugin "NMP" of type 0

Reply
7 Replies
tmac3
Contributor
Contributor
‎03-15-2012 06:10 PM

Did you find a solution to this? I just ran into the same issue.

Reply
0 Kudos
tmac3
Contributor
Contributor
‎03-15-2012 08:13 PM

Also noticed VMW_PSP_RR doesn't seem to stick on reboot. Changes back to Fixed.

Reply
0 Kudos
Josh26
Virtuoso
Virtuoso
‎03-15-2012 08:39 PM

Hi,

I believe "slow boot with iSCSI" is a known issue, resolved by application of the latest ESXi hotfixes.

Reply
0 Kudos
tmac3
Contributor
Contributor
‎03-19-2012 02:55 PM

This is fiber not iscsi. When the cmd 0x12 finally fails the LUN it's trying it on hangs up for about 10-20 seconds. This doesn't happen in 4.1

Reply
0 Kudos
Freaky201110141
Contributor
Contributor
‎03-26-2012 03:18 AM

Hi,

no I didn't find a solution. I used the image with the iSCSI patches included by the way (but as already pointed out, this is FC not iSCSI).

For now I've just upgraded 3.5. I read somewhere 4.0 (and perhaps 4.1 too then) should function fine.

Find the issue really weird tho' since 0x12 (inquiry) is a mandatory command (http://en.wikipedia.org/wiki/SCSI_commands) and I don't really believe it isn't supported thus.

Regards

Reply
0 Kudos
VDBG
Contributor
Contributor
‎04-28-2013 08:17 PM

tmac3.

You say the hang ups don't occur in ESXi 4.1. Have you tried Xyratex SAN with 4.1?

I have a Xyratex F6412E and am trying to see if I can provision it for ESXi 4.1.

I would prefer 5.1 but have heard that there are problems (just like the slows and hang ups you mention).

Reply
0 Kudos
Sreec
VMware Employee
VMware Employee
‎04-28-2013 10:23 PM

Hello Freaky,
             I believe naa.60050cc00020276c0000000000000007 is the boot lun which you are talking about?If so below points are correct.
SCSI inquiry commands are failing on a consistent basis,also i can see reservation errors logged.
H:0x0 D:0x2 P:0x0 Valid sense data indicates>When the target returns a Check Condition in response to a command it is indicating that it has entered a contingent allegiance condition. This means that an error occurred when it attempted to execute a SCSI command
1)Have you followed
https://www.vmware.com/pdf/vsphere4/r41/vsp_41_san_cfg.pdf to ensure boot lun is configured correctly
2)Understand that Storage is Xyratex ,please check the actual model number and do ensure it is supported as per http://www.vmware.com/resources/compatibility/search.php?deviceCategory=san
3)Do you see any errors logged against boot lun in the Storage array?
4)Did you tried presenting a new boot-lun and tried testing it ?
Note:You are right current version of ESX 3.5u3 is unsupported,so better get it upgraded to U5 or later.
Cheers,
Sree | VCIX-5X| VCAP-5X| VExpert 7x|Cisco Certified Specialist
Please KUDO helpful posts and mark the thread as solved if answered
Reply