Upgrade to 5.1 HP Custom Image unable to mount P2000 SAS Vdisk

[markbusu]

Hi,

We have recently upgraded (also performed a fresh install) to the VMware 5.1 HP Custom Image. One the installation is complete, and VMware starts booting, VMware becomes unresponsive and starts looping the following error in the VMkernal:

2012-09-29T01:11:47.750Z cpu10:4106)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.600***************": awaiting fast path state update before retrying failed command again...
2012-09-29T01:11:48.750Z cpu0:13406)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.600***************" - issuing command 0x4124008133c0
2012-09-29T01:11:48.750Z cpu10:4106)<4>hpsa 0000:09:00.0: Device:C1:B0:T1:L11 Command:0xc2 Command Invalid.
2012-09-29T01:11:48.750Z cpu10:4106)WARNING: LinScsi: SCSILinuxProcessCompletions:772:Error BytesXferred > Requested Length Marking transfer length as 0 - vmhba = vmhba0, Driver Name = hpsa, Requested length = 1024, Resid = 3012
2012-09-29T01:11:48.750Z cpu10:4106)WARNING: NMP: nmpCompleteRetryForPath:348:Retry cmd 0xc2 (0x4124008133c0) to dev "naa.600***************" failed on path "vmhba0:C0:T1:L11" H:0x1 D:0x0 P:0x0 Possible sense data: 0x0 0x0
0x0.
2012-09-29T01:11:48.750Z cpu10:4106)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.600***************": awaiting fast path state update before retrying failed command again...
2012-09-29T01:11:49.750Z cpu5:4651)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.600***************" - issuing command 0x4124008133c0
2012-09-29T01:11:49.750Z cpu10:4106)<4>hpsa 0000:09:00.0: Device:C1:B0:T1:L11 Command:0xc2 Command Invalid.
2012-09-29T01:11:49.750Z cpu10:4106)WARNING: LinScsi: SCSILinuxProcessCompletions:772:Error BytesXferred > Requested Length Marking transfer length as 0 - vmhba = vmhba0, Driver Name = hpsa, Requested length = 1024, Resid = 3012
2012-09-29T01:11:49.750Z cpu10:4106)WARNING: NMP: nmpCompleteRetryForPath:348:Retry cmd 0xc2 (0x4124008133c0) to dev "naa.600***************" failed on path "vmhba0:C0:T1:L11" H:0x1 D:0x0 P:0x0 Possible sense data: 0x0 0x0
0x0.
2012-09-29T01:11:49.750Z cpu10:4106)WARNING: NMP: nmpCompleteRetryForPath:378:Logical device "naa.600***************": awaiting fast path state update before retrying failed command again...
2012-09-29T01:11:49.789Z cpu0:4254)ScsiDeviceIO: 2316: Cmd(0x412400812ac0) 0xfe, CmdSN 0x28 from world 11234 to dev "naa.600c***************" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2012-09-29T01:11:49.789Z cpu0:4254)WARNING: NMP: nmp_DeviceStartLoop:721:NMP Device "naa.600***************" is blocked. Not starting I/O from device.
2012-09-29T01:11:49.789Z cpu4:11234)WARNING: HBX: 1968: Failed to initialize VMFS distributed locking on volume 4fa316dd-***************: Timeout
2012-09-29T01:11:49.789Z cpu4:11234)Vol3: 2359: Failed to get object 28 type 1 uuid 4fa316dd-*************** FD 0 gen 0 :Timeout
2012-09-29T01:11:49.789Z cpu4:11234)WARNING: Fil3: 2492: Failed to reserve volume f530 28 1 4fa316dd *************** 0 0 0 0 0 0 0
2012-09-29T01:11:49.789Z cpu4:11234)WARNING: NMP: nmp_DeviceStartLoop:721:NMP Device "naa.600***************" is blocked. Not starting I/O from device.
2012-09-29T01:11:50.750Z cpu0:13406)WARNING: NMP: nmpDeviceAttemptFailover:562:Retry world restore device "naa.600c***************" - no more commands to retry
2012-09-29T01:11:50.788Z cpu10:4106)<4>hpsa 0000:09:00.0: Device:C1:B0:T1:L11 Command:0xc2 Command Invalid.
2012-09-29T01:11:50.788Z cpu10:4106)WARNING: LinScsi: SCSILinuxProcessCompletions:772:Error BytesXferred > Requested Length Marking transfer length as 0 - vmhba = vmhba0, Driver Name = hpsa, Requested length = 1024, Resid = 3012
2012-09-29T01:11:50.788Z cpu10:4106)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0xc2 (0x412400728580, 11234) to dev "naa.600***************" on path "vmhba0:C0:T1:L11" Failed: H:0x1 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
Act:FAILOVER

I can see that the mpaths are both as active for this LUN, but it fails to mount the datastore. We did not encounter the same issue when on Version 5.0U1. The datastore is connecting to a P2000 G3 SAS Firmware T240... Does anyone have any feed back on this issue ?

[spravtek]

Just a stab in the dark here ...

Maybe it's the multipath policy? Was it changed ?

VMware recommends following:

http://partnerweb.vmware.com/comp_guide2/detail.php?deviceCategory=san&productid=17132&vcl=true

Did you also check the VAAI plugin (if you are using this).

[markbusu]

Seems like the issue was not related to the upgrade to 5.1 at all, after a downgrade to 5.0U1 the problem still remained..

VAAI was not installed and after we performaned the following KB the issue was resovled:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=200685...

I will install the VAAI plugin and re enable hardware locking, which should hopefully keep it right!

Thanks for the feedback!

[spravtek]

Interesting, thanks for getting back on this.

[markbusu]

Have done some further testing and it seems the issue did actually occur because the vaai plugin was installed and enabled... VAAI detected that the P2000 G3 did have Hackdware locking support (which technically it should) however it did not seem to function correctly... Once we disabled this, everything was fine and dandy

We would like to enable this feature, so we are investigating Smart Array Firmware upgrade for the hosts... The P2000 is running on the latest Firmware version. Will let you know if we manage to succesfully enable hardware locking.

thanks!