1 2 Previous Next 16 Replies Latest reply on Jan 4, 2011 2:50 PM by SomeJoe7777

    MD3000i Virtual Disk not on prefered path due to AVT/RDAC failover

    manfriday Hot Shot

       

      Hi,

       

       

      I am having some issues with my MD3000i failing over to an non-prefered path.

       

       

      The MD3000i throws the old "Virtual Disk not on prefered path due to AVT/RDAC failover" error several times a day.

       

       

      It happened once in a while with ESX 3.5, but is not happenging much more with version 4.

       

       

       

       

       

      I get the following in /var/log/messages:

       

       

      Jul 24 11:49:12 tdvserver1 vobd: Jul 24 11:49:12.926: 355597326227us: http://vprob.storage.redundancy.degraded Path redundancy to storage device naa.6001e4f000436ab80000067c48281b88 degraded. Path vmhba40:C2:T0:L2 is down. 2 remaining active paths. Affected datastores: "DataStore3 - MD3000i (1mb)".

      Jul 24 11:49:12 tdvserver1 vobd: Jul 24 11:49:12.934: 355597333792us: http://vprob.storage.redundancy.degraded Path redundancy to storage device naa.6001e4f000438fa20000068b482818af degraded. Path vmhba40:C2:T0:L3 is down. 2 remaining active paths. Affected datastores: "DataStore4 - MD3000i (8mb)".

      Jul 24 11:49:12 tdvserver1 vobd: Jul 24 11:49:12.942: 355597341892us: http://vprob.storage.redundancy.degraded Path redundancy to storage device naa.6001e4f000436ab8000017ae48d8991a degraded. Path vmhba40:C2:T0:L4 is down. 2 remaining active paths. Affected datastores: "DataStore1 - MD3000i (1mb)".

      Jul 24 11:49:12 tdvserver1 vobd: Jul 24 11:49:12.949: 355597348866us: http://vprob.storage.redundancy.degraded Path redundancy to storage device naa.6001e4f000438fa20000170448d8965d degraded. Path vmhba40:C2:T0:L5 is down. 2 remaining active paths. Affected datastores: "DataStore2 - MD3000i (1mb)".

      Jul 24 11:49:12 tdvserver1 vobd: Jul 24 11:49:12.995: 355597394915us: http://vprob.storage.redundancy.degraded Path redundancy to storage device naa.6001e4f000436ab800002d8149242090 degraded. Path vmhba40:C2:T0:L7 is down. 2 remaining active paths. Affected datastores: "VDIStore2 - MD3000i (1mb)".

      Jul 24 11:49:13 tdvserver1 vobd: Jul 24 11:49:13.004: 355597404112us: http://vprob.storage.redundancy.degraded Path redundancy to storage device naa.6001e4f000436ab800003d4c4a2ba941 degraded. Path vmhba40:C2:T0:L8 is down. 2 remaining active paths. Affected datastores: "VDIStore1 - MD3000i (2mb)".

      Jul 24 11:49:13 tdvserver1 vobd: Jul 24 11:49:13.013: 355597413405us: http://vprob.storage.redundancy.degraded Path redundancy to storage device naa.6001e4f000436ab800006a5e4a6474e4 degraded. Path vmhba40:C2:T0:L9 is down. 2 remaining active paths. Affected datastores: Unknown.

      Jul 24 11:49:13 tdvserver1 vobd: Jul 24 11:49:13.020: 355597420334us: http://vprob.storage.connectivity.lost Lost connectivity to storage device naa.6001e4f000436ab800006b684a65595d. Path vmhba40:C2:T0:L10 is down. Affected datastores: "DataStore5 - MD3000i (4mb)".

      Jul 24 11:49:17 tdvserver1 vobd: Jul 24 11:49:17.375: 355601775018us: http://vprob.vmfs.heartbeat.timedout 49248754-672e53f0-38b7-00151778736d VDIStore2 - MD3000i (1mb).

      Jul 24 11:49:22 tdvserver1 vobd: Jul 24 11:49:22.006: 355606406026us: http://vprob.vmfs.heartbeat.recovered 49248754-672e53f0-38b7-00151778736d VDIStore2 - MD3000i (1mb).

       

       

       

       

       

       

      And the following in /var/log/vmkernel:

       

       

      Jul 24 11:51:03 tdvserver1 vmkernel: 4:02:48:27.601 cpu7:4206)WARNING: NMP: nmp_DeviceAttemptFailover: Retry world failover device "naa.6001e4f000436ab800002d8149242090" - issuing command 0x41000812bc80

      Jul 24 11:51:03 tdvserver1 vmkernel: 4:02:48:27.601 cpu2:6691)NMP: nmp_CompleteRetryForPath: Retry world recovered device "naa.6001e4f000436ab800002d8149242090"

      Jul 24 11:51:06 tdvserver1 vmkernel: 4:02:48:30.549 cpu0:4107)NMP: nmp_CompleteCommandForPath: Command 0x12 (0x4100081c8300) to NMP device "mpx.vmhba32:C0:T0:L0" failed on physical path "vmhba32:C0:T0:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

      Jul 24 11:51:06 tdvserver1 vmkernel: 4:02:48:30.549 cpu0:4107)ScsiDeviceIO: 747: Command 0x12 to device "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

      Jul 24 11:51:06 tdvserver1 vmkernel: 4:02:48:30.693 cpu1:4097)NMP: nmp_CompleteCommandForPath: Command 0x12 (0x410008106500) to NMP device "mpx.vmhba33:C0:T0:L0" failed on physical path "vmhba33:C0:T0:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

      Jul 24 11:51:06 tdvserver1 vmkernel: 4:02:48:30.693 cpu1:4097)ScsiDeviceIO: 747: Command 0x12 to device "mpx.vmhba33:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

      Jul 24 11:51:13 tdvserver1 vmkernel: 4:02:48:38.059 cpu0:4107)NMP: nmp_CompleteCommandForPath: Command 0x12 (0x41000811f800) to NMP device "mpx.vmhba32:C0:T0:L0" failed on physical path "vmhba32:C0:T0:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

      Jul 24 11:51:13 tdvserver1 vmkernel: 4:02:48:38.059 cpu0:4107)ScsiDeviceIO: 747: Command 0x12 to device "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

      Jul 24 11:51:13 tdvserver1 vmkernel: 4:02:48:38.202 cpu1:4097)NMP: nmp_CompleteCommandForPath: Command 0x12 (0x4100081d2300) to NMP device "mpx.vmhba33:C0:T0:L0" failed on physical path "vmhba33:C0:T0:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

      Jul 24 11:51:13 tdvserver1 vmkernel: 4:02:48:38.202 cpu1:4097)ScsiDeviceIO: 747: Command 0x12 to device "mpx.vmhba33:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

       

       

       

       

       

      I have spoken with support folks from both Dell & Vmware.

       

       

      The VMWare support rep said it's a Dell issue.. that VMware is seeing a path problem and doing what it is supposed to do.

       

       

      The Dell guy says there is nothing wrong with the MD3000i and the issue is with VMWare.

       

       

      Neither of them seem interested in  helping me out any more.

       

       

       

       

       

      I have tried using MRU (the default) & Round-Robin pathing  policies and have configured iscsi on the esx hosts as per the following document:

       

       

      http://www.delltechcenter.com/page/VMwareESX4.0andPowerVault+MD3000i

       

       

       

       

       

      I can vmkping from all the hosts to all the IP's on the MD3000i with no problems.

       

       

       

       

       

      If anyone has any insight, I would be most appreciative.

       

       

       

       

       

      Thanks

       

       

       

       

       

      Jason

       

       

       

       

       

        1 2 Previous Next