Hello,
We put 13 identical VMware 5.5 hosts into production a few month ago. (IBM X3550 M4 with one Qlogic onboard Mez FCoE card and one PCIe Qlogic card) The fabrics is VDX swiches connected to an IBM V7000 with FCoE. 10 of the servers is running without any issues, but 3 of them is randomly dropping frames on the onboard FCoE cards. (PCIe is working perfectly) IBM already replaced the onboard FCoE cards several times. Tried creating a VMware support ticket however their answer was "it's a FC connectivity/Swtich problem". We got nothing from debugging the switches (no issues) Anyone seen this before?
vmhba5 = onboard FCoE card.
th "vmhba5:C0:T1:L7" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:41:30.903Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec3a1de00, 34047) to dev "naa.6005076802810f8f8800000000000007" on path "vmhba5:C0:T1:L7" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:41:31.070Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec3a1de00, 34047) to dev "naa.6005076802810f8f8800000000000007" on path "vmhba5:C0:T1:L7" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:41:31.237Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec3a1de00, 34047) to dev "naa.6005076802810f8f8800000000000007" on path "vmhba5:C0:T1:L7" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:41:31.253Z cpu21:33555)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6005076802810f8f8800000000000007" state in doubt; requested fast path state update...
2014-05-20T17:41:31.403Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec3a1de00, 34047) to dev "naa.6005076802810f8f8800000000000007" on path "vmhba5:C0:T1:L7" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:41:31.570Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec3a1de00, 34047) to dev "naa.6005076802810f8f8800000000000007" on path "vmhba5:C0:T1:L7" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:41:31.737Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec3a1de00, 34047) to dev "naa.6005076802810f8f8800000000000007" on path "vmhba5:C0:T1:L7" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:41:31.903Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec3a1de00, 34047) to dev "naa.6005076802810f8f8800000000000007" on path "vmhba5:C0:T1:L7" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:41:32.070Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec3a1de00, 34047) to dev "naa.6005076802810f8f8800000000000007" on path "vmhba5:C0:T1:L7" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:41:32.237Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec3a1de00, 34047) to dev "naa.6005076802810f8f8800000000000007" on path "vmhba5:C0:T1:L7" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:41:32.403Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec3a1de00, 34047) to dev "naa.6005076802810f8f8800000000000007" on path "vmhba5:C0:T1:L7" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:41:32.570Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec3a1de00, 34047) to dev "naa.6005076802810f8f8800000000000007" on path "vmhba5:C0:T1:L7" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:41:32.602Z cpu0:34047)WARNING: Fil3: 15068: Found invalid object on 52de99df-52c00ff3-b219-000e1e0e26e8 <FD c0 r0> expected <FD c0 r0>
2014-05-20T17:41:32.602Z cpu0:34047)Vol3: 2612: Failed to get object 28 type 2 uuid 52de99df-52c00ff3-b219-000e1e0e26e8 FD 4 gen 1 :Not found
2014-05-20T17:41:32.608Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec3a99d40, 34047) to dev "naa.6005076802810f8f8800000000000005" on path "vmhba5:C0:T1:L5" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:41:32.608Z cpu21:33555)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6005076802810f8f8800000000000005" state in doubt; requested fast path state update...
2014-05-20T17:41:32.780Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec3a99d40, 34047) to dev "naa.6005076802810f8f8800000000000005" on path "vmhba5:C0:T1:L5" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:41:32.952Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec3a99d40, 34047) to dev "naa.6005076802810f8f8800000000000005" on path "vmhba5:C0:T1:L5" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:41:33.120Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec3a99d40, 34047) to dev "naa.6005076802810f8f8800000000000005" on path "vmhba5:C0:T1:L5" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:41:33.253Z cpu21:33555)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6005076802810f8f8800000000000005" state in doubt; requested fast path state update...
2014-05-20T17:41:33.287Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec3a99d40, 34047) to dev "naa.6005076802810f8f8800000000000005" on path "vmhba5:C0:T1:L5" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:41:33.453Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec3a99d40, 34047) to dev "naa.6005076802810f8f8800000000000005" on path "vmhba5:C0:T1:L5" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:41:33.620Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec3a99d40, 34047) to dev "naa.6005076802810f8f8800000000000005" on path "vmhba5:C0:T1:L5" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:41:33.752Z cpu0:34047)WARNING: Fil3: 15068: Found invalid object on 52de99a4-cad373aa-17b7-000e1e0e26e8 <FD c0 r0> expected <FD c0 r0>
2014-05-20T17:41:33.752Z cpu0:34047)Vol3: 2612: Failed to get object 28 type 2 uuid 52de99a4-cad373aa-17b7-000e1e0e26e8 FD 4 gen 1 :Not found
2014-05-20T17:41:33.789Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec5020f80, 34047) to dev "naa.6005076802810f8f8800000000000005" on path "vmhba5:C0:T1:L5" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:41:33.961Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec5020f80, 34047) to dev "naa.6005076802810f8f8800000000000005" on path "vmhba5:C0:T1:L5" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:41:34.132Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec5020f80, 34047) to dev "naa.6005076802810f8f8800000000000005" on path "vmhba5:C0:T1:L5" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:41:34.303Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec5020f80, 34047) to dev "naa.6005076802810f8f8800000000000005" on path "vmhba5:C0:T1:L5" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:41:34.475Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec5020f80, 34047) to dev "naa.6005076802810f8f8800000000000005" on path "vmhba5:C0:T1:L5" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:41:34.647Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec5020f80, 34047) to dev "naa.6005076802810f8f8800000000000005" on path "vmhba5:C0:T1:L5" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:41:34.819Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec5020f80, 34047) to dev "naa.6005076802810f8f8800000000000005" on path "vmhba5:C0:T1:L5" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:41:34.952Z cpu0:34047)WARNING: Fil3: 15068: Found invalid object on 52de99a4-cad373aa-17b7-000e1e0e26e8 <FD c0 r0> expected <FD c0 r0>
2014-05-20T17:41:34.952Z cpu0:34047)Vol3: 2612: Failed to get object 28 type 2 uuid 52de99a4-cad373aa-17b7-000e1e0e26e8 FD 4 gen 1 :Not found
2014-05-20T17:41:34.955Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec3a7ad40, 34047) to dev "naa.6005076802810f8f8800000000000000" on path "vmhba5:C0:T3:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:41:34.955Z cpu21:33555)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6005076802810f8f8800000000000000" state in doubt; requested fast path state update...
2014-05-20T17:41:35.127Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec3a7ad40, 34047) to dev "naa.6005076802810f8f8800000000000000" on path "vmhba5:C0:T3:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:41:35.264Z cpu21:33555)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6005076802810f8f8800000000000000" state in doubt; requested fast path state update...
2014-05-20T17:41:35.299Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec3a7ad40, 34047) to dev "naa.6005076802810f8f8800000000000000" on path "vmhba5:C0:T3:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:41:35.470Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec3a7ad40, 34047) to dev "naa.6005076802810f8f8800000000000000" on path "vmhba5:C0:T3:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:41:35.636Z cpu0:34047)WARNING: Fil3: 15068: Found invalid object on 52cc103a-6f86766f-6164-000e1e0e2728 <FD c0 r0> expected <FD c0 r0>
2014-05-20T17:41:35.636Z cpu0:34047)Vol3: 2612: Failed to get object 28 type 2 uuid 52cc103a-6f86766f-6164-000e1e0e2728 FD 4 gen 1 :Not found
2014-05-20T17:41:35.639Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec0410a40, 34047) to dev "naa.6005076802810f8f8800000000000000" on path "vmhba5:C0:T3:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:41:35.706Z cpu6:34047)WARNING: Fil3: 15068: Found invalid object on 52cc103a-6f86766f-6164-000e1e0e2728 <FD c0 r0> expected <FD c0 r0>
2014-05-20T17:41:35.706Z cpu6:34047)Vol3: 2612: Failed to get object 28 type 2 uuid 52cc103a-6f86766f-6164-000e1e0e2728 FD 4 gen 1 :Not found
2014-05-20T17:41:40.025Z cpu4:35574)World: 14296: VC opID SWI-9648208 maps to vmkernel opID 40bdb4a
2014-05-20T17:41:41.267Z cpu18:32999)ScsiPath: 3620: Command 0xa3 (cmdSN 0x0, World 0) to path vmhba5:C0:T0:L7 timed out: expiry time occurs 16ms in the past
2014-05-20T17:41:41.267Z cpu18:32999)VMW_SATP_ALUA: satp_alua_issueCommandOnPath:651: Path "vmhba5:C0:T0:L7" (UP) command 0xa3 failed with status Timeout. H:0x3 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2014-05-20T17:41:43.266Z cpu0:33000)ScsiPath: 3620: Command 0xa3 (cmdSN 0x0, World 0) to path vmhba5:C0:T0:L5 timed out: expiry time occurs 15ms in the past
2014-05-20T17:41:43.266Z cpu0:33000)VMW_SATP_ALUA: satp_alua_issueCommandOnPath:651: Path "vmhba5:C0:T0:L5" (UP) command 0xa3 failed with status Timeout. H:0x3 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2014-05-20T17:41:44.251Z cpu21:33663)WARNING: iodm: vmk_IodmEvent:193: vmhba5: FRAME DROP event has been observed 400 times in the last one minute. This suggests a problem with Fibre Channel link/switch!.
2014-05-20T17:41:45.264Z cpu7:33003)ScsiPath: 3620: Command 0xa3 (cmdSN 0x0, World 0) to path vmhba5:C0:T0:L0 timed out: expiry time occurs 13ms in the past
2014-05-20T17:41:45.264Z cpu7:33003)VMW_SATP_ALUA: satp_alua_issueCommandOnPath:651: Path "vmhba5:C0:T0:L0" (UP) command 0xa3 failed with status Timeout. H:0x3 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2014-05-20T17:41:54.259Z cpu13:33004)ScsiPath: 3620: Command 0xa3 (cmdSN 0x0, World 0) to path vmhba5:C0:T0:L5 timed out: expiry time occurs 8ms in the past
2014-05-20T17:41:54.259Z cpu13:33004)VMW_SATP_ALUA: satp_alua_issueCommandOnPath:651: Path "vmhba5:C0:T0:L5" (UP) command 0xa3 failed with status Timeout. H:0x3 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2014-05-20T17:41:55.275Z cpu7:33003)ScsiPath: 3620: Command 0xa3 (cmdSN 0x0, World 0) to path vmhba5:C0:T3:L0 timed out: expiry time occurs 11ms in the past
2014-05-20T17:41:55.275Z cpu7:33003)VMW_SATP_ALUA: satp_alua_issueCommandOnPath:651: Path "vmhba5:C0:T3:L0" (UP) command 0xa3 failed with status Timeout. H:0x3 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2014-05-20T17:41:56.253Z cpu21:33555)WARNING: iodm: vmk_IodmEvent:193: vmhba5: FRAME DROP event has been observed 400 times in the last one minute. This suggests a problem with Fibre Channel link/switch!.
2014-05-20T17:42:00.023Z cpu6:34121)World: 14296: VC opID hostd-9eca maps to vmkernel opID db86b446
2014-05-20T17:42:08.160Z cpu21:33592)WARNING: iodm: vmk_IodmEvent:193: vmhba5: FRAME DROP event has been observed 400 times in the last one minute. This suggests a problem with Fibre Channel link/switch!.
2014-05-20T17:42:08.160Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec4f6dac0, 34047) to dev "naa.6005076802810f8f880000000000000b" on path "vmhba5:C0:T1:L1" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:42:08.160Z cpu21:33555)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6005076802810f8f880000000000000b" state in doubt; requested fast path state update...
2014-05-20T17:42:08.160Z cpu21:33555)ScsiDeviceIO: 2324: Cmd(0x412ec4f6dac0) 0x28, CmdSN 0x52a2 from world 34047 to dev "naa.6005076802810f8f880000000000000b" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2014-05-20T17:42:08.262Z cpu21:33555)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6005076802810f8f880000000000000b" state in doubt; requested fast path state update...
2014-05-20T17:42:08.330Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec4f6dac0, 34047) to dev "naa.6005076802810f8f880000000000000b" on path "vmhba5:C0:T1:L1" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:42:08.502Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec4f6dac0, 34047) to dev "naa.6005076802810f8f880000000000000b" on path "vmhba5:C0:T1:L1" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:42:08.569Z cpu6:34047)WARNING: Fil3: 15068: Found invalid object on 52fb6fcc-15803d7d-0616-000e1e0e2728 <FD c0 r0> expected <FD c0 r0>
2014-05-20T17:42:08.569Z cpu6:34047)Vol3: 2612: Failed to get object 28 type 2 uuid 52fb6fcc-15803d7d-0616-000e1e0e2728 FD 4 gen 1 :Not found
2014-05-20T17:42:08.572Z cpu21:33555)ScsiDeviceIO: 2324: Cmd(0x412ec5941480) 0x28, CmdSN 0x52a4 from world 34047 to dev "naa.6005076802810f8f880000000000000b" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2014-05-20T17:42:08.675Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec5941480, 34047) to dev "naa.6005076802810f8f880000000000000b" on path "vmhba5:C0:T1:L1" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:42:08.762Z cpu6:34047)WARNING: Fil3: 15068: Found invalid object on 52fb6fcc-15803d7d-0616-000e1e0e2728 <FD c0 r0> expected <FD c0 r0>
2014-05-20T17:42:08.762Z cpu6:34047)Vol3: 2612: Failed to get object 28 type 2 uuid 52fb6fcc-15803d7d-0616-000e1e0e2728 FD 4 gen 1 :Not found
2014-05-20T17:42:08.772Z cpu21:33555)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x28 (0x412ec3842c80, 34047) to dev "naa.6005076802810f8f8800000000000007" on path "vmhba5:C0:T1:L7" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2014-05-20T17:42:08.772Z cpu21:33555)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6005076802810f8f8800000000000007" state in doubt; requested fast path state update...
2014-05-20T17:42:08.772Z cpu21:33555)ScsiDeviceIO: 2324: Cmd(0x412ec3842c80) 0x28, CmdSN 0x5723 from world 34047 to dev "naa.6005076802810f8f8800000000000007" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.