VMware Cloud Community
AndreyKrushelni
Contributor
Contributor
‎09-19-2012 11:07 PM

ESXi 5.0 + P2000g3 iSCSI, ESXi periodically loses datastore.

I can not solve the problem, falls off storage from ESXi. =(((

Infrastructure:

1. iSCSI Storage: "P2000G3 iSCSI"

2. The array is connected to the Switch "HP A5820X-24XG-SFP+" on SFP+ direct attach cable.

3. Server (Supermicro) witch "ESXi 5.0 update 1" witch ethernet adapter "QLogic QLE3242 10Gb" dual port  is connected to the "HP A5820X-24XG-SFP+" on SFP+ direct attach cable.

The logs are messages ESXi:

2012-09-20T00:21:18.165Z cpu4:4100)ScsiDeviceIO: 1198: Device naa.600c0ff00011347d9b20f74f01000000 performance has improved. I/O latency reduced from 1222642 microseconds to 243409 microseconds.
2012-09-20T00:21:22.728Z cpu16:4112)ScsiDeviceIO: 1198: Device naa.600c0ff00011347d9b20f74f01000000 performance has improved. I/O latency reduced from 243409 microseconds to 55525 microseconds.
2012-09-20T00:22:10.827Z cpu0:4147)<6>qlcnic 0000:04:00.1: Freed vmnic5[1] irq.
2012-09-20T00:22:10.953Z cpu1:4147)<6>qlcnic 0000:04:00.0: Freed vmnic4[1] irq.
2012-09-20T00:40:06.493Z cpu19:6969)NetPort: 1427: disabled port 0x300000a
2012-09-20T00:40:15.756Z cpu3:4147)<6>qlcnic 0000:04:00.0: Feature 0x1 needed.
2012-09-20T00:40:16.152Z cpu11:4147)IDT: 991: 0x21 <vmnic4[1]> exclusive (entropy source), flags 0x10
2012-09-20T00:40:16.152Z cpu11:4147)VMK_VECTOR: 137: Added handler for  vector 33, flags 0x10
2012-09-20T00:40:16.152Z cpu11:4147)<6>qlcnic 0000:04:00.1: Feature 0x1 needed.
2012-09-20T00:40:16.507Z cpu1:4147)IDT: 991: 0x61 <vmnic5[1]> exclusive (entropy source), flags 0x10
2012-09-20T00:40:16.507Z cpu1:4147)VMK_VECTOR: 137: Added handler for  vector 97, flags 0x10
2012-09-20T00:40:31.827Z cpu20:7372)WARNING: ScsiDeviceIO: 1218: Device naa.600c0ff00011347d9b20f74f01000000 performance has deteriorated. I/O latency increased from average value of 28334 microseconds to 591034 microseconds.
2012-09-20T00:40:33.430Z cpu18:4114)ScsiDeviceIO: 1198: Device naa.600c0ff00011347d9b20f74f01000000 performance has improved. I/O latency reduced from 591034 microseconds to 114578 microseconds.
2012-09-20T00:40:34.508Z cpu7:4103)ScsiDeviceIO: 1198: Device naa.600c0ff00011347d9b20f74f01000000 performance has improved. I/O latency reduced from 114578 microseconds to 55214 microseconds.
2012-09-20T00:40:56.467Z cpu23:6970)NetPort: 2602: resuming traffic on DV port 404
2012-09-20T00:40:56.467Z cpu23:6970)NetPort: 1240: enabled port 0x300000a with mac 00:50:56:b2:29:73
2012-09-20T00:41:09.433Z cpu8:7162)WARNING: ScsiDeviceIO: 1218: Device naa.600c0ff00011347d9b20f74f01000000 performance has deteriorated. I/O latency increased from average value of 28343 microseconds to 582066 microseconds.
2012-09-20T00:41:10.912Z cpu21:4117)WARNING: ScsiDeviceIO: 1218: Device naa.600c0ff00011347d9b20f74f01000000 performance has deteriorated. I/O latency increased from average value of 28354 microseconds to 1179381 microseconds.
2012-09-20T00:41:16.469Z cpu9:4105)ScsiDeviceIO: 1198: Device naa.600c0ff00011347d9b20f74f01000000 performance has improved. I/O latency reduced from 1179381 microseconds to 231284 microseconds.
2012-09-20T00:41:17.827Z cpu23:4119)WARNING: ScsiDeviceIO: 1218: Device naa.600c0ff00011347d9b20f74f01000000 performance has deteriorated. I/O latency increased from average value of 28411 microseconds to 583574 microseconds.
2012-09-20T00:41:20.758Z cpu2:7161)ScsiDeviceIO: 1198: Device naa.600c0ff00011347d9b20f74f01000000 performance has improved. I/O latency reduced from 583574 microseconds to 115807 microseconds.
2012-09-20T00:42:15.818Z cpu0:4120)ScsiDeviceIO: 1198: Device naa.600c0ff00011347d9b20f74f01000000 performance has improved. I/O latency reduced from 115807 microseconds to 55092 microseconds.
2012-09-20T00:42:15.837Z cpu5:4147)<6>qlcnic 0000:04:00.0: Freed vmnic4[1] irq.
2012-09-20T00:42:16.190Z cpu3:4147)<6>qlcnic 0000:04:00.1: Freed vmnic5[1] irq.
2012-09-20T01:10:06.581Z cpu19:6967)NetPort: 1427: disabled port 0x300000a
2012-09-20T01:10:15.745Z cpu9:4147)<6>qlcnic 0000:04:00.0: Feature 0x1 needed.
2012-09-20T01:10:16.228Z cpu4:4147)IDT: 991: 0x21 <vmnic4[1]> exclusive (entropy source), flags 0x10
2012-09-20T01:10:16.228Z cpu4:4147)VMK_VECTOR: 137: Added handler for  vector 33, flags 0x10
2012-09-20T01:10:16.228Z cpu4:4147)<6>qlcnic 0000:04:00.1: Feature 0x1 needed.
2012-09-20T01:10:16.562Z cpu1:4147)IDT: 991: 0x61 <vmnic5[1]> exclusive (entropy source), flags 0x10
2012-09-20T01:10:16.562Z cpu1:4147)VMK_VECTOR: 137: Added handler for  vector 97, flags 0x10
2012-09-20T01:10:56.233Z cpu22:6969)NetPort: 2602: resuming traffic on DV port 404
2012-09-20T01:10:56.233Z cpu22:6969)NetPort: 1240: enabled port 0x300000a with mac 00:50:56:b2:29:73
2012-09-20T01:12:11.246Z cpu8:4147)<6>qlcnic 0000:04:00.1: Freed vmnic5[1] irq.
2012-09-20T01:12:11.410Z cpu6:4147)<6>qlcnic 0000:04:00.0: Freed vmnic4[1] irq.
2012-09-20T01:20:06.987Z cpu14:6970)NetPort: 1427: disabled port 0x300000a
2012-09-20T01:20:10.741Z cpu1:4147)<6>qlcnic 0000:04:00.0: Feature 0x1 needed.
2012-09-20T01:20:10.927Z cpu0:4147)IDT: 991: 0x21 <vmnic4[1]> exclusive (entropy source), flags 0x10
2012-09-20T01:20:10.927Z cpu0:4147)VMK_VECTOR: 137: Added handler for  vector 33, flags 0x10
2012-09-20T01:20:15.993Z cpu0:4147)<6>qlcnic 0000:04:00.0: Freed vmnic4[1] irq.
2012-09-20T01:20:23.884Z cpu21:5977)HBX: 2313: Waiting for timed out [HB state abcdef02 offset 3153920 gen 51 stampUS 65825493865 uuid 50596f1c-40094576-2894-000e1e059be0 jrnl <FB 5361600> drv 14.54] on vol 'P2000g3_R10'
2012-09-20T01:20:24.884Z cpu7:5970)HBX: 2313: Waiting for timed out [HB state abcdef02 offset 3153920 gen 51 stampUS 65825493865 uuid 50596f1c-40094576-2894-000e1e059be0 jrnl <FB 5361600> drv 14.54] on vol 'P2000g3_R10'
2012-09-20T01:20:26.885Z cpu12:5978)HBX: 2313: Waiting for timed out [HB state abcdef02 offset 3153920 gen 149 stampUS 65828494092 uuid 50596f1c-40094576-2894-000e1e059be0 jrnl <FB 1550000> drv 14.54] on vol 'P2000g3_R6'
2012-09-20T01:20:27.371Z cpu6:52361)ScsiCore: 63: Starting taskmgmt handler world 52361/2
2012-09-20T01:20:29.373Z cpu12:52363)ScsiCore: 63: Starting taskmgmt handler world 52363/3
2012-09-20T01:20:29.888Z cpu0:5979)HBX: 2313: Waiting for timed out [HB state abcdef02 offset 3153920 gen 149 stampUS 65828494092 uuid 50596f1c-40094576-2894-000e1e059be0 jrnl <FB 1550000> drv 14.54] on vol 'P2000g3_R6'
2012-09-20T01:20:31.395Z cpu10:4803)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba35:CH:1 T:3 L:0 : Task mgmt "Abort Task" with itt=0xc2490 (refITT=0xc2484) timed out.
2012-09-20T01:20:31.395Z cpu2:4290)ScsiCore: 97: Stopping taskMgmt handler world 42902
2012-09-20T01:20:35.986Z cpu0:4200)HBX: 2313: Waiting for timed out [HB state abcdef02 offset 3153920 gen 51 stampUS 65825493865 uuid 50596f1c-40094576-2894-000e1e059be0 jrnl <FB 5361600> drv 14.54] on vol 'P2000g3_R10'
2012-09-20T01:20:36.695Z cpu21:4803)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba35:CH:1 T:3 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
2012-09-20T01:20:36.695Z cpu21:4803)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d00000a TARGET: iqn.1986-03.com.hp:storage.p2000g3.112512ec9b TPGT: 1 TSIH: 0]
2012-09-20T01:20:36.695Z cpu21:4803)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 10.0.10.16:56488 R: 10.0.10.10:3260]
2012-09-20T01:20:36.695Z cpu21:52361)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x412440383f00, 4120) to dev "naa.600c0ff00011347d9b20f74f01000000" on path "vmhba35:C1:T3:L0" Failed: H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2012-09-20T01:20:36.695Z cpu0:4805)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x88 (0x41244129d8c0, 6578) to dev "naa.600c0ff00011347d9b20f74f01000000" on path "vmhba35:C1:T3:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2012-09-20T01:20:36.695Z cpu21:52361)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.600c0ff00011347d9b20f74f01000000" state in doubt; requested fast path state update...
2012-09-20T01:20:36.695Z cpu0:4805)ScsiDeviceIO: 2309: Cmd(0x41244129d8c0) 0x88, CmdSN 0x100 from world 6578 to dev "naa.600c0ff00011347d9b20f74f01000000" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2012-09-20T01:20:36.695Z cpu21:52361)ScsiDeviceIO: 2322: Cmd(0x412440383f00) 0x2a, CmdSN 0x93d21 from world 4120 to dev "naa.600c0ff00011347d9b20f74f01000000" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2012-09-20T01:20:36.695Z cpu0:4805)ScsiDeviceIO: 2309: Cmd(0x412441091b00) 0x88, CmdSN 0xd1 from world 6578 to dev "naa.600c0ff00011347d9b20f74f01000000" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2012-09-20T01:20:36.695Z cpu0:4805)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x88 (0x4124403dc7c0, 6578) to dev "naa.600c0ff00011347d9b20f74f01000000" on path "vmhba35:C1:T3:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2012-09-20T01:20:36.695Z cpu0:4805)ScsiDeviceIO: 2309: Cmd(0x4124403dc7c0) 0x88, CmdSN 0xd2 from world 6578 to dev "naa.600c0ff00011347d9b20f74f01000000" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2012-09-20T01:20:36.695Z cpu21:4803)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba35:CH:1 T:3 L:0 : Task mgmt "Abort Task" with itt=0xc2492 (refITT=0xc2482) timed out.
2012-09-20T01:20:36.695Z cpu0:4805)ScsiDeviceIO: 2309: Cmd(0x4124411e31c0) 0x88, CmdSN 0xfb from world 6578 to dev "naa.600c0ff00011347d9b20f74f01000000" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2012-09-20T01:20:36.695Z cpu0:4805)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x88 (0x4124403a9900, 6578) to dev "naa.600c0ff00011347d9b20f74f01000000" on path "vmhba35:C1:T3:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2012-09-20T01:20:36.695Z cpu0:4805)ScsiDeviceIO: 2309: Cmd(0x4124403a9900) 0x88, CmdSN 0xc4 from world 6578 to dev "naa.600c0ff00011347d9b20f74f01000000" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2012-09-20T01:20:36.695Z cpu0:4805)ScsiDeviceIO: 2309: Cmd(0x4124403c86c0) 0x88, CmdSN 0xcb from world 6578 to dev "naa.600c0ff00011347d9b20f74f01000000" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2012-09-20T01:20:36.695Z cpu0:4805)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x88 (0x412441918440, 6578) to dev "naa.600c0ff00011347d9b20f74f01000000" on path "vmhba35:C1:T3:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2012-09-20T01:20:36.695Z cpu0:4805)ScsiDeviceIO: 2309: Cmd(0x412441918440) 0x88, CmdSN 0xfc from world 6578 to dev "naa.600c0ff00011347d9b20f74f01000000" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2012-09-20T01:20:36.695Z cpu0:4805)ScsiDeviceIO: 2309: Cmd(0x4124415263c0) 0x8a, CmdSN 0xed from world 6771 to dev "naa.600c0ff00011347d9b20f74f01000000" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2012-09-20T01:20:36.695Z cpu0:4805)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x8a (0x41244191ed40, 6771) to dev "naa.600c0ff00011347d9b20f74f01000000" on path "vmhba35:C1:T3:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2012-09-20T01:20:36.695Z cpu0:4805)ScsiDeviceIO: 2309: Cmd(0x41244191ed40) 0x8a, CmdSN 0xcf from world 6771 to dev "naa.600c0ff00011347d9b20f74f01000000" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2012-09-20T01:20:36.695Z cpu23:4803)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba35:CH:5 T:3 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
2012-09-20T01:20:36.695Z cpu23:4803)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d00000e TARGET: iqn.1986-03.com.hp:storage.p2000g3.112512ec9b TPGT: 3 TSIH: 0]
2012-09-20T01:20:36.695Z cpu23:4803)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 10.0.10.16:56543 R: 10.0.10.11:3260]
2012-09-20T01:20:36.695Z cpu11:4805)ScsiDeviceIO: 2309: Cmd(0x4124403be5c0) 0x88, CmdSN 0xf7 from world 6578 to dev "naa.600c0ff00011347d9b20f74f01000000" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2012-09-20T01:20:36.695Z cpu11:4805)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x88 (0x412441095b00, 6578) to dev "naa.600c0ff00011347d9b20f74f01000000" on path "vmhba35:C5:T3:L0" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2012-09-20T01:20:36.695Z cpu11:4805)ScsiDeviceIO: 2309: Cmd(0x412441095b00) 0x88, CmdSN 0xd4 from world 6578 to dev "naa.600c0ff00011347d9b20f74f01000000" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

and on all the time:

2012-09-20T02:54:27.262Z cpu4:7367)NMP: nmp_PathDetermineFailure:2084: SCSI cmd RESERVE failed on path vmhba35:C7:T3:L1, reservation state on device naa.600c0ff000113366f57c095001000000 is unknown.
2012-09-20T02:54:27.275Z cpu4:7367)NMP: nmp_PathDetermineFailure:2084: SCSI cmd RESERVE failed on path vmhba35:C7:T3:L1, reservation state on device naa.600c0ff000113366f57c095001000000 is unknown.
2012-09-20T02:54:27.287Z cpu4:7367)NMP: nmp_PathDetermineFailure:2084: SCSI cmd RESERVE failed on path vmhba35:C7:T3:L1, reservation state on device naa.600c0ff000113366f57c095001000000 is unknown.
2012-09-20T02:54:27.300Z cpu4:7367)NMP: nmp_PathDetermineFailure:2084: SCSI cmd RESERVE failed on path vmhba35:C7:T3:L1, reservation state on device naa.600c0ff000113366f57c095001000000 is unknown.
2012-09-20T02:54:27.312Z cpu4:7367)NMP: nmp_PathDetermineFailure:2084: SCSI cmd RESERVE failed on path vmhba35:C7:T3:L1, reservation state on device naa.600c0ff000113366f57c095001000000 is unknown.
2012-09-20T02:54:27.325Z cpu4:7367)NMP: nmp_PathDetermineFailure:2084: SCSI cmd RESERVE failed on path vmhba35:C7:T3:L1, reservation state on device naa.600c0ff000113366f57c095001000000 is unknown.
2012-09-20T02:54:27.338Z cpu4:7367)NMP: nmp_PathDetermineFailure:2084: SCSI cmd RESERVE failed on path vmhba35:C7:T3:L1, reservation state on device naa.600c0ff000113366f57c095001000000 is unknown.
2012-09-20T02:54:27.350Z cpu4:7367)NMP: nmp_PathDetermineFailure:2084: SCSI cmd RESERVE failed on path vmhba35:C7:T3:L1, reservation state on device naa.600c0ff000113366f57c095001000000 is unknown.
2012-09-20T02:54:27.363Z cpu4:7367)NMP: nmp_PathDetermineFailure:2084: SCSI cmd RESERVE failed on path vmhba35:C7:T3:L1, reservation state on device naa.600c0ff000113366f57c095001000000 is unknown.
2012-09-20T02:54:27.376Z cpu4:7367)NMP: nmp_PathDetermineFailure:2084: SCSI cmd RESERVE failed on path vmhba35:C7:T3:L1, reservation state on device naa.600c0ff000113366f57c095001000000 is unknown.
2012-09-20T02:54:27.376Z cpu4:7367)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x16 (0x4124003d9540, 0) to dev "naa.600c0ff000113366f57c095001000000" on path "vmhba35:C7:T3:L1" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x2 0x3a 0x1. Act:EVAL

After a while he hangs on ESXi is dead, can only go to SSH.

If anyone faced a similar problem, prompt decision.

Sorry for my bad English.

Thank you!

0 Kudos
9 Replies
tlyczko
Enthusiast
Enthusiast
‎10-18-2012 05:27 AM

We have similar messages like this:

Any idea why??

Thank you, Tom

2012-10-18T10:40:20.667Z   cpu4:2052)WARNING: ScsiDeviceIO: 1218: Device   naa.600c0ff00013b5658f9b924f01000000 performance has deteriorated. I/O   latency increased from average value of 4066 microseconds to 88884   microseconds.

2012-10-18T00:38:19.009Z   cpu0:2056)WARNING: ScsiDeviceIO: 1218: Device   naa.600c0ff00013b565a99b924f01000000 performance has deteriorated. I/O   latency increased from average value of 2633 microseconds to 122357   microseconds.

2012-10-18T00:35:06.915Z   cpu4:3995017)WARNING: ScsiDeviceIO: 1218: Device   naa.600c0ff00013b565a99b924f01000000 performance has deteriorated. I/O   latency increased from average value of 2631 microseconds to 251749   microseconds.

0 Kudos
a_pflueger
Contributor
Contributor
‎07-29-2013 11:16 AM

Hallo Andrey,

we have the same environment and the same problem. Switch "HP A5820X-24XG-SFP+" on SFP+ direct attach cable and QLogic QLE3242.  (ESXi 5.1 with current patch level)

Every 30 seconds: link up, link down

How have you solved your problem?

Best regards

Andy

0 Kudos
AndreyKrushelni
Contributor
Contributor
‎07-29-2013 11:27 PM

Hello!

I solved this problem having switched off Jumba Frames in HP P2000G3.

It helped me. 😃

0 Kudos
MKguy
Virtuoso
Virtuoso
‎07-30-2013 01:08 AM

Is your NIC really actually losing the physical link?

Sounds like it might be a driver problem, try updating the qlcnic driver for your QLE3242 NIC:

http://www.vmware.com/resources/compatibility/detail.php?deviceCategory=io&productid=19404&deviceCat...

https://my.vmware.com/group/vmware/details?downloadGroup=DT-ESXI5X-QLOGIC-QLCNIC-51157&productId=285

-- http://alpacapowered.wordpress.com
0 Kudos
a_pflueger
Contributor
Contributor
‎07-30-2013 01:24 AM

we are using this driver version: 5.1.157

0 Kudos
AndreyKrushelni
Contributor
Contributor
‎07-30-2013 01:30 AM

try disabling Jumbo Frames (MTU=9000) and check.

0 Kudos
a_pflueger
Contributor
Contributor
‎07-30-2013 01:51 AM

I changed the MTU Size to 1500. But same issue. Link up, link down, link up (every 30 seconds).

We are using the firmware version 4.9.34 from QLE3242 (this is not the latest). We try an update from the firmware.

0 Kudos
a_pflueger
Contributor
Contributor
‎07-30-2013 05:29 AM

after updating the qle3242 firmware from 4.9.34 to 4.12.1 the problem is solved.

0 Kudos
cypherx
Hot Shot
Hot Shot
‎04-08-2014 08:22 AM

Were having the same issue but using NFS storage presented to vmware.  Up / down / up / down and its randomly.  Then you get into a random issue where both paths go down and all vms on it go grey and the datastore shows (inaccessable).

I thought maybe because we migrated everything from vSwitches to dvSwitches and there was a config issue, but nope, vmware support webex'd on and verified our networking was setup correctly.  The latest firmware is now 4.16.34.  our adapters came with 4.9.34.  Also we were using driver 5.0.727.  Tried updating to driver 5.1.178 and kept flapping even worse.  But once the firmware was updated, then the newer driver worked fine.

So far so good, but because this thing could run perfectly for 24 hours before showing a problem, I'm not marking it solved just yet.  I'm running iometer stress test to a datastore off this newer firmware/driver combination and seeing if the generated load remains stable.  I'm very thankful to find this thread and seeing that the firmware update resolved it for you, and I'm really hoping it resolves it for us as well.

We are on ESX 5.0 update 3 patch level 1489271 and yes we are also using jumbo frames with 9000 mtu.

0 Kudos