VMware Cloud Community
MartinSvec
Enthusiast
Enthusiast

ESXi 5 silently re-enables DelayedAck?

Hello,

I have a problem with DelayedAck option on ESXi 5 hosts in conjunction with an iSCSI storage. We use EqualLogic PS6010 arrays over a 10GE SAN network together with an ESXi software iSCSI initiator. To get a reasonable SAN performance without terrible latencies, it is necessary to set DelayedAck=0 in iSCSI advanced options. On ESXi 4.1, once I set the option everything works fine. But on ESXi 5 it seems to me that the kernel sometimes "forget" that DelayedAck is set to zero and the SAN performance returns to iops/latency values as if DelayedAck would be set to 1, although vSphere Client and CLI still show that DelayedAck=0.

After a number of desperate tests, I guess that it is in some way related to a reconfiguration and/or a reconnection of the iSCSI adapter or a kernel port. I usually trigger the problem when I apply a host profile to a host or when I make iSCSI setup changes on the host. But at least one time I hit the bug when ESXi temporarily lost all connections to SAN switches due to network topology changes. (Of course, the host profile has DelayedAck set to zero and the other changes also didn't touch this option.)

To get the original performance, it usually suffices to reboot the host. But sometimes I also have to make a change of the DelayedAck option like turn it on and off, followed by a reboot.

I tried to set DelayedAck=0 globally in iSCSI storage adapter options, on a per-dynamic-target basis and also per-static-target basis, but there is no difference between them, the problem occurs despite the level on which DelayedAck=0 is set.

The problem is clearly in ESXi hosts and not in SAN or storage, because when I migrate a running VM with iometer to an unaffected host, the performance immediately increases after host switchover. (Hosts are equally installed blade servers with identical SAN setup and connections to blade switches.) I'm also quite sure that the described performance drop is related to the delayed ack feature, because the measured iometer performance statistics perfectly correspond to the ones measured when DelayedAck is intentionally turned on.

Does anybody have the same problem or is there something that I do wrong?

Best regards.

Martin

Reply
0 Kudos
19 Replies
PinkishPanther
Contributor
Contributor

Hi Martin,

We also have an environment with a Dell EqualLogic 6010E in a vSphere 5 setup.  We have have had significant issues with VM Applications running with high IO load causing LUN disconnects.  Also, when doing a storage vMotion to or from the EqualLogic to local disks it fails about 50% of the time, when it doesn't fail there are errors similar to the following in the vmkernel log:

2011-11-29T13:33:09.760Z cpu5:42841)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x4124408056c0) to dev "naa.6090a0b8004fee9cefeaa406e849f21e" on path "vmhba36:C1:T2:L0"

2011-11-29T13:33:09.760Z cpu5:42841)ScsiDeviceIO: 2305: Cmd(0x4124408056c0) 0x2a, CmdSN 0x204eba to dev "naa.6090a0b8004fee9cefeaa406e849f21e" failed H:0x0 D:0x2 P:0x0 Val

2011-11-29T13:33:47.536Z cpu10:2717)WARNING: iscsi_vmk: iscsivmk_TaskMgmtAbortCommands: vmhba36:CH:1 T:2 L:0 : Abort task response indicates task with itt=0x214821 has bee

2011-11-29T13:33:47.536Z cpu15:2210)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x4124410b6a40) to dev "naa.6090a0b8004fee9cefeaa406e849f21e" on path "vmhba36:C1:T2:L0"

2011-11-29T13:33:47.536Z cpu15:2210)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.6090a0b8004fee9cefeaa406e849f21e" state in doubt; requested fast pat

2011-11-29T13:33:47.536Z cpu15:2210)ScsiDeviceIO: 2316: Cmd(0x4124410b6a40) 0x2a, CmdSN 0x204eb9 to dev "naa.6090a0b8004fee9cefeaa406e849f21e" failed H:0x5 D:0x0 P:0x0 Pos

2011-11-29T13:33:47.536Z cpu13:1311204)WARNING: J3: 2644: Error committing txn callerID: 0xc1d00006 to slot 0: Timeout

2011-11-29T13:33:47.536Z cpu13:1311204)Fil3: 13341: Max timeout retries exceeded for caller Fil3_FileIO (status 'Timeout')

In addition, we have just confirmed that the delayed ack setting also reversed itself (this was an early recommendation by VMWare to help with the issue described above).  Have you discovered a method to ensure it's off?  It is very difficult for us to tell based on performance since we have these other issues.

Did you have anything similar to what we're seeing?  If so, did you find a fix?  What NICs are you using?  We have Intel 520's.

Thanks in advance - I'll post if we find anything else out related to delayed ack.

Reply
0 Kudos
MartinSvec
Enthusiast
Enthusiast

Hi,

We don't see issues like LUN disconnects or vmkernel errors similar to yours. But our environment is still not in production so all results are obtained only by syntetic performance tests.

Regarding the DelayedAck and high latencies, I've found no exact way to fix it. In fact, I even wasn't able to find exact steps that would reproduce the problem and prove that it's really related to the delayed ack feature. Right now, I have three Dell PE M710 blades with ESXi 5 and everything seems to be o.k. -- I can no longer trigger the bug, god knows why :-(( But there was a number of changes in our SAN, including firmware upgrade of all EqualLogic boxes to v5.1.2, firmware upgrade of all PC8024 SAN switches, changes in flow control settings etc. I'm not sure if one of these upgrades fixed the problem or the problem is still there and only temporarily disappeared.

Our blades have two Broadcom 57711 10GE NICs, with bnx2x device firmware 6.4.5 and bnx2x driver 2.0.15g.v50.11-5vmw. I disabled everything that was possible in Broadcom BIOS and also unloaded bnx2i iSCSI offload module from vmkernel. DelayedAck is disabled at dynamic discovery level.

I'll let you know if we find something new.

Martin

Reply
0 Kudos
PinkishPanther
Contributor
Contributor

You mentioned flow control changes, are you able to indicate what you changed?  The 8024F (which is what we have) has flow control enabled by default and I have not yet found specific commands to customize it by port or switch.

This may be relevant?

http://monolight.cc/2011/08/flow-control-flaw-in-broadcom-bcm5709-nics-and-bcm56xxx-switches/

The symptom listed in this link could be similar to what we are seeing, we see disconnects at time of very high traffic usage (>300MB/sec usually with a sequential read situation - backups and storage vMotion).

Reply
0 Kudos
MartinSvec
Enthusiast
Enthusiast

There were inconsistencies in reported and observed flow control settings between top-of-rack 8024F switches, blade M8024 switches and ESXi hosts. Sorry I'm not sure of details now, but I definitely recommend you to upgrade 8024F firmware to the latest available version. In October, Dell released 4.1.1.9 firmware and according to the release notes, it should fix the flow control bug described in http://monolight.cc/2011/08/flow-control-flaw-in-broadcom-bcm5709-nics-and-bcm56xxx-switches/.

(Be careful, upgrade from 2.x/3.x to 4.x firmware is a two-step process. Read upgrade instructions or you will recover your switches with a serial cable :-))

Martin

Reply
0 Kudos
PinkishPanther
Contributor
Contributor

It's not that issue then, I upgraded the switchs to 4.1.1.9 before we put them in service.


I'll keep looking, this has been on-going with Dell, EqualLogic and VMWare since Nov 10th, need a resolution soon since we've got our EqualLogic just sitting around doing nothing while we wait to determine the root cause!

Thanks for the responses!

Reply
0 Kudos
PinkishPanther
Contributor
Contributor

In case others run into this really odd problem thought I would post what we've found to be the most probably root cause.

We are using CommVault on a Dell DL2200, which runs windows.  We had our EqualLogic LUNs attached and logged in to the EQL to allow SAN level backups, yesterday we discovered that we could log them out and revert a LUN with the nmp_ThrottleLogForDevice events in the vmkernel log back to stable.

Towards the original post for this thread, we have proven through EQL logs, that delayed ack is not getting disabled properly following the published directions.  We believe this has been escalated to VM engineers and will be corrected with a patch - a side benefit to our complex problem.  (complex in that the pattern was very, very difficult to find).

Thanks!

Reply
0 Kudos
ScottCirrus9
Contributor
Contributor

Hi Everybody.  I work with PinkishPanther and would like to share some news with you.  We have been pushing this issue very hard and have learned today that the vmware engineers have been able to confirm that this is an actual issue with vsphere 5. We have been told that they can reproduce the issue by disabling delayed ack both globally via the iscsi initiator and via individual connections.   Disabling delayed ack and rebooting the server appears to only partially implement the change.   You can use esxcfg-info  filtering for delayed  and you will see that delayed ack is still on.(true).  The issue has been esclated and as soon as we get an update, we will post it here for you.This issue affects all customers who need to turn off delayed ack as recommended by their storage vendor.

Reply
0 Kudos
a_p_
Leadership
Leadership

Discussion moved from VMware ESXi™ 4 to VMware ESXi 5

Reply
0 Kudos
anujtyagi
Contributor
Contributor

Hi Martin,

I am running a setup very similar to yours.

1) Dell PS6010 SAN (Latest Firmware 5.1.2)

2) Dell Rackmount servers R710 with ESXi5.0 Build  latest release 515841

3) 2 * Broadcom 57711 10GB adaptors (with latest drivers)

4) Dell 8024 Power Connect switches with Latest firmware (4.2.0.4)

5) Using Softare iSCSI with Dell PSP plugin.

However for some reason at weird intervals high latency figures can be observerd(with or without Jumbo frames). I haven't configured delayed ACK yet. I understand there is a KB article  suggesting the same. Would you be able to confirm if there is any other reference/best practice document which also recommends  "disable delayedACK" advanced settings.  Bit concerned over changing advanced settings feature Smiley Happy as previously i never had to set it.

Thanks,


Anuj

Reply
0 Kudos
PinkishPanther
Contributor
Contributor

Annujtyagi,

Your setup is actually identical to ours except we have Intel 520-2's where you have Broadcom.

We upgraded the PS6010 to 5.2.0 on the weekend, because we had a drive failure that ended up showing a stuck slot which was only fixed by upgrading.  All looks good so far.

Delayed Ack - we have a ticket still open with VMWare, with ESXi 5 it seems that disabling it shows it disabled but according to EqualLogic tier2 it remains enabled.  This is still in progress.

To answer your concern however, we have received a definitive answer from EqualLogic - delayed ack is recommended to be off.  I've not seen this in writing in spite of asking a few times but I have it in emails and verbal.

Hope that helps, long story short - wait until the results of our 7 week old ticket come in.

John

Reply
0 Kudos
MartinSvec
Enthusiast
Enthusiast

Hi Anuj,

I'm not aware of any EqualLogic guide that recommends to set DelayedAck=0, which is surprising to me (without this setting, PS6010 was nearly unusable when stressed by large sequential reads and writes). However, EqualLogic is mentioned in kb.vmware.com/kb/1002598 and there is a number of other discussions about SAN latencies resolved by DelayedAck. As PinkishPanther said, users with latency issues usually receive this recommendation directly from EqualLogic support. Just try it and you will see, you can always turn it on again (don't forget to reboot the host). But if you're worried about it, simply ask EqualLogic support.

Martin

Reply
0 Kudos
anujtyagi
Contributor
Contributor

Thanks PinkishPanther/Martin for sharing.

Quick note : Equallogics are running 5.2.0 (incorrectly typed 5.1.2)

I also have an case opened with Equallogic support. In the meantime I will try with disabling "delayedACK" one of the hosts on the cluster with test/non-critical vms on it to see the behaviour.

@PinkishPanther : let me know how your ticket goes through.

Will post results soon..

Anuj

Reply
0 Kudos
ScottCirrus9
Contributor
Contributor

Anuj

Pinkish_Panther and I are on i think our 3rd escalation with this issue and have finally gotten someone to ask for usable information.  I have a vmware tcpdump command that will capture iscsi traffic that vmware will look at to determine if tcpdump is really on or not.  you would need to run it while in a dir that you can write lots of data do though, and i changed into one of the local datastores to do it while ssh'ed into the server.

tcpdump-uw -i <vmk iscsi interface> -X -n -ttt not port ssh and not arp –w filename.ext

where vmk iscsi interface is the vmk#   of one of your vmkernel ports used for iscsi.

We did this yesterday and sent the information off for review. I will let you know what comes of it.

Reply
0 Kudos
anujtyagi
Contributor
Contributor

Hi Scott/PinkishPanther/Martin,

Thanks for your inputs. Much appreciated. Disabling DelayedACK has massively reduced the Latency figures. I haven't been able to look at the concerned vmware environment since last month in detail. But logged in remotely to see the latency readings and they are back to acceptable, no more 90000 millisecond spikes :smileygrin:

Would be great if "disabling delayedACK" is given a mention in VMware/Equallogic Best practice document by DELL.

Rgds,
Anuj

Reply
0 Kudos
dgrace
Enthusiast
Enthusiast

We may be seeing this same issue so I am currently going through my hosts turning off delayed ack. We are running 4.1 but the plan was to do the V5 upgrade next week. I've been waiting on the next set of 4.1 patches for a VMWare Tool bug fix. The upgrade to 5 was to bypass it. Argh.

Thanks for keeping us up to date on the V5 status!

Reply
0 Kudos
Lax617
Contributor
Contributor

If you can figure out how to disable delayed ACK from the CLI you might be able to add the command to rc.local so it is persistant across reboots.

Reply
0 Kudos
s_buerger
Contributor
Contributor

esxcli iscsi adapter param set --adapter=vmhba<n> --key=DelayedAck --value=false

I checked my 2 hosts again and only one had it enabled again (the g7 with sdcard).

On the old dl380g5 with install on hdd it was still disabled.

Both already updated to u1.

Reply
0 Kudos
w_estermann
Contributor
Contributor

Hello All

Am hoping for some advice please:

Current Setup:

- 5 * Dell R710 servers

- 1 * dual port 10GB Broadcom 57711 NIC in each server

- 2 * EqualLogic PS6510E (firmware 5.2.2) configured in a single group with all disks running RAID 10.

- ESXi 5.0 Update 1 - 623860

- each nic is bound to a iscis port group as per forum suggestions

- Am using Dell MEM driver for advanced multipathing and load balancing

Have beein using VM Labs IO analyzer to compare results, test being 'Max_Throughput.icf'

Based on everyones experience, what is preferred options currently being used with best performance:

- hardware or software ISCSI?

- Jumbo frams off or on?

- Delayed Ack enabled or disabled?

- storage IO control enabled or disabled?

- what range of IOPS and MBps would you expect to see?

My tests are being very random and not 100% sure how to understand the results. Currently I am seeing:

- software iSCSI on, jumbo frames on, delayed Ack on = 952 IOPS / 472 MBReads/S

- software iSCSI on, jumbo frames off, delayed Ack on = 1169 IOPS / 579 MBReads/S

- software iSCSI on, jumbo frames on, delayed Ack off = 1505 IOPS / 747 MBReads/S

- software iSCSI on, jumbo frames off, delayed Ack off = 181 IOPS / 90 MBReads/S

Is my third test the best results I could expect?

How does that compare with the rest of the community?

Is there any other settings I should be using / changing to improve my results.

Worry is that these results are based on only 1 test machine being run. When I run all 5 test machines (1 on each host) the overall performance drops VERY low.

Thanks in advance for any help / replies

Warren Estermann

Reply
0 Kudos
mgerolami
Contributor
Contributor

I am having this problem currently and I was wondering if anyone found a solution to the problem? 

Reply
0 Kudos