Gateway : increased ping latency depending on the ESXi version from the web interface (not from SSH)

[gauthi3r]

Hello everyone,

I am currently experiencing a somewhat strange problem that I cannot fully understand.

the context :

• ESXi host: Supermicro SYS-E300-8D
• Firewall Virtual Appliance: PfSense 2.4.4-P3
• Tested with E1000E and VMXNET3 for network interface same issue
• Supermicro SYS-E300-8D BIOS and Firmware are up to date
• Open-VM-Tools for FreeBSD up to date

The problem:

I regularly update my ESXi host by following vmware updates. Since I updated 14141615 I have a problem with the latency of my FTTH gateway.

The latest version of ESXi with which I have no problem is version 13981272.

Following various tests, I determined that the problem was most likely software and related to the way pfsense displays gateway monitoring.my normal ping latency is between 6.6 to 7ms.when i have

The problem, my latency ping is very unstable (between 6.6ms to 18ms!)

Here are two screenshots:

The first with last esxi build 151xxxxxx the problem between latency from webui and latency from ssh ping :

https://i.imgur.com/TgUHs9A.png

The second with the esxi build 13981272 with no difference between latency from webui and from ssh ping :

https://i.imgur.com/AIbMguK.jpg

Others things :

• My 2 cisco switches are up to date
• About the screens my firewall ip is : 172.16.2.1 and DNS is firewall.house.lan
• The bug appear only with my Supermicro SYS-E300-8D with build 14141615 minimum.
• No matter which build of ESXi is used haven’t the bug with my second ESXi (Dell hardware)

On my Supermicro SYS-E300-8D with ESXi build at least 14141615 :

• I have the same bug on fresh installation of PfSense 2.4.4-llation of OPNSense (fork of PfSense – maybe the both firewall use the same code for gateway monitoring).

• I have no other ideas to solve my problem except stay on build 13981272.

• I don’t know if VMware has change any driver with next build of 13981272, with the screen I’m sure isn’t a pure networking problem or network driver problem and maybe a pfsense bug ?

• I want to share my experience with you to solve my problem as soon as possible, hoping that people can help me.  I also created topics on netgate forums and in the pfsense category of reddit.

Thanks you for your help !

[Teddy092]

Hi gauthi3r​,

This is indeed a weird issue.

I really don't see here what it could be, except a network driver issue with your Supermicro ESXi server.

You might need to go back to a previous driver and why not check on Supermicro's forum if that issue is known.

You could try also to use the native ESXi build from VMware just to see how behaves your host.

Otherwise, I really know what else you could do.

Good luck

[gauthi3r · Accepted solution]

Hi Teddy092​

Thank you for taking the time to answer me, after having cold re-analyzed the results, I started looking for a patch for my different network cards (Intel I210 & Intel I350)

and I found this one: https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=io&productid=19798

However, despite the driver upgrade, the problem was still present.

so I decided to completely remove the VIBs associated with the Intel IGBN driver :

https://i.imgur.com/YhQSqvY.jpg

And surprise ... I have no problem. It has been working for several hours now with very stable latency.

I'm not sure what to think about it, but for the moment, I have the latest version of ESXi without IGBN driver and no more problems.

The result should be confirmed within days but I think everything is ok. If necessary for the next updates, I will re-delete the driver if necessary.

[Teddy092]

This is good news!

Happy to help

[gauthi3r]

I bring a little more information on my problem.

my problem was related to the native driver, by deleting the driver so I switched to the vmlinux driver which is not affected by the problem.

i think i use now the igb driver here : https://my.vmware.com/fr/group/vmware/details?downloadGroup=DT-ESXI60-INTEL-IGB-533&productId=491

Screen with driver network difference : https://i.imgur.com/0t01Etj.png

i don't know if the problem come from intel developers or vmware integration.

Anyone know how create a bug ticket ?