VMware Cloud Community
Fabse91
Contributor
Contributor

Extreme Latency Spikes in ESXi 6.5, but not in ESXi 5.5

Hello everbody,

we have two different vSphere environments - both are running on Cisco UCS and Hitachi HDS Storage.

One environment has new hosts with the Cisco ESXi 6.5 U1 ISO, VSCA 6.5.

The second environment is running with ESXi 5.5 U3 Hosts, VSCA 6.5.

Now we have many latency alerts from the first environment with read latency up to 7 minutes. But there is no impact within the guest os or other VMs running on this host or datastore.

We see this latencies in veeam and in the vcenter.

Reply
0 Kudos
11 Replies
MartinGustafsso
VMware Employee
VMware Employee

Firmware/driver combination supported for ESXi 6.5?

Reply
0 Kudos
jlo4realz
Contributor
Contributor

I just saw you post about the extreme latency spikes and I am seeing the same issue. I have multiple clients running VMware 6.5, UCS Mini, and IBM Storage. I have tried both the latest versions of Cisco's custom ESXi and the standard VMware ESXi. We are seeing spikes of 10+ years however guest machines are showing no issues. I've worked with Cisco, IBM, and VMware and they all are reporting that the issue is with the UI. I still have an open ticket with VMware but they just keep bouncing me between their storage group and their application team. There most resent response was don't worry about it because its not real. Use esxtop to see what going on. So frustrating!!!

Reply
0 Kudos
mcclainje
Contributor
Contributor

We are also seeing this issue in 6.5.  We have had VMware support look at our hosts and they could not find anything that would explain those values. 

Reply
0 Kudos
dr_uk
Contributor
Contributor

Seeing the same issue with ESXi 6.5 build 7967591

Cisco C220 M4 with firmware 3.0(3f)

Fnic driver 1.6.0.34

Reply
0 Kudos
sjesse
Leadership
Leadership

Look at the IO of the virtual machine, is there no IO, but a large latency spike? If so it is a bug

VMware Knowledge Base

I've seen latency metrics up to 1 year, which obviously aren't correct. I have no confidence in vmwares metrics.

Reply
0 Kudos
sjesse
Leadership
Leadership

Hmm, maybe not, it does say it's fixed in 6.5.x.

Reply
0 Kudos
dr_uk
Contributor
Contributor

VMware support case opened - they say engineering is aware and will be fixed in 6.5 U2

Reply
0 Kudos
ashvetsov
Enthusiast
Enthusiast

I have the same peaks at different hosts (6.0, 6.5 with actiual patches) with different storages.

All peaks are due creating-removing snapshots for backups.

Vmware support can't resolve that issue, they told that it should be storages problem. 10 years latency... they must be kidding.

Reply
0 Kudos
mayanh8
Contributor
Contributor

Seeing the same thing. Apparently one of our VM's has been attempting to access data on our SAN since before Jesus was born.

We're about to upgrade to 6.7 so I'm hoping it goes away then.

Reply
0 Kudos
tman24
Enthusiast
Enthusiast

Noticing this on a new ESXi 6.7 cluster connected to a flash array too using multipathed 1Gb iSCSI, but the latency spike only shows on the guest, and not on the host if you monitor the same timeframe. The guests don't appear to have a problem, but the UI certainly shows read and write latency stretching into many years!

Reply
0 Kudos
Raj1988
Enthusiast
Enthusiast

VMware Knowledge Base

Issue would be fixed in 6.5 U3 and 6.7 U2 versions respectively. Subscribe to the KB . It would be updated when the fix is available for public.

Regards,

AJ

Reply
0 Kudos