VMware Cloud Community
franvicente
Contributor
Contributor

ESXi 6.7 U1 realtime performance gaps/lost

Hello,

I'm having random problems with ESX performance charts. From the point of view of the ESXi server they appear as lost, and from the vCenter they show gaps on the daily report.

2019-01-08 15_08_56-Greenshot image editor - Copy.png

I've checked all the suggestions on VMware Knowledge Base KB: 1003878 but without success. The vpxa.log and hostd.log didn't show any error at the point where the performance chart was lost.

The vCenter cluster is composed by 3 Dell R730 servers, all o them with the same ESXi image and the same load, but the problem happens frequently on two of them (more than 10 times each day), and rarely on the third (once a day). It happens

Anybody knows how to fix it?

Kind regards,

Fran

13 Replies
sjesse
Leadership
Leadership

From the host view you only see so far in the past, I think its an hour, and from the host view you will only see statistics from when it moved there. That article is out of date, see if anything on, the following link helps. Its when metrics stopped all togther

VCSA 6.5 Management - Performance charts "no data to display"

Reply
0 Kudos
franvicente
Contributor
Contributor

Thanks, but the problem is not related to the VCSA Management performance charts, the appliance performance charts work, the health checks of the appliance also pass without problems.

The problem only occurs on the ESXi servers (it shows on the management interface of the ESXi which I shown on the screenshot, and also from the vCenter). As you can see the host interface shows 0 Memory usage before 15:03, the same happens for CPU/Disk/etc. and the host was powered on with a constant load, so it couldn't be 0Mb memory usage.

From the vCenter the gaps shows like this on "Last day report":

2019-01-15 16_52_03-vSphere - 192.168.16.1 - Overview.png

But on the "Realtime" report the graph shows like on the ESXi interface when the data gets lost.

Reply
0 Kudos
bjstick82
Contributor
Contributor

Did you get anywhere on this? I'm seeing the same thing across our 6.7U1 install base... multiple vCenters, different host hardware vendors, but all the same 6.7U1 version. VMware support has been less than stellar thus far when I've pressed them on there being a known issue.

Reply
0 Kudos
pj101
Contributor
Contributor

I've just upgraded from 6.5 to 6.7U2. And i have the same problem for 2 hosts of 4. Four completely identical servers (HPE ProLiant DL380 Gen9  with E5-2667v3 CPU)  in the same cluster, for two of them the charts are ok and for two others i have exactly the same gap pattern as you. I have completely reinstalled one of the nodes having this problem. It did not help. Even after a vanilla install of ESXi 6.7U2 without adding to cluster/dVS etc the problem persists. I will do a VMWare support ticket.

Everything worked fine in version 6.5 (all updates) on all 4 servers.

bjstick82
Contributor
Contributor

Upgrading to 6.7 Update 2 did not fix the issue for us. VMware also advised completely reinstalling from scratch as a troubleshooting step. We have a rather large environment and this would be a huge timesink.

Reply
0 Kudos
bmrkmr
Enthusiast
Enthusiast

same here on 8 of 240 (Dell) hosts on 6.7U1

please update this thread once you have a fix

Best regards

Bernhard

Reply
0 Kudos
BBittner
Enthusiast
Enthusiast

Same here...

Any news on that ticket?

Reply
0 Kudos
JimWelsh
Contributor
Contributor

Same here on 2 out of 4 hosts.  VCSA is Version 6.7.0.30000 Build 13010631, and hosts are all 6.7.0, Build 13006603.

+1 the "Any news on that ticket?" question.

Reply
0 Kudos
smart900
Contributor
Contributor

I have the same problem on HP Custom Image 6.7.0 Update 2 (Build 13006603) , and same on updated Build 13981272.

The ESXi 6.7 Update 3 |  AUG 20 2019 | ISO Build 14320388 solved the problem!

VMware ESXi 6.7 Update 3 Release Notes

PR 2295425: Performance charts of ESXi hosts might intermittently disappear from any web-based application that connects to the vCenter Server system

You might intermittently stop seeing performance charts of ESXi hosts in the embedded VMware Host Client or any web-based application that connects to the vCenter Server system, including the vSphere Client and vSphere Web Client. An internal issue causes hostd to delete available performance counters information for some ESXi hosts.

Reply
0 Kudos
BBittner
Enthusiast
Enthusiast

I can confirm this, after 6.7 U3 the problem is solved

Reply
0 Kudos
burge71
Contributor
Contributor

Been running update 3 for a while problem is there. running VMware ESXi, 6.7.0, 14320388

pastedImage_1.png

I don't see this across all vm's at the same time, maybe this is vm tools related. running latest version 10346, version makes no difference, see across several tools versions too.

Reply
0 Kudos
RAMint
Contributor
Contributor

We're having the same issue, same build (VMware ESXi, 6.7.0, 14320388).  If looking at Monitor/Performance/Overview of the cluster, when these gaps occur, we will be able to find a host (or more than one) where we are unable to view the Performance/Overview charts of that host - It will just say 'Loading...'.   When this occurs, we are unable to perform Vsphere operations on any VMs on the effected host(s).  e.g.  Snapshots wont start, Can't launch a remote console, etc... until the performance charts start working again.

Is anyone else seeing these symptoms (delay in issuing Vsphere operations) when these gaps occur?

Reply
0 Kudos
mviel
Enthusiast
Enthusiast

We're having the same issue, (VMware ESXi, 6.7.0, 15160138 )

If looking at Monitor/Performance/Overview of the cluster, when these gaps occur, we will be able to find a host (or more than one) where we are unable to view the Performance/Overview charts of that host - It will just say 'Loading...'.   When this occurs, we are unable to perform Vsphere operations on any VMs on the effected host(s).  e.g.  Snapshots wont start, Can't launch a remote console, etc... until the performance charts start working again.

I will open up a Call with VMWARE as for me that Problem is ongoing . ( Initialy the System was setup with DellEMC - ESXI -6.7U2 -13981272 -A03 )

Reply
0 Kudos