VMware Cloud Community
djb801
Contributor
Contributor

ESXi 6.7 Performance graphs only show the last few minutes

This is my second install of ESXi 6.7 U1 and I'm seeing the same problem. The performance monitoring graph on the host and on individual VMs only shows the last few minutes, no matter how long it has been up. This is the case with all counters (CPU/Network/Memory/Disk). Screenshots attached. Running the free license. I have tried restarting hostd, running with/without NTP, and, obviously, a full re-install. Any suggestions?

33 Replies
MattGagliardi
Enthusiast
Enthusiast

Same problem here, I'm only seeing about 2-3 minute of perf. data in the "Realtime" charts in one host of a two host deployment (Enterprise Plus license), the other host is A-OK.  Currently running the HPE-customized install at build 13473784.  Doesn't matter if I look in vCenter or log in directly to the host at "/ui".  FWIW historical data/rollups seem to be perfectly fine...for example looking at Last Day in vCenter does show me the last day with no gaps.  FWIW I also checked the size of the actual in-mem stats collection file (out at /var/lib/vmware/hostd/stats) and it's clear (both hosts) based on timestamps that the file is being updated...also the file sizes on both hosts are roughly the same size.

Based on above (Last Day working fine, file updates taking place, approx. same amount of data in both files) I'd guess this is a display issue, not a data collection issue.

Reply
0 Kudos
Gravis14
Contributor
Contributor

In my case, the performance stats are only lasting about 5 minutes even when viewed from ESXi directly.

If I refresh the screen, the stats collection graphs clear themselves of all but about 1 minute of that 5 minutes of data.

VCSA 6.7.0 10244857


Running on single CPU AMD EPYC. Anyone else running AMD's? I have used this VCSA for other systems (Intel-based) and have not had this issue. Seems host-specific.

Reply
0 Kudos
DanielSvensson
Contributor
Contributor

Hi

My performance history started working again after i fixed my scratch location. I created a new folder and pointet ScratchConfig.ConfiguredScratchLocation to that.

VMware Knowledge Base

I hope it helps.

manfriday
Enthusiast
Enthusiast

We are also having this issue.

6.7U2 (build 131644319) installed on HPE DL580's (gen 9 and 10)

I tried setting a persistent scratch location but that did not resolve the issue for me.

I have been working with VMWare support on this, but they seem as baffled as we are.

They want me to rebuild a host from scratch, which I guess I will need to do one of these days... 😕

Reply
0 Kudos
vmwarewubing
Contributor
Contributor

We are having same issue.

HPE customized ESXi ISO 6.7U1, was not having issue on HPE customized ESXi ISO 6.0U3.'

Lenovo hosts using Lenovo customized ISO had no issues.

Reply
0 Kudos
Gravis14
Contributor
Contributor

I also created a new scratch location and rebooted... however this isn't helping and my performance data only has about 5-10 minutes of history still.

Reply
0 Kudos
manfriday
Enthusiast
Enthusiast

I rebuilt a host from scratch, using the HPE image, and applied all patches.

It seemed to be working fine. Left it in Maintenance mode for a couple days and saw no gaps in the graphs.

Took it out of maintenance mode and as soon as a few VMs migrated over, the gaps re-appeared.

Funny thing is that once the gaps start appearing, removing VMs from the host and putting it in maintenance mode does not alleviate the issue.

Reply
0 Kudos
Gravis14
Contributor
Contributor

I've upgraded to U2, and I'm still having the issue.

One thing I found interesting, is that performance counters at the cluster level will show a history beyond the 5-10 minutes BUT there are clear gaps between the datapoints. It's like the data isn't contiguous, and the host-level charts aren't able to skip over any gaps in the data.

Tobinski
Contributor
Contributor

Same problem here. The garphs are only visable in a few minute periode!

6.7.0 Update 2 (Build 13006603)

Vmware please help us!

Reply
0 Kudos
DanielSvensson
Contributor
Contributor

Hi I, don’t know if it is relevant or not. I didn’t think it was initially. In addition to my original post. Besides moving the scratch location (to a local SSD). I also configured all the settings under Swap, to Yes, and an SSD datastore (I needed to set them in order to successfully do manual updates). And I changed Time & date settings to manual and disabled NTP service. After all the changes performance history seems to be saved for the entire last hour. I tested to enable NTP today again and medially the performance data was erased for me. I could not see that my host current time was noticeable changed by the NTP service. So it was very close to correct already.

If any of the settings is relevant to the problem I feel it is a bug. For example small time corrections by NTP should not cause such problems.

I’m on a standalone, white box, host running ESXI 6.7 U2 B13644139. I also have a LAB host, IBM server, at work with the same problem but I have not had time to try to fix the performance logs on that yet.

I hope it helps.

Raj1988
Enthusiast
Enthusiast

Well ESXi's would have realtime performance and can be seen for an hour only.

ESXi 6.7 had issues with the performance charts failing and this will be fixed in a future version. ETA is August/Sept this year.

Regards,

AJ

JamPie
Contributor
Contributor

So they know what it is and why it happens ....

any workaround or manual fix for it?

Reply
0 Kudos
LuisCastroS
Contributor
Contributor

I had the same issue. I moved the scratch location... did not work.

Then, after reading DanielSvenssonpost, I stopped the NTP client... did not work again. I disabled the NTP client, rebooted the host, and now it worked. I have 1 hour of logs in my free Hypervisor.

NTP client should not interfere with performance logging, but I guess this is a bug. Hope VMware will fix this soon.

Reply
0 Kudos
kalmadarin
Contributor
Contributor

vmware 6.7 update3 - the bug is still there Smiley Sad

Reply
0 Kudos