VMware Cloud Community
franckehret
Enthusiast
Enthusiast

vRealize Operations 8.6 causing VM CPU peaks

Hi there,

This evening, I wanted to update vRealize Operations to v8.6 in my lab. Update itself went fine and I didn't had to worry about.

But later, I noticed that cluster DRS score seemed pretty low compared to standard and I started to check my VMs CPU usage.

I found out one of my DC peaking every 5 minutes but trying to figure out why, I noticed all Windows VMs were impacted.

franckehret_0-1635797533916.png

The process involved was MS Defender but then I asked myself "what did change?", answer: vrOps!

I stopped the VM to confirm it and when it was confirmed, I rolled back the VM to 8.5 (snapshot - always) and everything was back to normal...

Did anybody experienced the same issue? Thanks for your feedback! 😉

Labels (3)
Reply
0 Kudos
14 Replies
KabirAli82
Expert
Expert

I've no experiance with vROps 8.6 (yet). VMware Tools is becoming more and more important for vROps. I could be 8.6 is trying to use VMware Tools a bit more and thus the spike.

 

How is your setup? Which VMware Tools version and Windows version do you have?


Was I helpful? Give a kudo for appreciation!
Braindumping @ http://kablog.nl/
Tweeting @ https://twitter.com/_Kabir_Ali_
Reply
0 Kudos
franckehret
Enthusiast
Enthusiast

Hi,

Most of my VMs run Windows 2019 core (fully patched), some normal version with GUI. I have also a couple of 2016 with GUI and I'm starting Windows 2022 migration slowly but surely! 😉

VMtools is probably the latest available (it seems 11.3.5 build 18557794 when we speak).

I do auto-update on all of them so they are at least updated once a month when Windows Update will trigger the reboot.

Reply
0 Kudos
antt1
Contributor
Contributor

Hi!

Did this problem ever solve? I see a similar problem in our environment. Also version 8.6

Reply
0 Kudos
sxnxr
Commander
Commander

I am running vrops 8.6.2 in production and have 2 separate vrops instances pulling from the same vcenters and i am not seeing this behavior in my environment 

Reply
0 Kudos
franckehret
Enthusiast
Enthusiast

Hi,

I did the update to 8.6.2 last night and started experiencing the same behavior again straight away! 

I might stage a new one from 8.6.2 but it is REALLY annoying...

But this time I noticed it was all Windows VMs, which I didn't pay attention in the first place. Linuxes are fine.

Reply
0 Kudos
franckehret
Enthusiast
Enthusiast

Some graphs:

Single Windows VM right after vROps rollback (big peaks disapear):

franckehret_0-1641386769238.png

The same VM during last day (CPU usage starts EXACTLY when vROps comes back online after update):

franckehret_1-1641386822713.png

And I confirm : all Windows VMs present exactly the same pattern.

Reply
0 Kudos
sxnxr
Commander
Commander

This must be specific to you as i am not seeing any of this. All i can sugest is look at the windows guest before and after and see what process is using the extra cpu cycles that may point you in a direction. Other than that a cal to support may be needed

 

on a side note terminology is key.

ESXi physical server = host

windows VM = windows VM or windows Guest

Linux VM = Linux VM or Linux Guest

the only time you can have a windows host is if you are running Hyper-V and lets be honest why would you do that. In VMware host should not be used when referring to a VM.

Saying host when you mean VM can be confusing

Reply
0 Kudos
franckehret
Enthusiast
Enthusiast

Hi,

I corrected the terminology... 😉

The process is wmiprvse.exe but I didn't find anything following this guide : https://docs.microsoft.com/en-us/troubleshoot/windows-server/system-management-components/high-cpu-u...

But on each VM, I have the following events in WMI-Activity:

Id = {00000000-0000-0000-0000-000000000000}; ClientMachine = SERVERNAME; User = NT AUTHORITY\SYSTEM; ClientProcessId = XXXXX; Component = Unknown; Operation = Start IWbemServices::ExecQuery - ROOT\CIMV2 : SELECT ExecutablePath FROM Win32_Process WHERE processid="XXXXX"; ResultCode = 0x80041017; PossibleCause = Unknown

They didn't completely disappeared after the rollback but I had a lot with 8.6.X running. I couldn't correlate process ID with something relevant for now (svchost or inetpub)

Reply
0 Kudos
Salmari
Contributor
Contributor

Have same issue with vROps 8.6.2 when Credential-Less Service Discovery enabled. Disabling feature solved issue, but would love to know if there is way to make discovery behave better.

lukaslang
Enthusiast
Enthusiast

Sorry for unburying this thread, but we had the exact same issue in our DEV environment. Disabling the application discovery also solved this for us. I have not seen any specific mention in the release notes from 8.6.3 but maybe someone has already tested it? Analyzing the WMI Logs of the windows machines, it seems that the WMI query is faulty and causes the CPU spikes every 5 minutes.

Reply
0 Kudos
franckehret
Enthusiast
Enthusiast

Hi,

 

I have to admit, I didn't retry the migration since then, had some priorities in the meantime, but this is interesting! 

Thanks for sharing 😉

Reply
0 Kudos
rm_bk
Enthusiast
Enthusiast

8.6.2 here. Same issue! Searching for a fix...

Reply
0 Kudos
jhanekom
Virtuoso
Virtuoso

Seeing same here. What AV are you running, if any? McAfee here, yet to look deeper but at a casual glance that was a high consumer in the one or two VMs we checked before stopping Application Discovery.

Reply
0 Kudos
franckehret
Enthusiast
Enthusiast

Hello,

I'm running now MS AV but I run McAfee in the past. I don't have the feeling it made a difference because I tried over the time.

But since latest version 8.10.1, I don't see this behaviour anymore. You should try this one.

KR

Franck

Reply
0 Kudos