vRops 7.0 - Higher resource usage / Parts of the GUI are slow / Communication Error: Timeout reached while trying to communicate with the server.

[carvaled]

hey everyone!!

I have just upgraded my home lab (4vCPU/ 16GB mem) from 6.7 to 7.0 went smooth no issues with both OS and APP pak's, before the upgrade it used to chug along just fine, very snappy, etc... but now the thing hardly functions... I also noticed that it's CPU usage increased significantly... from about 20% average to 70-80% average! (trying to get some stats to upload soon)

the lab is a small environment, 4 hosts, 25 VM's... NSX, vCenter, vSAN vRA, vRB, vRO & vRLI solutions running just like before the upgrade.

So for example when I try to load All Metrics, I get this error after a few minutes:

Communication Error: Timeout reached while trying to communicate with the server.

Try refreshing the UI manually.

has anyone else noticed this since upgrading?? glad i didn't upload any of my customers yet!

Cheers

vMan

[RickVerstegen]

Did you also perform a reboot after the upgrade of vROps?

[carvaled]

yep i did, 2 actually!

[daphnissov]

Were you smart and snapshot your vROps beforehand?

[carvaled]

lol yes.... but..... after it all worked and looked OK i deleted the snapshots as I am low on storage! (facepalm!)

[daphnissov]

Dang, well I've been through this rodeo many times, and this is why--even in my lab--I don't upgrade to GA releases until a few weeks have passed. :smileycry:

Installing anew and letting run side-by-side is one way to get around it (depending on resources). I know it's not helpful to you now, but...

[carvaled]

Before and after the upgrade, let you guess when the upgrade happened! (charts are large so you will need to click on them to see the full timeline)

CPU / MEM %

Read and Write Disk IOPS

Net usage KBPS

DISK Usage KBPS

[carvaled]

So after checking TOP on the vRops server it appears the increased CPU usage was from the vRLI agent on the vRops box (4.7.0-9983377)... after stopping liagentd the usage dropped to about ~30% ... I wonder if its because i was still using the vRops 6.7 Content Pack / config on vRops 7.

Happy that the CPU has dropped to a more reasonable level... but unfortunately I still have the timeout error in the vRops GUI... I am able to pull metric data out with the API  tho...

[carvaled · Accepted solution]

So the great folks over at VMware found my issue.

It was log related which explains why vRLI agent went nuts on CPU... even tho I had all the logs in Analytics set to "Warning" It still has "Root Logger level" set to DEBUG. (facepalm)

So it means vRops 7.0 Analytics "Debug" is generating even more logs than 6.7 did or some setting has changed which is now a bottle neck... Anyway after changing "Root Logger level" to Warning the errors went away and CPU dropped to about 10% on average.

What I can say 100% is this is not a setting I changed before the upgrade to vRops 7, I might have changed it months and months ago while troubleshooting my Custom Rest Plugin on vRops 6.6! even before I upgraded to 6.7... what was even more frustrating what that I couldn't find the timeout error anywhere in all the logs it was collecting LOL...

Anyway no point crying over spilt milk.... In case anyone else runs into this issue check your log levels in vROPS!!

Cheers

vMan.

[siddiqui_au]

thank you vMan.