VMware vSphere

Hi All.

I hope that this has been posted in the correct forum section, I would like
to find out if anyone else is seeing the same thing within each of your own
environment.

On Thursday 22nd Feb 2018 we noticed extremely high CPU usage within our 1 vmWare
cluster which consist of 3 physical hosts. After further investigation we
noticed the same in our other clusters.

When signing into the cluster we could not account for the high utilization.
Each individual virtual servers usage did not account for the high utilization,
something else was causing this.

We then signed into each host directly and what we found was rather
disturbing. A virtual server on that host that our team did not provision or
had any idea about. When we connect to this virtual server we noted ubuntu OS
and this virtual server had 16GB RAM and 32vCPU assigned to it. The CPU’s where
operating at peaking 100%. After further investigation each host within our
organization had 1 unknown virtual server on it. All these virtual servers had
16GB RAM and 32vCPU’s running at at 100%. These virtual servers we’re somehow
hidden from the main cluster. These servers all had the word LAB in it's name.

Has anyone else picked this up?

Sorry forgot to mention that we're running vmware esxi 6.0.

What versions are your hosts/vcenter running? I've noticed high CPU usage from a host but it was on an isolated LAN

We're running esxi 6.0

Are you sure none in your institution created lab vms?

I wonder if there was some logs, task/events for this systems so you can trace from where they was created.

Hello cve_ZA

Yes, it is possible to prevent VMs from being visible from a vCenter/vSphere level:

http://www.virten.net/2015/10/how-to-hide-a-virtual-machine/

@klabiak

"I wonder if there was some logs, task/events for this systems so you can trace from where they was created."

Yes there *should* be connection logs from users and for such tasks but whether they are still present depends on log retention and when this was initiated.

If no-one that had/has access to your infrastructure that would have created such VMs then the naming of 'LAB' is potentially part of the ruse - evaluate your current security configuration for vulnerabilities and remediate any holes such as compromised users.

Bob

Hi All.

I can confirm that this was NOT created by anyone in our organization, also
note that the company who does have access to our environment that monitors our
vmWare infrastructure for us has noted that what we have experienced has in
fact being experienced by about 5 other clients of theirs all
within South Africa.