Virtual Machine Objects under Health Badges are showing unknown.

[Ephos]

Hi All,

Fairly new to vCOPs, running 5.8 build 1537842 licensed as Advanced .  I've been running it for about a month now and learning what I can in some spare time.  I have noticed that any of my health related badges a large subset of virtual machines are showing unknown.  I haven't found too much specifically related that pointed in me in the right direction so far.  I attached a screen shot as well.  I wasn't sure if this was an overall vCOPs issue or if it was related to the individual VM or possibly my configuration.  This doesn't affect the heat maps in the other badges just Health and its minor badges. 

Thanks!

[gradinka]

anything specific on those VMs?

if they are powered-off, and were powered-off even before vcops was installed, you might have that '?'

another aspect is the access permissions

[Ephos]

Having checked some of them a lot of the unknown machines were powered on when vcops was rolled out.  Almost all the affected machines are powered up currently as well.  In regards to permissions.  The service account that vcops uses to access vCenter has a role assigned with the permissions we had setup according the guide.  The account has the following permissions defined in its role. 

Global -> HealthGlobal -> vCenter Operation Manager User

Storage Views -> View

Virtual Machine -> Interaction -> Console Interaction

Virtual Machine -> Interaction -> Guest operating system management by VIX API

[gradinka]

do you have some other adapters, or plugins (like vin, vcm, etc)?

as to permissions, any chance those are defined for a specific datacenter/cluster, which does not include the "?" VMs?

[Ephos]

Thanks for the quick replies gradinka!  Yes we do have the vin and vcm adapters/plugins. As for the permissions almost all of affected virtual machines are within 1 virtual center (though most of the virtuals in the VC are working).  The permissions are defined at the virtual center object and propagated down.  All of the affected machines fall into different areas within VC however the vcops account should be inherited from the top for these, spot checking some of ~480 machines that are unknown they do have the permission inherited.  

[mark_j]

There's a possibility that the VMs that are "unknown" are duplicates or were created from VMs that are not removed. Some backup solutions cause a problem with this, as the placeholder VMs are collected and created a significant amount of irrelevant resources representative of "VMs".

How many VMs do you have in the your environment right now, powered on? Compare that to the quantity of VMs in the screenshot you showed me - but instead use the "members" tab in the vSphere UI to count them up. I'm thinking you may be seeing VMs that are either powered off OR no longer exist.

There also lies the possibility that hosts are not reporting stats for VMs (this did happen due to some bugs in certain versions). Find a VM that is showing nodata/?, then go vCenter for that VM. Check to see if you can get perf stats.

[Ephos]

I just checked, also specifically checked one of the machines.  Its unknown in vcops however it is both powered on and reporting performance stats in vCenter.  I've also validated VM tools on these machines to make sure they are up to date just to rule that out.  As far as powered on vs powered off, I can easily filter out the powered off VM's (which aren't too many) by hitting the buttons on the top of the environment tab.  If I filter out the powered off VM's then i get the same count in both places (1644).

[mark_j]

Have you restarted the vC Ops vApp? It looks like the badge just isn't calculating, even thought it is getting metrics (sparkline shows data).

[gradinka]

I'd suggest to:

1) stop the the VIN collection for 1-2 of the VMs, wait up to an hour, see if the issue is resolved

if nothing's changed, then

2) unregister the VCM adapter and see what happens

that may sound a bit drastic, so make snapshots etc.; but it will isolate the unknowns

[Ephos · Accepted solution]

Hi All - I just wanted to give you guys an update as we tried something that seemed to work.  Though I had originally spot checked the permissions within the hosts and clusters view I hadn't checked under the VMs and Templates view.  I know in the past for other permission issues we have seen where subsets of VMs in the VMs and Templates view would lose some of their permissions and I think this was the case here.

I re applied  the vcops permissions at the vCenter level in the VM's and templates view for the 1 vCenter where the affected VM's resided and as soon as the permission was re applied the unknown servers became known again.

Things seem to be back to normal now.  I didn't end up restarting the vApp or having to pull adapters. 

[mark_j]

Ah ha! Collector permissions strike again!

[gradinka]

Good news, everyone