100% CPU-Load with Get-Cluster

[lenzker]

Hallo,

Probably someone here in the community can help me or has observed similar symptoms.

A short description of the setup. We have 2 vCenter in place. On a third machine PowerCLI is installed. This third machine (let's call it automation center) is doing a lot of non-interactive tasks against the two vCenter.

We have now observed that in 1 out of 100 times strange symptoms are occurring.

the automation center is connected to both vCenter.

during a > get-cluster 'CLUSTERNAME'

the powershell process seems to hang (no cluster object is returned), the CPU load of the powershell process is reaching and remaining at 100% (until the process is killed manually). 

Has anyone observed similar things as well? I have never observed it while working interactive on the 2 vCenter.

Since it is hard to reproduce the issue (and I am not aware of fancy Powershell Troubleshooting tools) I have troubles getting to the root cause of it. Has anyone here have an idea?

Thanks a lot and best regards,

[lenzker]

A few more information on the Version:

Authentication: Integrated AD.

Versions:

$PSVersion

Name                           Value

----                           -----

CLRVersion                     2.0.50727.5466

BuildVersion                   6.1.7601.17514

PSVersion                      2.0

WSManStackVersion              2.0

PSCompatibleVersions           {1.0, 2.0}

SerializationVersion           1.1.0.1

PSRemotingProtocolVersion      2.1

Get-VIToolkitVersion

PowerCLI Version

----------------

   VMware vSphere PowerCLI 5.1 Release 2 build 1012425

---------------

Snapin Versions

---------------

   VMware vSphere PowerCLI Component 5.1 build 1012428

[LucD]

I haven't seen anything similar, but I notice you are running somewhat older versions of PowerShell and PowerCLI.

Is there a specific reason for these specific versions ?

[lenzker]

Hey LucD,

thanks for the answer. It is indeed an outdated version of Powershell/PowerCLI.

Our plan was indeed to upgrade the Powershell/PowerCLI version. I don't want to upgrade Powershell/PowerCLI without any concrete testing plan in case. And since we haven't had this problem until we attached the second vCenter, we did not care that much about the version (since it was working fine and productive).

I will need to rewrite some code if we update from Powershell 2 to 4 and PowerCLI 5.1 to 5.8.

I know you can't give an answer here. But do you have any experiences on issues after updating the Powershell and PowerCLI version? I don't believe that everything in PS4 will be compatible, but this is just a wild guess...

[LucD]

I didn't encounter any noteworthy issues when I upgraded from PSv2 to PSv3 or PSv4.

Remember, you always start PowerShell v3 or v4 with the -Version 2 option.

Are the 2 vCenters set up in linked-mode ?

Perhaps that feature could cause the high CPU usaeg you seem to have.

[lenzker]

nope..we have are 2 independent vCenters. Just a quick question. Is the inventory service used by PowerCLI (e.g. get-cluster) or is it only used by the Webclient? I haven't found a concrete information on this so far. I always though it's for web-client usage only...

Thanks for the tip with -Version 2.

I need to go into the dev-environment and plan the update... I will let you know here about the progress... But to be honest i don't believe any changes will be allowed until next year 😉

[LucD]

Afaik, the Inventory Service is used as a cache for all client requests.

In What’s New in VMware vSphere 5.1–VMware vCenter Server it is stated "..., vCenter Inventory Service reduces direct client requests to the vCenter server with query caching, reducing the load on core vCenter Server processes"