Key commands take long time and max out CPU on Win2k8R2 64-bit

[marc045]

Hi All,

I am experiencing significant performance issues with PowerCLI around key commands like connnect-viserver and get-vm, these cmdlets pin the CPU at 100% for the entire duration of their first run.

Here is my setup:

Running VMware Workstation 7.1 on my desktop (Host OS = Windows 7 64-bit)
Desktop specs are 4 CPU Cores (Core i7 2Ghz)/16GB RAM.

INSIDE VMware Workstation, I am running a Win2k8 R2 64-bit VM with vCenter + Powershell. (configured with 1 vCPU/4GB RAM)
PowerCli 4.1.1
PowerShell 2 ($host.version -> Major=2,Minor=0,Build=-1,Revision=-1)

The PowerCLI shell launches relatively quickly, at around 6-7 seconds on the first run, and around 5 seconds on subsequent runs after the first run.

The issues are around the FIRST time I invoke a particular command.

connect-viserver takes 20 seconds no matter what I try (I am using -server localhost -user user -pass pass notation, also tried -credential, and -protocol http, -server IP, -server 127.0.0.1 and resuming an existing session)
get-vm takes OVER 2 MINS on the FIRST time (even though there are less than 10 VMs in the vCenter inventory). (get-vm is fast on subsequent runs)

Using process explorer, it appears PowerCLI is getting stuck inside the plumbing of .NET.  Both for connect-viserver and get-vm, process explorer shows PowerCLI at 100% CPU inside this DLL for the entire duration:

mscorwks.dll!InitializeFusion (version 4927).  I also updated to version 4952 but same issue, I have not yet tried downgrading to an earlier version.

For the get-vm cmdlet, the working set of the process increases steadily from 100MB to eventually topping 500MB right before the command completes execution.

Memory/disk/network activity is low to non-existent, only CPU at 100%.

The version of .NET installed PowerCLI is using is the one that ships with 2008 R2:

2.0.50727.4927 (64-bit)

I have actioned several recommendations from the web including:
* disabled checking if publisher's cert is revoked
* disabled proxy via set-vitoolkitconfiguration
* disabled proxy autodetect in powershell.config (as per http://vinternals.wordpress.com/2010/08/24/powershell-and-proxy-auto-detect-problem/)
* ngen all compilations are up to date

None of these have had a significant impact.

For comparison, I installed PowerCLI onto a far less powerful Win XP Pro VM and connected to the vCenter from there instead.  The differences are significant:

connect-viserver: 6 seconds (compared to 20 seconds on the win2k8r2 vCenter VM)
get-vm: 40 seconds (compared to over 2 minutes on the win2k8r2 vCenter VM)

When running these commands from the XP Pro VM, the vCenter VM's CPU/disk/network/memory was low at all times.

The actual PowerCLI prompt takes longer to come up on the Win XP VM (about 15 seconds compared to 6 seconds).

Whether I've hit a bug, or running win2k8r2 as a VM under vmware workstation is causing some issues, or whether only win2k8 (not R2) is supported, I am posting this message here in case others are experiencing the same issues.

I was also wondering if others can share further techniques to optimise the speed of PowerShell/.NET/PowerCLI (fundamental configuration changes, etc... not merely optimising bad code), and also, what kind of duration (seconds) are others getting on their first run of a simple "get-vm" command?

The fact it is fast the on subsequent runs is a small comfort, as I need to invoke scripts on an ad-hoc basis thus hitting the 20 second connect & 2 mins+ get-vm issue regularly.

What other setups do people have in terms of precise OS version, precise .NET version etc that is really high performing?  I want to get all commands down to 5 seconds or less even on first run.

Regards
marco

[RvdNieuwendijk]

Hi Marco,

welcome to the VMware Communities.

I noticed that your vCenter Server has only one vCPU. The ESXi Installable and vCenter Server Setup Guide ESXi 4.1 Installable vCenter Server 4.1 page 15 says that the minimum requirements for a vCenter Server is "two 64-bit CPUs or one 64-bit dual-core processor". So your vCenter Server might run faster if you give it a second vCPU.

The PowerCLI Get-VM cmdlet is know to be slow. In my test environment it took 7.5 seconds to retrieve 12 VM's. You gain much speed if you use:

Get-View -ViewType VirtualMachine

In my test environment this took only 1.4 seconds.

The Get-View cmdlet returns another type of object than Get-VM does. Get-VM returns a VMware.VimAutomation.ViCore.Impl.V1.Inventory.VirtualMachineImpl object. And Get-View returns a VMware.Vim.VirtualMachine object. But you can easily convert the returned object to same type that Get-VM gives you, using the Get-VIObjectByVIView cmdlet.

While the Get-VM cmdlet returns only VM's, the Get-View statement given above also returns templates. You can prevent this using a filter:

Get-View -ViewType VirtualMachine -Filter @{"Config.Template"="False"}

There is a way to gain additional speed using the Get-View cmdlet. If you use the -Property parameter and specify only the properties that you want to retrieve, the cmdlet doesn't need to return all the other properties and is much faster.

Regards, Robert

[marc045]

THanks Robert,

I am still not able to get the same kind of times when I run get-vm on my test environment with similar amount of VMs.  Again, the slow execute times are for the FIRST run of the command only, subsequent runs are speedy.

For the server that you are running PowerCLI on (where you got the 7 seconds / 1.4 seconds Get-VM / Get-View results), are you able to share the following details:

[1]
What precise OS and bits (32 or 64)?

[2]
What precise version of .NET 2.0? (via C:\windows\Microsoft.NET\Framework\v2.0.50727 -> right click mscorwks.dll > properties > details > file version (2.0.50727.XXXX) - what is the value of XXXX?

[3]
What version of PowerShell and is it 32-bit or 64-bit?

[4]
The '7 second get-vm' result, was that on the first run of the command?

[5]
The '1.4 second get-view result', did you run that immediately after the get-vm command, or did you close & relaunch PowerCLI before running the get-view command?

[6]

I am now using 32-bit Powershell and getting much faster times (get-vm from 2mins+ is now down to 16 seconds!) - However, on my setup, on the FIRST RUN of either "get-vm" or "Get-View -ViewType VirtualMachine", these BOTH come in at 16 seconds - I am not getting a significant difference in times like you are (7 secs vs 1.4 secs) - any thoughts or insight on this point?

Thanks Robert, I hope to be able to use your information to build a similar environment which produces faster results.

Regards
marc0

PS: Update on my previous post, I find using 32-bit Powershell is markedly faster than 64-bit
PPS: I'm trying to award you points for a helpful response but I can't seem to find the option for this

[wprather]

I am running into the same issue with 64-bit Powershell taking forever to execute the conenct-viserver and get-vm cmdlets.  I thought I would post what details I could of my environment.  Running a Windows 7 64-bit VM with 2 GB RAM, 2x 2.66Ghz Xeon processors, and PowerCLI 4.1.1.  This is on a much more powerful server the details of which I don't believe are important.  In 64-bit Powershell, I run a script that takes 106 seconds and pegs the process at 100% with private working set of Memory soaring.  By doing nothing more than running the script in 32-bit PowerShell, the script finishes in 22 seconds.  The processor hit 20% usage and the private workign set of memory finished at 63k.

The script uses the connect-viserver once and then uses the get-vm cmdlet twice overall.  That's where the majority of my script execution time was coming from.

Not a resolution by a long shot, simply trying to add some additional information.

[LucD]

It won't help your case of course, but I also noticed that there is a serious performance difference between running PowerCLI in 32- or 64-bit mode.

It looks as if PowerCLI is not really fine-tuned for 64-bit platforms.

Since 64-bit platforms are becoming the standard, I agree there is some work to be done here.

[RvdNieuwendijk]

A new blogpost by the PowerCLI team gives an explanation and a solution for the problem that PowerCLI commands run much slower on  the first run. Read How to speed-up the execution of the first PowerCLI cmdlet for more information.

Message was edited by: RvdNieuwendijk Corrected the hyperlink.