VMware Cloud Community
vijaysaiv
Contributor
Contributor

Powercli 6.5 memory leak

Folks,

I am new to power cli, and started using it from couple of months. I was very happy with PowerCLI 6.3 but I recently upgraded to PowerCLI 6.5 (Powershell 4.0 or 5.0) as our vcenter is upgraded to 6.5.

The moment I upgraded, I started running into memory issues.

Each end every command of powercli leaks memory and not freed for ever until I force garbage collect.

Connect-VIserver, get-vm etc, all commands are leaking memory big time. Have you seen this kind of issue any time? is there anyway we can contain this leak.

Here is my envivonment: Windows 2012 R2 server, .net 4.0, PowerShell 4.0/5.0, PowerCLI 6.5, Vcenter 6.5.

Appreciate your help.

Regards

Vijay

27 Replies
LucD
Leadership
Leadership

No, haven't seen that behavior.
I'm currently on PowerShell 5.1 and PowerCLI 6.5.3, but I don't see any exceptional memory use, nor memory leaks.

Could you perhaps give an example of such a memory leak you discovered?

A screenshot is ok.

You could perhaps also upgrade to PowerShell 5 (not sure what you mean by PowerShell 4/5 btw) and PowerCLI 6.5.3.
And check if you still have the issue.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
vijaysaiv
Contributor
Contributor

Hi, Thank you for your reply.

I am using powershell 4 and powershell 5 as well.

Here is the information that you had asked.

I have experimented with PowerCLI 6.3r1 and PowerCLI 6.5.3 (latest). If you can go through the below notes, you can find that there is steep increase in the memory consumption incase of powercli 6.5.3. And the memory never comes down. The change in the environment between powercli 6.3.r1 and powercli 6.5.3 is, the windows OS version and powershell.

So, if I run get-VM and any other commands of power CLI using 6.5.3, the memory keeps on increase and never comes down.

1/ PowerCLI 6.3r1, with powershell 4.0, .Net 4.5, windows 7 sp1

powercli-6.3r1-image1, indicates the memory status after I connected to the VI Server, the memory shows that 89MB

powercli-6.3r1-image1.png

powercli-6.3r1-image2, indicates the memory status after I executed get-vm 3 times, the memory shows that 88MB

powercli-6.3r1-image2.png

powercli-6.3r1-image2, indicates the memory status after I executed get-vm <<vm_name> 10  times . the memory shows that 79 MB.

powercli-6.3r1-image3.png

If you see, memory highlighted is pretty much stable and it went down once the command got executed,

2/ PowerCLI 6.5.3, with powershell 5.0, .Net 4.5, windows 2012 r2 standard.

powercli-6.3r1-image1, indicates the memory status after I connected to the VI Server, the memory shows that 93MB

powercli-6.5.3-image1.png

powercli-6.3r1-image2, indicates the memory status after I executed get-vm 3 times, the memory shows that 131 MB

powercli-6.5.3-image2.png

powercli-6.3r1-image2, indicates the memory status after I executed get-vm <<vm_name> 10  times . the memory shows that 155 MB.

powercli-6.5.3-image3.png

Reply
0 Kudos
LucD
Leadership
Leadership

I assume that under 2) when you mention PowerCLI 6.3R1, that that is a typo, and should be PowerCLI 6.5.3

Imho you are comparing apples and pears. The OS is different, the .Net is different and the PS version is different.

I find your conclusion that PowerCLI has a memory leak a bit premature.

Granted PowerCLI uses memory from the PowerShell address space, but you can't compare these numbers (between completely different environments) like you do.

Just to mention one point that is totally different, in PowerCLI 6.5.3, there are only modules, no more PSSnapin.

And with the module auto-load feature, the moment you use a cmdlet, the corresponding module(s) will be loaded.

I did a small test on memory consumption in my environment (.Net 4.5, PS v5.1, PCLI 6.5.3), and I really don't see anything to get worried about.

Mind, I'm not saying there might not be a memory leak in PowerCLI, but afaik that is nothing to worry about as far as I can see.

From my test I see nothing out of the ordinary.

 

ActionMemoryMB
PS start30
Connect-VIServer117
Get-VM (x  times)114
Disconnect-VIServer117
Remove-Module  (PCLI)117
[System.GC]::Collect()105
Remove-Module  (all) + [System.GC]::Collect()93

That the memory consumption never gets back to what is was when I started the PS session, is normal imho, and is to be attributed to the working of PS, the garbage collection in .Net....

Btw, don't do this garbage collection in your code, let .Net decide when and how it should be started.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
vijaysaiv
Contributor
Contributor

I replied through email but my response is not appearing here, posting the same reply here .

---------_--------------------

That's a typo, I realised after posting it. Sorry about it. Thanks for your prompt response.

I gave powercli 6.5.3 as an example but I started with powercli 6.5.0.  I can confirm that through my tests it is happening in same way for both these versions.

I have a .net application, that queries 100s VMs using powercli and does various operations.

On the system I mentioned as windows 7, power CLI 6.3, the application runs with out any issues for days together as memory is getting freed up pretty quickly. I process 100s of VMs in one loop. And the memory consumption of my application at the end of the run doesn't even cross 200 MB.

On the other system I mentioned as windows 2012 R2 , powercli 6.5.0 or powercli 6.5.3, it won't even allow me to process 100 VMs as I get out of memory exception consistently. Either it is leaking memory or the freeing up memory is pretty slow as I  hit a wall before garbage collection happens. I have no clue what else to do.

I don't think these garbage collection policies be so different between two OS types.

I am not using forced garbage collection yet but I thought about it.

Also, I assume that the auto module.load is not available with powercli 6.5.0.

Another point is, these two machines are virtual machines on ESxi server.

Vijay

Reply
0 Kudos
LucD
Leadership
Leadership

If you really suspect a PowerCLI memory leak, I suggest you do some more fact finding.

For example with a tool like .Net Memory Profiler.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
vijaysaiv
Contributor
Contributor

We coudlnt do the profiling but we are pretty much certain that the new version of PowerCLI has issues. Apart from Power CLI 6.3, we also tried powercli 6.0, which works very well and memory is managed pretty well. At the end of our application run we could see that only 250 MB is consumed by our application. It is 2GB with PowerCLI 6.5.x.

PowerCLI 6.0 is officially not supported with VCenter 6.5, by VMware. So, I am still looking for a solution.

Reply
0 Kudos
LucD
Leadership
Leadership

You seem to ignore that you are also using different PowerShell and .Net versions, at least that is what I deduce from your previous entries.

I'm not saying that the latest releases of PowerShell, .Net and PowerCLI are not requiring more memory, but I find it a bit premature to immediately attribute the issue to PowerCLI.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
vijaysaiv
Contributor
Contributor

I really dont think those are the factors that will impact at this scale as we are talking in terms GBs.

I have not mentioned this specifically as I did not convince myself.

However, I did little more testing to answer your question too.

1./

We have a windows 7 machine, on which we have power cli 6.5, PowerShell 5.0, .Net 4.5.

On this we see our application is consuming over 2GB at the end of the run

2/

Same machine, windows 7. Uninstalled 6.5 and installed 6.0, powershell 5.0 and .Net 4.5 remains same.

At the end of the run our application consumes just 250 MB.

Server side things remain same for both the above scenarios.

Something is seriously changed between these two versions of PowerCLI. I am unable to figure out if there could be some changes in the command usage or initialisation.

Reply
0 Kudos
LucD
Leadership
Leadership

What stays allocated at the end of the script can be an indication, but did you also measure the maximum memory consumption during the running of the script on the two platforms?

And there is an important difference between PowerCLI 6 and 6.5, the latter is modules only, while the the 6 version still has PSSnapin.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
vijaysaiv
Contributor
Contributor

Maximum consumption by the application was 2.5 GB, and it comes down to 2 GB and stays there, for PowerCLI 6.5 platform

For the PowerCLI 6.0 platform, it goes to 450 MB max, and then comes down to 250 MB.

I am aware of this difference between PowerCLI 6.0 and PowerCLI 6.5, however, do you think this will impact our memory usage?

Reply
0 Kudos
LucD
Leadership
Leadership

Yes, that will definitely impact memory consumption.

Can you share the script (anonymized of course) that you are using for your measurements?
I would like to run it through the .Net Memory Profiler (under PCLI 6 and 6.5) to see what is actually consuming that memory.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
vijaysaiv
Contributor
Contributor

>>Yes, that will definitely impact memory consumption.

May I know how this is going to impact the memory consumption?

The script has many VM operations that we are doing through out the application life cycle, and we are doing things for at least 100 VMs. I cant copy the script here but basically, we power on VMs, wait for VMs to come up, check network on the VM if it is up and running, and then we reboot the VM.  Would this be sufficient

Reply
0 Kudos
LucD
Leadership
Leadership

I'm afraid that without seeing the actual code, it is hard to determine if your claim is justified.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
vijaysaiv
Contributor
Contributor

I need to verify if I can share the script here, let me work on that. However, can you please tell me how PSSnapin and modules will have changes in memory consumption ?

Reply
0 Kudos
kruddy
Enthusiast
Enthusiast

Can we get an output of the following command when you're experiencing the memory leak: Get-Module

Please, don't forget the awarding points for "helpful" and/or "correct" answers.
Reply
0 Kudos
vijaysaiv
Contributor
Contributor

Here it is:

PowerCLI C:\> Get-Module

ModuleType Version    Name                                ExportedCommands

---------- -------    ----                                ----------------

Script     0.0        Initialize-VMware.VimAutomation....

Script     0.0        Initialize-VMware.VimAutomation....

Script     0.0        Initialize-VMware.VimAutomation....

Script     0.0        Initialize-VMware_DeployAutomation

Script     0.0        Initialize-VMware_VimAutomation_Cis

Script     0.0        Initialize-VMware_VumAutomation

Manifest   3.1.0.0    Microsoft.PowerShell.Management     {Add-Computer, Add-Content, Checkpoint-Computer, Clear-Content...}

Manifest   3.1.0.0    Microsoft.PowerShell.Utility        {Add-Member, Add-Type, Clear-Variable, Compare-Object...}

Binary     6.0.0.0    VMware.DeployAutomation             {Add-DeployRule, Add-ProxyServer, Add-ScriptBundle, Copy-DeployRule...}

Binary     6.0.0.0    VMware.ImageBuilder                 {Add-EsxSoftwareDepot, Add-EsxSoftwarePackage, Compare-EsxImageProfile, Export-EsxImageProfile...}

Binary     6.5.0.4... VMware.VimAutomation.Cis.Core       {Connect-CisServer, Disconnect-CisServer, Get-CisService}

Binary     6.5.0.4... VMware.VimAutomation.Cloud          {Add-CIDatastore, Connect-CIServer, Disconnect-CIServer, Get-Catalog...}

Manifest   6.5.0.4... VMware.VimAutomation.Common

Binary     6.5.0.2... VMware.VimAutomation.Core           {Add-PassthroughDevice, Add-VirtualSwitchPhysicalNetworkAdapter, Add-VMHost, Add-VMHostNtpServer...}

Binary     6.0.0.0    VMware.VimAutomation.HA             Get-DrmInfo

Binary     7.0.2.4... VMware.VimAutomation.HorizonView    {Connect-HVServer, Disconnect-HVServer}

Binary     6.5.0.4... VMware.VimAutomation.License        Get-LicenseDataManager

Binary     6.5.0.4... VMware.VimAutomation.PCloud         {Connect-PIServer, Disconnect-PIServer, Get-PIComputeInstance, Get-PIDatacenter}

Manifest   6.5.0.4... VMware.VimAutomation.Sdk            Get-PSVersion

Binary     6.5.0.4... VMware.VimAutomation.Storage        {Copy-VDisk, Export-SpbmStoragePolicy, Get-NfsUser, Get-SpbmCapability...}

Binary     6.5.0.4... VMware.VimAutomation.Vds            {Add-VDSwitchPhysicalNetworkAdapter, Add-VDSwitchVMHost, Export-VDPortGroup, Export-VDSwitch...}

Binary     6.5.0.4... VMware.VimAutomation.vROps          {Connect-OMServer, Disconnect-OMServer, Get-OMAlert, Get-OMAlertDefinition...}

Binary     6.0.0.0    VMware.VumAutomation                {Add-EntityBaseline, Copy-Patch, Get-Baseline, Get-Compliance...}

Reply
0 Kudos
kruddy
Enthusiast
Enthusiast

Is that the setup that's leaking memory?

If so, that's not using PowerCLI 6.5.3.

Your module output should look a little more like this:

PS C:\Users\kruddy> Get-Module

ModuleType Version    Name

---------- -------    ----

Manifest   3.1.0.0    Microsoft.PowerShell.Management

Manifest   3.1.0.0    Microsoft.PowerShell.Utility

Script     1.2        PSReadline

Binary     6.5.1.5... VMware.DeployAutomation

Binary     6.5.1.5... VMware.ImageBuilder

Manifest   6.5.3.6... vmware.powercli

Binary     6.5.3.6... VMware.VimAutomation.Cis.Core

Binary     6.5.1.5... VMware.VimAutomation.Cloud

Manifest   6.5.1.5... VMware.VimAutomation.Common

Binary     6.5.2.6... VMware.VimAutomation.Core

Binary     6.0.0.5... VMware.VimAutomation.HA

Binary     7.1.0.5... VMware.VimAutomation.HorizonView

Binary     6.5.1.5... VMware.VimAutomation.License

Binary     2.0.0.6... VMware.VimAutomation.Nsxt

Binary     6.5.1.5... VMware.VimAutomation.PCloud

Manifest   1.0.0.5... VMware.VimAutomation.Sdk

Binary     6.5.1.5... VMware.VimAutomation.Srm

Binary     6.5.1.5... VMware.VimAutomation.Storage

Script     1.0        VMware.VimAutomation.StorageUtility

Binary     6.5.1.5... VMware.VimAutomation.Vds

Binary     6.5.1.5... VMware.VimAutomation.vROps

Binary     6.5.1.5... VMware.VumAutomation

I would recommending uninstalling your current version of PowerCLI, installing PowerCLI 6.5.3, and trying your script again.

Couple notes:

Please, don't forget the awarding points for "helpful" and/or "correct" answers.
Reply
0 Kudos
vijaysaiv
Contributor
Contributor

This is the setup where we have PowerCLI 6.5.0 installed. As I mentioned earlier, we see this problem through out PowerCLI 6.5 series. We tried 6.5.1 and 6.5.3, all of them have the same issue.

The get-module output was from a machine with PowerCLI 6.5.0.

Reply
0 Kudos
kruddy
Enthusiast
Enthusiast

Unfortunately, I'm not able to replicate the memory leak with PowerCLI 6.5.3.

The attached image shows multiple points where I open up PowerShell, connect to the vCenter Server, then run the 'Get-VM' command through a loop 1,000 times.

The memory fluctuated between 110MB and 202MB, but ended at 113.5MB.

At this point, I think it would be best to open a support request. Pick a single cmdlet, like Get-VM, and include those results in the support request stating that you believe there's a memory leak. Please also include a link to this thread in the SR.

If you can then also post the SR number in this thread, I'll be able to help the VMware support team get to the bottom of this.

Thank you for your patience!

Please, don't forget the awarding points for "helpful" and/or "correct" answers.
Reply
0 Kudos