Folks,
I am new to power cli, and started using it from couple of months. I was very happy with PowerCLI 6.3 but I recently upgraded to PowerCLI 6.5 (Powershell 4.0 or 5.0) as our vcenter is upgraded to 6.5.
The moment I upgraded, I started running into memory issues.
Each end every command of powercli leaks memory and not freed for ever until I force garbage collect.
Connect-VIserver, get-vm etc, all commands are leaking memory big time. Have you seen this kind of issue any time? is there anyway we can contain this leak.
Here is my envivonment: Windows 2012 R2 server, .net 4.0, PowerShell 4.0/5.0, PowerCLI 6.5, Vcenter 6.5.
Appreciate your help.
Regards
Vijay
No, haven't seen that behavior.
I'm currently on PowerShell 5.1 and PowerCLI 6.5.3, but I don't see any exceptional memory use, nor memory leaks.
Could you perhaps give an example of such a memory leak you discovered?
A screenshot is ok.
You could perhaps also upgrade to PowerShell 5 (not sure what you mean by PowerShell 4/5 btw) and PowerCLI 6.5.3.
And check if you still have the issue.
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Hi, Thank you for your reply.
I am using powershell 4 and powershell 5 as well.
Here is the information that you had asked.
I have experimented with PowerCLI 6.3r1 and PowerCLI 6.5.3 (latest). If you can go through the below notes, you can find that there is steep increase in the memory consumption incase of powercli 6.5.3. And the memory never comes down. The change in the environment between powercli 6.3.r1 and powercli 6.5.3 is, the windows OS version and powershell.
So, if I run get-VM and any other commands of power CLI using 6.5.3, the memory keeps on increase and never comes down.
1/ PowerCLI 6.3r1, with powershell 4.0, .Net 4.5, windows 7 sp1
powercli-6.3r1-image1, indicates the memory status after I connected to the VI Server, the memory shows that 89MB
powercli-6.3r1-image2, indicates the memory status after I executed get-vm 3 times, the memory shows that 88MB
powercli-6.3r1-image2, indicates the memory status after I executed get-vm <<vm_name> 10 times . the memory shows that 79 MB.
If you see, memory highlighted is pretty much stable and it went down once the command got executed,
2/ PowerCLI 6.5.3, with powershell 5.0, .Net 4.5, windows 2012 r2 standard.
powercli-6.3r1-image1, indicates the memory status after I connected to the VI Server, the memory shows that 93MB
powercli-6.3r1-image2, indicates the memory status after I executed get-vm 3 times, the memory shows that 131 MB
powercli-6.3r1-image2, indicates the memory status after I executed get-vm <<vm_name> 10 times . the memory shows that 155 MB.
I assume that under 2) when you mention PowerCLI 6.3R1, that that is a typo, and should be PowerCLI 6.5.3
Imho you are comparing apples and pears. The OS is different, the .Net is different and the PS version is different.
I find your conclusion that PowerCLI has a memory leak a bit premature.
Granted PowerCLI uses memory from the PowerShell address space, but you can't compare these numbers (between completely different environments) like you do.
Just to mention one point that is totally different, in PowerCLI 6.5.3, there are only modules, no more PSSnapin.
And with the module auto-load feature, the moment you use a cmdlet, the corresponding module(s) will be loaded.
I did a small test on memory consumption in my environment (.Net 4.5, PS v5.1, PCLI 6.5.3), and I really don't see anything to get worried about.
Mind, I'm not saying there might not be a memory leak in PowerCLI, but afaik that is nothing to worry about as far as I can see.
From my test I see nothing out of the ordinary.
Action | MemoryMB |
PS start | 30 |
Connect-VIServer | 117 |
Get-VM (x times) | 114 |
Disconnect-VIServer | 117 |
Remove-Module (PCLI) | 117 |
[System.GC]::Collect() | 105 |
Remove-Module (all) + [System.GC]::Collect() | 93 |
That the memory consumption never gets back to what is was when I started the PS session, is normal imho, and is to be attributed to the working of PS, the garbage collection in .Net....
Btw, don't do this garbage collection in your code, let .Net decide when and how it should be started.
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
I replied through email but my response is not appearing here, posting the same reply here .
---------_--------------------
That's a typo, I realised after posting it. Sorry about it. Thanks for your prompt response.
I gave powercli 6.5.3 as an example but I started with powercli 6.5.0. I can confirm that through my tests it is happening in same way for both these versions.
I have a .net application, that queries 100s VMs using powercli and does various operations.
On the system I mentioned as windows 7, power CLI 6.3, the application runs with out any issues for days together as memory is getting freed up pretty quickly. I process 100s of VMs in one loop. And the memory consumption of my application at the end of the run doesn't even cross 200 MB.
On the other system I mentioned as windows 2012 R2 , powercli 6.5.0 or powercli 6.5.3, it won't even allow me to process 100 VMs as I get out of memory exception consistently. Either it is leaking memory or the freeing up memory is pretty slow as I hit a wall before garbage collection happens. I have no clue what else to do.
I don't think these garbage collection policies be so different between two OS types.
I am not using forced garbage collection yet but I thought about it.
Also, I assume that the auto module.load is not available with powercli 6.5.0.
Another point is, these two machines are virtual machines on ESxi server.
Vijay
If you really suspect a PowerCLI memory leak, I suggest you do some more fact finding.
For example with a tool like .Net Memory Profiler.
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
We coudlnt do the profiling but we are pretty much certain that the new version of PowerCLI has issues. Apart from Power CLI 6.3, we also tried powercli 6.0, which works very well and memory is managed pretty well. At the end of our application run we could see that only 250 MB is consumed by our application. It is 2GB with PowerCLI 6.5.x.
PowerCLI 6.0 is officially not supported with VCenter 6.5, by VMware. So, I am still looking for a solution.
You seem to ignore that you are also using different PowerShell and .Net versions, at least that is what I deduce from your previous entries.
I'm not saying that the latest releases of PowerShell, .Net and PowerCLI are not requiring more memory, but I find it a bit premature to immediately attribute the issue to PowerCLI.
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
I really dont think those are the factors that will impact at this scale as we are talking in terms GBs.
I have not mentioned this specifically as I did not convince myself.
However, I did little more testing to answer your question too.
1./
We have a windows 7 machine, on which we have power cli 6.5, PowerShell 5.0, .Net 4.5.
On this we see our application is consuming over 2GB at the end of the run
2/
Same machine, windows 7. Uninstalled 6.5 and installed 6.0, powershell 5.0 and .Net 4.5 remains same.
At the end of the run our application consumes just 250 MB.
Server side things remain same for both the above scenarios.
Something is seriously changed between these two versions of PowerCLI. I am unable to figure out if there could be some changes in the command usage or initialisation.
What stays allocated at the end of the script can be an indication, but did you also measure the maximum memory consumption during the running of the script on the two platforms?
And there is an important difference between PowerCLI 6 and 6.5, the latter is modules only, while the the 6 version still has PSSnapin.
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Maximum consumption by the application was 2.5 GB, and it comes down to 2 GB and stays there, for PowerCLI 6.5 platform
For the PowerCLI 6.0 platform, it goes to 450 MB max, and then comes down to 250 MB.
I am aware of this difference between PowerCLI 6.0 and PowerCLI 6.5, however, do you think this will impact our memory usage?
Yes, that will definitely impact memory consumption.
Can you share the script (anonymized of course) that you are using for your measurements?
I would like to run it through the .Net Memory Profiler (under PCLI 6 and 6.5) to see what is actually consuming that memory.
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
>>Yes, that will definitely impact memory consumption.
May I know how this is going to impact the memory consumption?
The script has many VM operations that we are doing through out the application life cycle, and we are doing things for at least 100 VMs. I cant copy the script here but basically, we power on VMs, wait for VMs to come up, check network on the VM if it is up and running, and then we reboot the VM. Would this be sufficient
I'm afraid that without seeing the actual code, it is hard to determine if your claim is justified.
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
I need to verify if I can share the script here, let me work on that. However, can you please tell me how PSSnapin and modules will have changes in memory consumption ?
Can we get an output of the following command when you're experiencing the memory leak: Get-Module
Here it is:
PowerCLI C:\> Get-Module
ModuleType Version Name ExportedCommands
---------- ------- ---- ----------------
Script 0.0 Initialize-VMware.VimAutomation....
Script 0.0 Initialize-VMware.VimAutomation....
Script 0.0 Initialize-VMware.VimAutomation....
Script 0.0 Initialize-VMware_DeployAutomation
Script 0.0 Initialize-VMware_VimAutomation_Cis
Script 0.0 Initialize-VMware_VumAutomation
Manifest 3.1.0.0 Microsoft.PowerShell.Management {Add-Computer, Add-Content, Checkpoint-Computer, Clear-Content...}
Manifest 3.1.0.0 Microsoft.PowerShell.Utility {Add-Member, Add-Type, Clear-Variable, Compare-Object...}
Binary 6.0.0.0 VMware.DeployAutomation {Add-DeployRule, Add-ProxyServer, Add-ScriptBundle, Copy-DeployRule...}
Binary 6.0.0.0 VMware.ImageBuilder {Add-EsxSoftwareDepot, Add-EsxSoftwarePackage, Compare-EsxImageProfile, Export-EsxImageProfile...}
Binary 6.5.0.4... VMware.VimAutomation.Cis.Core {Connect-CisServer, Disconnect-CisServer, Get-CisService}
Binary 6.5.0.4... VMware.VimAutomation.Cloud {Add-CIDatastore, Connect-CIServer, Disconnect-CIServer, Get-Catalog...}
Manifest 6.5.0.4... VMware.VimAutomation.Common
Binary 6.5.0.2... VMware.VimAutomation.Core {Add-PassthroughDevice, Add-VirtualSwitchPhysicalNetworkAdapter, Add-VMHost, Add-VMHostNtpServer...}
Binary 6.0.0.0 VMware.VimAutomation.HA Get-DrmInfo
Binary 7.0.2.4... VMware.VimAutomation.HorizonView {Connect-HVServer, Disconnect-HVServer}
Binary 6.5.0.4... VMware.VimAutomation.License Get-LicenseDataManager
Binary 6.5.0.4... VMware.VimAutomation.PCloud {Connect-PIServer, Disconnect-PIServer, Get-PIComputeInstance, Get-PIDatacenter}
Manifest 6.5.0.4... VMware.VimAutomation.Sdk Get-PSVersion
Binary 6.5.0.4... VMware.VimAutomation.Storage {Copy-VDisk, Export-SpbmStoragePolicy, Get-NfsUser, Get-SpbmCapability...}
Binary 6.5.0.4... VMware.VimAutomation.Vds {Add-VDSwitchPhysicalNetworkAdapter, Add-VDSwitchVMHost, Export-VDPortGroup, Export-VDSwitch...}
Binary 6.5.0.4... VMware.VimAutomation.vROps {Connect-OMServer, Disconnect-OMServer, Get-OMAlert, Get-OMAlertDefinition...}
Binary 6.0.0.0 VMware.VumAutomation {Add-EntityBaseline, Copy-Patch, Get-Baseline, Get-Compliance...}
Is that the setup that's leaking memory?
If so, that's not using PowerCLI 6.5.3.
Your module output should look a little more like this:
PS C:\Users\kruddy> Get-Module
ModuleType Version Name
---------- ------- ----
Manifest 3.1.0.0 Microsoft.PowerShell.Management
Manifest 3.1.0.0 Microsoft.PowerShell.Utility
Script 1.2 PSReadline
Binary 6.5.1.5... VMware.DeployAutomation
Binary 6.5.1.5... VMware.ImageBuilder
Manifest 6.5.3.6... vmware.powercli
Binary 6.5.3.6... VMware.VimAutomation.Cis.Core
Binary 6.5.1.5... VMware.VimAutomation.Cloud
Manifest 6.5.1.5... VMware.VimAutomation.Common
Binary 6.5.2.6... VMware.VimAutomation.Core
Binary 6.0.0.5... VMware.VimAutomation.HA
Binary 7.1.0.5... VMware.VimAutomation.HorizonView
Binary 6.5.1.5... VMware.VimAutomation.License
Binary 2.0.0.6... VMware.VimAutomation.Nsxt
Binary 6.5.1.5... VMware.VimAutomation.PCloud
Manifest 1.0.0.5... VMware.VimAutomation.Sdk
Binary 6.5.1.5... VMware.VimAutomation.Srm
Binary 6.5.1.5... VMware.VimAutomation.Storage
Script 1.0 VMware.VimAutomation.StorageUtility
Binary 6.5.1.5... VMware.VimAutomation.Vds
Binary 6.5.1.5... VMware.VimAutomation.vROps
Binary 6.5.1.5... VMware.VumAutomation
I would recommending uninstalling your current version of PowerCLI, installing PowerCLI 6.5.3, and trying your script again.
Couple notes:
This is the setup where we have PowerCLI 6.5.0 installed. As I mentioned earlier, we see this problem through out PowerCLI 6.5 series. We tried 6.5.1 and 6.5.3, all of them have the same issue.
The get-module output was from a machine with PowerCLI 6.5.0.
Unfortunately, I'm not able to replicate the memory leak with PowerCLI 6.5.3.
The attached image shows multiple points where I open up PowerShell, connect to the vCenter Server, then run the 'Get-VM' command through a loop 1,000 times.
The memory fluctuated between 110MB and 202MB, but ended at 113.5MB.
At this point, I think it would be best to open a support request. Pick a single cmdlet, like Get-VM, and include those results in the support request stating that you believe there's a memory leak. Please also include a link to this thread in the SR.
If you can then also post the SR number in this thread, I'll be able to help the VMware support team get to the bottom of this.
Thank you for your patience!