1 2 Previous Next 27 Replies Latest reply on May 18, 2019 9:05 AM by vijaysaiv

    Powercli 6.5 memory leak

    vijaysaiv Novice

      Folks,

       

      I am new to power cli, and started using it from couple of months. I was very happy with PowerCLI 6.3 but I recently upgraded to PowerCLI 6.5 (Powershell 4.0 or 5.0) as our vcenter is upgraded to 6.5.

      The moment I upgraded, I started running into memory issues.

       

      Each end every command of powercli leaks memory and not freed for ever until I force garbage collect.

      Connect-VIserver, get-vm etc, all commands are leaking memory big time. Have you seen this kind of issue any time? is there anyway we can contain this leak.

       

      Here is my envivonment: Windows 2012 R2 server, .net 4.0, PowerShell 4.0/5.0, PowerCLI 6.5, Vcenter 6.5.

       

      Appreciate your help.

       

      Regards

      Vijay

        • 1. Re: Powercli 6.5 memory leak
          LucD Guru
          User ModeratorsvExpertCommunity Warriors

          No, haven't seen that behavior.
          I'm currently on PowerShell 5.1 and PowerCLI 6.5.3, but I don't see any exceptional memory use, nor memory leaks.

           

          Could you perhaps give an example of such a memory leak you discovered?

          A screenshot is ok.

           

          You could perhaps also upgrade to PowerShell 5 (not sure what you mean by PowerShell 4/5 btw) and PowerCLI 6.5.3.
          And check if you still have the issue.

          • 2. Re: Powercli 6.5 memory leak
            vijaysaiv Novice

            Hi, Thank you for your reply.

             

            I am using powershell 4 and powershell 5 as well.

            Here is the information that you had asked.

             

            I have experimented with PowerCLI 6.3r1 and PowerCLI 6.5.3 (latest). If you can go through the below notes, you can find that there is steep increase in the memory consumption incase of powercli 6.5.3. And the memory never comes down. The change in the environment between powercli 6.3.r1 and powercli 6.5.3 is, the windows OS version and powershell.

            So, if I run get-VM and any other commands of power CLI using 6.5.3, the memory keeps on increase and never comes down.

             

            1/ PowerCLI 6.3r1, with powershell 4.0, .Net 4.5, windows 7 sp1

             

            powercli-6.3r1-image1, indicates the memory status after I connected to the VI Server, the memory shows that 89MB

            powercli-6.3r1-image1.png

            powercli-6.3r1-image2, indicates the memory status after I executed get-vm 3 times, the memory shows that 88MB

            powercli-6.3r1-image2.png

            powercli-6.3r1-image2, indicates the memory status after I executed get-vm <<vm_name> 10  times . the memory shows that 79 MB.

            powercli-6.3r1-image3.png

            If you see, memory highlighted is pretty much stable and it went down once the command got executed,

             

            2/ PowerCLI 6.5.3, with powershell 5.0, .Net 4.5, windows 2012 r2 standard.

             

            powercli-6.3r1-image1, indicates the memory status after I connected to the VI Server, the memory shows that 93MB

            powercli-6.5.3-image1.png

             

            powercli-6.3r1-image2, indicates the memory status after I executed get-vm 3 times, the memory shows that 131 MB

            powercli-6.5.3-image2.png

             

            powercli-6.3r1-image2, indicates the memory status after I executed get-vm <<vm_name> 10  times . the memory shows that 155 MB.

            powercli-6.5.3-image3.png

            • 3. Re: Powercli 6.5 memory leak
              LucD Guru
              User ModeratorsCommunity WarriorsvExpert

              I assume that under 2) when you mention PowerCLI 6.3R1, that that is a typo, and should be PowerCLI 6.5.3

               

              Imho you are comparing apples and pears. The OS is different, the .Net is different and the PS version is different.

              I find your conclusion that PowerCLI has a memory leak a bit premature.

               

              Granted PowerCLI uses memory from the PowerShell address space, but you can't compare these numbers (between completely different environments) like you do.

               

              Just to mention one point that is totally different, in PowerCLI 6.5.3, there are only modules, no more PSSnapin.

              And with the module auto-load feature, the moment you use a cmdlet, the corresponding module(s) will be loaded.

               

              I did a small test on memory consumption in my environment (.Net 4.5, PS v5.1, PCLI 6.5.3), and I really don't see anything to get worried about.

              Mind, I'm not saying there might not be a memory leak in PowerCLI, but afaik that is nothing to worry about as far as I can see.

               

              From my test I see nothing out of the ordinary.

               

              ActionMemoryMB
              PS start30
              Connect-VIServer117
              Get-VM (x  times)114
              Disconnect-VIServer117
              Remove-Module  (PCLI)117
              [System.GC]::Collect()105
              Remove-Module  (all) + [System.GC]::Collect()93

               

              That the memory consumption never gets back to what is was when I started the PS session, is normal imho, and is to be attributed to the working of PS, the garbage collection in .Net....

               

              Btw, don't do this garbage collection in your code, let .Net decide when and how it should be started.

              • 4. Re: Powercli 6.5 memory leak
                vijaysaiv Novice

                I replied through email but my response is not appearing here, posting the same reply here .

                ---------_--------------------

                 

                That's a typo, I realised after posting it. Sorry about it. Thanks for your prompt response.

                 

                I gave powercli 6.5.3 as an example but I started with powercli 6.5.0.  I can confirm that through my tests it is happening in same way for both these versions.

                 

                I have a .net application, that queries 100s VMs using powercli and does various operations.

                 

                On the system I mentioned as windows 7, power CLI 6.3, the application runs with out any issues for days together as memory is getting freed up pretty quickly. I process 100s of VMs in one loop. And the memory consumption of my application at the end of the run doesn't even cross 200 MB.

                 

                On the other system I mentioned as windows 2012 R2 , powercli 6.5.0 or powercli 6.5.3, it won't even allow me to process 100 VMs as I get out of memory exception consistently. Either it is leaking memory or the freeing up memory is pretty slow as I  hit a wall before garbage collection happens. I have no clue what else to do.

                 

                I don't think these garbage collection policies be so different between two OS types.

                 

                I am not using forced garbage collection yet but I thought about it.

                 

                Also, I assume that the auto module.load is not available with powercli 6.5.0.

                 

                Another point is, these two machines are virtual machines on ESxi server.

                 

                Vijay

                • 5. Re: Powercli 6.5 memory leak
                  LucD Guru
                  Community WarriorsvExpertUser Moderators

                  If you really suspect a PowerCLI memory leak, I suggest you do some more fact finding.

                  For example with a tool like .Net Memory Profiler.

                  • 6. Re: Powercli 6.5 memory leak
                    vijaysaiv Novice

                    We coudlnt do the profiling but we are pretty much certain that the new version of PowerCLI has issues. Apart from Power CLI 6.3, we also tried powercli 6.0, which works very well and memory is managed pretty well. At the end of our application run we could see that only 250 MB is consumed by our application. It is 2GB with PowerCLI 6.5.x.

                    PowerCLI 6.0 is officially not supported with VCenter 6.5, by VMware. So, I am still looking for a solution.

                    • 7. Re: Powercli 6.5 memory leak
                      LucD Guru
                      Community WarriorsUser ModeratorsvExpert

                      You seem to ignore that you are also using different PowerShell and .Net versions, at least that is what I deduce from your previous entries.

                      I'm not saying that the latest releases of PowerShell, .Net and PowerCLI are not requiring more memory, but I find it a bit premature to immediately attribute the issue to PowerCLI.

                      • 8. Re: Powercli 6.5 memory leak
                        vijaysaiv Novice

                        I really dont think those are the factors that will impact at this scale as we are talking in terms GBs.

                        I have not mentioned this specifically as I did not convince myself.

                        However, I did little more testing to answer your question too.

                         

                        1./

                        We have a windows 7 machine, on which we have power cli 6.5, PowerShell 5.0, .Net 4.5.

                        On this we see our application is consuming over 2GB at the end of the run

                         

                        2/

                        Same machine, windows 7. Uninstalled 6.5 and installed 6.0, powershell 5.0 and .Net 4.5 remains same.

                        At the end of the run our application consumes just 250 MB.

                         

                        Server side things remain same for both the above scenarios.

                         

                        Something is seriously changed between these two versions of PowerCLI. I am unable to figure out if there could be some changes in the command usage or initialisation.

                        • 9. Re: Powercli 6.5 memory leak
                          LucD Guru
                          vExpertUser ModeratorsCommunity Warriors

                          What stays allocated at the end of the script can be an indication, but did you also measure the maximum memory consumption during the running of the script on the two platforms?

                           

                          And there is an important difference between PowerCLI 6 and 6.5, the latter is modules only, while the the 6 version still has PSSnapin.

                          • 10. Re: Powercli 6.5 memory leak
                            vijaysaiv Novice

                            Maximum consumption by the application was 2.5 GB, and it comes down to 2 GB and stays there, for PowerCLI 6.5 platform

                            For the PowerCLI 6.0 platform, it goes to 450 MB max, and then comes down to 250 MB.

                             

                            I am aware of this difference between PowerCLI 6.0 and PowerCLI 6.5, however, do you think this will impact our memory usage?

                            • 11. Re: Powercli 6.5 memory leak
                              LucD Guru
                              Community WarriorsvExpertUser Moderators

                              Yes, that will definitely impact memory consumption.

                               

                              Can you share the script (anonymized of course) that you are using for your measurements?
                              I would like to run it through the .Net Memory Profiler (under PCLI 6 and 6.5) to see what is actually consuming that memory.

                              • 12. Re: Powercli 6.5 memory leak
                                vijaysaiv Novice

                                >>Yes, that will definitely impact memory consumption.

                                May I know how this is going to impact the memory consumption?

                                 

                                The script has many VM operations that we are doing through out the application life cycle, and we are doing things for at least 100 VMs. I cant copy the script here but basically, we power on VMs, wait for VMs to come up, check network on the VM if it is up and running, and then we reboot the VM.  Would this be sufficient

                                • 13. Re: Powercli 6.5 memory leak
                                  LucD Guru
                                  vExpertUser ModeratorsCommunity Warriors

                                  I'm afraid that without seeing the actual code, it is hard to determine if your claim is justified.

                                  • 14. Re: Powercli 6.5 memory leak
                                    vijaysaiv Novice

                                    I need to verify if I can share the script here, let me work on that. However, can you please tell me how PSSnapin and modules will have changes in memory consumption ?

                                    1 2 Previous Next