1 2 Previous Next 19 Replies Latest reply on Dec 20, 2019 2:00 AM by Scarlito

    Slow login/Performance App volumes with Symantec client installed

    Hmunning Novice

      We`ve been troubleshooting slow login and poor application performance on our Non Persistent VDI for a while now. App Volumes and Symantec Endpoint Protection doesn`t seem to like each other.

       

      Without a SEP cliënt installed everything is performing well and user experience feels like a persistent VDI. When SEP is installed including all obvious exceptions and even using the virtual image exception tool no significant change in performance is noticed. We`ve been testing all scenario`s disabling components of symantec. Only disabling "Application & Device Control" seems to improve login and application performance.

       

      Login times :

       

      Symantec all components on + 3 Appstacks                        2,36 Min

      Symantec Application & Device Control off + 3 AppStacks  2,06 Min

       

      Symantec all components on + 1 Appstack                          1,51 Min

      Symantec Application & Device Control off + 1 Appstack    1,25 Min

       

      No Symantec cliënt installed + 3 Appstacks                         0,50 Min

       

      Anyone dealing the same issues? We can`t skip Symantec or move to another AV vendor. (large enterprise)

       

      Horizon View 7.8

      App Volumes 2.16 (computer assigned)

      UEM 9.6

      SEP RU1 14.2.3335.1000

        • 1. Re: Slow login/Performance App volumes with Symantec client installed
          Hmunning Novice

          Forgot another question : I can`t capture boot logging because the c:\ drive is refreshed after a reboot. (sort of readonly when a appstack is attached?)  Tools i`ve been trying store files during capture on c drive . Can`t change this. 

           

          Anyone knows how i can do this?

           

          - Process Monitor (procmon)

          - Microsoft Windows Performance Toolkit- Xperf

          - WPR (Windows performance recorder)

          • 2. Re: Slow login/Performance App volumes with Symantec client installed
            Ray_handels Master
            vExpertCommunity Warriors

            We use an agent-less scanner (McAfee Move) but I do understand that you cannot go that route.

            How many CPU's do you have? What happens if you try to add some more CPU's? You will need at least 2 vCPU's, otherwise Appvolumes and Symantec are both fighting for resources during logon.

             

            And what exclusions do you have? Did you try adding the c:\snapvolumestemp\Mountpoints as an exclusion??

            • 3. Re: Slow login/Performance App volumes with Symantec client installed
              Hmunning Novice

              We use 2 vCPU`s and 4 GB memory. We`ve tested with 4vCPU`s and 8GB memory. No change in login performance.  

               

              We have the following exclusions :

               

              mdf,ndf,ldf

              log,jrs,chk,edb,sdb,SHAREDTA

              c:\ws\

              c:\thinapps\

              C:\pagefile.sys

              %[USER_PROFILE]%\appdata\roaming\thinstall\

              %[PROGRAM_FILES]%VMware\

              %[PROGRAM_FILES]%CloudVolumes\

              %[COMMON_APPDATA]%VMware UEM\FlexSync\

              %[PROGRAM_FILES]%Immidio\Flex Profiles\FlexEngine.exe

              %[PROGRAM_FILES]%Immidio\Flex Profiles\FlexService.exe

              C:\PROGRAM FILES (X86)\BIGFIX ENTERPRISE\BES CLIENT\BESCLIENT.EXE

              C:\PROGRAM FILES\VMWARE\VMWARE VIEW\AGENT\BIN\V4PA_AGENT.EXE

              C:\PROGRAM FILES\IMMIDIO\FLEX PROFILES\FLEXSERVICE.EXE

              C:\SnapVolumesTemp\

              C:\SVROOT\

              %[WINDOWS]%SoftwareDistribution\DataStore\

              %[COMMON_APPDATA]%NTUser.pol

              %[COMMON_APPDATA]%VMware\VDM\Logs\

              %[PROGRAM_FILES]%CloudVolumes\Agent\svservice.exe

              %[PROGRAM_FILES]%Immidio\Flex Profiles\Flex+ Self-Support.exe

              • 4. Re: Slow login/Performance App volumes with Symantec client installed
                Ray_handels Master
                vExpertCommunity Warriors

                That should be enough looking at the exclusions. I would suggest using more then 4GB for a VDI machines but that has got nothing to do with virusscanning and slow logon times, just a best practice.

                To be honest I don't now how to fix this. Would it be an option to try and run Windows Defender just as a reference?? It might be the way Symantec works maybe? You will always receive a logon penalty from virus scanners but it should not be this much.

                • 5. Re: Slow login/Performance App volumes with Symantec client installed
                  Anobix67 Enthusiast

                  What happens if you try user-assigned instead of computer-assigned?

                   

                  We also have SEP 14.2.3335.1000 running in our environment with the same exlusions set on our SEPM servers and login times are a minute or less with a couple appstacks applied.

                   

                  VMs are currently 2CPU/6GB RAM and we have machines on both Windows 10 1709 and 1809 LTSC.

                   

                  Horizon 7.8

                  AppVol 2.17

                  UEM 9.8

                  • 6. Re: Slow login/Performance App volumes with Symantec client installed
                    BenWilliamsIOM Lurker

                    We are seeing the same issue here. Removing SEP altogether has shown a x5 increase in almost anything we use. Logon, AppVolumes attachment, AppV launch and cache, general usability are all dramatically improved.

                     

                    Are there any additional settings over Application and Device control that can be set to improve performance?

                    • 7. Re: Slow login/Performance App volumes with Symantec client installed
                      Scarlito Novice

                      Hi everyone.

                       

                      I'm sorry to say that Hmunning, but I'm happy someone else is facing the same problem, as I was starting to become crazy and feeling alone !

                      I'm fighting with this issue for more than a year (almost 2 actually) on Windows 7 machines. On Windows 10, I haven't tested a lot for now as we purchased licences recently.

                      By the way, Hmunning, you didn't tell us what Windows version you're using.

                       

                      I currently use the following versions :

                      - Windows 7 SP1 x64

                      - AppVolumes : 2.18

                      - Horizon View Agent 7.6

                      - SEP 14.2.4814.1101

                      - UEM (now called DEM) 9.9

                       

                      Interesting fact : I notice this problem only appears after I apply Microsoft Security KB4056897 or later (and of course, with SEP agent installed and active and AppStacks mounted)

                      This means the problem is not only with SEP + AppVolumes, but SEP + AppVolumes + MS Updates (starting january 2018 and all the Intel security breaches fixes).

                      If I remove ANY ONE of these 3 elements, everything works well.

                      Until now, no Monthly security updates from Microsoft has solved anything.

                       

                      I had a case at VMware and another one at Symantec, but none of them solved the problem.

                      I sent them dumps, videos of the user experience with and without the problem, logs... We had WebEx sessions, phone call, emails...

                      After months, we had this answer from VMware :

                      ------------

                      Our backline engineers provided feedback about the procmon log files provided. They confirmed that the issue is related with a performance drop down that is making the access to registry key take longer. They could confirm that Symantec is the component that is making our driver work slow.

                       

                      They suggested you contact Symantec and confirm if they have a way to solve the performance drop and/or to exclude App Volumes driver.

                      ------------

                       

                      We were about to organize a call between VMware and Symantec, but finally we closed the cases, unresolved, hoping for a better experience with Windows 10...

                       

                      And now I'm working on Windows 10 migration, and I still have to test AppVolumes and SEP... And still hoping for this issue to not rise!

                      KB4056897

                      • 8. Re: Slow login/Performance App volumes with Symantec client installed
                        sWORDs Enthusiast
                        VMware Employees

                        Hi Scarlito,

                         

                        What are the case numbers with VMware and Symantec, the customer above is using Windows 8.1 and had similar issues with Windows 10 as far as I know.

                        • 9. Re: Slow login/Performance App volumes with Symantec client installed
                          Scarlito Novice

                          Hi sWORDs.

                          Here are the cases numbers :

                          VMware : 18862278007 and 18997393111

                          Symantec : 15162388

                          Best regards,

                          Carlos.

                          • 10. Re: Slow login/Performance App volumes with Symantec client installed
                            nburton935 Enthusiast

                            We actually had to ditch SEP altogether because of this specific issue. We never couldn’t get them to play nicely together. We saw around 1 minute per simple AppStack when SEP was involved.

                            • 11. Re: Slow login/Performance App volumes with Symantec client installed
                              Scarlito Novice

                              Hi nburton935.

                              That's what I'm fighting for, but I doubt it will be possible unfortunately.

                              Just out of curiosity, what did you replace it with ?

                              • 12. Re: Slow login/Performance App volumes with Symantec client installed
                                nburton935 Enthusiast

                                It was between Endgame and Defender; we ended up going with Endgame. We also have NSX doing micro-segmentation, which can help if you're in an AV debate with InfoSec.

                                • 13. Re: Slow login/Performance App volumes with Symantec client installed
                                  Scarlito Novice

                                  OK. We unfortunately don't use NSX, and it's not supposed to happen soon...

                                  When you say you were not able to make SEP work well with AppVolumes, what were the paths you went on ? Did you get help from VMware and Symantec, or did you try to solve this by yourself ?

                                  Are there any Support Case that we can refer to ?

                                  • 14. Re: Slow login/Performance App volumes with Symantec client installed
                                    Hmunning Novice

                                    Hi, everybody,

                                     

                                    I am a little pleased to hear that we are not the only ones with this problem.

                                     

                                    After a few months of troubleshooting we have only achieved small results. From the current support calls of Symantec and VMware no rootcause has been found yet. We have delivered many gb of logfiles and videos. Procmon, WPA,XPerf, Wireshark. You name it... 

                                     

                                    Both Symantec and VMware indicated that no other support calls are registred with this specific problem. Well that was not true, but maybe difficult to find?  For now the only thing what is seen is there are extremely high CPU spikes and Symantec scans a lot of registry entries. The filter drivers of Symantec, App Volumes and DEM all want to use their resources probably at the same time. Due to the spectre and meltdown patch the performance degradation in this combination is severe. 

                                     

                                    By accident we found out that symantec didn't work at all !!  Every thing looked fine. The client was healthy from de management server point of view, but stopping and then starting the smc.exe resulted in a crash. A simple EICAR virus test was not even detected !! Through exceptions in the snapvol.cfg we got symantec working properly.  We want to have these exceptions validated by VMware. There is a PR opened up for this.

                                     

                                    Since Symantec is working now we see better (not optimal) startup times of thinapps in an app stack . Login times unfortunately not. We declared all de collected log files to be unreliable, because the symantec client did not work at all. And so the exceptions and exclusions may not have worked at all.  We collected all log files again recently.

                                     

                                    I will update you when we hear something back from Symantec or VMware.

                                    1 2 Previous Next