Slow login/Performance App volumes with Symantec client installed

[Hmunning]

We`ve been troubleshooting slow login and poor application performance on our Non Persistent VDI for a while now. App Volumes and Symantec Endpoint Protection doesn`t seem to like each other.

Without a SEP cliënt installed everything is performing well and user experience feels like a persistent VDI. When SEP is installed including all obvious exceptions and even using the virtual image exception tool no significant change in performance is noticed. We`ve been testing all scenario`s disabling components of symantec. Only disabling "Application & Device Control" seems to improve login and application performance.

Login times :

Symantec all components on + 3 Appstacks                        2,36 Min

Symantec Application & Device Control off + 3 AppStacks  2,06 Min

Symantec all components on + 1 Appstack                          1,51 Min

Symantec Application & Device Control off + 1 Appstack    1,25 Min

No Symantec cliënt installed + 3 Appstacks                         0,50 Min

Anyone dealing the same issues? We can`t skip Symantec or move to another AV vendor. (large enterprise)

Horizon View 7.8

App Volumes 2.16 (computer assigned)

UEM 9.6

SEP RU1 14.2.3335.1000

[Hmunning]

Forgot another question : I can`t capture boot logging because the c:\ drive is refreshed after a reboot. (sort of readonly when a appstack is attached?)  Tools i`ve been trying store files during capture on c drive . Can`t change this. 

Anyone knows how i can do this?

- Process Monitor (procmon)

- Microsoft Windows Performance Toolkit- Xperf

- WPR (Windows performance recorder)

[Ray_handels]

We use an agent-less scanner (McAfee Move) but I do understand that you cannot go that route.

How many CPU's do you have? What happens if you try to add some more CPU's? You will need at least 2 vCPU's, otherwise Appvolumes and Symantec are both fighting for resources during logon.

And what exclusions do you have? Did you try adding the c:\snapvolumestemp\Mountpoints as an exclusion??

[Hmunning]

We use 2 vCPU`s and 4 GB memory. We`ve tested with 4vCPU`s and 8GB memory. No change in login performance.  

We have the following exclusions :

mdf,ndf,ldf

log,jrs,chk,edb,sdb,SHAREDTA

c:\ws\

c:\thinapps\

C:\pagefile.sys

%[USER_PROFILE]%\appdata\roaming\thinstall\

%[PROGRAM_FILES]%VMware\

%[PROGRAM_FILES]%CloudVolumes\

%[COMMON_APPDATA]%VMware UEM\FlexSync\

%[PROGRAM_FILES]%Immidio\Flex Profiles\FlexEngine.exe

%[PROGRAM_FILES]%Immidio\Flex Profiles\FlexService.exe

C:\PROGRAM FILES (X86)\BIGFIX ENTERPRISE\BES CLIENT\BESCLIENT.EXE

C:\PROGRAM FILES\VMWARE\VMWARE VIEW\AGENT\BIN\V4PA_AGENT.EXE

C:\PROGRAM FILES\IMMIDIO\FLEX PROFILES\FLEXSERVICE.EXE

C:\SnapVolumesTemp\

C:\SVROOT\

%[WINDOWS]%SoftwareDistribution\DataStore\

%[COMMON_APPDATA]%NTUser.pol

%[COMMON_APPDATA]%VMware\VDM\Logs\

%[PROGRAM_FILES]%CloudVolumes\Agent\svservice.exe

%[PROGRAM_FILES]%Immidio\Flex Profiles\Flex+ Self-Support.exe

[Ray_handels]

That should be enough looking at the exclusions. I would suggest using more then 4GB for a VDI machines but that has got nothing to do with virusscanning and slow logon times, just a best practice.

To be honest I don't now how to fix this. Would it be an option to try and run Windows Defender just as a reference?? It might be the way Symantec works maybe? You will always receive a logon penalty from virus scanners but it should not be this much.

[Anobix67]

What happens if you try user-assigned instead of computer-assigned?

We also have SEP 14.2.3335.1000 running in our environment with the same exlusions set on our SEPM servers and login times are a minute or less with a couple appstacks applied.

VMs are currently 2CPU/6GB RAM and we have machines on both Windows 10 1709 and 1809 LTSC.

Horizon 7.8

AppVol 2.17

UEM 9.8

[BenWilliamsIOM]

We are seeing the same issue here. Removing SEP altogether has shown a x5 increase in almost anything we use. Logon, AppVolumes attachment, AppV launch and cache, general usability are all dramatically improved.

Are there any additional settings over Application and Device control that can be set to improve performance?

[Scarlito]

Hi everyone.

I'm sorry to say that Hmunning, but I'm happy someone else is facing the same problem, as I was starting to become crazy and feeling alone !

I'm fighting with this issue for more than a year (almost 2 actually) on Windows 7 machines. On Windows 10, I haven't tested a lot for now as we purchased licences recently.

By the way, Hmunning, you didn't tell us what Windows version you're using.

I currently use the following versions :

- Windows 7 SP1 x64

- AppVolumes : 2.18

- Horizon View Agent 7.6

- SEP 14.2.4814.1101

- UEM (now called DEM) 9.9

Interesting fact : I notice this problem only appears after I apply Microsoft Security KB4056897 or later (and of course, with SEP agent installed and active and AppStacks mounted)

This means the problem is not only with SEP + AppVolumes, but SEP + AppVolumes + MS Updates (starting january 2018 and all the Intel security breaches fixes).

If I remove ANY ONE of these 3 elements, everything works well.

Until now, no Monthly security updates from Microsoft has solved anything.

I had a case at VMware and another one at Symantec, but none of them solved the problem.

I sent them dumps, videos of the user experience with and without the problem, logs... We had WebEx sessions, phone call, emails...

After months, we had this answer from VMware :

------------

Our backline engineers provided feedback about the procmon log files provided. They confirmed that the issue is related with a performance drop down that is making the access to registry key take longer. They could confirm that Symantec is the component that is making our driver work slow.

They suggested you contact Symantec and confirm if they have a way to solve the performance drop and/or to exclude App Volumes driver.

------------

We were about to organize a call between VMware and Symantec, but finally we closed the cases, unresolved, hoping for a better experience with Windows 10...

And now I'm working on Windows 10 migration, and I still have to test AppVolumes and SEP... And still hoping for this issue to not rise!

KB4056897

[sWORDs]

Hi Scarlito,

What are the case numbers with VMware and Symantec, the customer above is using Windows 8.1 and had similar issues with Windows 10 as far as I know.

[Scarlito]

Hi sWORDs.

​Here are the cases numbers :

​VMware : 18862278007 and 18997393111

​Symantec : 15162388

​Best regards,

​Carlos.

[nburton935]

We actually had to ditch SEP altogether because of this specific issue. We never couldn’t get them to play nicely together. We saw around 1 minute per simple AppStack when SEP was involved.

[Scarlito]

Hi nburton935.

That's what I'm fighting for, but I doubt it will be possible unfortunately.

Just out of curiosity, what did you replace it with ?

[nburton935]

It was between Endgame and Defender; we ended up going with Endgame. We also have NSX doing micro-segmentation, which can help if you're in an AV debate with InfoSec.

[Scarlito]

OK. We unfortunately don't use NSX, and it's not supposed to happen soon...

When you say you were not able to make SEP work well with AppVolumes, what were the paths you went on ? Did you get help from VMware and Symantec, or did you try to solve this by yourself ?

Are there any Support Case that we can refer to ?

[Hmunning]

Hi, everybody,

I am a little pleased to hear that we are not the only ones with this problem.

After a few months of troubleshooting we have only achieved small results. From the current support calls of Symantec and VMware no rootcause has been found yet. We have delivered many gb of logfiles and videos. Procmon, WPA,XPerf, Wireshark. You name it... 

Both Symantec and VMware indicated that no other support calls are registred with this specific problem. Well that was not true, but maybe difficult to find?  For now the only thing what is seen is there are extremely high CPU spikes and Symantec scans a lot of registry entries. The filter drivers of Symantec, App Volumes and DEM all want to use their resources probably at the same time. Due to the spectre and meltdown patch the performance degradation in this combination is severe. 

By accident we found out that symantec didn't work at all !!  Every thing looked fine. The client was healthy from de management server point of view, but stopping and then starting the smc.exe resulted in a crash. A simple EICAR virus test was not even detected !! Through exceptions in the snapvol.cfg we got symantec working properly.  We want to have these exceptions validated by VMware. There is a PR opened up for this.

Since Symantec is working now we see better (not optimal) startup times of thinapps in an app stack . Login times unfortunately not. We declared all de collected log files to be unreliable, because the symantec client did not work at all. And so the exceptions and exclusions may not have worked at all.  We collected all log files again recently.

I will update you when we hear something back from Symantec or VMware.

[P0yRaZz]

Hi Scarlito,

I was wondering what would happen if you tried to stop your SEP client with the smc.exe -stop command and then try to start it again (smc.exe -start)?

When I do this in our environment I see that the SEP client will not start up anymore.

I'm not exactly sure but I think the service was running in service.msc, but the client said something different. The only way to start the client was rebooting.

We also saw that a simple EICAR test virus was not detected even when the SEP client was running and the GUI indicating that the computer was protected. Can you try this as well?

Because of all these behavior we added some exceptions in the snapvol.cfg for the SEP client. These exceptions have solved the problem that the client could be restarted/stopped and also that a test virus was detected again.

As a result of these actions the performance improved when starting applications (30+ to 14~18 seconds) and very minimal when logging in. Maybe it reacts differently for you.

By the way, is your case closed at Symantec or is it still under investigation?

I look forward to your response.

[Scarlito]

Hi P0yRaZz.

I just did the EICAR test, and indeed, I can download the files without any warning.

I stopped and restarted SEP using smc -stop, and I have the same warning telling me that some services are stopped...

I'm curious about the exceptions you added in the snapvol.cfg !

Here are the current exclusions regarding SEP in the snapvol.cfg :

exclude_path=\ProgramData\Symantec

exclude_path=\Program Files\Symantec

exclude_path=\Program Files\Common Files\Symantec

exclude_path=\Program Files (x86)\Symantec

exclude_path=\Program Files (x86)\Common Files\Symantec

I think it's time for VMware and Symantec to start working on this issue !

[Hmunning]

Hi, scarlito,

P0yRaZz is a colleague of mine. I would like to respond to your question. We have included the following symantec exclusions in the snapvol.cfg. On top of the exclusions you already mentioned. We simply don't know what exactly is needed and that's why we've excluded everything related to symantec. This is the reason why we want it validated by VMware.

Disclaimer: I would like to warn you and everyone else that this is at your own risk. On the other hand, without these exclusions the virus scanner probably didn't work at all !

>

# Custom Exclusion Symantec Performance Issues

exclude_registry=\REGISTRY\MACHINE\SOFTWARE\Symantec

exclude_registry=\REGISTRY\MACHINE\SOFTWARE\Wow6432Node\Symantec

exclude_registry=\REGISTRY\MACHINE\SYSTEM\ControlSet001\services\BHDrvx64

exclude_registry=\REGISTRY\MACHINE\SYSTEM\ControlSet001\services\eeCtrl

exclude_registry=\REGISTRY\MACHINE\SYSTEM\ControlSet001\services\EraserUtilRebootDrv

exclude_registry=\REGISTRY\MACHINE\SYSTEM\ControlSet001\services\IDSVia64

exclude_registry=\REGISTRY\MACHINE\SYSTEM\ControlSet001\services\SepMasterService

exclude_registry=\REGISTRY\MACHINE\SYSTEM\ControlSet001\services\SNAC

exclude_registry=\REGISTRY\MACHINE\SYSTEM\ControlSet001\services\SRTSP

exclude_registry=\REGISTRY\MACHINE\SYSTEM\ControlSet001\services\SRTSPX

exclude_registry=\REGISTRY\MACHINE\SYSTEM\ControlSet001\services\SyDvCrtl

exclude_registry=\REGISTRY\MACHINE\SYSTEM\ControlSet001\services\SymEFASI

exclude_registry=\REGISTRY\MACHINE\SYSTEM\ControlSet001\services\SymELAM

exclude_registry=\REGISTRY\MACHINE\SYSTEM\ControlSet001\services\SymEvent

exclude_registry=\REGISTRY\MACHINE\SYSTEM\ControlSet001\services\SymIRON

exclude_registry=\REGISTRY\MACHINE\SYSTEM\ControlSet001\services\SYMNETS

exclude_registry=\REGISTRY\MACHINE\SYSTEM\ControlSet001\services\SysMain

exclude_registry=\REGISTRY\MACHINE\SYSTEM\ControlSet001\services\SysPlant

exclude_registry=\REGISTRY\MACHINE\SYSTEM\ControlSet001\services\Teefer2

exclude_registry=\REGISTRY\MACHINE\SYSTEM\ControlSet001\services\Eventlog\Application\Symantec Antivirus

exclude_registry=\REGISTRY\MACHINE\SYSTEM\ControlSet001\services\Eventlog\Application\Symantec Endpoint Protection

exclude_registry=\REGISTRY\MACHINE\SYSTEM\ControlSet001\services\Eventlog\Application\Symantec Network Protection

exclude_registry=\REGISTRY\MACHINE\SYSTEM\ControlSet001\services\Eventlog\Application\Symantec WSS Traffic Redirection

exclude_registry=\REGISTRY\MACHINE\SYSTEM\ControlSet001\services\Eventlog\Symantec Endpoint Protection Client

exclude_path=\Program Files\Common Files\Symantec Shared

exclude_path=\Program Files (x86)\Common Files\Symantec Shared

exclude_process_name=ccSvcHst.exe

exclude_process_name=SmcGui.exe

exclude_process_name=SISIDSService.exe

exclude_process_name=SISIPSService.exe

exclude_process_name=SISIPSUtil.exe

exclude_process_name=sepWscSvc64.exe

[Scarlito]

Hi Hmunning.

Thanks a lot for your reply.

I'm gonna test this on our test desktop pools and let you know.

[Scarlito]

Hello everyone.

I've made a few tests, but unfortunately, it seems that the exclusions don't help so much in terms of performance. I also had my VMs getting stuck at login when mounting several AppStacks (it did this with 3 stacks, not with 2, and I didn't try with more than 3 stacks).

My security colleague re-opened the case at Symantec. Maybe if some of you have a case number to share so we can ask them to link those cases, it might help.

Regards,

Carlos.