1 2 Previous Next 17 Replies Latest reply on Sep 21, 2015 9:17 AM by JoJoGabor

    Symantec Endpoint Protection slows file transfer and network speed by fourth in VMware environment

    Sumit975 Lurker

      We are experiencing significant reduction in file transfer rate and network speed with in our LAN between Virtual Machines running Windows 7 and Windows Server 2008 R2 guest OSs, Symantec Endpoint Protection (SEP) installed is 12.1.2015.2015.

       

      VMs with all features of SEP installed have file transfer speed of about 30 MB/sec vs 120 MB/sec with no SEP installed.

      Network speeds measured using the iperf utility shows a similar speed degradation of 4 times, 350 Mb/sec vs 1400 Mb/sec.

       

      To simplify and exclude all extraneous factors we performed file transfer and network speed test where all VMs are hosted on the same VMware ESXi virtualization hosts (Version ESXi 5.1.0 Build 1117900). All VMs are x64 and the ethernet adapters are VMXNET 3, VMWare tools are installed and updated to the latest versions. Virtualization Host CPU usage is 20% and Memory Usage is 40% during the test. No AV scans are running during the test.

       

      The only article I found on the subject was https://www-secure.symantec.com/connect/forums/sep-121-ru2-windows-server-2012-vm-singnificantly-reduced-performance. We already had the power setting to high performance so the solution did not help our case.

       

      We tried enabling only the relevant features of SEP, it did not result in any significant improvement. Only installing SEP Core or unistalling SEP completely seem to be the only solution.

       

      This seems to be a much bigger trade off between Security and Network Speed than anticipated. There must be millions of users of SEP in VMware environment and it is hard to believe that it is a common issue. If our case is unique then there should be some configuration/exclusion rule etc that can help us. Any suggestions and comments are welcome.

        • 1. Re: Symantec Endpoint Protection slows file transfer and network speed by fourth in VMware environment
          mithunsanghavi Novice

          Hello,

          We are aware of your Thread on Symantec Forums:

          https://www-secure.symantec.com/connect/forums/symantec-endpoint-protection-slows-file-transfer-and-network-speed-vms-vmware-environment-fac

          Could you please let us know if this file transfer issue occurying from Server to client machine or vice versa or both ways?

          Secondly, could you try installing the AV/AS component only and disable the symtdi.sys driver from the machines and check if that helps.

          Hello,

          Could you please let us know if this file transfer issue occurying from Server to client machine or vice versa or both ways?

          Secondly, could you try installing the AV/AS component only and disable the symtdi.sys driver from the machines and check if that helps.

          The SEP firewall components will not protect a VMware guest operating system.

          If the VMware guest operating system requires SEP protection, it must be installed directly to the VMware guest Operating System.

          For Vmware Environment, check these Articles:

          Guidelines for installing and running the Symantec Endpoint Protection Manager (SEPM) in a VMware image.

          http://www.symantec.com/docs/TECH132456

          Best Practices for Symantec Endpoint Protection in Virtual Environments

          http://www.symantec.com/docs/TECH95300

          Using Symantec Endpoint Protection in virtual infrastructures

          http://www.symantec.com/docs/HOWTO81060

          Best Practice for Symantec Endpoint Protection Scheduled Scans in VMWare

          http://www.symantec.com/docs/TECH95928

          SEPM: poor database performance

          http://www.symantec.com/docs/TECH155046

          Hope that helps!!

          • 2. Re: Symantec Endpoint Protection slows file transfer and network speed by fourth in VMware environment
            Sumit975 Lurker

            Mithun,

            None of the articles you have mentioned are pertinent to the question I am posting:

            >Guidelines for installing and running the Symantec Endpoint Protection Manager (SEPM) in a VMware image.

            >http://www.symantec.com/docs/TECH132456

            The SEPM is currently installed on a VM as close to given guidelines as possible. We have not encountered any issues with the administration, reporting, updates of SEPM or the managed client machines.

            >Best Practices for Symantec Endpoint Protection in Virtual Environments

            >http://www.symantec.com/docs/TECH95300

            This article talks about Best Practices for optimizing Virus Definition Updates and Scheduled Scans. As stated in original question the SEP are not running scans when the slow network speed has been measured. The tests have been done at so many various times that the Virus updates can also be eliminated as the determining factor.

            >Using Symantec Endpoint Protection in virtual infrastructures

            >http://www.symantec.com/docs/HOWTO81060

            This article talks about Shared Insight Cache, Virtual Image Exception Tool, and non-persistent virtual desktop infrastructure feature. I don't see how these topics are applicable to the issue at hand because, no scans are running during the test hogging the resources. Virtual Image exceptions are again meant to skip scanning the baseline image files, NOT APPLICABLE. Neither, we have non-persistent Virtual Desktop infrastructure.

            >Best Practice for Symantec Endpoint Protection Scheduled Scans in VMWare

            >http://www.symantec.com/docs/TECH95928

            NOT APPLICABLE again as no scans are not the issue.

            >SEPM: poor database performance

            >http://www.symantec.com/docs/TECH155046

            NOT APPLICABLE and not an issue either.

            >Could you please let us know if this file transfer issue occurying from Server to client machine or vice versa or both ways?

            >Secondly, could you try installing the AV/AS component only and disable the symtdi.sys driver from the machines and check if that helps.

            To answer these two questions see the attached stats I have collected after HOURS of installing uninstalling SEP features. Though I have not tried it after disabling symtdi.sys driver.

                                                                                                                                                                             

            Win 7 SEP

            Win 7

            Win 2008

            Win 7 SEP

            File Transfer

            Source

            37

            18

            65

            in MBps

            30

            Source

            120

            20

             

            27

            133

            Source

            20

            Iperf

            Server

            386

            361

            307

            in Mbps

            1440

            Server

            3340

            2365

             

            1401

            3461

            Server

            1853

             

            249

            329

            388

            Server

            Win 7 SEP Core

            Win 7

            Win 2008

            Win 7 SEP Core

            File Transfer

            Source

            97

            210

            165

            in MBps

            95

            Source

            105

            115

             

            160

            225

            Source

            195

            Iperf

            Server

            1464

            2539

            1136

            in Mbps

            3328

            Server

            6584

            7792

             

            5253

            5908

            Server

            2713

             

            2283

            2734

            2867

            Server

            Win 7 SEP Core

            Win 7

            Win 2008

            Win 7 SEP Core

            Win 7 SEP

            File Transfer

            19

            21

            23

            21

            Source

            in MBps

            33

            30

            34

            39

            Destination

            Iperf

            272

            281

            206

            242

            Server

            in Mbps

            1873

            1300

            2344

            1781

            Client

            Win 7 SEP Core

            Win 7

            Win 2008

             

            Win 7 SEP

            File Copy with in same machine in MBps

            120

            120

            95

             

            37

             

             

                                                                                                                                                                                                                                                                                                                                                                                                       

            Win 2008

            Win 7 SEP Core

            Installed Symantec EP Features

            File Copy with in same machine in MBps

             

            110

            Virus, Spyware and Basic Download   Protection

            File Transfer

            115

             

             

            Iperf

            2641

            Server

             

            in Mbps

            3246

            Client

             

            Win 2008

            Win 7 SEP Core

            Installed Symantec EP Features

            File Copy with in same machine in MBps

             

            67

            Virus, Spyware and Basic Download   Protection

            File Transfer

            58

             

            Advanced Download Protection

            Iperf

            964

            Server

             

            in Mbps

            2764

            Client

             

            Win 2008

            Win 7 SEP Core

            Installed Symantec EP Features

            File Copy with in same machine in MBps

             

            60

            Virus, Spyware and Basic Download   Protection

            File Transfer

            50

             

            Advanced Download Protection

            Iperf

            1025

            Server

            Outlook Scanner

            in Mbps

            2775

            Client

             

            Win 2008

            Win 7 SEP Core

            Installed Symantec EP Features

            File Copy with in same machine in MBps

             

            72

            Virus, Spyware and Basic Download   Protection

            File Transfer

            45

             

            Advanced Download Protection

            Iperf

            625

            Server

            Outlook Scanner

            in Mbps

            2119

            Client

            Notes Scanner

            Win 2008

            Win 7 SEP Core

            Installed Symantec EP Features

            File Copy with in same machine in MBps

             

            66

            Virus, Spyware and Basic Download   Protection

            File Transfer

            48

             

            Advanced Download Protection

            Iperf

            992

            Server

            Outlook Scanner

            in Mbps

            6338

            Client

            Notes Scanner

             

             

             

            POP3/SMTP Scanner

            Win 2008

            Win 7 SEP Core

            Installed Symantec EP Features

            File Copy with in same machine in MBps

             

            60

            Virus, Spyware and Basic Download   Protection

            File Transfer

            43

             

            Advanced Download Protection

            Iperf

            607

            Server

            Outlook Scanner

            in Mbps

            5273

            Client

            Notes Scanner

             

            POP3/SMTP Scanner

             

             

             

            Proactive Threat Protection

            Win 2008

            Win 7 SEP Core

            Installed Symantec EP Features

            File Copy with in same machine in MBps

             

            66

            Virus, Spyware and Basic Download   Protection

            File Transfer

            46

             

            Advanced Download Protection

            Iperf

            700

            Server

            Outlook Scanner

            in Mbps

            3840

            Client

            Notes Scanner

             

            POP3/SMTP Scanner

             

            Proactive Threat Protection

             

             

             

            SONAR

            Win 2008

            Win 7 SEP Core

            Installed Symantec EP Features

            File Copy with in same machine in MBps

             

            34

            Virus, Spyware and Basic Download   Protection

            File Transfer

            37

             

            Advanced Download Protection

            Iperf

            772

            Server

            Outlook Scanner

            in Mbps

            3336

            Client

            Notes Scanner

             

            POP3/SMTP Scanner

             

            Proactive Threat Protection

             

            SONAR

             

             

             

            Application and Device Control

            Win 2008

            Win 7 SEP Core

            Installed Symantec EP Features

            File Copy with in same machine in MBps

             

            41

            Virus, Spyware and Basic Download   Protection

            File Transfer

            35

             

            Advanced Download Protection

            Iperf

            520

            Server

            Outlook Scanner

            in Mbps

            2703

            Client

            Notes Scanner

             

            POP3/SMTP Scanner

             

            Proactive Threat Protection

             

            SONAR

             

            Application and Device Control

             

             

             

            Network Threat Protection

            Win 2008

            Win 7 SEP Core

            Installed Symantec EP Features

            File Copy with in same machine in MBps

             

            55

            Virus, Spyware and Basic Download   Protection

            File Transfer

            36

             

            Advanced Download Protection

            Iperf

            913

            Server

            Outlook Scanner

            in Mbps

            2385

            Client

            Notes Scanner

             

            POP3/SMTP Scanner

             

            Proactive Threat Protection

             

            SONAR

             

            Application and Device Control

             

            Network Threat Protection

             

             

             

            Intrusion Prevention

            Win 2008

            Win 7 SEP Core

            Installed Symantec EP Features

            File Copy with in same machine in MBps

             

            54

            Virus, Spyware and Basic Download   Protection

            File Transfer

            32

             

            Advanced Download Protection

            Iperf

            277

            Server

            Outlook Scanner

            in Mbps

            1392

            Client

            Notes Scanner

             

            POP3/SMTP Scanner

             

            Proactive Threat Protection

             

            SONAR

             

            Application and Device Control

             

            Network Threat Protection

             

            Intrusion Prevention

             

             

             

            Firewall

             

             

            >The SEP firewall components will not protect a VMware guest operating system.

            I am intrigued by your above comment. You mean to say that SEP firewall component does not play any part on Windows 7 VM in VMware environment? Or do you mean to say that even with SEP firewall on VMware guest OS there are alternate ways to breach the firewall.

            >If the VMware guest operating system requires SEP protection, it must be installed directly to the VMware guest Operating System.

            I am in agreement on this point…

             

            I am interested in knowing if any other user can do a simple test and confirm my findings. It is very much possible that our set up is an outlier. But another user on Symantec community did an independent test and confirmed my findings.

            http://www.symantec.com/connect/forums/symantec-endpoint-protection-slows-file-transfer-and-network-speed-vms-vmware-environment-fac#comment-9035671

            All comments are welcome.

            Sumit

            • 3. Re: Symantec Endpoint Protection slows file transfer and network speed by fourth in VMware environment
              Surti Lurker

              Hello,

              I am having the exactly the same problem. Did you find a solution. I also installed the lates SEP version (12.1.4). No difference.

              Iperf performance test results is extremely bad. When I disable network threat protection, everything is well.

               

              Regards,

              Surti

              • 4. Re: Symantec Endpoint Protection slows file transfer and network speed by fourth in VMware environment
                Sumit975 Lurker

                No solution yet. Surprisingly very few users seem to be bothered by the problem. Symantec made few half hearted attempts for couple of months. They seem to have given up on my ticket.

                 

                Sumit

                • 5. Re: Symantec Endpoint Protection slows file transfer and network speed by fourth in VMware environment
                  mithunsanghavi Novice

                  Hello,

                   

                  Your case is currently being worked by Symantec Backline and Engineering Teams.

                   

                  Please get in touch with the Symantec Technical Support Team for more information.

                   

                  Regards,

                   

                  Mithun Sanghavi

                  • 6. Re: Symantec Endpoint Protection slows file transfer and network speed by fourth in VMware environment
                    ScottMSC Lurker

                    We recently upgraded our SEP from 11.x to 12.1.4 and we are also having significantly slower network speeds when NTP is enabled. I spent most of today testing file transfers and application performance with NTP both enabled and disabled. We are using VMware with Server 2008r2 guests and Windows 7 x64 workstations.

                    • 7. Re: Symantec Endpoint Protection slows file transfer and network speed by fourth in VMware environment
                      Surti Lurker

                      I am having the same problem for a long time. This problem is not only in virtual machines but also in physical machines and no solution yet.

                       

                      Regards,

                      • 8. Re: Symantec Endpoint Protection slows file transfer and network speed by fourth in VMware environment
                        iamxCPx Enthusiast

                        Let's bring this thread to life!!!

                         

                        I'm in the same boat with version 12.1.3001.165.

                        Users were complaining about transfer rate and after troubleshooting with VMware View engineer today, we came to conclusion that SEP reducing the transfer rate because when we disabled it, it was fine.

                         

                        I'm glad my 3-years contract about to end next month.

                        Looks like it's time to jump ship if they don't do anything about it based on the a lot feedback that I've read.

                         

                        Cheers.

                        • 9. Re: Symantec Endpoint Protection slows file transfer and network speed by fourth in VMware environment
                          sbridle Lurker

                          I'm seeing the same issue with SEP 12.1.671.4971. Doesn't seem to be too much information or help from Symantec, might have to look at pushing clients away from Symantec. I'm seeing speeds less than 100Mb from Windows 7 VM to Windows 7 VM on the same host with VMXNET3 adapters.

                          • 10. Re: Symantec Endpoint Protection slows file transfer and network speed by fourth in VMware environment
                            Surti Lurker

                            It is really unbelievable that this problem still exists. We are thinking to leave the SEP and use a new software.

                            • 11. Re: Symantec Endpoint Protection slows file transfer and network speed by fourth in VMware environment
                              invisiblekid Lurker

                              Having the same issue here on a Win7 x64 SP1 host with all different guests (Win7, Server 2k8, Linux).

                              Running SEP 12.1.4013.4013 and VMware workstation 9.0.3 build-1410761

                              Also happened on SEP 12.1.3001.165 and VMware Workstation 9.0.2 build-1031769 (and combinations of the different versions as I tried upgrading each to mitigate the problem)

                               

                              Turning off SEP's firewall seems to fix it. But, depending on your policies, it'll turn itself back on after short time. I'm trying to narrow down what exactly in the SEP firewall is causing it.

                               

                              What's very interesting is several of my co-workers have combinations of these versions of SEP and Workstation installed and don't have the issue. They are in the same SEP container as me, so they'll have the same policies.

                               

                              I actually completely wiped my host the other day and reinstalled everything, my issue still persists. I did, however, just do a file > open on my VMs after I reinstalled Workstation. So possibly something configured with them is causing the issue? I've already tried all the suggested ideas I could find (change to vmxnet3, disable offload on the NIC)

                              • 12. Re: Symantec Endpoint Protection slows file transfer and network speed by fourth in VMware environment
                                apmorey Lurker

                                Hello.

                                 

                                I have been experiencing the same problems with slow backups.

                                I have just replaced the vmxnet3 adapter with E1000 and the results are favourable.

                                 

                                Not ideal I know.

                                The other obvious fix was to disable Symantec NTP.

                                 

                                Kind Regards.

                                • 13. Re: Symantec Endpoint Protection slows file transfer and network speed by fourth in VMware environment
                                  JoJoGabor Expert

                                  Did anyone find a resolution to this issue? I have it also. on Windows Server 2012 R2 guests, running SEP 12.1.4013.4013 on VMware ESXi 5.0 Update 3, using Intel 10Gb cards. iperf goes from about 1.4Gbps to over 6 Gbps by removing SEP on both machines. Removing it just on the iperf server client results in a bit more sporadic transfer rates of between 3Gbps and 5.2 Gbps. I'm going to log a call with Symantec but not holding my breath.

                                  • 14. Re: Symantec Endpoint Protection slows file transfer and network speed by fourth in VMware environment
                                    Texiwill Guru
                                    User ModeratorsvExpert

                                    Hello,

                                     

                                    The real question I would start to ask, is there a better way of doing A/V or if A/V is really necessary? Can you use another tool to gain the same level of protection? One that does not sit in each VM or where SEP currently sits in your environment. It can sit on the network and each VM. Not sure what your configuration is. You may also want to limit the scope of where SEP is installed. Look at your security policy and see if it is for 'ALL' systems or for 'specific' types of systems. If specific, then install only in those. Policy should mention the need for AV not the type of AV to be used.

                                     

                                    Are there alternatives? Yes.

                                     

                                    I would look at some of the tools around segmentation for workloads and for limiting access out of the box (sandboxing, etc.) Symantec has Data Center Protection (used to be critical system protection) which does that. It may be faster than A/V by preventing virus/malware spread and uses different algorithms to detect it. A/V can then be used to remove it.

                                     

                                    There are other tools that live on the network, not the VMs and do A/V scans of data heading into and out of your VMs as well. Still others that have small shims in the VM.

                                     

                                    What is absolutely needed by policy, if you can turn off features to gain that, you may be able to find what is causing the slow down as well.

                                     

                                    Best regards,
                                    Edward L. Haletky
                                    VMware Communities User Moderator, VMware vExpert 2009, 2010, 2011,2012,2013,2014

                                    Author of the books 'VMWare ESX and ESXi in the Enterprise: Planning Deployment Virtualization Servers', Copyright 2011 Pearson Education. 'VMware vSphere and Virtual Infrastructure Security: Securing the Virtual Environment', Copyright 2009 Pearson Education.

                                    Virtualization and Cloud Security Analyst: The Virtualization Practice, LLC -- vSphere Upgrade Saga -- Virtualization Security Round Table Podcast

                                    1 2 Previous Next