We are experiencing significant reduction in file transfer rate and network speed with in our LAN between Virtual Machines running Windows 7 and Windows Server 2008 R2 guest OSs, Symantec Endpoint Protection (SEP) installed is 12.1.2015.2015.
VMs with all features of SEP installed have file transfer speed of about 30 MB/sec vs 120 MB/sec with no SEP installed.
Network speeds measured using the iperf utility shows a similar speed degradation of 4 times, 350 Mb/sec vs 1400 Mb/sec.
To simplify and exclude all extraneous factors we performed file transfer and network speed test where all VMs are hosted on the same VMware ESXi virtualization hosts (Version ESXi 5.1.0 Build 1117900). All VMs are x64 and the ethernet adapters are VMXNET 3, VMWare tools are installed and updated to the latest versions. Virtualization Host CPU usage is 20% and Memory Usage is 40% during the test. No AV scans are running during the test.
The only article I found on the subject was https://www-secure.symantec.com/connect/forums/sep-121-ru2-windows-server-2012-vm-singnificantly-red.... We already had the power setting to high performance so the solution did not help our case.
We tried enabling only the relevant features of SEP, it did not result in any significant improvement. Only installing SEP Core or unistalling SEP completely seem to be the only solution.
This seems to be a much bigger trade off between Security and Network Speed than anticipated. There must be millions of users of SEP in VMware environment and it is hard to believe that it is a common issue. If our case is unique then there should be some configuration/exclusion rule etc that can help us. Any suggestions and comments are welcome.
Hello,
We are aware of your Thread on Symantec Forums:
Could you please let us know if this file transfer issue occurying from Server to client machine or vice versa or both ways?
Secondly, could you try installing the AV/AS component only and disable the symtdi.sys driver from the machines and check if that helps.
Hello,
Could you please let us know if this file transfer issue occurying from Server to client machine or vice versa or both ways?
Secondly, could you try installing the AV/AS component only and disable the symtdi.sys driver from the machines and check if that helps.
The SEP firewall components will not protect a VMware guest operating system.
If the VMware guest operating system requires SEP protection, it must be installed directly to the VMware guest Operating System.
For Vmware Environment, check these Articles:
Guidelines for installing and running the Symantec Endpoint Protection Manager (SEPM) in a VMware image.
http://www.symantec.com/docs/TECH132456
Best Practices for Symantec Endpoint Protection in Virtual Environments
http://www.symantec.com/docs/TECH95300
Using Symantec Endpoint Protection in virtual infrastructures
http://www.symantec.com/docs/HOWTO81060
Best Practice for Symantec Endpoint Protection Scheduled Scans in VMWare
http://www.symantec.com/docs/TECH95928
SEPM: poor database performance
http://www.symantec.com/docs/TECH155046
Hope that helps!!
Mithun,
None of the articles you have mentioned are pertinent to the question I am posting:
>Guidelines for installing and running the Symantec Endpoint Protection Manager (SEPM) in a VMware image.
>http://www.symantec.com/docs/TECH132456
The SEPM is currently installed on a VM as close to given guidelines as possible. We have not encountered any issues with the administration, reporting, updates of SEPM or the managed client machines.
>Best Practices for Symantec Endpoint Protection in Virtual Environments
>http://www.symantec.com/docs/TECH95300
This article talks about Best Practices for optimizing Virus Definition Updates and Scheduled Scans. As stated in original question the SEP are not running scans when the slow network speed has been measured. The tests have been done at so many various times that the Virus updates can also be eliminated as the determining factor.
>Using Symantec Endpoint Protection in virtual infrastructures
>http://www.symantec.com/docs/HOWTO81060
This article talks about Shared Insight Cache, Virtual Image Exception Tool, and non-persistent virtual desktop infrastructure feature. I don't see how these topics are applicable to the issue at hand because, no scans are running during the test hogging the resources. Virtual Image exceptions are again meant to skip scanning the baseline image files, NOT APPLICABLE. Neither, we have non-persistent Virtual Desktop infrastructure.
>Best Practice for Symantec Endpoint Protection Scheduled Scans in VMWare
>http://www.symantec.com/docs/TECH95928
NOT APPLICABLE again as no scans are not the issue.
>SEPM: poor database performance
>http://www.symantec.com/docs/TECH155046
NOT APPLICABLE and not an issue either.
>Could you please let us know if this file transfer issue occurying from Server to client machine or vice versa or both ways?
>Secondly, could you try installing the AV/AS component only and disable the symtdi.sys driver from the machines and check if that helps.
To answer these two questions see the attached stats I have collected after HOURS of installing uninstalling SEP features. Though I have not tried it after disabling symtdi.sys driver.
Win 7 SEP | Win 7 | Win 2008 | Win 7 SEP | ||
File Transfer | Source | 37 | 18 | 65 | |
in MBps | 30 | Source | 120 | 20 | |
| 27 | 133 | Source | 20 | |
Iperf | Server | 386 | 361 | 307 | |
in Mbps | 1440 | Server | 3340 | 2365 | |
| 1401 | 3461 | Server | 1853 | |
| 249 | 329 | 388 | Server | |
Win 7 SEP Core | Win 7 | Win 2008 | Win 7 SEP Core | ||
File Transfer | Source | 97 | 210 | 165 | |
in MBps | 95 | Source | 105 | 115 | |
| 160 | 225 | Source | 195 | |
Iperf | Server | 1464 | 2539 | 1136 | |
in Mbps | 3328 | Server | 6584 | 7792 | |
| 5253 | 5908 | Server | 2713 | |
| 2283 | 2734 | 2867 | Server | |
Win 7 SEP Core | Win 7 | Win 2008 | Win 7 SEP Core | Win 7 SEP | |
File Transfer | 19 | 21 | 23 | 21 | Source |
in MBps | 33 | 30 | 34 | 39 | Destination |
Iperf | 272 | 281 | 206 | 242 | Server |
in Mbps | 1873 | 1300 | 2344 | 1781 | Client |
Win 7 SEP Core | Win 7 | Win 2008 |
| Win 7 SEP | |
File Copy with in same machine in MBps | 120 | 120 | 95 |
| 37 |
Win 2008 | Win 7 SEP Core | Installed Symantec EP Features | |
File Copy with in same machine in MBps |
| 110 | Virus, Spyware and Basic Download Protection |
File Transfer | 115 |
|
|
Iperf | 2641 | Server |
|
in Mbps | 3246 | Client |
|
Win 2008 | Win 7 SEP Core | Installed Symantec EP Features | |
File Copy with in same machine in MBps |
| 67 | Virus, Spyware and Basic Download Protection |
File Transfer | 58 |
| Advanced Download Protection |
Iperf | 964 | Server |
|
in Mbps | 2764 | Client |
|
Win 2008 | Win 7 SEP Core | Installed Symantec EP Features | |
File Copy with in same machine in MBps |
| 60 | Virus, Spyware and Basic Download Protection |
File Transfer | 50 |
| Advanced Download Protection |
Iperf | 1025 | Server | Outlook Scanner |
in Mbps | 2775 | Client |
|
Win 2008 | Win 7 SEP Core | Installed Symantec EP Features | |
File Copy with in same machine in MBps |
| 72 | Virus, Spyware and Basic Download Protection |
File Transfer | 45 |
| Advanced Download Protection |
Iperf | 625 | Server | Outlook Scanner |
in Mbps | 2119 | Client | Notes Scanner |
Win 2008 | Win 7 SEP Core | Installed Symantec EP Features | |
File Copy with in same machine in MBps |
| 66 | Virus, Spyware and Basic Download Protection |
File Transfer | 48 |
| Advanced Download Protection |
Iperf | 992 | Server | Outlook Scanner |
in Mbps | 6338 | Client | Notes Scanner |
|
|
| POP3/SMTP Scanner |
Win 2008 | Win 7 SEP Core | Installed Symantec EP Features | |
File Copy with in same machine in MBps |
| 60 | Virus, Spyware and Basic Download Protection |
File Transfer | 43 |
| Advanced Download Protection |
Iperf | 607 | Server | Outlook Scanner |
in Mbps | 5273 | Client | Notes Scanner |
| POP3/SMTP Scanner | ||
|
|
| Proactive Threat Protection |
Win 2008 | Win 7 SEP Core | Installed Symantec EP Features | |
File Copy with in same machine in MBps |
| 66 | Virus, Spyware and Basic Download Protection |
File Transfer | 46 |
| Advanced Download Protection |
Iperf | 700 | Server | Outlook Scanner |
in Mbps | 3840 | Client | Notes Scanner |
| POP3/SMTP Scanner | ||
| Proactive Threat Protection | ||
|
|
| SONAR |
Win 2008 | Win 7 SEP Core | Installed Symantec EP Features | |
File Copy with in same machine in MBps |
| 34 | Virus, Spyware and Basic Download Protection |
File Transfer | 37 |
| Advanced Download Protection |
Iperf | 772 | Server | Outlook Scanner |
in Mbps | 3336 | Client | Notes Scanner |
| POP3/SMTP Scanner | ||
| Proactive Threat Protection | ||
| SONAR | ||
|
|
| Application and Device Control |
Win 2008 | Win 7 SEP Core | Installed Symantec EP Features | |
File Copy with in same machine in MBps |
| 41 | Virus, Spyware and Basic Download Protection |
File Transfer | 35 |
| Advanced Download Protection |
Iperf | 520 | Server | Outlook Scanner |
in Mbps | 2703 | Client | Notes Scanner |
| POP3/SMTP Scanner | ||
| Proactive Threat Protection | ||
| SONAR | ||
| Application and Device Control | ||
|
|
| Network Threat Protection |
Win 2008 | Win 7 SEP Core | Installed Symantec EP Features | |
File Copy with in same machine in MBps |
| 55 | Virus, Spyware and Basic Download Protection |
File Transfer | 36 |
| Advanced Download Protection |
Iperf | 913 | Server | Outlook Scanner |
in Mbps | 2385 | Client | Notes Scanner |
| POP3/SMTP Scanner | ||
| Proactive Threat Protection | ||
| SONAR | ||
| Application and Device Control | ||
| Network Threat Protection | ||
|
|
| Intrusion Prevention |
Win 2008 | Win 7 SEP Core | Installed Symantec EP Features | |
File Copy with in same machine in MBps |
| 54 | Virus, Spyware and Basic Download Protection |
File Transfer | 32 |
| Advanced Download Protection |
Iperf | 277 | Server | Outlook Scanner |
in Mbps | 1392 | Client | Notes Scanner |
| POP3/SMTP Scanner | ||
| Proactive Threat Protection | ||
| SONAR | ||
| Application and Device Control | ||
| Network Threat Protection | ||
| Intrusion Prevention | ||
|
|
| Firewall |
>The SEP firewall components will not protect a VMware guest operating system.
I am intrigued by your above comment. You mean to say that SEP firewall component does not play any part on Windows 7 VM in VMware environment? Or do you mean to say that even with SEP firewall on VMware guest OS there are alternate ways to breach the firewall.
>If the VMware guest operating system requires SEP protection, it must be installed directly to the VMware guest Operating System.
I am in agreement on this point…
I am interested in knowing if any other user can do a simple test and confirm my findings. It is very much possible that our set up is an outlier. But another user on Symantec community did an independent test and confirmed my findings.
All comments are welcome.
Sumit
Hello,
I am having the exactly the same problem. Did you find a solution. I also installed the lates SEP version (12.1.4). No difference.
Iperf performance test results is extremely bad. When I disable network threat protection, everything is well.
Regards,
Surti
No solution yet. Surprisingly very few users seem to be bothered by the problem. Symantec made few half hearted attempts for couple of months. They seem to have given up on my ticket.
Sumit
Hello,
Your case is currently being worked by Symantec Backline and Engineering Teams.
Please get in touch with the Symantec Technical Support Team for more information.
Regards,
Mithun Sanghavi
We recently upgraded our SEP from 11.x to 12.1.4 and we are also having significantly slower network speeds when NTP is enabled. I spent most of today testing file transfers and application performance with NTP both enabled and disabled. We are using VMware with Server 2008r2 guests and Windows 7 x64 workstations.
I am having the same problem for a long time. This problem is not only in virtual machines but also in physical machines and no solution yet.
Regards,
Let's bring this thread to life!!!
I'm in the same boat with version 12.1.3001.165.
Users were complaining about transfer rate and after troubleshooting with VMware View engineer today, we came to conclusion that SEP reducing the transfer rate because when we disabled it, it was fine.
I'm glad my 3-years contract about to end next month.
Looks like it's time to jump ship if they don't do anything about it based on the a lot feedback that I've read.
Cheers.
I'm seeing the same issue with SEP 12.1.671.4971. Doesn't seem to be too much information or help from Symantec, might have to look at pushing clients away from Symantec. I'm seeing speeds less than 100Mb from Windows 7 VM to Windows 7 VM on the same host with VMXNET3 adapters.
It is really unbelievable that this problem still exists. We are thinking to leave the SEP and use a new software.
Having the same issue here on a Win7 x64 SP1 host with all different guests (Win7, Server 2k8, Linux).
Running SEP 12.1.4013.4013 and VMware workstation 9.0.3 build-1410761
Also happened on SEP 12.1.3001.165 and VMware Workstation 9.0.2 build-1031769 (and combinations of the different versions as I tried upgrading each to mitigate the problem)
Turning off SEP's firewall seems to fix it. But, depending on your policies, it'll turn itself back on after short time. I'm trying to narrow down what exactly in the SEP firewall is causing it.
What's very interesting is several of my co-workers have combinations of these versions of SEP and Workstation installed and don't have the issue. They are in the same SEP container as me, so they'll have the same policies.
I actually completely wiped my host the other day and reinstalled everything, my issue still persists. I did, however, just do a file > open on my VMs after I reinstalled Workstation. So possibly something configured with them is causing the issue? I've already tried all the suggested ideas I could find (change to vmxnet3, disable offload on the NIC)
Hello.
I have been experiencing the same problems with slow backups.
I have just replaced the vmxnet3 adapter with E1000 and the results are favourable.
Not ideal I know.
The other obvious fix was to disable Symantec NTP.
Kind Regards.
Did anyone find a resolution to this issue? I have it also. on Windows Server 2012 R2 guests, running SEP 12.1.4013.4013 on VMware ESXi 5.0 Update 3, using Intel 10Gb cards. iperf goes from about 1.4Gbps to over 6 Gbps by removing SEP on both machines. Removing it just on the iperf server client results in a bit more sporadic transfer rates of between 3Gbps and 5.2 Gbps. I'm going to log a call with Symantec but not holding my breath.
Hello,
The real question I would start to ask, is there a better way of doing A/V or if A/V is really necessary? Can you use another tool to gain the same level of protection? One that does not sit in each VM or where SEP currently sits in your environment. It can sit on the network and each VM. Not sure what your configuration is. You may also want to limit the scope of where SEP is installed. Look at your security policy and see if it is for 'ALL' systems or for 'specific' types of systems. If specific, then install only in those. Policy should mention the need for AV not the type of AV to be used.
Are there alternatives? Yes.
I would look at some of the tools around segmentation for workloads and for limiting access out of the box (sandboxing, etc.) Symantec has Data Center Protection (used to be critical system protection) which does that. It may be faster than A/V by preventing virus/malware spread and uses different algorithms to detect it. A/V can then be used to remove it.
There are other tools that live on the network, not the VMs and do A/V scans of data heading into and out of your VMs as well. Still others that have small shims in the VM.
What is absolutely needed by policy, if you can turn off features to gain that, you may be able to find what is causing the slow down as well.
Best regards,
Edward L. Haletky
VMware Communities User Moderator, VMware vExpert 2009, 2010, 2011,2012,2013,2014
Author of the books 'VMWare ESX and ESXi in the Enterprise: Planning Deployment Virtualization Servers', Copyright 2011 Pearson Education. 'VMware vSphere and Virtual Infrastructure Security: Securing the Virtual Environment', Copyright 2009 Pearson Education.
Virtualization and Cloud Security Analyst: The Virtualization Practice, LLC -- vSphere Upgrade Saga -- Virtualization Security Round Table Podcast
For us, last year I ended up switching to TrendMicro Deep Security Virtual Appliance solution that utilize vShield because I'm tired of wasting my time troubleshooting with Symantec Support.
I did try deploying the SEP utilizing vShield to fix the "slowness" issue but it wasn't as easy to manage/monitor and require pre-req steps prior to the install process so I said enough is enough.
But, it might be improved it by now...
Nothing was perfect also with TrendMicro as far as the first install/setup. They require the host to be rebooted after the installation process or after installing new updates/patches where Symantec or Sophos* doesn't require it so that was pita.
Support also wasn't so great unless you get the premium package and a few more other rants. lol
But I don't have the speed issues anymore.
*Prior to TrendMicro, I tried out Sophos and they have speed issues with Office Macros file even when utilizing the vShield technology.
FWIW..
We too are experiencing this issue. I have started a discussion on the Symantec Communities.
Slow 10Gbps network when Symantec Endpoint Client is installed | Symantec Connect
Curious if anyone else has a permanent solution that doesn't leave our operating system vulnerable. We are considering scratching Symantec and trying another solution but would like to use what we already have in place if we can. I currently have an open case with Symantec and it is in the process of being escalated.
Bottom line is with SEP installed our network throughput is slow, uninstalling Symantec clears out this issue.
Darren
No, my call got escalated several times and ended up with engineering, eventually Symantec basically said to me that this was within acceptable performance boundaries and they would not escalate any higher. Very poor show in my opinion. If it was up to me I would assess new AV vendors