VMware Cloud Community
Sumit975
Contributor
Contributor

Symantec Endpoint Protection slows file transfer and network speed by fourth in VMware environment

We are experiencing significant reduction in file transfer rate and network speed with in our LAN between Virtual Machines running Windows 7 and Windows Server 2008 R2 guest OSs, Symantec Endpoint Protection (SEP) installed is 12.1.2015.2015.

VMs with all features of SEP installed have file transfer speed of about 30 MB/sec vs 120 MB/sec with no SEP installed.

Network speeds measured using the iperf utility shows a similar speed degradation of 4 times, 350 Mb/sec vs 1400 Mb/sec.

To simplify and exclude all extraneous factors we performed file transfer and network speed test where all VMs are hosted on the same VMware ESXi virtualization hosts (Version ESXi 5.1.0 Build 1117900). All VMs are x64 and the ethernet adapters are VMXNET 3, VMWare tools are installed and updated to the latest versions. Virtualization Host CPU usage is 20% and Memory Usage is 40% during the test. No AV scans are running during the test.

The only article I found on the subject was https://www-secure.symantec.com/connect/forums/sep-121-ru2-windows-server-2012-vm-singnificantly-red.... We already had the power setting to high performance so the solution did not help our case.

We tried enabling only the relevant features of SEP, it did not result in any significant improvement. Only installing SEP Core or unistalling SEP completely seem to be the only solution.

This seems to be a much bigger trade off between Security and Network Speed than anticipated. There must be millions of users of SEP in VMware environment and it is hard to believe that it is a common issue. If our case is unique then there should be some configuration/exclusion rule etc that can help us. Any suggestions and comments are welcome.

Reply
0 Kudos
17 Replies
mithunsanghavi
Contributor
Contributor

Hello,

We are aware of your Thread on Symantec Forums:

https://www-secure.symantec.com/connect/forums/symantec-endpoint-protection-slows-file-transfer-and-...

Could you please let us know if this file transfer issue occurying from Server to client machine or vice versa or both ways?

Secondly, could you try installing the AV/AS component only and disable the symtdi.sys driver from the machines and check if that helps.

Hello,

Could you please let us know if this file transfer issue occurying from Server to client machine or vice versa or both ways?

Secondly, could you try installing the AV/AS component only and disable the symtdi.sys driver from the machines and check if that helps.

The SEP firewall components will not protect a VMware guest operating system.

If the VMware guest operating system requires SEP protection, it must be installed directly to the VMware guest Operating System.

For Vmware Environment, check these Articles:

Guidelines for installing and running the Symantec Endpoint Protection Manager (SEPM) in a VMware image.

http://www.symantec.com/docs/TECH132456

Best Practices for Symantec Endpoint Protection in Virtual Environments

http://www.symantec.com/docs/TECH95300

Using Symantec Endpoint Protection in virtual infrastructures

http://www.symantec.com/docs/HOWTO81060

Best Practice for Symantec Endpoint Protection Scheduled Scans in VMWare

http://www.symantec.com/docs/TECH95928

SEPM: poor database performance

http://www.symantec.com/docs/TECH155046

Hope that helps!!

Reply
0 Kudos
Sumit975
Contributor
Contributor

Mithun,

None of the articles you have mentioned are pertinent to the question I am posting:

>Guidelines for installing and running the Symantec Endpoint Protection Manager (SEPM) in a VMware image.

>http://www.symantec.com/docs/TECH132456

The SEPM is currently installed on a VM as close to given guidelines as possible. We have not encountered any issues with the administration, reporting, updates of SEPM or the managed client machines.

>Best Practices for Symantec Endpoint Protection in Virtual Environments

>http://www.symantec.com/docs/TECH95300

This article talks about Best Practices for optimizing Virus Definition Updates and Scheduled Scans. As stated in original question the SEP are not running scans when the slow network speed has been measured. The tests have been done at so many various times that the Virus updates can also be eliminated as the determining factor.

>Using Symantec Endpoint Protection in virtual infrastructures

>http://www.symantec.com/docs/HOWTO81060

This article talks about Shared Insight Cache, Virtual Image Exception Tool, and non-persistent virtual desktop infrastructure feature. I don't see how these topics are applicable to the issue at hand because, no scans are running during the test hogging the resources. Virtual Image exceptions are again meant to skip scanning the baseline image files, NOT APPLICABLE. Neither, we have non-persistent Virtual Desktop infrastructure.

>Best Practice for Symantec Endpoint Protection Scheduled Scans in VMWare

>http://www.symantec.com/docs/TECH95928

NOT APPLICABLE again as no scans are not the issue.

>SEPM: poor database performance

>http://www.symantec.com/docs/TECH155046

NOT APPLICABLE and not an issue either.

>Could you please let us know if this file transfer issue occurying from Server to client machine or vice versa or both ways?

>Secondly, could you try installing the AV/AS component only and disable the symtdi.sys driver from the machines and check if that helps.

To answer these two questions see the attached stats I have collected after HOURS of installing uninstalling SEP features. Though I have not tried it after disabling symtdi.sys driver.

                                                                                                                                                                 

Win 7 SEP

Win 7

Win 2008

Win 7 SEP

File Transfer

Source

37

18

65

in MBps

30

Source

120

20

27

133

Source

20

Iperf

Server

386

361

307

in Mbps

1440

Server

3340

2365

1401

3461

Server

1853

249

329

388

Server

Win 7 SEP Core

Win 7

Win 2008

Win 7 SEP Core

File Transfer

Source

97

210

165

in MBps

95

Source

105

115

160

225

Source

195

Iperf

Server

1464

2539

1136

in Mbps

3328

Server

6584

7792

5253

5908

Server

2713

2283

2734

2867

Server

Win 7 SEP Core

Win 7

Win 2008

Win 7 SEP Core

Win 7 SEP

File Transfer

19

21

23

21

Source

in MBps

33

30

34

39

Destination

Iperf

272

281

206

242

Server

in Mbps

1873

1300

2344

1781

Client

Win 7 SEP Core

Win 7

Win 2008

Win 7 SEP

File Copy with in same machine in MBps

120

120

95

37

                                                                                                                                                                                                                                                                                                                                                                                           

Win 2008

Win 7 SEP Core

Installed Symantec EP Features

File Copy with in same machine in MBps

110

Virus, Spyware and Basic Download   Protection

File Transfer

115

Iperf

2641

Server

in Mbps

3246

Client

Win 2008

Win 7 SEP Core

Installed Symantec EP Features

File Copy with in same machine in MBps

67

Virus, Spyware and Basic Download   Protection

File Transfer

58

Advanced Download Protection

Iperf

964

Server

in Mbps

2764

Client

Win 2008

Win 7 SEP Core

Installed Symantec EP Features

File Copy with in same machine in MBps

60

Virus, Spyware and Basic Download   Protection

File Transfer

50

Advanced Download Protection

Iperf

1025

Server

Outlook Scanner

in Mbps

2775

Client

Win 2008

Win 7 SEP Core

Installed Symantec EP Features

File Copy with in same machine in MBps

72

Virus, Spyware and Basic Download   Protection

File Transfer

45

Advanced Download Protection

Iperf

625

Server

Outlook Scanner

in Mbps

2119

Client

Notes Scanner

Win 2008

Win 7 SEP Core

Installed Symantec EP Features

File Copy with in same machine in MBps

66

Virus, Spyware and Basic Download   Protection

File Transfer

48

Advanced Download Protection

Iperf

992

Server

Outlook Scanner

in Mbps

6338

Client

Notes Scanner

POP3/SMTP Scanner

Win 2008

Win 7 SEP Core

Installed Symantec EP Features

File Copy with in same machine in MBps

60

Virus, Spyware and Basic Download   Protection

File Transfer

43

Advanced Download Protection

Iperf

607

Server

Outlook Scanner

in Mbps

5273

Client

Notes Scanner

POP3/SMTP Scanner

Proactive Threat Protection

Win 2008

Win 7 SEP Core

Installed Symantec EP Features

File Copy with in same machine in MBps

66

Virus, Spyware and Basic Download   Protection

File Transfer

46

Advanced Download Protection

Iperf

700

Server

Outlook Scanner

in Mbps

3840

Client

Notes Scanner

POP3/SMTP Scanner

Proactive Threat Protection

SONAR

Win 2008

Win 7 SEP Core

Installed Symantec EP Features

File Copy with in same machine in MBps

34

Virus, Spyware and Basic Download   Protection

File Transfer

37

Advanced Download Protection

Iperf

772

Server

Outlook Scanner

in Mbps

3336

Client

Notes Scanner

POP3/SMTP Scanner

Proactive Threat Protection

SONAR

Application and Device Control

Win 2008

Win 7 SEP Core

Installed Symantec EP Features

File Copy with in same machine in MBps

41

Virus, Spyware and Basic Download   Protection

File Transfer

35

Advanced Download Protection

Iperf

520

Server

Outlook Scanner

in Mbps

2703

Client

Notes Scanner

POP3/SMTP Scanner

Proactive Threat Protection

SONAR

Application and Device Control

Network Threat Protection

Win 2008

Win 7 SEP Core

Installed Symantec EP Features

File Copy with in same machine in MBps

55

Virus, Spyware and Basic Download   Protection

File Transfer

36

Advanced Download Protection

Iperf

913

Server

Outlook Scanner

in Mbps

2385

Client

Notes Scanner

POP3/SMTP Scanner

Proactive Threat Protection

SONAR

Application and Device Control

Network Threat Protection

Intrusion Prevention

Win 2008

Win 7 SEP Core

Installed Symantec EP Features

File Copy with in same machine in MBps

54

Virus, Spyware and Basic Download   Protection

File Transfer

32

Advanced Download Protection

Iperf

277

Server

Outlook Scanner

in Mbps

1392

Client

Notes Scanner

POP3/SMTP Scanner

Proactive Threat Protection

SONAR

Application and Device Control

Network Threat Protection

Intrusion Prevention

Firewall

>The SEP firewall components will not protect a VMware guest operating system.

I am intrigued by your above comment. You mean to say that SEP firewall component does not play any part on Windows 7 VM in VMware environment? Or do you mean to say that even with SEP firewall on VMware guest OS there are alternate ways to breach the firewall.

>If the VMware guest operating system requires SEP protection, it must be installed directly to the VMware guest Operating System.

I am in agreement on this point…

I am interested in knowing if any other user can do a simple test and confirm my findings. It is very much possible that our set up is an outlier. But another user on Symantec community did an independent test and confirmed my findings.

http://www.symantec.com/connect/forums/symantec-endpoint-protection-slows-file-transfer-and-network-...

All comments are welcome.

Sumit

Reply
0 Kudos
Surti
Contributor
Contributor

Hello,

I am having the exactly the same problem. Did you find a solution. I also installed the lates SEP version (12.1.4). No difference.

Iperf performance test results is extremely bad. When I disable network threat protection, everything is well.

Regards,

Surti

Reply
0 Kudos
Sumit975
Contributor
Contributor

No solution yet. Surprisingly very few users seem to be bothered by the problem. Symantec made few half hearted attempts for couple of months. They seem to have given up on my ticket.

Sumit

Reply
0 Kudos
mithunsanghavi
Contributor
Contributor

Hello,

Your case is currently being worked by Symantec Backline and Engineering Teams.

Please get in touch with the Symantec Technical Support Team for more information.

Regards,

Mithun Sanghavi

Reply
0 Kudos
ScottMSC
Contributor
Contributor

We recently upgraded our SEP from 11.x to 12.1.4 and we are also having significantly slower network speeds when NTP is enabled. I spent most of today testing file transfers and application performance with NTP both enabled and disabled. We are using VMware with Server 2008r2 guests and Windows 7 x64 workstations.

Reply
0 Kudos
Surti
Contributor
Contributor

I am having the same problem for a long time. This problem is not only in virtual machines but also in physical machines and no solution yet.

Regards,

Reply
0 Kudos
iamxCPx
Enthusiast
Enthusiast

Let's bring this thread to life!!!

I'm in the same boat with version 12.1.3001.165.

Users were complaining about transfer rate and after troubleshooting with VMware View engineer today, we came to conclusion that SEP reducing the transfer rate because when we disabled it, it was fine.

I'm glad my 3-years contract about to end next month. Smiley Happy

Looks like it's time to jump ship if they don't do anything about it based on the a lot feedback that I've read.

Cheers.

Reply
0 Kudos
sbridle
Contributor
Contributor

I'm seeing the same issue with SEP 12.1.671.4971. Doesn't seem to be too much information or help from Symantec, might have to look at pushing clients away from Symantec. I'm seeing speeds less than 100Mb from Windows 7 VM to Windows 7 VM on the same host with VMXNET3 adapters.

Reply
0 Kudos
Surti
Contributor
Contributor

It is really unbelievable that this problem still exists. We are thinking to leave the SEP and use a new software.

Reply
0 Kudos
invisiblekid
Contributor
Contributor

Having the same issue here on a Win7 x64 SP1 host with all different guests (Win7, Server 2k8, Linux).

Running SEP 12.1.4013.4013 and VMware workstation 9.0.3 build-1410761

Also happened on SEP 12.1.3001.165 and VMware Workstation 9.0.2 build-1031769 (and combinations of the different versions as I tried upgrading each to mitigate the problem)

Turning off SEP's firewall seems to fix it. But, depending on your policies, it'll turn itself back on after short time. I'm trying to narrow down what exactly in the SEP firewall is causing it.

What's very interesting is several of my co-workers have combinations of these versions of SEP and Workstation installed and don't have the issue. They are in the same SEP container as me, so they'll have the same policies.

I actually completely wiped my host the other day and reinstalled everything, my issue still persists. I did, however, just do a file > open on my VMs after I reinstalled Workstation. So possibly something configured with them is causing the issue? I've already tried all the suggested ideas I could find (change to vmxnet3, disable offload on the NIC)

Reply
0 Kudos
apmorey
Contributor
Contributor

Hello.

I have been experiencing the same problems with slow backups.

I have just replaced the vmxnet3 adapter with E1000 and the results are favourable.

Not ideal I know.

The other obvious fix was to disable Symantec NTP.

Kind Regards.

Reply
0 Kudos
JoJoGabor
Expert
Expert

Did anyone find a resolution to this issue? I have it also. on Windows Server 2012 R2 guests, running SEP 12.1.4013.4013 on VMware ESXi 5.0 Update 3, using Intel 10Gb cards. iperf goes from about 1.4Gbps to over 6 Gbps by removing SEP on both machines. Removing it just on the iperf server client results in a bit more sporadic transfer rates of between 3Gbps and 5.2 Gbps. I'm going to log a call with Symantec but not holding my breath.

Reply
0 Kudos
Texiwill
Leadership
Leadership

Hello,

The real question I would start to ask, is there a better way of doing A/V or if A/V is really necessary? Can you use another tool to gain the same level of protection? One that does not sit in each VM or where SEP currently sits in your environment. It can sit on the network and each VM. Not sure what your configuration is. You may also want to limit the scope of where SEP is installed. Look at your security policy and see if it is for 'ALL' systems or for 'specific' types of systems. If specific, then install only in those. Policy should mention the need for AV not the type of AV to be used.

Are there alternatives? Yes.

I would look at some of the tools around segmentation for workloads and for limiting access out of the box (sandboxing, etc.) Symantec has Data Center Protection (used to be critical system protection) which does that. It may be faster than A/V by preventing virus/malware spread and uses different algorithms to detect it. A/V can then be used to remove it.

There are other tools that live on the network, not the VMs and do A/V scans of data heading into and out of your VMs as well. Still others that have small shims in the VM.

What is absolutely needed by policy, if you can turn off features to gain that, you may be able to find what is causing the slow down as well.

Best regards,
Edward L. Haletky
VMware Communities User Moderator, VMware vExpert 2009, 2010, 2011,2012,2013,2014

Author of the books 'VMWare ESX and ESXi in the Enterprise: Planning Deployment Virtualization Servers', Copyright 2011 Pearson Education. 'VMware vSphere and Virtual Infrastructure Security: Securing the Virtual Environment', Copyright 2009 Pearson Education.

Virtualization and Cloud Security Analyst: The Virtualization Practice, LLC -- vSphere Upgrade Saga -- Virtualization Security Round Table Podcast

--
Edward L. Haletky
vExpert XIV: 2009-2023,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
Reply
0 Kudos
iamxCPx
Enthusiast
Enthusiast

For us, last year I ended up switching to TrendMicro Deep Security Virtual Appliance solution that utilize vShield because I'm tired of wasting my time troubleshooting with Symantec Support.

I did try deploying the SEP utilizing vShield to fix the "slowness" issue but it wasn't as easy to manage/monitor and require pre-req steps prior to the install process so I said enough is enough.

But, it might be improved it by now...

Nothing was perfect also with TrendMicro as far as the first install/setup. They require the host to be rebooted after the installation process or after installing new updates/patches where Symantec or Sophos* doesn't require it so that was pita.

Support also wasn't so great unless you get the premium package and a few more other rants. lol

But I don't have the speed issues anymore.

*Prior to TrendMicro, I tried out Sophos and they have speed issues with Office Macros file even when utilizing the vShield technology.

FWIW..

Reply
0 Kudos
dlane1975
Contributor
Contributor

We too are experiencing this issue.  I have started a discussion on the Symantec Communities.

Slow 10Gbps network when Symantec Endpoint Client is installed | Symantec Connect

Curious if anyone else has a permanent solution that doesn't leave our operating system vulnerable.  We are considering scratching Symantec and trying another solution but would like to use what we already have in place if we can.  I currently have an open case with Symantec and it is in the process of being escalated.

Bottom line is with SEP installed our network throughput is slow, uninstalling Symantec clears out this issue.

Darren

Reply
0 Kudos
JoJoGabor
Expert
Expert

No, my call got escalated several times and ended up with engineering, eventually Symantec basically said to me that this was within acceptable performance boundaries and they would not escalate any higher. Very poor show in my opinion. If it was up to me I would assess new AV vendors

Reply
0 Kudos