Symantec anti-virus definition updates causing heavy I/O on our SAN, better configuration or product?

[shepnasty]

This Subject was posted before by another user but now we are running into the same issue. All of our 400 VM's are going out and getting the definition update and loading it at the same time (about a 15 minute window). It is killing our SAN, basically bringing it to it's knees. We are running 8 IBM 3850 M2 servers with 128 GB of mem each with (2) 4gb QLogic HBA's, Brocade Fiber Fabric (48K Directors), IBM SVC with DS 8100 storage behind it (majority of datastores - 34) and Netapp V3140 delivering NFS datastores (10) and fiber datastores(1). We are on ESX 3.5 update 5 patched up to the most recent critical patches as of today.

I would like to know if anyone is using Symantec (version 10.1.6) anti-virus in their environment and how they are using it so it doesn't clobber their SAN when the definitions are pulled and loaded. I am not too excited about the idea of throwing them into smaller groups and having to manage them that way, making the client go to Live Update for their definitions randomly. That is why we bought an Enterprise solution, so we could manage it. The Symantec management tools are poor to say the least, in being granular to let machines pull randomly. And, I heard SEP introduces new issues. We were thinking about upgrading but as I have researched this more, heart beats become the pain point on the SAN.

If you aren't using Symantec for AV in your large VMWare environments, what are you using? Do you see any heavy I/O on your SAN because of it?

Thanks!

[Rumple]

The issue you are running into is probably the setting that forces the AV to perform a scan after it downloads the updates. the Updates themselves shouldn't make any difference..its basicalyl 400 hosts performing a scan all at once causing it.

Disable this setting and set different scheduled scans for different machines so as to not have too many machines on a single lun performing a scan at once...

[VMmatty]

I've seen this exact problem with McAfee in a VDI environment but the result was the same - all workstations downloaded pattern updates at the same time and it drove too much I/O for the SAN to handle. Here is what we did:

1) Make sure the VMs don't perform virus scans after receiving an update, as another poster already stated.

2) Set the VMs to check for pattern updates starting at 2:00AM instead of whenever they want.

3) Created a "randomization" pattern of 3 hours so that all the VMs don't download their patterns at 2:00AM. Instead they randomly check in starting at 2:00AM until 5:00AM.

Doing those things have helped out tremendously. I'm not sure if that version of Symantec supports #3 though. You might want to look into upgrading to Endpoint Protection as that is the latest version and I believe the issue of high I/O during pattern updates is much better in that version. I'm working with several clients using SEP and I haven't seen it kill the SAN when properly configured.

[shepnasty]

Thank you for your responses. Here is a quick follow up:

1 - We don't perform virus scans after receiving the def update, this would sink the ship for hours instead of the 15 minute slowdown window.

2 - We are a 24 hour shop so even having this ocurr during the middle of the night is bad for us.

3 - Yeah you can't do this with our version of Symantec and I have heard upgrading to SEP causes new issues with the "heart beat" monitor. We could group VM's and let them go out to LiveUpdate randomly to get their update but doesn't that kind of defeat the point and may create another issue on my internet traffic side? We have VDI users that work from home coming in at all hours of the day so we try to keep the internet pipe as clean as possible.

At this point I am thinking that it will be important to go with an AV providor that has the smallest footprint. The smaller the footprint the less impact overall. I remember Sophos being the smallest foot print a few years back but am not sure if this is still the case.

[VMmatty]

You should definitely call Symantec and explain the situation. I have multiple clients running SEP and honestly don't see the high I/O during updates, nor have I seen anything with the heartbeat.

I wouldn't change your AV vendor without first reaching out to Symantec to see if they can address the issue in your current version or see if it is resolved in the latest version.

When I've spread out the updates over 3 hours the load on the SAN is minimal. Only a few desktops are actually updating at any one time so the SAN has no issue with high I/O. There are brief spikes but nothing like the sustained utilization I saw when they all updated at once.

[shepnasty]

Went to VMWorld 2010 and they said this has been a big problem, especially with VDI environments. Hence the new product anouncement for endpoint protection. Should be interesting.