ESXi 5.5 vShield Endpoint Blue Screen Issues.

[dakid255]

Hello,

Has anyone else dealt with the following issue or know of a workaround? We have this exact issue in our production environment and have had to revert from using vshield endpoint antivirus. I noticed the Resolution has changed from "no workaround" to "contact vmware support" Unfortunately they will not speak to me because I don't have an active support contract with vmware directly.

VMware KB: Windows virtual machine installed with vShield Endpoint Thin Agent (vsepflt.sys) and vShi...

Any information appreciated!

[lakshya32]

Hi

Welcome to communities.

Most of the time blue screen comes due to bad memory , hard disk and driver

Please find out one by one.

[mikejroberts]

We had several issues and got a private "fix", that didn't really fix anything.  We abandoned endpoint antivirus after what was supposed to be the fix for one issue ended having a memory leak.  Per KB article 2077302, the issues are supposed to be resolved in the latest patches, but we will not be testing it again until next year.

[dakid255]

mikejroberts,

Thanks for your reply. Unfortunately the latest release (5.5.0 Build 1892794, VMware tools version 9.4.6 build-1770165) does not fix the problem as the KB article you mentioned suggest. It seems to be an issue with the VMware tools as the servers we have running version 9.0.5, build-1065307 don't have the problem even running on the new esxi version. Suspiciously the KB article I mentioned is now gone... Really Wishing we hadn't purchased the vShield product as it has been nothing but trouble.

If anyone else has heard of the custom fix and had some success with it I would be more than grateful for comments or any solutions.

Thanks,

[mikejroberts]

Sounds par for the course.  VMware has gotten really sloppy with 5.x.  Be careful with custom fixes.  The one we tried installed new vShield endpoint drivers as a separate install, next to the VMware Tools install in Add/Remove Programs, and that broke future automated upgrades.  With that fix, we had to manually uninstall it before we could modify or update the full tools package.  That means when they actually do fix it, you will be required to pull out the fix before you can roll out the new tools.

[TUS361]

Thanks for posting, this helped me discover the root cause of issues we've been dealing with since our 5.5 upgrade.  We use Trend Deep Security and have been having random VM's lockup and blue screen for 3 months across a multi-tenant environment.  You did more with one post than Trend Micro support has done in 3 months. 

We recently experienced an increase in the problem with 9.4.6 tools, so I rolled back to 9.4.5-1598834 and it seems to be more stable.  We mainly have seen issues on busy Windows web servers, not so much on Linux or Windows SQL or Exchange VM's.  Since the IIS w3wp processes use a lot of npp memory, I think that explains why we experience the issue more on those types of servers.  It manifests itself as a complete lockup of the VM, black screen in the console, or in some cases it actually blue screens and leaves a dump file.  If you catch it early enough you can sometimes stop it from completely locking up if you vmotion the VM to another host, or deactivate and reactivate your endpoint antivirus. 

Has anyone upgraded to 5.5 Update 2 yet and seen evidence that the issue is resolved?

[dakid255]

Yes it was kinda a headache for us to find out as well. I probably only noticed it was an issue because one of our primary file servers randomly blue screened a day or so after the vmtools upgrade. Which by the way we were upgrading our file servers first because of another vshield issue concerning excel files and network shares.  After Checking out the crashdumps it was pretty evident. Like you suggested it can fail in multiple ways and seems to be different by what is running on the server. We also have a fairly stagnant AD Server in our dev cluster that locks up and maxes out the cpu fairly often,without blue screening. Luckily because i caught the issue in time I didn't upgrade the vmware tools to the rest of our environment.

I am really surprised that this issue has not seen more light. although there is nothing in the release notes for update2 other than a rebranding of the vshield driver (we can only hope they have made some application changes as well!) I am planning on upgrading our dev cluster tomorrow or early next week. I'll post back here with my findings after the upgrade.

[TUS361]

Just a quick update on this issue.  Because of customer impact, we pushed out an upgrade of all of our virtual environments to 5.5 Update 2 vCenter, ESXi and VMtools.  Before the upgrade we were seeing various blue screens and VM lockups on several VM's per day.  Afterwards, we have seen it only on a couple of VM's over the course of 3 days.  It is entirely possible those VM's were already suffering memory leakage prior to the upgrade because we did the tools upgrade without a reboot following on many of them.  Looking better overall, but I'm still not entirely convinced it is "fixed".  Time will tell.....

If anyone else has upgraded, please share your results.

[MatteoMarchetti]

Hi,

I'm upgrading VMtools in ESX 5.5 build 2302651 and some of the Windows VMs using vShield Endpoint driver randomly blue-screened.

This is not something I can easily reproduce because if I revert the snapshot and I re-apply the VMtools upgrade it simply works fine.

VMware support didn't give any useful answer except for "this is a one-off issue".

Anyone else facing the same problem?

[Shawnho_Taiwan]

TUS361, I wonder if your machines are ok now after applying vSphere 5.5 U2? My clients are also facing the same problem but are reluctant to apply vSphere 5.5 U2 since they don't believe it would really "fix" anything. Your share would be valuable for me as well as my clients. Thanks in advance!

[CSLSL]

HI,

We have recently (4 weeks) upgraded from VMWare 5.1 to 5.5 update 2 and have just come across this issue.

We have also installed VShield: this version: "ksv-3.0.0-92c.x86_64.sles_signed.ova".  It is installed on all of out ESX hosts.  Its been installed about 2 weeks.

We have been updating VMWare tools on our entire estate to the latest version (About 50%) through.  - Version 9.4.12, build 2627939

As part of the Tools upgrade we are also installing the Intraspection drivers (to allow communication with the VShield Anti-virus (We use Kaspersky))

We rebooted the machines after the tools upgrade.

On Monday I upgraded the tools on 3 of our Virtual 2008R2 Domain Controllers.  Thursday evening All 3 of our domain controllers hung (black screen) and we are trying to troubleshoot this issue, but it appears that it points toward the Intraspection drivers/VShield.

So just to answer the question above if it is fixed in 5.5 update 2... it appears not.

Any help with this issue is appreciated.

Thanks

[Bleeder]

Did you see there is a different blue screen issue that the newer VMware Tools introduce?

https://communities.vmware.com/message/2488343

[joeyalex82]

I just wanted to chime in and say that we are seeing this in our vSphere 6 (latest version of everything) environment, so apparently it still isn't fixed.  My primary fileserver blue screens every few days now if I have the anti-virus turned on.

I had to upgrade the tools version because (as noted below) the older versions have a weird problem with Excel files and network shares, so now I either can't run anti-virus at all, people can't save Excel files, or I get random blue screens.

I really wish I had seen this before I upgraded to 6, and I can't believe this hasn't been fixed yet!

[grbvmw]

vSphere 6 shipped with VMTools 10.0.0 which has issues with Endpoint, see

Resolved Issues

•   Virtual machine performance issues after upgrading VMware tools version to 10.0.x in NSX and VMware vCloud Networking and Security 5.5.x
  While upgrading VMware Tools version to 10.0x in a NSX 6.x and VMware vCloud Networking and Security 5.5.x environment, the performance of the guest operating system in the virtual machine becomes slow and unresponsive. A number of operations like, logging in and logging off through an RDP session, response for an IIS website and launching applications become slow and unresponsive.
  This issue occured due to a known issue with VMware Tools version 10.0.x. This issue is resolved in this release. For more information see KB 2144236.

VMware Tools 10.0.8 Release Notes

If you're having issues with VMs on vSphere 6.0 that leverage Endpoint update to 10.0.8.

[dhanarajramesh]

I had same issue and found this solution

....: Complete installation of VMware tools caused Windows 2008 r2 server unexpected rebooted