BLUESCREEN 'BAD_POOL_CALLER' in Windows 2003 Guest on ESXi 5.5u2 : possibly caused by 'vsepflt.sys'

[MatthiasSchmidt]

Hello !

Since some time, I think :smileyalert: since the beginning of this year, I have problems with my Windows 2003 Guest on a ESXI 5.5u2 host with current patches (ESXi550-201501001, and now, ESXi550-201502001) applied:

After some days of running, the guest "throws" a "BAD_POOL_CALLER" bluescreen, Windows kernel Debugger shows the following:

----------------------------------

BugCheck C2, {7, 121a, 2130002, 87c46bd8}

Probably caused by : vsepflt.sys ( vsepflt+ec80 )

Followup: MachineOwner

...

BAD_POOL_CALLER (c2)
The current thread is making a bad pool request.  Typically this is at a bad IRQL level or double freeing the same allocation, etc.
Arguments:
Arg1: 00000007, Attempt to free pool which was already freed
Arg2: 0000121a, (reserved)
Arg3: 02130002, Memory contents of the pool block
Arg4: 87c46bd8, Address of the block of pool being deallocated

---------------------------------

-> Are there any known problems with this filter driver "vsepflt.sys" since the first update of this year ?

I found knowledge base articles about a memory leak, KB2077302 (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=207730...) and other failures, KB2081616 (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=208161...), but both issues should be resolved with update 02, and the failure in my guest has other symptoms ("double freeing the same allocation" vs. "memory leak", etc.).

I will try to disable this filter driver now as described here, KB2034490 (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=203449...), and hope it prevent further bluescreens.

Other ideas ?

Thanks.

[qsnow]

I just noticed the same issue this morning.  Active Directory running on a Windows 2003 server.  Pulled the memory dump file and it points to the vsepflt.sys driver.

My ESX build is 5.5 build 2067769 (which I believe is post U2).  vmTools version on the guest:  9.4.11 build: 2400950.  I believe I am patched past where the KB articles mention I should be to have this issue resolved.

SYMBOL_NAME:  vsepflt+ec80

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: vsepflt

IMAGE_NAME:  vsepflt.sys

DEBUG_FLR_IMAGE_TIMESTAMP:  54664599

FAILURE_BUCKET_ID:  0xc2_7_VFil_vsepflt+ec80

BUCKET_ID:  0xc2_7_VFil_vsepflt+ec80

BAD_POOL_CALLER (c2)

The current thread is making a bad pool request.  Typically this is at a bad IRQL level or double freeing the same allocation, etc.

Arguments:

Arg1: 00000007, Attempt to free pool which was already freed

Arg2: 0000121a, (reserved)

Arg3: 02130009, Memory contents of the pool block

Arg4: 892c4610, Address of the block of pool being deallocated

Debugging Details:

------------------

POOL_ADDRESS:  892c4610 Nonpaged pool

FREED_POOL_TAG:  VFil

BUGCHECK_STR:  0xc2_7_VFil

DEFAULT_BUCKET_ID:  DRIVER_FAULT

PROCESS_NAME:  System

CURRENT_IRQL:  0

LAST_CONTROL_TRANSFER:  from 808947bb to 80827f7d

[qsnow]

Created a case with vmWare -- Initial thought is they a new memory leak has been introduced in the latest version of the endpoint driver.  They took my memory.dmp file and information and sent it to development.  They also gave me the .sys drivers for the previous version to load onto machines experiencing the issue.  No BSOD last night.  Time will tell.

[Fabio978]

Hello, we are experiencing same issue, but we cannot disable vShield protection..

Are there any news from VMware?

[qsnow]

No.  Apparently the 'previous' version of the vsepflt.sys they gave us was the wrong version as well and they are suppose to be supplying us with another 'previous' version.  I would really expect vmWare to be prioritizing this as BSODing VMs sure doesn't sit well.

[Fabio978]

I also tried to use "previous" driver version but it didn't work cos it returned an error on windows boot. BSODs on our 2003 servers are increasing, we really hope vmware will find a fix very soon..

[MarcoCrv]

We are suffering the same issue,

The situations apper to be exactly as described by Matthias Schmidt.

We upgraded from ESXI 5.1U1 to ESXI 5.5U2 patch.

We use Kaspersky SVM for antivirus protection and we need the driver working.

Have you already opened a PR to VMWare?

Thanks.

[MatthiasSchmidt]

Hello,

After disabling this filter driver (as described in KB2034490), the BSOD doesn't occur anymore, so I'm looking forward for an official patch.

@MarcoCrv: Sorry, I'm only a user of the free hypervisor - I can't submit official problem reports, I think.

[MarcoCrv]

Hi Matthias,

I opened UP an SR and collecting information about it.

What do you use the Endpoint for? I mean, for me it interfaces the Kaspersky SVM.

Thanks.

[qsnow]

We are also using it with Kaspersky SVMs.  For us, however, it doesn't matter if we have Kaspersky configured to scan those machines or not -- the driver still eats NonPaged Memory.  The previous version they rolled us back to appears to have stabilized our BSODs.  The version that appears to be working for us is:  5.5.2.0 build-1904019.  An official patch does need to get out.  Seems to me this should be a pretty high priority from vmWare.

[Fabio978]

Did you need to rollback the whole vmtools package or only the vsepflt driver?

[MarcoCrv]

Hi,

I don't think it is possible to rollback only the vsepflt.sys driver I only disabled it on some VMs as needed.

I'm in contact with vmware, I hope they will respond soon.

Thanks

[qsnow]

TThe install they gave us was an .msi file that just drops the two .sys files.  these two file go into systrm32/drivers and then the server needs a reboot.  All of the tools does not need to rollback -- just a swap of these two .sys driver files.

[MarcoCrv]

You will put the installation in a non supported configuration that way.

I agree that it might work as a workaround, but unless there is a specific KB from VMWare (I didn't search for it) we can't consider it acceptable.

Thanks.

[Fabio978]

Which are the drivers? vsepflt.sys and vnetflt.sys?

I tried swapping them with an older version (5.2.0 build 2012768) but then they fail to start on system boot..

[qsnow]

TThe previous version of the drivers and the process came directly from VMware support, so I consider this a supported workaround. As for which drivers, yes it is those two, however the only ones that worked without the memory leak for us was the version number I supplied in a previous post. the first version support gave us didn't work they then gave us the one I listed. Keep in mind there is a 32 and 64 bit version of the drivers. If they don't start maybe you put the wrong version down?  We didn't have any issues with them not starting 

[Fabio978]

You're right! I didn't pay attention to x86-x64 versions. Copied them from another 2003 - 32bit and it seems ok. Let's see if we won't have BSODs again.. waiting for an official resolution.