Windows 10 file corruption

[bjohn]

We created a Windows 10 1607 pool recently. If I run a /sfc scannow on the master image, it completes without errors. We created a pool of machines from this master image. Out of the blue, we have machines that are blue screening, start menu stops working, programs stops working etc... When we run sfc /scannow on the linked clones, we are finding errors. The storage is on a VNX 5300, and according to the storage group, there are no errors on the disk.

Wondering if anyone has any clue on what could be causing file corruption. We recently upgraded to 7.2.0, but the agent on this pool is still 7.0.2. The issues were happening prior to us upgrading to 7.2.0. Thanks

[BenFB]

We are seeing a similar issue on 7.1. Are you by chance using any application layering tools like VMware App Volumes or Liquidware FlexApp/ProfileUnity?

[bjohn]

Yes, we are using Profile Unity Version 6.5.5896 e54314e. Haven't seen this issue on Windows 7 pools though. Actually haven't run sfc on windows 7 to see if it errors.

[Jeroen235]

Maybe unrelated to your issue but we have also experienced file corruptions in our Windows 10 environment.

With us It turned out to be Trend Micro Deep security that was causing the file corruption.

It (or rather vmware) uses a filter driver in Windows that works in conjunction with the Trend micro deep security appliances.

Turning off the anti virus made the file corruption disappear.

[bjohn]

No trend, but we do use Symantec.

[BenFB]

We also have FlexAPP/ProfileUnity and determined that FlexApp was the cause. The team that manages FlexApp/ProfileUnity is still working through resolving the issue but once we disabled FlexApp the issue went away.

[bjohn]

Thanks BenFB We do have some FlexApp applications, but the users in this Windows 10 pool does not have any FlexApp applications assigned to them. How can I disable FlexApp for this specific pool of Windows 10 machines?

[bjohn]

BenFB​

Just following up to see if you have any further input.

Thanks!

[BenFB]

At this time they still have it disabled and will revisit it at another time. The feeling is that the NAS where the images were located was not keeping up/disconnecting. This resulted in the VHD dropping and Windows thinking that a hard disk had corrupted.

[bjohn]

BenFB​

Did you guys ever open a ticket with Liquidware? I opened a ticket and they are not apparently aware of this issue. I'm not how we are the only two people with this issue.

Thanks

[bjohn]

BenFB You have any further details on this issue? Thanks

[BenFB]

bjohn The team that manages FlexApp/ProfileUnity put the deployment of FlexApp on hold and will revisit it later this year. We are using ProfileUnity without issue. Since removing FlexApp we are no longer seeing the issue. You might look into Jeroen235​ suggestion of removing anti-virus from your image as a test. After working with VMware support they said it was likely caused by a filter driver. In our environment our only filter drivers are FlexApp/ProfileUnity and anti-virus.

[bjohn]

BenFB​ Appreciate the reply.

When you say you put the deployment of flexapp on hold, what exactly does that mean? How did you remove flexapp?

I contacted liquidware and they told me to uninstall the container service. Basically, followed Windows 10 unusable after ProfileUnity Client tools are installed with Unidesk – Liquidware Customer...

Uninstalled PU and reinstalled referencing the edited config file. The cbfltfs4 file system driver is no longer present.

Even after doing this, I'm still seeing corruption.

The file system drivers present after doing this are:

FsDepends

BHDrvx64

eeCtrl

SRTSP

SymEFASI

storqosflt

wcifs

FileCrypt

luafv

npsvctrig

FileInfo

Wof

The only one that is out of the norm is FsDepends (compared to a machine without PU)

P.S. What version of PU are you on?

Thank You

[BenFB]

I passed along that article and they've applied it to see if it makes a difference. They have also opened a ticket with Liquidware support.

[Jeroen235]

Here the reaction of Trend Micro concerning the file corruption in Windows 10 using DSVA appliances on the 23th of november 2017:

I have been investigating and with your information and other details been able to find a potential cause correlation.

The Issue seems to stem from the changes made between the Win 7 and Server 2016/Win 10.
This relates to the TDI (Transport Driver Interface) which seems to have been removed/modified between iterations.
https://technet.microsoft.com/en-us/library/cc939977.aspx

Our engineering team and engaged VMware and they have opened an internal bug (BUG / 2000210).

Obviously we cannot unload this driver as this would remove the functionality of the DSVA, one potential test would be to use agent based protection on a windows 10  machine to see if we notice the same behavior.

Unfortunatly no further response and I am unable to locate any information on the status of the internal BUG report at VMware.

[BenFB]

Thank you for the update. I'll have our account team and our escalation engineer look into the bug and see if it's related to our issue.

[BenFB]

Our VMware team looked into that issue and it unfortunately(fortunately?) does not apply to us. We do not install FileIntrospection or NetworkIntrospection with VMware Tools since we don't have NSX.

Maybe bjohn​ has that in his install and it will at least help him.

[bjohn]

We use Symantec endpoint protection, no trend micro We also don't use NSX so those components are not installed as far as I know. I'm still seeing corruption, even after removing ProfileUnity completely.

[BenFB]

Are you doing a scripted install of VMware Tools on your golden image? If you are doing a GUI install you may have to specifically exclude the introspection components.

[BenFB]

VMware Tools 10.2.5 was released a few weeks ago and one of the resolved issues might help you.

• vnetflt.sys causes active directory problems resulting in an authentication failure.
  Attempts to log in fail due to problems in the active directory. Network Introspection driver vnetflt.sys, which uses deprecated TDI framework causes this failure. The driver is installed during the installation of VMware Tools.
  vnetflt.sys driver is replaced with the new WFP-based vnetWFP driver for Windows 7 and later.
  This issue is resolved in this release.

VMware Tools 10.2.5 Release Notes