bravobrawi
Contributor
Contributor

Windows 7 and Windows2008R2 VM BSOD ntfs.sys

Hello,

since the update to ESXi 5.5.0 Build 2456374 some VM crash with BSOD ( ntfs.sys ) when we start to clone the VM for backup.

The VMware Tools are up to date ( typical installation not full ).

Are there any known issues?

Kind regards,

bravobrawi

51 Replies
drc0106
Contributor
Contributor

I talked in detail with the Escalation Engineer that is working the case. Looks like the Microsoft Hotfix does not correct the issue. They are actively working on a resolution and I am now working directly with the EE to do some testing for them. My recommendation is to open a ticket with VMWare and have it tied to PR1437136 so you get updates on the issue.

0 Kudos
thaynes2015
Contributor
Contributor

thanks I have had a case open with them for about 2 weeks on this already.  Hopefully I can get them pointed in the right direction.

0 Kudos
drc0106
Contributor
Contributor

I did some testing with the EE Yesterday. Its not a 100% proven yet but you may want to try uninstalling VMWare Tools and Re-installing without the VSS Writer. Windows will default to the Windows Sync Driver for your backups. I know this might be difficult if you have a large number of servers like we do but the workaround seems to work.

0 Kudos
thaynes2015
Contributor
Contributor

Any luck yet?  I know your working with VMWARE pretty heavily on this.  Crossing fingers for some good news soon.

0 Kudos
drc0106
Contributor
Contributor

The EE has narrowed down the issue. They are working on a hotfix now. It may take a few weeks to get written and go through quality control.

0 Kudos
drc0106
Contributor
Contributor

I got an update from the EE I am working with and am free to share some information now.

For ESX 5.5 Patch version 4 VMWare updated the VSS Writer in the VMWare Tools to correct some false errors in the error log. It appears this "fix" is what is causing the BSOD. I was able to test with VMWare both removing the VSS Writer and downgrading VMWare tools and were able to prove that the issue was with this code. VMWare will be rolling back to the old VSS Writer in the next patch set that is scheduled for Mid-June. This does not fix the original issue with the false error messages in the event viewer but will fix the BSOD issue until they can find a root cause in the new code and fix it.

MattHPN
Contributor
Contributor

Any word on this?  Does Patch 5 fix the issue?

0 Kudos
chicaneau
Contributor
Contributor

Hi All,

Just had a look at the KB article - the affected versions (and what the correlate to) are:

  

8.6.15          8399        esx/5.0p11                             8.6.15        2509828
9.0.15          9231        esx/5.1u3                              9.0.15        2323236 
9.4.11         9355        esx/5.5ep06                            9.4.11        2456374
9.10.0         9216        esx/6.0                                9.10.0        2476743

# (Column 0 = affected versions listed in KB - VMware KB: Quiescing operations cause a Windows virtual machine to panic with a Stop 24 error on ntf...

# Column 1: Tools version on NGC/VI Client
# Column 2: ESXi server version
# Column 3: Tools version on guest Setup/About page (same as column 0)
# Column 4: ESXi server build number

From: https://packages.vmware.com/tools/versions

It appears the latest version of VMware Tools for vSphere 5.5 is 9.4.12.

The question is, has this issue been fixed in 9.4.12, or is the KB out of date referring to 9.4.11 as being the affected version of VMware Tools for 5.5?

Cheers,

Glynn

0 Kudos
drc0106
Contributor
Contributor

I Have been told by engineering that the fix will come in late July with the next patch roll out. We just disabled quencing from our backups but you can also re-install vmware tools and remove the vss writer. That will fix the BSOD issure

0 Kudos
DrWhy
Enthusiast
Enthusiast

Has Microsoft released a hotfix for this issue yet?  If so, please post a link.

0 Kudos
chicaneau
Contributor
Contributor

I contacted VMware support about this and they said they would get the KB updated to reflect the issue affects 9.4.12 (as well as the documented 9.4.11).

It appears VMware will be bringing out a patch to fix it (rather than MS patching it from their side)..

Copy/Paste from the VMware support email:
--
Related VMware Bug
1437136 is still open and  the KB will get updated as well (this has not happened) 😕

From my previous e-mail FIX will be available in the next release of 5.5 U3 (tentative release date end of July 2015)  .

Your mentioned (latest version ) VMware Tools 9.4.12 for  is a branch of ESXi 5.5p05
Workaround for now : Install an older version of vmware tools ( unaffected version ).

--


Cheers

0 Kudos
drc0106
Contributor
Contributor

Last I heard it will be the end of July VMWare will be sending out an update that will reverse the change they made to the VSS Writer. In the meantime you can remove the VSS Writer from VMWare tools and Windows will revert to its native Sync Driver. I have not heard back from Alex in a while so I will email him and get an update.

0 Kudos
Bleeder
Hot Shot
Hot Shot

The native Sync driver has had its own share of problems (database corruptions), just so people are aware..  Gostev from Veeam recently reminded everyone of this in last week's Veeam community forums digest.

0 Kudos
DrWhy
Enthusiast
Enthusiast

Will this patch be for ESXi 6 as well?

0 Kudos
Aaron_C
Contributor
Contributor

Any update on patch release?

0 Kudos
drc0106
Contributor
Contributor

My understanding it will be in late July with the next patch release.

0 Kudos
DrWhy
Enthusiast
Enthusiast

Looks like the patch has been released. 

VMware ESXi 6.0.0b Release Notes

0 Kudos
MattHPN
Contributor
Contributor

This looks to be the patch for 6.0.  5.5 patch still has not been released as most of us in this thread are looking for that version.

0 Kudos
drc0106
Contributor
Contributor

Not sure about 6.0. I talked with VMWare earlier this week and for VMWare 5.5 it will come out with Update 3 scheduled for the second week of August baring there are no quality control issues with their testing.

0 Kudos
Aaron_C
Contributor
Contributor

I spoke with support on the 8th of this month (July) and was advised that a patch is scheduled for release sometime between late July and early August.

Was offered the possibility of a hot patch if logs and BSOD dump files can be provided.

0 Kudos