since the update to ESXi 5.5.0 Build 2456374 some VM crash with BSOD ( ntfs.sys ) when we start to clone the VM for backup.
The VMware Tools are up to date ( typical installation not full ).
Are there any known issues?
I talked in detail with the Escalation Engineer that is working the case. Looks like the Microsoft Hotfix does not correct the issue. They are actively working on a resolution and I am now working directly with the EE to do some testing for them. My recommendation is to open a ticket with VMWare and have it tied to PR1437136 so you get updates on the issue.
I did some testing with the EE Yesterday. Its not a 100% proven yet but you may want to try uninstalling VMWare Tools and Re-installing without the VSS Writer. Windows will default to the Windows Sync Driver for your backups. I know this might be difficult if you have a large number of servers like we do but the workaround seems to work.
I got an update from the EE I am working with and am free to share some information now.
For ESX 5.5 Patch version 4 VMWare updated the VSS Writer in the VMWare Tools to correct some false errors in the error log. It appears this "fix" is what is causing the BSOD. I was able to test with VMWare both removing the VSS Writer and downgrading VMWare tools and were able to prove that the issue was with this code. VMWare will be rolling back to the old VSS Writer in the next patch set that is scheduled for Mid-June. This does not fix the original issue with the false error messages in the event viewer but will fix the BSOD issue until they can find a root cause in the new code and fix it.
Just had a look at the KB article - the affected versions (and what the correlate to) are:
8.6.15 8399 esx/5.0p11 8.6.15 2509828
9.0.15 9231 esx/5.1u3 9.0.15 2323236
9.4.11 9355 esx/5.5ep06 9.4.11 2456374
9.10.0 9216 esx/6.0 9.10.0 2476743
# (Column 0 = affected versions listed in KB - VMware KB: Quiescing operations cause a Windows virtual machine to panic with a Stop 24 error on ntf...
# Column 1: Tools version on NGC/VI Client
# Column 2: ESXi server version
# Column 3: Tools version on guest Setup/About page (same as column 0)
# Column 4: ESXi server build number
It appears the latest version of VMware Tools for vSphere 5.5 is 9.4.12.
The question is, has this issue been fixed in 9.4.12, or is the KB out of date referring to 9.4.11 as being the affected version of VMware Tools for 5.5?
I Have been told by engineering that the fix will come in late July with the next patch roll out. We just disabled quencing from our backups but you can also re-install vmware tools and remove the vss writer. That will fix the BSOD issure
I contacted VMware support about this and they said they would get the KB updated to reflect the issue affects 9.4.12 (as well as the documented 9.4.11).
It appears VMware will be bringing out a patch to fix it (rather than MS patching it from their side)..
Copy/Paste from the VMware support email:
Related VMware Bug 1437136 is still open and the KB will get updated as well (this has not happened) 😕
From my previous e-mail FIX will be available in the next release of 5.5 U3 (tentative release date end of July 2015) .
Your mentioned (latest version ) VMware Tools 9.4.12 for is a branch of ESXi 5.5p05
Workaround for now : Install an older version of vmware tools ( unaffected version ).
Last I heard it will be the end of July VMWare will be sending out an update that will reverse the change they made to the VSS Writer. In the meantime you can remove the VSS Writer from VMWare tools and Windows will revert to its native Sync Driver. I have not heard back from Alex in a while so I will email him and get an update.
The native Sync driver has had its own share of problems (database corruptions), just so people are aware.. Gostev from Veeam recently reminded everyone of this in last week's Veeam community forums digest.
Not sure about 6.0. I talked with VMWare earlier this week and for VMWare 5.5 it will come out with Update 3 scheduled for the second week of August baring there are no quality control issues with their testing.
I spoke with support on the 8th of this month (July) and was advised that a patch is scheduled for release sometime between late July and early August.
Was offered the possibility of a hot patch if logs and BSOD dump files can be provided.