VM crashes during snapshot

[Ruth_dG]

We are setting up a new ESXi 6.5 systems with new shared storage both Huawei connected to an existing VCenter 6.5 with ESXi 6 hosts. While testing a new backup solution (Huawei BCManager) via quiesced vm snapshots we sometimes have vm's that crash directly after a snapshot was triggered. The machines reboots and we either get a "File system specific implementation of LookupAndOpen[file] failed" after which the vm stops, or it boots but we get a "No operating system was found"

My first thought was that it had to do with the following kb VMware Knowledge Base , but after patching the hosts, the problems remains.

It looks like it only happens after a storage vmotion to anther datastore

Attached the vmware.log of a failing VM, the log begins during the storage vmotion and around 2019-01-25T11:20:28.302Z the no operating was found errors appears.

Any idea what the reason for this could be?

[vmrale]

Hi,

what operating system is installed in that VM? What VM Tools version is installed?

Did You try to take a quiesced snapshot directly from GUI on this VM, to eleminate backup software issue?

Did You try to take a snapshot without quiesing, to eliminate VSS/SYNC issues?

[Ruth_dG]

It's windows 2012 R2 with a mix of vmware tools 10341 and 10305

I haven't been able to reproduce with taking snapshots from the GUI in VCenter, but the backup software takes dozens of snapshots without issues, so I cannot eliminate the backup software totally.

I maybe can test to take scripted snapshots in VCenter to get the same amount as the backup software does.

I haven't test without the quiesing, but as this behaviour happens randomly, it's hard to eliminate things. I'll try with some vm's without the quiesing.

[vmrale]

Check the folder of this VM on datastore if there're any unwanted snapshots left, not deleted by backup software after backup job completion.

Why this VM has mix of VMware Tools? Did You restart it after recent VMware Tools update?

Why You can't take a snapshot of this VM from Web/vSphere Client?

Did You investigate this?

[Ruth_dG]

There seem to be some snapshot files left, even when the Snapshot manager show no active snapshots.

I spinned up some test-vm's, to reproduce the behaviour, for which I didn't bother to update the vmware tools. That is the reason for the mix.

I can create snapshots from the GUI but not the amount the backupsoftware takes (every 15 minutes) That's why I'm going to script it, to have a bigger chance to reproduce.

[vmrale]

If there're some snapshot (*-Snapshot*.vmsn and -delta.vmdk/-sesparse.vmdk) You don't see in Snapshot Manager, You have to do a consolidation. If snapshot consolidation won't work I recommend to use VMware Converter and convert this VM to base disks (without snapshots).

There's a faster procedure, but not supported by VMware.