Accepted Solutions
Hi René,
Hmmm... There is something very, very wrong with that NVRAM file. It looks like the power outage has corrupted the NVRAM file in a way that confuses the firmware unrecoverably. I'll look into this further next week (next year!) when I'm back in the office, and will file an internal bug report if I find something that we can make more robust.
Although I generally do not advise this for EFI virtual machines, in your situation your best plan of action is probably to power off the VM, delete the .nvram file from the VM's directory (or rename it to something else if you want to be extra careful), and then power it back on again. A fresh .nvram file will be created automatically when the VM starts. For most reasonably normal Windows installations, the VM should then automatically boot into the installed OS.
(For the benefit of other forum readers who might come across this thread, I must emphasize: Do not delete the NVRAM of an EFI virtual machine unless you understand the consequences! It looks like it is appropriate to delete NVRAM in this particular situation and very few others.)
Please let me know if that gets things going again!
And I'll save one of our forum regulars, WoodyZ, from having to make his regular comment for this situation: If you haven't already done so, you may wish to consider investing in an uninterruptible power supply (UPS) to keep your system running through power failures or to at least allow an orderly shutdown instead of a hard crash. 
Cheers,
--
Darius
Hi René,
Hmmm... There is something very, very wrong with that NVRAM file. It looks like the power outage has corrupted the NVRAM file in a way that confuses the firmware unrecoverably. I'll look into this further next week (next year!) when I'm back in the office, and will file an internal bug report if I find something that we can make more robust.
Although I generally do not advise this for EFI virtual machines, in your situation your best plan of action is probably to power off the VM, delete the .nvram file from the VM's directory (or rename it to something else if you want to be extra careful), and then power it back on again. A fresh .nvram file will be created automatically when the VM starts. For most reasonably normal Windows installations, the VM should then automatically boot into the installed OS.
(For the benefit of other forum readers who might come across this thread, I must emphasize: Do not delete the NVRAM of an EFI virtual machine unless you understand the consequences! It looks like it is appropriate to delete NVRAM in this particular situation and very few others.)
Please let me know if that gets things going again!
And I'll save one of our forum regulars, WoodyZ, from having to make his regular comment for this situation: If you haven't already done so, you may wish to consider investing in an uninterruptible power supply (UPS) to keep your system running through power failures or to at least allow an orderly shutdown instead of a hard crash.
Cheers,
--
Darius
Hi,
I run tests during all day,
I have set up a new fresh w2012 vm.
Started lot of snapshot ( quiesced and not quiesced ) from VI client : no issue occured.
Started a "continuous" backup job from veeam : no issue after 1 hour ( nearly 30 snapshots ) ( using application aware guest processing )
Started a "continuous" replication job : no issue after 30 replication. ( using application aware guest processing )
Since issue happens only ( right now ) on my 2012 AD Domain Controller, I setup the vm as DC.
again no issue after the same turing for 4 hours.
ok not the best test protocol but hey, I had some work to do
meanwhile 1 time of 5 ( approx. ) my "real" dc get their efi corrupted....
maybe it's not related to snapshot after all...
What we know so far is that something is causing the EFI "Boot Options" (i.e. the entries visible in the EFI Boot Manager) to replicate until eventually the VM's EFI NVRAM runs out of space and the VM fails to boot. It might be a few snapshots or a few VM reboot cycles before the problem can be first seen as multiple entries in EFI Boot Manager, and it might be many more snapshots or reboot cycles before the VM fails to boot due to insufficient NVRAM space.
About that I can say that reboot seems not to be the culprit since my corrupted vm are my DC and they almost never get rebooted.
For instance, I had my vm corrupted, fixed the nvram, boot it up and get it corrupted in my veeam lab. after some backup. It was also corrupted on the host since I tried to reboot it to see what will happen and tada ! nvram was dead.
I'm so glad I found this thread! I have been banging my head with this problem for the past week now (ever since updating to VEEAM 7.0.)
A little background on our problem. One host is using local storage. We have about 8 VM's. 5-6 of them are Windows 2012. All but one uses legacy BIOS. The one that has EFI BIOS is also a domain controller. About a month ago, we had thought the VM crashed and when it rebooted we got that boot manager menu and no matter what we did we had to boot it manually using that NO VOLUME LABEL option. We didn't have any problems with VEEAM backups or replications after that, but whenever we rebooted that VM we had to manually boot it up.
This past week, I updated VEEAM to 7.0 R2. That's when all hell broke loose. During VEEAM backup/replication jobs, our host would pink screen. I opened a case with VEEAM, and they said there's not much they could do because their logs don't reveal anything that's going on with the vmware host side. So I opened a case with VMWARE to see if they could help ( SR 13403911911 ). The tech working with me really didn't have any ideas after looking at the logs because we didn't have a scratch partition set up so when the host crashed and rebooted it was pretty much empty. Meanwhile, the VEEAM engineer pointed me towards the article that states there is a bug with ESXI 5.x concerning e1000e vnic drivers along with Windows 2012 and how they can crash hosts causing the PSOD. So I changed all the vnics on the Windows 2012 VM's to vmxnet3. Since then we have had no more pink screens, however, the veeam backup replication job seems to come to a crawl when replicating the ONE vm that has the EFI boot menu issue. The VEEAM server (which is also a VM) constantly loses it's network connectivity when the job is running. It actually brings the host and all other VM's to a crawl too. As soon as I stop the job from running, everything goes back to normal.
It seems to me that all of our problems relate to this one VM that uses EFI BIOS. I'm thinking maybe I should try deleting the NVRAM file as stated above? Any thoughts? Ideas? 
Thanks.