11 Replies Latest reply on Mar 16, 2020 4:44 PM by sgorbach

    Guests being powered off in Workstation Pro 15.1

    WinArchitect Lurker

      This is a new one for me. I've never seen behavior like this. Anyone else?

       

      Workstation Pro 15; VMs are being powered off (not Shutdown) every few hours (completely random, could be 2 hours, could be 8). Host is stable and not resetting, Workstation console itself does not close, but all VMs are being powered off simultaneously. Event viewer in the guest OS shows that the previous system shutdown was unexpected so this was not a graceful shutdown. Guests are running combination of Windows Server 2016, 2019, and Windows 10 1809 & 1903. All VMs are powered off at the same time. VMs were copied from another machine and the VMs do not behave this way on that device.

       

      The new host is an Intel NUC with and i7 processor, 32 GB RAM, a 256 GB SSD for OS and a 1 TB NVMe drive for the VMs. It is running Windows Server 2019 (fully patched) with data deduplication enabled on the NVMe drive. I am running Workstation Pro 15.1.

       

      Any ideas?

        • 1. Re: Guests being powered off in Workstation Pro 15.1
          haiweiz Enthusiast
          VMware Employees

          Hello WinArchitect:

           

          Could you upload the support bundle to us? Beside that, could you also upload the kernel dump in guest as well?

          We never see this problem during our testing, it maybe caused by hardware.

           

          Thanks.

          • 2. Re: Guests being powered off in Workstation Pro 15.1
            WinArchitect Lurker

            Here is the support bundle. There is no kernel dump because it was a power event, not a blue screen.

            • 3. Re: Guests being powered off in Workstation Pro 15.1
              continuum Guru
              Community WarriorsvExpert

              Wow - 3 VMs that report corrupted vmdks that need repair at the same time is indeed something special.

              Do you have any strange filesystem used for V: ?
              And with strange I mean anything other than plain uncompressed NTFS ...

              • 4. Re: Guests being powered off in Workstation Pro 15.1
                haiweiz Enthusiast
                VMware Employees

                Hi, WinArchitect:

                 

                From your support bundle, I can see you have one NVME disk mounted as V:, all your VM is running here.

                The log always show there have disk IO problem, I am wandering your driver V: have some problem, would you mind move your VMs into another hard drive and try again?

                In your VM folder, you should find some vmware-vmx-xxxx.dmp, upload one for us to make sure the root cause.

                 

                Thanks.

                • 5. Re: Guests being powered off in Workstation Pro 15.1
                  WinArchitect Lurker

                  Volume V: is a 1TB M.2 NVMe drive with just the one partition. That drive has data deduplication enabled in Windows Server 2019.

                  • 7. Re: Guests being powered off in Workstation Pro 15.1
                    haiweiz Enthusiast
                    VMware Employees

                    Thanks WinArchitect.

                     

                    From the dump, the error happen because IN_PAGE_ERROR, "Inpage operation failed at 000002a3f44aebb0, due to I/O error 00000000c0000010", normally this can happen due to the OS failed to read data into memory from disk/pagefile.

                    I would suggest you move the VM to another hard drive and try again. Also, run disk repair for all your VMs, you need backup the VM before you run repair.

                    Ref this KB to repair your disk:

                    VMware Knowledge Base

                    • 8. Re: Guests being powered off in Workstation Pro 15.1
                      WinArchitect Lurker

                      Hey Haiweiz,

                       

                      I moved one of the VMs to the SSD (without Data Dedupe) and so far the VM has been running for 24 hours. I did not run any repairs or anything on that disk and it is still running fine.

                       

                      I ran the repairs you linked to and the Windows repair disk in the guest itself on the same VM as above, on the NVMe drive (with Disk Dedupe), and within 3 hours it crashed again.

                       

                      It seems pretty clear that the Disk Dedupe from Windows Server 2019 is the issue here. Doing a Google search, I found a couple of other that listed the same thing. Dedupe in Server 2012 R2 & 2016 work fine but 2019 no go. Shame because that means that I will have to move to Hyper-V instead of VMware because I can't afford to lose that space. Sad day for me... :`(

                       

                      Thank you for your assistance. Wish that it was a better result but not everything works everywhere.

                      • 9. Re: Guests being powered off in Workstation Pro 15.1
                        WinArchitect Lurker

                        Ok, so I turned off deduplication on the second drive and then ran the VMs from that drive again and everything is running normal now. I am fully convinced that the Deduplication feature in Windows Server 2019 is not compatible with VMware Workstation. This is a huge disappointment since I will have to move to Hyper-V and rebuild my entire lab now. Hopefully a future version of Workstation Pro will support this feature. It is rumored that Deduplication will be coming to Windows 10 so that will really be a big issue.

                         

                        ---------------------------------------------------------------------------------------------------------

                        Was it helpful? Let us know by completing this short survey here.

                         

                         

                        • 10. Re: Guests being powered off in Workstation Pro 15.1
                          jaybird283 Novice

                          Does VMWare have an official stance on using Windows Server Deduplication with Workstation Pro?

                           

                          I am using Windows Server 2016 with VMWare Workstation Pro 15.5 on a PowerEdge T420 with spinning SAS drives and having the exact same issues. So i am wondering if Workstation even supports running from a Dedup Volume.

                          • 11. Re: Guests being powered off in Workstation Pro 15.1
                            sgorbach Lurker

                            I was trying to troubleshoot the same problem on a customer with both Windows Server 2019 and Windows Server 2016, both running testing labs over Workstation 15.5 and reached the same conclusion, whenever the Dedup feature is scheduled one of three things could happen: 1) Nothing at all, dedup job runs to completion with the VMs still running and vmdks safe; 2) VMs shutdown spontaneously during dedup job, sometimes with vmdk corruption; 3) VMs keep running but the vmdks are somehow "disconnected" from the VM so they keep showing IO errors and most of the times ends up in vmdk corruption. Disabling Dedup immediately fixes the problem. I saw the same problems running the VMs both as "Shared" on the background and "manually" on the vmware workstation console, same results. During the debugging sessions vms were configured to use the vmdks as virtual sata/ide/scsi disks, all with the same result.

                             

                            So I can confirm the findings of the original poster, something is very broken between Server 2016/2019 Dedup and VMWare Workstation 15+. This customer used to run Server 2008R2 and Server 2012 before, with older Workstation versions and basically the same configuration overall with no problems, started using Dedup in Server 2012 and found no crashes like this before, so it seems to be related to newer/Windows 10-based Server versions and/or newer VMware versions specifically.