Could you upload the support bundle to us? Beside that, could you also upload the kernel dump in guest as well?
We never see this problem during our testing, it maybe caused by hardware.
Wow - 3 VMs that report corrupted vmdks that need repair at the same time is indeed something special.
Do you have any strange filesystem used for V: ?
And with strange I mean anything other than plain uncompressed NTFS ...
From your support bundle, I can see you have one NVME disk mounted as V:, all your VM is running here.
The log always show there have disk IO problem, I am wandering your driver V: have some problem, would you mind move your VMs into another hard drive and try again?
In your VM folder, you should find some vmware-vmx-xxxx.dmp, upload one for us to make sure the root cause.
Volume V: is a 1TB M.2 NVMe drive with just the one partition. That drive has data deduplication enabled in Windows Server 2019.
From the dump, the error happen because IN_PAGE_ERROR, "Inpage operation failed at 000002a3f44aebb0, due to I/O error 00000000c0000010", normally this can happen due to the OS failed to read data into memory from disk/pagefile.
I would suggest you move the VM to another hard drive and try again. Also, run disk repair for all your VMs, you need backup the VM before you run repair.
Ref this KB to repair your disk:
I moved one of the VMs to the SSD (without Data Dedupe) and so far the VM has been running for 24 hours. I did not run any repairs or anything on that disk and it is still running fine.
I ran the repairs you linked to and the Windows repair disk in the guest itself on the same VM as above, on the NVMe drive (with Disk Dedupe), and within 3 hours it crashed again.
It seems pretty clear that the Disk Dedupe from Windows Server 2019 is the issue here. Doing a Google search, I found a couple of other that listed the same thing. Dedupe in Server 2012 R2 & 2016 work fine but 2019 no go. Shame because that means that I will have to move to Hyper-V instead of VMware because I can't afford to lose that space. Sad day for me... :`(
Thank you for your assistance. Wish that it was a better result but not everything works everywhere.
Ok, so I turned off deduplication on the second drive and then ran the VMs from that drive again and everything is running normal now. I am fully convinced that the Deduplication feature in Windows Server 2019 is not compatible with VMware Workstation. This is a huge disappointment since I will have to move to Hyper-V and rebuild my entire lab now. Hopefully a future version of Workstation Pro will support this feature. It is rumored that Deduplication will be coming to Windows 10 so that will really be a big issue.
Was it helpful? Let us know by completing this short survey here.
Does VMWare have an official stance on using Windows Server Deduplication with Workstation Pro?
I am using Windows Server 2016 with VMWare Workstation Pro 15.5 on a PowerEdge T420 with spinning SAS drives and having the exact same issues. So i am wondering if Workstation even supports running from a Dedup Volume.
I was trying to troubleshoot the same problem on a customer with both Windows Server 2019 and Windows Server 2016, both running testing labs over Workstation 15.5 and reached the same conclusion, whenever the Dedup feature is scheduled one of three things could happen: 1) Nothing at all, dedup job runs to completion with the VMs still running and vmdks safe; 2) VMs shutdown spontaneously during dedup job, sometimes with vmdk corruption; 3) VMs keep running but the vmdks are somehow "disconnected" from the VM so they keep showing IO errors and most of the times ends up in vmdk corruption. Disabling Dedup immediately fixes the problem. I saw the same problems running the VMs both as "Shared" on the background and "manually" on the vmware workstation console, same results. During the debugging sessions vms were configured to use the vmdks as virtual sata/ide/scsi disks, all with the same result.
So I can confirm the findings of the original poster, something is very broken between Server 2016/2019 Dedup and VMWare Workstation 15+. This customer used to run Server 2008R2 and Server 2012 before, with older Workstation versions and basically the same configuration overall with no problems, started using Dedup in Server 2012 and found no crashes like this before, so it seems to be related to newer/Windows 10-based Server versions and/or newer VMware versions specifically.