Windows (7 & 2012R2) corrupting hard drive data when they're a VM on ESXi 6.7.0, but not on bare metal

[TaoeDude]

ESXi version: 6.7.0 Update 2 (Build 13006603)
Motherboard: SuperMicro X10SRH-CF BIOS v 3.2 (latest)
OS VMs - WIndows 7 Ultimate and 2012R2, with all mandatory updates installed
Hard drives: 1TB SanDIsk SSD for ESXi attached to motherboard SATA, and various 6TB data drives

I've only just started experimenting with using VMs and so far I like it.

But last week I was copying very large files (90GB or larger) between a USB3.1 hard drive and a WD SATA 6TB drive attached to the motherboard's controller on a Windows 7 VM, and discovered that the WD 6TB was corrupting the data (and NTFS' various file structures, according to chkdsk). I assumed it was a problem with the drive (though there were no indications in the SMART data that there was a problem) and swapped it out.

The new drive started having the same problems so I decided it must be a cable or controller issue. Swapping cables made no difference so I tried mounting the drive via an LSI SAS controller and SATA breakout cable. No problems. OK, so it appeared to be an issue with the motherboard's SATA controller. I installed SuperMicro's latest chipset drivers and plugged the drive back into the SATA controller but that made no difference. The same problems were exhibited under a Win 2012R2 VM, too. I tried sfc /scannow on both VMs, as well as chkdsk on the system drives, but they no found no issues.

I upgraded the motherboard's BIOS but that made no difference. On a hunch I swapped the ESXi boot drive for a Windows 2012R2 drive so I could run that on the bare metal and... no problems with the drive using the motherboard SATA controller. I copied terabytes of data around over the course of a couple of days and no issues whatsoever. When I replugged in the ESXi and booted that, I immediately started having the corruption and NTFS directory structure problems again.

So I am obliged to conclude the problem is with ESXi.

Does ESXi have the equivalent of sfc /scannow so I can check the integrity of the installation? Is there a way to re-install its chipset/SATA drivers? Any other ideas about what might be going on?

[Ardaneh]

Hi

I can recommend you to upgrade to the latest version of ESXi 6.7 (Build 15160138 released in 12/05/2019). sometimes these kinds of strange problems will be solved by upgrading to the latest version.

Please share if your problem was gone

Hope this could be helpful

[TaoeDude]

Thanks for the suggestion.

I thought the problem might be that I'd "re-used" the VMDK file from one 6TB drive to another that didn't have exactly the same geometry, but no; I deleted the old vmdk file, rebooted the machine, and created a new vmdk file using vmkfstools -z. I'm still getting the same problem: when a WIndows VM accesses a drive through the ESXi vmdk, it appears corrupt, but not when it access the same disk via a SAS controller using PCI passthru.

I'll try updating ESXi and seeing if that solves the problem.

[TaoeDude]

OK, I've updated to Build 15160138. It hasn't made any difference.

So, to recap, the NTFS file system of this drive appears fine to Windows when it's addressing it directly through hardware, but not through an ESXi vmdk.

[HendersonD]

Win7 is end of life so I would ditch it and give win10 a go

[TaoeDude]

The same problem manifests in Win 2012R2

[continuum]

Hi

Do you have anything to work with ? vmware.logs or something like that ?

By the way - I have seen the same. My first idea would be the USB 3.1 with Windows 7 combination.
Anyway - please provide some data then I will look into it.

Ulli

[TaoeDude · Accepted solution]

The problem is when the drive is connected via the motherboards SATA controller, so I don't think the USB3.1 is an issue. Besides, that's being used via passthrough.

As it happens, I'm beginning to think the issue is with the motherboards C612 chipset, which doesn't appear to be supported by vmware (Supermicro says it's supported by ESXi 5.5U1, but I can't find any verification of this on VmWare's site)

Anyway, I installed ESXi 5.5 and had a play, and voila! no disc corruption using the motherboard's SATA controller!

But now I can't seem to passthrough the onboard SAS 3008 to Windows 2012r2. Windows hangs on boot  😛

[TaoeDude]

The SAS3008 controller issue turned out to need  pciPassthru0.maxMSIXvectors be set to 31

see:
https://kb.vmware.com/s/article/2032981