Windows 10 VM hangs 6-7 minutes on boot - ESXi 7.0U1

[khloros]

Hello

One of the VMs on my ESXi hangs for 6-7 minutes on boot.

I have found some threads suggesting VMFS5 as a solution but I already tried that and It didn't solved the issue. Also all these threads are old and for ESXi 6.5 so I think VMFS is not the issue in my case.

Windows10-1809 boot time takes 5-6 minutes  

windows 10 1809 slow 

Re: windows 10 1809 slow 

Below you can see some part of the log where the hang occurs. Full log available as an attachment. I have did some research but I couldn't find any solution to fix this issue.

2021-03-26T14:33:42.319Z| vcpu-0| I005: Guest MSR write (0x49: 0x1)
2021-03-26T14:33:42.418Z| vcpu-0| I005: Hyper-V guest: management OS identified as 0x0001040a:0x0000271b.
2021-03-26T14:33:42.470Z| vcpu-0| I005: Preparing for SPEC_CTRL Guest MSR write (0x48) passthrough.
2021-03-26T14:33:43.297Z| vcpu-1| I005: CPU reset: soft (mode Interp)
2021-03-26T14:33:43.300Z| vcpu-2| I005: CPU reset: soft (mode Interp)
2021-03-26T14:33:43.304Z| vcpu-3| I005: CPU reset: soft (mode Interp)

#### ~7 minutes of nothing ####

2021-03-26T14:40:10.919Z| vcpu-2| I005: SVGA: Unregistering IOSpace at 0x1070
2021-03-26T14:40:10.920Z| vcpu-2| I005: SVGA: Unregistering MemSpace at 0xe8000000(0xe8000000) and 0xfe000000(0xfe000000)
2021-03-26T14:40:10.920Z| vcpu-2| I005: SVGA: FIFO is already mapped
2021-03-26T14:40:10.922Z| vcpu-2| I005: SVGA: Registering IOSpace at 0x1070
2021-03-26T14:40:10.922Z| vcpu-2| I005: SVGA: Registering MemSpace at 0xe8000000(0xe8000000) and 0xfe000000(0xfe000000)
2021-03-26T14:40:10.923Z| vcpu-2| I005: SVGA: FIFO is already mapped

ESXi Details
(Updated) ESXi-7.0U1c-17325551-standard (VMware, Inc.)

VM Details
Guest OS Microsoft Windows 10 (64-bit) VBS not enabled
Compatibility ESXi 7.0 U1 virtual machine

OS Details
Edition Windows 10 Pro N Version 20H2 OS build 19042.868
Experience Windows Feature Experience Pack 120.2212.551.0

[vbondzio]

When you check esxtop during that time and expand the GID of the VM, what do you see? Is the VM burning CPU? %VMWAIT for any vCPUs?

[khloros]

VM using %99-100 CPU until it finish booting. But I didn't see any large amount of %VMWAIT for vCPUs on boot. Only ~0.01 for couple of seconds.

[vbondzio]

Hmm, that is most likely the OS doing some work (it could be the monitor but that is less likely), Windows boot troubleshooting is your best bet here. Is the VM doing any IO?

If you want to profile it with help from the ESXi side, you could configure "monitor_control.log_vmsample = true", boot the VM, suspend towards the end of the "hang" and use e.g. awk / uniq -c / sort -nr on the vmware.log the get the highest hitting RIPs for the relevant time frame. Then use https://flings.vmware.com/vmss2core to convert the vmem / vmss to a WinDbg readable dump, configure symbols and "unassemble" the RIPs to get a better idea what Windows is doing.

[khloros]

I have got the highest hitting RIPs.

246 rip=0xfffff80338894bf5
141 rip=0xfffff80338dadbd1
124 rip=0xfffff8451703b343
115 rip=0xfffff80338dadd5c
76 rip=0xfffff80338894c5a
66 rip=0xfffff80338894c49
61 rip=0xfffff8033886c62a
55 rip=0xfffff80338894c36
48 rip=0xfffff80338936320
47 rip=0xfffff80338894be0
36 rip=0xfffff80338dadbd4
22 rip=0xfffff80338dadd5f
22 rip=0xfffff803389b1d79
22 rip=0xfffff8033886ca2e
17 rip=0xfffff803389b1df4
16 rip=0xfffff80338932b29
15 rip=0x2de6
12 rip=0xfffff8033886c62e
11 rip=0xfffff803389b1e3d
11 rip=0xfffff80338894c63

Downloaded vmss and vmem files from datastore and tried to convert these to WinDbg readable dump. But I got following errors. I did some research online and some says some windows version is not compatible with this tool. Maybe my windows (Windows 10 Pro N Version 20H2 OS build 19042.868) is also not compatible?

vmss2core with -W (directly finishes with no dump file)

vmss2core with -W8 (just stops after some time with empty memory.dmp file)

[vbondzio]

Hmm, does it exit or crash? Can you check with e.g. procdump? I.e. is it trying to print "Couldn't find encoded DDB" or is it failing before ? (i.e. it isn't something stupid like not enough space / write permission etc.) ... Can you run vmss2core -C *vmss | findstr cr3?

[vbondzio]

Oh, since your guest actually boots, you could just let it finish and use LiveKd with WinDbg. I'd still be curious about the vmss2core failure though (not curious enough ATM to repro though).