One of the VMs on my ESXi hangs for 6-7 minutes on boot.
I have found some threads suggesting VMFS5 as a solution but I already tried that and It didn't solved the issue. Also all these threads are old and for ESXi 6.5 so I think VMFS is not the issue in my case.
Below you can see some part of the log where the hang occurs. Full log available as an attachment. I have did some research but I couldn't find any solution to fix this issue.
2021-03-26T14:33:42.319Z| vcpu-0| I005: Guest MSR write (0x49: 0x1) 2021-03-26T14:33:42.418Z| vcpu-0| I005: Hyper-V guest: management OS identified as 0x0001040a:0x0000271b. 2021-03-26T14:33:42.470Z| vcpu-0| I005: Preparing for SPEC_CTRL Guest MSR write (0x48) passthrough. 2021-03-26T14:33:43.297Z| vcpu-1| I005: CPU reset: soft (mode Interp) 2021-03-26T14:33:43.300Z| vcpu-2| I005: CPU reset: soft (mode Interp) 2021-03-26T14:33:43.304Z| vcpu-3| I005: CPU reset: soft (mode Interp) #### ~7 minutes of nothing #### 2021-03-26T14:40:10.919Z| vcpu-2| I005: SVGA: Unregistering IOSpace at 0x1070 2021-03-26T14:40:10.920Z| vcpu-2| I005: SVGA: Unregistering MemSpace at 0xe8000000(0xe8000000) and 0xfe000000(0xfe000000) 2021-03-26T14:40:10.920Z| vcpu-2| I005: SVGA: FIFO is already mapped 2021-03-26T14:40:10.922Z| vcpu-2| I005: SVGA: Registering IOSpace at 0x1070 2021-03-26T14:40:10.922Z| vcpu-2| I005: SVGA: Registering MemSpace at 0xe8000000(0xe8000000) and 0xfe000000(0xfe000000) 2021-03-26T14:40:10.923Z| vcpu-2| I005: SVGA: FIFO is already mapped
(Updated) ESXi-7.0U1c-17325551-standard (VMware, Inc.)
Guest OS Microsoft Windows 10 (64-bit) VBS not enabled
Compatibility ESXi 7.0 U1 virtual machine
Edition Windows 10 Pro N Version 20H2 OS build 19042.868
Experience Windows Feature Experience Pack 120.2212.551.0
Hmm, that is most likely the OS doing some work (it could be the monitor but that is less likely), Windows boot troubleshooting is your best bet here. Is the VM doing any IO?
If you want to profile it with help from the ESXi side, you could configure "monitor_control.log_vmsample = true", boot the VM, suspend towards the end of the "hang" and use e.g. awk / uniq -c / sort -nr on the vmware.log the get the highest hitting RIPs for the relevant time frame. Then use https://flings.vmware.com/vmss2core to convert the vmem / vmss to a WinDbg readable dump, configure symbols and "unassemble" the RIPs to get a better idea what Windows is doing.
I have got the highest hitting RIPs.
Downloaded vmss and vmem files from datastore and tried to convert these to WinDbg readable dump. But I got following errors. I did some research online and some says some windows version is not compatible with this tool. Maybe my windows (Windows 10 Pro N Version 20H2 OS build 19042.868) is also not compatible?
vmss2core with -W (directly finishes with no dump file)
vmss2core with -W8 (just stops after some time with empty memory.dmp file)
Hmm, does it exit or crash? Can you check with e.g. procdump? I.e. is it trying to print "Couldn't find encoded DDB" or is it failing before ? (i.e. it isn't something stupid like not enough space / write permission etc.) ... Can you run vmss2core -C *vmss | findstr cr3?
Oh, since your guest actually boots, you could just let it finish and use LiveKd with WinDbg. I'd still be curious about the vmss2core failure though (not curious enough ATM to repro though).