VMware Communities
cyberopsit
Contributor
Contributor

HAL_INITIALIZATION_FAILED After Upgrading Host to Windows 10 Pro

Good morning, I've upgraded my Host operating system to Windows 10 Pro and now my guests are throwing the HAL_INITIALIZATION_FAILED error once or twice a day.  Please help.

Pre Upgrade

Host:  Dell Precision Workstation

Host:  Windows 7 Professional

Host:  VMWare Workstation 10

Hosting 2 - Windows 7 Pro Virtual Machines

This configuration has been stable for about 3 years without any real issues.

Upgrade Stage 1

Host:  Dell Precision Workstation

Host:  Windows 7 Professional

Host:  VMWare Workstation 10

Hosting 2 - Windows 10 Pro Virtual Machines

For the first stage I upgraded the virtual machines to Windows 10 Pro using the in place upgrade and Windows Media Creation tool.  Once the VMs were upgraded this configuration was stable for about 2 months.

Upgrade Stage 2

Host:  Dell Precision Workstation

Host:  Windows 10 Professional

Host:  VMWare Workstation 10

Hosting 2 - Windows 10 Pro Virtual Machines

For the second stage I upgraded the Host computer to Windows 10 Pro using the in place upgrade and Windows Media Creation tool.  After upgrading the Host operating system I installed the latest version of VMWare Workstation and imported the existing virtual machines.  Since upgrading the Host computer the VMs have been extremely unstable, and they generally crash once or twice a day with the HAL_INITIALIZATION_FAILED blue screen error.  This has been going on for several weeks.

Remediation Steps Attempted

1.  I have tried building a completely new Windows 10 VM from the upgraded host, without running any in place upgrade on the VM, and this new VM is still unstable.

2.  I have disabled hibernation and sleep options on the Host machine and the two VM Guests.

Most of the articles I've seen on this say this is because the guests are going into hibernation, but I've disabled hibernation in the control panel/power options.  Should I be looking somewhere else?  Any advice would be greatly appreciated.  Thanks in advance.

0 Kudos
10 Replies
Mits2020
Hot Shot
Hot Shot

I hate to see two days passed without a reply - yesterday I started posting some ideas but I cancelled them. I have never seen a blue screen on my virtual machines and your findings are bugging me. The VMware virtual hardware is tested in a vast array scenarios by millions of users, so something else is going on. So, today it's a new day and new ideas emerge...

First is the hardware. Since everything seemed to work OK for 3 years (and then for 2 months on WIn10), you may have a hardware degradation issue.

Are you sure that the hard disk(s) where the host and guests are installed has good health (no bad sectors) and that the filesystem is intact (chkdsk)?

Have you ever tested your RAM for defects with Memtest? How much RAM do you use and how much do you allocate to the guest(s)? Is the RAM you use on your motherboard's Qualified Vendor List?

Since you have a vanilla Dell workstation I am assuming there is no exotic hardware in your PC like e.g the latest generation of Intel or AMD CPUs, weird network cards, scsi controllers etc, correct?

Is your motherboard's BIOS updated to the latest version? Are you using any kind of overclocking?

Finally, do you have a decent power supply (or has it perhaps aged) that can handle all your hardware properly?

Then is the software. Since everything seemed stable with Windows 7, then Windows 10 may be the issue,.

Please provide details on how you are creating the Windows 10 virtual machines. Are you using original iso files from MS (checked with checksums), or are you using a CD or USB? What build/flavor is this Windows? Have you tried using the latest ISO from Microsoft?

Are you using any exotic software on the host that requires intensive 3D graphics? Have you tried disabling 3D acceleration on the guests?

Finally, are you using default options for the CPU virtualization? In Virtual Machine Settings - Processors the three checkboxes for the Virtualisation Engine should be unchecked. Also how many cores has your CPU host and how many do you use in the VMs? Have you tried reducing the number of cores for the VMs (e.g 2 instead of 4) to allow possibly better scheduling for the host?

That's all I can think for now. I hope somebody else has more ideas.

0 Kudos
cyberopsit
Contributor
Contributor

Hi Mits2020 and thank you for taking the time to write your reply.  I've been away on vacation and here is what I've done.

Hardware: 

Dell Precision Tower 3420

Processor: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz

Physical Cores: 4

Memory: 31.9 GB

Display Adapter:  Intel(R) HD Graphics 530 AMD FirePro W2100 (FireGL V)

BIOS Name: BIOS Date: 06/14/19 02:44:12 Ver: 2.13.1

BIOS Version: 2.13.1

BIOS Release Date: 2019-06-13

Motherboard: Dell Inc. 08K0X7

1. First is the hardware. Since everything seemed to work OK for 3 years (and then for 2 months on WIn10), you may have a hardware degradation issue.  Are you sure that the hard disk(s) where the host and guests are installed has good health (no bad sectors) and that the filesystem is intact (chkdsk)?  Have you ever tested your RAM for defects with Memtest? How much RAM do you use and how much do you allocate to the guest(s)? Is the RAM you use on your motherboard's Qualified Vendor List?

Response:  I ran Dell full hardware diagnostics on all components, including exhaustive tests. It ran for hours.  I also ran chkdsk, and checked the event logs for any obvious hardware, disk or file system errors and didn't see anything obvious.

2.  Since you have a vanilla Dell workstation I am assuming there is no exotic hardware in your PC like e.g the latest generation of Intel or AMD CPUs, weird network cards, scsi controllers etc, correct?

Response:  Correct, this is a standard Dell build, no custom hardware or components.

3.  Is your motherboard's BIOS updated to the latest version? Are you using any kind of overclocking?

Response:  Bios and all hardware drivers are completely updated, and no overclocking or any custom fans or anything.  See Bios info above

4.  Finally, do you have a decent power supply (or has it perhaps aged) that can handle all your hardware properly?

Response:  This is the same basic 350 Watt power supply that shipped with the system.  I have no reason to believe it's faulty, since these problems basically began after I upgraded the Host OS to Windows 10 Professional.

5.  Then is the software. Since everything seemed stable with Windows 7, then Windows 10 may be the issue,.  Please provide details on how you are creating the Windows 10 virtual machines. Are you using original iso files from MS (checked with checksums), or are you using a CD or USB?

Response:  For the Host I ran an in place upgrade using the Microsoft Windows Media Creation Tool.  I've used this tool a bazillion times with no problems and the Host OS seems completely stable.

6.  What build/flavor is this Windows? Have you tried using the latest ISO from Microsoft?

Response:  The Host and VMs are running Windows 10 Professional.  Remember that I upgraded the VMs from Windows 7 to Windows 10 first, and then let them run for weeks before performing the upgrade on the Host.  So the in place upgrade on the VMs was successful and the VMs were stable for weeks after the Windows 10 upgrade.  However, after we upgraded the host and stared having problems with the VMs, I actually created an entirely new Windows 10 Professional VM from scratch and it's crashing as well.  I don't believe it's the upgrade or installation media.

7.  Are you using any exotic software on the host that requires intensive 3D graphics?

Response:  No exotic software that is graphic intensive on the Host or the VMs. 

8.  Have you tried disabling 3D acceleration on the guests?

Response:  Are you talking about in the VM configuration, disabling 3D Acceleration?  No I have not disabled 3D Acceleration, I'll try this tonight.

9.  Finally, are you using default options for the CPU virtualization? In Virtual Machine Settings - Processors the three checkboxes for the Virtualisation Engine should be unchecked. Also how many cores has your CPU host and how many do you use in the VMs? Have you tried reducing the number of cores for the VMs (e.g 2 instead of 4) to allow possibly better scheduling for the host?

Response:  This is how the VM guests are configured.

8GB RAM

Number of Processors:  1  (This was set to 2, I changed it to 1 but still crashing)

Number of Cores per processor:  2

Total Processor Cores:  2

Virtualization Intel VT-x/EPT or AMD-V/RVI is deselected

Virtualize CPU Performance counters is deselected

Additional Information:  I also disabled VSS on the system drive so the VSS service is no longer writing recovery points for the drive the guests are on.

i appreciate the help

0 Kudos
Susie201110141
VMware Employee
VMware Employee

Hi cyberopsit

What's your current workstation version ?Could you upgrade to latest 15.5.5 version?

We have fixed several related HAL_INITIALIZATION_FAILED issues in latest 15.5.5 release.

I can update here if you still have the issue with latest 15.5.5 release.

0 Kudos
Mits2020
Hot Shot
Hot Shot

I should have seen this earlier. I'm afraid we both lost valuable time... Sorry Smiley Sad

In VMware Compatibility Guide - Guest/Host Search  if you search for Guest compatibility for Win10, you'll see that Win10 is supported as a guest from Workstation version 14.0 on.

Win10guestVM.jpg

Therefore, the chances of running Win10 properly on Workstation v10 that you are using are slim. The HAL problems you experience are probably due this guest incompatibility. Therefore, consider upgrading Workstation to v15.5.2 (don't use v15.5.5 yet, it's too fresh).

0 Kudos
cyberopsit
Contributor
Contributor

Thank you Susie and Mits2020. 

After I performed the Windows 10 Professional Upgrade on the Host I upgraded to VMWare Workstation v15.5.2

This morning I upgraded to Workstation 15.5.5 as per Susie's recommendation.  I'll report back on the status soon.

Thanks for your help

0 Kudos
pizzalord
Contributor
Contributor

Hi I just wanted to chime in that I have been having this problem for about a year. I am running VMWare Workstation 15.5.7 on an HP Z640 workstation,

Win 10 Pro 64 bit Build 1909

Intel Xeon E5-2623 v3
32 GB RAM

 

A coworker has the same problem and I wonder if our company's IT management is somehow conflicting with VMWare. The problem seems to occur if I remote desktop into the PC that is running vmware. Typically I remote desktop into the VMs when I am using them, but I need to remote into the host PC fairly frequently to perform other tasks. I am usually only running 1 or 2 VMs at a time, each VM is given 8 GB of RAM, 1 processor / 4 cores, 100 GB of storage space on a 1 TB NVMe SSD. The VMs perform otherwise with no issue aside from the frequent "HAL_INITIALIZATION_FAILED" blue screen. If the VM blue screened and rebooted it would be annoying, but unfortunately the VM just goes non responsive until I remote into the host PC and manually reboot it.

0 Kudos
codegrinder89
Contributor
Contributor

In our company we have the exact same behavior on many different host machines (Win 10, Server 2019). When using remote desktop to connect to the host and you let the connection open for a while (more than 30 minutes or an hour, cannot say an exact time) without moving the mouse, on all open VMs you see the same blue screen. The VMs themselves are Windows 10, when analyzing the  Crash Dump with WinDbg, we can see that the error is related to the VMWare Tools:

-----------------------------

Microsoft (R) Windows Debugger Version 10.0.19041.685 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:\Temp\MEMORY.DMP]
Kernel Bitmap Dump File: Kernel address space is available, User address space may not be available.


************* Path validation summary **************
Response Time (ms) Location
Deferred SRV*c:\symbols*http://msdl.microsoft.com/download/symbols
Symbol search path is: SRV*c:\symbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows 10 Kernel Version 18362 MP (4 procs) Free x64
Product: WinNt, suite: TerminalServer SingleUserTS
Built by: 18362.1.amd64fre.19h1_release.190318-1202
Machine Name:
Kernel base = 0xfffff802`4a200000 PsLoadedModuleList = 0xfffff802`4a648150
Debug session time: Fri May 7 20:54:39.318 2021 (UTC + 2:00)
System Uptime: 1 days 5:33:55.507
Loading Kernel Symbols
...............................................................
............................Page 14dcfe not present in the dump file. Type ".hh dbgerr004" for details
....................................
...........................Page 127191 not present in the dump file. Type ".hh dbgerr004" for details
..........................
Loading User Symbols
PEB is paged out (Peb.Ldr = 0000000d`61761018). Type ".hh dbgerr001" for details
Loading unloaded module list
.......
For analysis of this file, run !analyze -v
2: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

HAL_INITIALIZATION_FAILED (5c)
Arguments:
Arg1: 0000000000000115
Arg2: fffff7e680015d30
Arg3: 000000000000172f
Arg4: ffffffffc0000001

Debugging Details:
------------------


KEY_VALUES_STRING: 1

Key : Analysis.CPU.Sec
Value: 4

Key : Analysis.DebugAnalysisProvider.CPP
Value: Create: 8007007e on MCHN16920

Key : Analysis.DebugData
Value: CreateObject

Key : Analysis.DebugModel
Value: CreateObject

Key : Analysis.Elapsed.Sec
Value: 4

Key : Analysis.Memory.CommitPeak.Mb
Value: 70

Key : Analysis.System
Value: CreateObject


VIRTUAL_MACHINE: VMware

BUGCHECK_CODE: 5c

BUGCHECK_P1: 115

BUGCHECK_P2: fffff7e680015d30

BUGCHECK_P3: 172f

BUGCHECK_P4: ffffffffc0000001

ADDITIONAL_DEBUG_TEXT: Halfailures structure in bugcheck.cpp needs updated. Please information OCA (alias:werka).

BLACKBOXBSD: 1 (!blackboxbsd)


BLACKBOXNTFS: 1 (!blackboxntfs)


BLACKBOXPNP: 1 (!blackboxpnp)


BLACKBOXWINLOGON: 1

PROCESS_NAME: vmtoolsd.exe

STACK_TEXT:
ffffbe81`854d8eb8 fffff802`4ad078eb : 00000000`0000005c 00000000`00000115 fffff7e6`80015d30 00000000`0000172f : nt!KeBugCheckEx
ffffbe81`854d8ec0 fffff802`4ad07252 : 000000fc`a6bbe34f 00000000`0000172e fffff7e6`80023ac0 fffff802`4ad23048 : hal!HalpVpptUpdatePhysicalTimer+0x143
ffffbe81`854d8f00 fffff802`4acb83f6 : 000000fc`a6bbe34f ffff8f8e`7f7654c0 ffff8f8e`7bc3f510 00000000`00000000 : hal!HalpVpptAcknowledgeInterrupt+0x102
ffffbe81`854d8f30 fffff802`4a22cc75 : 000000f7`d0849551 ffffe60c`c20ba200 ffffe60c`c20ba2b0 ffffa05f`c9925c4d : hal!HalpTimerClockInterrupt+0x36
ffffbe81`854d8f60 fffff802`4a3c3e1a : ffff8f8e`7f765540 ffffe60c`c20ba200 00000000`00000000 ffffe60c`c20ba200 : nt!KiCallInterruptServiceRoutine+0xa5
ffffbe81`854d8fb0 fffff802`4a3c4387 : 00000000`00000000 ffff8f8e`7f765540 ffff8f8e`7f765540 ffffffff`94b62e00 : nt!KiInterruptSubDispatchNoLockNoEtw+0xfa
ffff8f8e`7f7654c0 00007ffe`77bfa7b0 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiInterruptDispatchNoLockNoEtw+0x37
0000000d`618ff208 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x00007ffe`77bfa7b0


SYMBOL_NAME: nt!KiCallInterruptServiceRoutine+a5

MODULE_NAME: nt

IMAGE_NAME: ntkrnlmp.exe

STACK_COMMAND: .thread ; .cxr ; kb

BUCKET_ID_FUNC_OFFSET: a5

FAILURE_BUCKET_ID: 0x5C_nt!KiCallInterruptServiceRoutine

OS_VERSION: 10.0.18362.1

BUILDLAB_STR: 19h1_release

OSPLATFORM_TYPE: x64

OSNAME: Windows 10

FAILURE_ID_HASH: {8e155cd0-c545-25d5-7ee1-936e57893c8c}

Followup: MachineOwner
------------------------

 

 

0 Kudos
vfandrea
Contributor
Contributor

Same behaviour , anyone find a solution ? 

0 Kudos
fpivot95
Contributor
Contributor

Did anyone fix the issue? I'm having the same problem

0 Kudos
glassyant
Contributor
Contributor

For a temporary fix, you could add an auto RDP logoff after 15 minutes of idle time.

0 Kudos