Windows Server 2000 BSOD Stop 0x0000001e after applying Patch 8 Build 4179633

[joboyle]

Quite by accident I went to reboot a Windows Server 2000 VM on an ESXi 5.5 host after applying the August ESXi550-201608001 patch release, build 4179633. This also is called Patch 8 on the build correlation web page. And I am getting a BSOD Stop 0x0000001e - worker_thread_returned_at_bad_IRQL. I can boot it into both Safe Mode and Safe Mode with Networking, but not do a regular boot. It will boot OK on an Express Patch 10 ESXi 5.5 host (Build 3568722) and I can migrate it to the Patch 8 host and it will run OK, just not boot.

I have a backup copy of this vm, made prior to installing VMware Tools, and it has the same problem, so it's not a Tools related issue. I tried opening a case with VM Technical Support, but they won't touch it 'cause Win 2K.

What I'm guessing is that somewhere along the boot process it's probing a certain device and that's the problem. What device is the question. I've tried uninstalling and removing things that I could find, but no luck. This VM was a p2v conversion of an IBM Server from years ago, so there's probably something there from that conversion.

I do have an Windows Server 2000 installation CD. Interestingly the installer hung during the scan and configuration of a new vm on the Patch 8 host. I was able to run it on the Express Patch 10 host and create a vanilla W2K vm. And that boots OK on the Patch 8 system.

So, I've been guessing that this might be due to the USB stuff that was patched, but I have no real way of telling. We're a small shop, with only three ESXi hosts, so leaving a host at Patch 8 is not really a good option. This particular vm (and a second one similar to it) is part of the expense reporting system, so it's fairly important. And I'm not sure we have the necessary install materials to reinstall the application from scratch or I'd just rebuild the vm.

So, thought I'd see if anyone had anything to offer. Thanks for reading.

[joboyle]

It occurs to me to add the these vm's are version 8, in case that makes a difference. I will probably try different versions, just in case.

[joboyle · Accepted solution]

I thought I'd update this, in case anyone finds this. Nothing worse than finding something that matches your problem and finding out that it wasn't updated to reflect the outcome.

I did try creating a vm version 7, and while this solved the problem of the Windows 2000 installer hanging, it did not fix the stop E1 issue.

The long and short of it was that the problem was being caused by a driver twgsysin.sys, which is part of the IBM Director Agent installation. In this case v4.11.0 was what was installed. We long since stopped using IBM Director, but the uninstaller that comes with the Agent installation doesn't always cleanly complete and a lot of stuff is left hanging around. Renaming twgsysin.sys to twgsysin.old was enough to get the system to boot on the newly patched ESXi hosts.

IBM released an uninstall script called dirclean. I found a version 2.1.7 on the web that covered the version of the Director Agent installed. This is a script that runs and removes the stuff that the regular uninstaller failed to remove. Apparently this was a common enough issue to do this. Running this deletes the twgsysin driver among a lot of other stuff in the registry.

Two other programs that were very helpful in resolving this was driverwiew and bluescreenview. The first will list the drivers loaded and running. I used this to create a list of drivers running in safe mode and then a list of the drivers running when the system was normally booted. Then crossing off the drivers common to both lists, gave me a list of drivers that were suspect. twgsysin.sys was one of the ones on this list. bluescreen interprets the contents of the minidump created when a BSOD occurs. Among other things, it gives a list of drivers loaded, I think in the order they were loaded, as twgsysin.sys was the last driver in the list. So that made it pretty obvious.

So, hopefully this helps someone down the road.

[dariusd]

Superb effort with troubleshooting this, and thank you for taking the time to post back with the outcome.

I thought I'd update this, in case anyone finds this. Nothing worse than finding something that matches your problem and finding out that it wasn't updated to reflect the outcome.

An uncommon and problematic third-party driver can be a troubleshooting nightmare.  I can't help but think that you've prevented this: xkcd: Wisdom of the Ancients

Windows 2000 has a few defects which can cause hangs during installation.  One of the more common hangs we've seen is caused by a defect in the guest OS's USB drivers, which sometimes requires that you start a fresh installation (or simply configure the VM with no USB controller) to get past it.  Using the last service packs will reduce (but still not eliminate) the risk of a hang during installation.

Cheers,

--

Darius