I will try and post a complete syllabus here:
Started day #0 with 2 hosts running esxi4 latest update. All was well. I have about a dozen Windows 2000 terminal servers running Citrix Metaframe XP that host my primary ERP as well as a few other very specific apps. Ive been trucking along like that without change for a solid 2 years (aside from VMWare updates / patches). I have a few other servers, but they are 2008 R2 for the most part. Every other server OS has been problem free and is not part of this scenario.
day #1 (upgraded 1 HOST to esxi5 and replaced the other HOST with new hardware, installed esxi5 from scratch). My current setup on this cluster is 2 hosts, 1 Dell R610, latest everything, 1 Dell R710, latest everything. Identical CPU's (to the stepping) and identical RAM. Did the updates throughout the night. Storage is a NetApp FAS2020. Switching is Cisco 3750G. I got the hosts to esxi 5, and updated the guest VMWare tools (change user /install on the TS boxes) but did not upgrade hardware version on anything.. just in case. There were no infrastructure changes, no hardware changes, and all host settings were retained identically. The only hardware change was the Dell server replacement with greater horsepower, identical CPU and more RAM.
Next morning, 0800 sharp, phone is ringing off the hook. Primary ERP application is unusable.
I spent the next 4 days of my life doing everything under the sun. I mean everything to the point of even rebuilding the terminal servers. After 4 days of not sleeping, and my end users loading their muskets, I called VMware. The first thing they had me do was obviously update hardware version to 8. That did not resolve my problem at all, and I was then in a real pickle.
I managed to find this thread, and I saw that someone suggested they had resolved the problem by rolling back to esxi4. At this point, I just needed to solve the problem before the rioters broke down my cardboard barrier.
Because I had upgraded to HW8, I had to spend another sleepless night moving the host back to esxi4, and then using converter to bring the servers back to HW7. We then built 4 additional terminal servers so we had a larger pool with which to play with. (This allowed us to have enough TS's to run a working production system, as well as a complete test system where we could initiate incremental changes).
The next day, the phone calls stopped, and the people rejoiced. But Im still on esxi4, and I cannot use automated DRS...
From there it went to gathering logs back and forth with VMWare, but never finding much to work with. They did suggest bandwidth / NFS traffic contention, but Im still a little suspicious as simply moving everything back to esxi4 resolved the problem. That was about 4 weeks ago, so it clearly did resolve the problem.
Im now on day#2 of testing after changing the monitor mode to HWMMU. That server is operational and has not had a complaint yet. I migrated another of the TS (without switching monitor mode) onto the esxi5 host last night, and within 40 minutes I had phone calls reporting it was unresponsive. So clearly the problem still exists, but it would appear to not be affecting my server with the monitor mode changed.
Im going to change the HWMMU settings on a second guest tonight, and upgrade the hardware version on the already repaired server.
The behavior seen on the guest when they are failing is that they become unresponsive. When I can actually get a process monitor, its not like the server is actually doing anything. I have a couple nice screen shots I took, where our of the ~80 processes running, the entire task manager shows every process running like at a minimum of 3% CPU. This makes no sense. Its not like anything is happening, its like the server is simply unable to continue processing tasks. It builds and builds and builds, and then slowly becomes unresponsive until it KO's. RAM is not even used. Both VMWare and the local Windows task manager show CPU @100% eventually. At that point, it is usually easier to just power off the box, rather than try and log in to safely restart.
I will try and reiterate the fix tomorrow. I will post back.
because the workaround mentioned here and in the KB2012205 did not work for me (I'm using software virtualization 😞 not having any hardware virtualization capabilities...)