Disable HA until your all good with everything. I've had HA give me fits doing this type of thing before.
Use esxupdate instead of update manager to install the patch
I did 1 host. There's no reboot of the host? I've got one eye on the Olympics and one on VC. Could I have missed the reboot?
As a VMware Enterprise Partner and VMware Authorized Consultant
I can tell you this IS a big deal for VMware to release a product that has such grave consequences for even a relatively small portion of the total VMware user population. A small percentage does not diminish the severity of problem for affected users and the upmost urgency is expected from a company that caters to enterprise customers who don’t have “downtime” in their corporate dictionary anymore.
As said previously, bugs happen. However, I believe this could have been prevented by not rushing an update to market which was intended to be free and compete with Hyper V. VMware ran face first into the very hurdle it was trying to clear by releasing a free version of its hypervisor to compete with Microsoft’s recent release. This will no doubt teach VMware a lesson and unfortunately will cast doubt about the reliability of VMware in the enterprise. It’s a shame a clearly superior product is going to get bad publicity from this oversight.
Virtualization is here to stay, and VMware has been the leader in this arena for good reason. Let’s give them credit and hope they learn from their mistakes.
I did not see the new build# until I rebooted...
I have successfully fixed the fist host using the "Update Manager" process. If you have the Update Manager plugin and installed in your VC, you can update as follows:
1. Click on "Update Manager" icon in toolbar
2. Click the "Plugins" menu item and select "Update Manager > Schedule Update Download"
3. Make sure "ESX Server Updates" is selected, click "Next"
4. Change Frequency to "Once" and then select start time of "Now"
5. Click Next > Next > Finish
Watch in the status window at bottom of screen for that to complete and then:
6. Click on "Inventory" icon, click on the impacted host and shut down all the guests
7. Once all guests are shut down put it in maint mode
8. Set the time back to the correct time if you made this workaround.
9. Right click on host and select "scan for updates"
10. Watch in the status window and once complete, right click on host and select "Remediate"
11. Make sure critical updates is selected and click next. You should notice here that there is a red cross next to critical updates stating there is one missing.
Server will now patch, once Complete, exit maint mode and start the guests back up. I did not have to reboot.
I havn't checked VMotion is working, but I did see the new version number without rebooting.. HTH...
My Vmotion issue is resolved - totally unrelated someone messed up my vmotion network today - ahh
To share my experience:
I had a host with VMs on it that could tolerate limited downtime, so went for it. I had SSH connected with the esxupdate command ready to go, and then shutdown the VMs and put the host in maintenance mode. I applied the patch, restarted vmware-hostd, and then restarted the servers. I believe the downtime was under 10 minutes.
I just tested, and VMotion works to that server now from my unpatched servers, so I can put the others in Maintenance mode and patch under less pressure.
I updated one of my 5 hosts and everything appears to be fine so far. As bad timing has it I upgraded the first 2 yesterday and this third one this morning which pretty much left it in a broken state since I could not power anything on with it nor could I vmotion anything over. I have disabled HA so if I lose one of my U1 servers it does not try to start VMs on U2 broken servers. I put my broken server in to maint mode and patched it using Update Manager. It worked fine although it takes a while for the post update scan.
After taking it out of maint mode I fired up a bunch of test VMs and vmotioned one over to a U1 node and back to the patched node successfully. I'm going to let this box run overnight and if there are no bad reports in the morning from you folks on the other side of the world I'll update my other U2 hosts and reenable HA.
Thanks for all the hard work getting this patch out. Unfortunately I think you've gotten a serious black eye today. We were finally getting management happy with the idea of using ESX for production servers and this set us back a little bit.
Letter from CEO Paul Maritz has just been posted:
[VMware Communities User Moderator|http://communities.vmware.com/docs/DOC-2444][/i]
The express patches have been posted. This thread is long.
I'd normally just lock this thread, but there are a lot of incoming links and a lot of people with email notifications on this one.
However, it's really unmanageable for most people, so please let's move the conversations to the above two threads.
I'm having a serious problem with the patch. I patched one host and it was OK as I started machines on it, but then I see infinite loops of machine state changes from ON to RECONFIGURING and back to ON when I look at the latest hostd.log. They are coming from VirtualCenter Server, but I can't figure out why or how to stop it from happening. It's keeping hostd too busy to do anything else, so I can't move machines to that host.
The difference between the word "licensing" and "time bomb" is semantics in this case....
Agreed. Simply put, what the point is, is that in order to combat some envisioned piracy scenario, VMware installed into ESX a deliberate SPF whose as-designed failure mode is capable of turning your data center into a sea of chaos.
And, the appropriate response to even the mere existance of such a potentially devastating logic-bomb in what is supposed to be enterprise-class software can be summarized even more simply: UNACCEPTABLE.