Hi Joerg,

I don't know how you designed your environment, but despite of our highly redundand hardware (8 NIC, 8 SAN), UPS ,multi-path everywhere, there are still a number of failures you can not provide redundancy for (i.e. motherboard failure, CPU failure, local RAID controller itself), plus failure of ESX virtualization layer itself in this case.

To circumvent such kind of issues, VMware designed HA/DRS/VMotion. But this is not enough to protect you fully. You have to improve global redundancy at the host level by keeping N+1 redundancy. If your environment is Production only and critical, you should be able to loose one server in the cluster and restart the VM's on the remaining ones.

I'm the first to recognize that such a bug is a shame for VMware but don't blame them for poor design.

If you want to be proactive, I suggest you find the ESX server hosting the less domageable VM's you have (maybe one hosting only dev servers, or the one with the least VM's or the one where customers are the most tolerant) and stop all VM's on it (an orderly shutdown is always better than a crash), restart them elsewhere on 3.5U2 and downgrade the empty box to 3.5U1. You can then use VMotion to move VM's from another ESX 3.5U2 server (it is working, i tested it) and perform a "rolling downgrade" of your infrastructure. Re-installing ESX should not take more than one hour and you will be in a much better situation when VMware release the fix, regardless of the form it will have, ISO, RPM ...

Hope this helps, best regards,

Ludovic

I agree with your, however, I did not Update to this "update 2" level manually.

Apparently, major update releases are put in place, automatically by Update manager!
I was unaware that Update Manager would do this, i thought it only put out interim patches, and not major OS updates!

We have (now HAD) Update Manager running the updates monthly.
Apparently this got put in the update sequence for installation and I didn't see it.

I have Change Control documents in the pipeline for U2, and had expected to install in phases.
(the right way)

But now, i'm apparently all updated. (Except for VirtualCenter and VCB).

Actually, one of the VMware customers I work with suggested this enhancement request:
Remove the license server from VMware completely. Most customers are honest anyway, and if someone wants to cheat he can find ways to do so, anyway.

Update from kb-article:

An issue with ESX/ESXi 3.5 Update 2 causes the product license to expire on August 12, 2008. VMware engineering has isolated the root cause of this issue and {color:#ff0000}will reissue the various upgrade media including the ESX 3.5 Update 2 ISO, ESXi 3.5 Update 2 ISO, ESX 3.5 Update 2 upgrade tar and zip files by noon, PST on August 13. These will be available from the page: {color:#000000}http://www.vmware.com/download/vi. Until then, VMware advises against upgrading to ESX/ESXi 3.5 Update 2.
The Update patch bundles will be released separately later in the week. This KB article will be updated as soon as more information is available, check back frequently for updates and additions.

Daniel

Even worse, the 36-hr solution will be for ISO and upgrade TARs only... Patches are to arrive even later than that... Somewhere this week

I've just tried it. Still broken on the 13th.

A little bit of calm please. This was a major screw up, but 36 hours is a really short time to fix this particular issue. As I stated in a previous post, this fix is most likely going to touch many many places in the code-base. Do you really want them to compound the problem by rushing out a broken set of patches that could be an even bigger disaster? This bug doesn't lose data, there are some simple workarounds that will allow you to boot your VM's without too much impact.

Right now the engineers are having to create a new branch off of the 3.5u2 code-base, apply the fixes for this problem, compile the entire payload, pull all the QA people from every country that ESX is released in (all different time zones) off of what they were doing to make sure they didn't break something else and put everything up on the site (making sure to remove all the legacy broken stuff.) This is usually a 2-3 week process, 36 hours is better than nothing. And if you really think Microsoft will serve you better and more effectively, well maybe you should give their product a try and see how far that gets you. We'll be here when you're done with that...