I think you'll see it shortly, they want to make sure it is tested before releasing it. It wouldn't do any good to release something that may introduce other problems. A few of us were contacted about 4 hours ago to see if we wanted to help test the patch. Hopefully it appears soon, I'm sure VMware will make an announcement to this thread as soon as it's ready.

Eric Siebert
VMware Communities User Moderator
-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=-
Check out my website: VMware-land
Read my virtualization blog: SSV Blog
-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=-

Dear VMware Customers,

Please find the latest update about the product expiration issue. We are staging the express patches and expect it to complete in an hour. When the staging is done, we will send out a communication with more details.

Please see FAQ 1) for details about these express patches. In FAQ 2), we describe what upgrade media and update patch bundles to be release later are for. These are the updates since our last communication.

Complete information on the ESX/ESXi 3.5 Update 2 issue follows:

Problem:

An issue has been discovered by many VMware customers and partners with ESX/ESXi 3.5 Update 2 where Virtual Machines fail to power on or VMotion successfully. This problem began to occur on August 12, 2008 for customers that had upgraded to ESX 3.5 Update 2. The problem is caused by a build timeout that was mistakenly left enabled for the release build.

The following message is displayed in the vmware.log file for the virtual machine:

This product has expired. Be sure that your host machine's date and time are set correctly.
There is a more recent version available at the VMware web site: http://www.vmware.com/info?id=4.

---

Module License Power on failed.

Affected Products:

•- VMware ESX 3.5 Update 2 & ESXi 3.5 Update 2.

• - The problem will be seen ifESX350-200806201-UG is applied to a system.
• - No other VMware products are affected.

What has been done?

• - VMware removed the ESX 3.5 Update and ESXi 3.5 Update 2 binaries from the download pages in the evening of August 11, 2008 PST.
• - VMware Engineering teams have isolated the cause of the problem and are working around the clock to deliver updated builds and patches for impacted customers.
• - A Knowledgebase article has been published (http://kb.vmware.com/kb/1006716) and is being refreshed regularly.

Resolution:

VMware Engineering has produced express patches for impacted customers that will resolve the issue.

FAQ:

1. What will the express patches do?

There are two express patches: one for ESX 3.5 Update 2 and one for ESXi 3.5 Update 2. They are specifically targeted for customers who have installed or fully upgraded to ESX/ESXi 3.5 Update 2 or who have applied the ESX350-200806201-UG patch to ESX/ESXi 3.5 or ESX/ESX 3.5 Update 1 hosts. For customers who haven't done either, these express patches should not be applied.

To be noted is that these patches have been validated to work with esxupdate. However, testing with the VMware Update Manager is still under way. In subsequent communications, we will provide confirmation whether the patches work with VMware Update Manger or if a re-spin is required.

To apply the patches, no reboot of ESX/ESXi hosts is required. One can VMotion off running VMs, apply the patches and VMotion the VMs back. If VMotion capability is not available, VMs need to be powered off before the patches are applied and powered back on afterwards.

We are currently testing an option to apply the patch without requiring VMotion or VM power-off and re-power-on at the point of patch application. To immediately refresh vmx on the VM, one can VMotoin off running VMs, apply the patches and VMotion the VMs back. If VMotion capability is not available, VMs can be powered off before the patches are applied and powered back on afterwards.

2. When will VMware reissue the upgrade media and patch bundles?

VMware plans to reissue upgrade media by 6pm, August 13 PST and all update patch bundles later in the week. We will provide an ETA for the update patch bundles subsequently.\

NOTE:

• An upgrade media refers to ESX 3.5 Update 2 ISO, ESXi 3.5 Update 2 ISO, ESX 3.5 Update 2 upgrade tar and zip files. They are for customers who haven't installed or upgraded to ESX/ESXi 3.5 Update 2 but wish to.
• The "patch bundles" here refer to those released at GA. They are for customers who do not wish to do a full upgrade to ESX/ESXi 3.5 Update 2, but apply patches that are deemed necessary to hosts running ESX/ESXi 3.5 or ESX/ESXi 3.5 Update 1. They are not the same as the express patch which is described above.

3. Why does VMware plan to reissue the upgrade media before the patch bundles?

Since we can complete building and testing of the upgrade media before the patch bundles, we want to make that available to customers right away instead of reissuing all the binaries later in the week.

4. Can VMware issue a patch that opens the licensing backdoor in the next hour as a critical measure?

There is no licensing backdoor in our code.

5. Does this issue affect VC 2.5 Update 2?

No.

I don't see why everyone is so surpised, agreed the issue is pretty bad but as with any software occasionally you will find bugs (some small, some big :P ouch! ), because ultimately you have people who write the software and people who either test or build the regression test routines/scripts for software. To expect software to always be 100% reliable from version to version or patch to patch is a bit of an unrealistic strech.

The most interesting fallout from this from my perspective is not so much the bug but the trade-off an organisation/corporation/etc... needs to accept by standarding on a singlular platform for the sake of increased operational efficiency VERSUS the risk such a model carries when all your services are bundled in the same basket.

The case and point can also be seen in GMAIL which services also went down this week due to a bug, check out: http://gmailblog.blogspot.com/2008/08/we-feel-your-pain-and-were-sorry.html

Not sure there is an easy fix, because chances are future disruptions to various degrees will happen again and will just need to be managed via plan A, plan B, plan C, etc...

Otherwise VMWARE still rules, they'll learn a lot out of this and i'm pretty sure it'll be a long time before another bug of even half the magnitude appears, for our environment the servers that we were trialing update 2 on, we just rolled back the environment to ESXi Update 1 and all is well.

Thanks, its makes sense now that the way the official release states its.

RParker you misinterpret what I meant...

I'm saying that the bug isn't with ESX, but the fact ESX can't get a valid license (it's thinks it's expired).... this is why it stops working... fix the license issue (yes it's in code, I get it) and the ESX works again...

In my opinion licensing shouldnt stop the core function of booting a server... it's so dangerous in general to have that in place.. why not do it like MS and cripple it or something... maybe run the VM at half speed or restrict some features, but stopping the VM from booting (which is the entire point of the software) is crazy....

we are in this position because VMWare want to protect their software from pirates, not because of a fault with the product. Do it a better way.... protect the customers who actually pay a great deal of money for it... ESX is enterprise based software, not some graphics program or document editor that is peddled around on torrent sites... you need specialised hardware to run it which is not in the domain of the general pirate trying to fleece your software... this fact alone means most compaines will license it... your not going to spend 50k plus on hardware only to rip off your copy of vmware are you??....

If this happenned on photoshop or something it would be far less critical.. the fact a license issue stops an entire production network that generates millions of dollers for some compaines is just outrageous... for the most part people can put the workaround in as described and it's not a big issue, but other companies like ecommerce / banks / lottery etc this is not an option.. there are laws to stop tampering with dates on systems and this a sector this enterprise product is aimed out - and one that vmware even uses to promote the reliability of it's products on its website...

remove the 2 bit licensing model from it and do it another way... it's hard to tell management you cant boot the servers that make the money because of a license expiry issue when your all paid up.... the fact the "shareware" version of code was implemented in the update itself is laughable...

Hi awbc-au,

I second your opinion on this, so to describe it in not so many words: IT SUCKS.

Everyone is getting this confused with a licensing problem, which it is not. It is a Beta-Build time-bomb, which companies typically put in to prevent thier Beta builds from running after a certian date. This practice has been in place for years and is usually commented very plainly. This is a case where that time-bomb wasn't removed for what ever reason, and instead of just shutting down the hypervisor, it just crippled it by not allowing Vmotion to the target, this we know because Vmotion from a box still works, but what good is that if the host cannot accept the guest. Since the guest VM is on a host that is managed by VC, you must have the proper licensing, which an expired product will not take the proper licensing. If you remove the host from VC, I would bet that the VM's will start up no problem with the host in stand-alone mode.

So it is very easy and logical from a programmer stand-point to see how this happened, and yes, QA could've caught it and should have, but then to err is to be human. Also, it isn't stopping the VM's from running, it's just making the host act like a bad GSX server ( I did have one that the VM's would not restart on properly), at least in my humble opinion.

Also, someone stated earlier about thier manager breathing down thier neck; one thing I've learned is even a tech savvy manager may not know the core of the technology and reacts based on your reaction, so if you are calm and explain it rationaly, then the manager will be calm. I understand this is a freaky situation, I definatly got anxious this morning, but as I grabbed the trusty dry-erase markers, got the tropps together and did some brain-storming, I realized that it is not a show stopper, just a major inconvienance. I can bring up a VM host from a cold-boot, I just have to take a couple extra steps to do this. Yes, the time will be off on the VM, then you just log in as a local administrator and reset the time, no biggie. That takes an extra 5 minutes of your time to do this, and with planning, hopefully no body is shutting VM's down and needing to do this.

I personally am watchinig my Virtual Center like a hawk, but sometimes you just have to pull up the sleeves, forget about 9 to 5, and get a little dirty.