BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

[mattjk]

The express patches have been posted. This thread is long.

Please post technical experiences here and non-technical feedback here. --JohnTroyer

Hi all,

We've just encountered a serious bug with our ESX cluster - serious enough that I thought I should post about it here as a prior warning for others running ESX 3.5 Update 2.

The VMWare tech support person we spoke to wouldn't 100% confirm whether this was / would be affecting all ESX3.5u2 installs, but he strongly alluded that it was widespread. For others sake I hope I'm wrong and it's limited.

The bug:

Starting this morning, we could not power on nor VMotion any of our Virtual Machines. The VI Client threw the error "A general system error occurred: Internal Error".

Further digging lead us to messages like this one in /var/log/vmware/hostd.log, and the log file for any virtual machine we tried to power on or VMotion:

Aug 12 10:40:10.792: vmx| This product has expired.

Aug 12 10:40:10.792: vmx| Be sure that your host machine's date and time are set correctly.

Aug 12 10:40:10.792: vmx| There is a more recent version available at the VMware Web site: "http://www.vmware.com/info?id=4".

A call to tech support confirmed this as a known problem with a temporary workaround.

The work-around:

Turn off NTP (if you're using it), and then manually set the date of all ESX 3.5u2 hosts back to 10th of August. This can be done either through the VI Client (Host -> Configuration -> Time Configuration) or by typing date -s "08/10/2008" at the Service Console command line on the ESX hosts.

As soon as the date was reset to the 10th - problem solved.

Note that running VMs were operating fine, this only seems to affect initial VM power-on (including from suspended state) and VMotion.

So, it sounds like a serious licensing bug has crept into 3.5u2. Further testing shows that the problem begins as soon as the date hits 12th August - 10th is fine, 11th is fine, 12th and the problem appears.

There wasn't any real reference to similar problems in the forums as far as I could see, but it's quite possible we're seeing this before most of the rest of the world as we're in Australia, and therefore the date here ticked over to the 12th "before" those in Europe, America, etc.

Hope this helps others... took us a couple of hours to get this far - at least we can power on VMs again though!

Cheers,

Matt Kilham

Stratton Car Finance

Message was edited by: JohnTroyer to add new thread links.

[KlinikenLB]

Seems, myates entered the wrong door

[COS]

You're english is fine. Just another rogue arbitrary poster with no rhyme or reason in the forum...

[Trystam]

No he just has a very weird prespective of the problem at hand .. but im glad his servers are fine though

[RonWBrown]

by the way, this is the most publicized forum on the internet today

oh,

and I burnt 3 disks with nero......Wow LOL

RB

[Trystam]

edawg,

That is correct, your not affected by the bug.

Keep your servers in a glass dome regarding upgrades until this issue is resolved, and you might also want to promote one of your servers to a "test rat" level to test drive into mass spreading "version updates"

Cheers

[Trystam]

And i dont think it's something we should be proud, and we should keep this thread the cleaner the possible for doubts that are really worth it and for the topic at hand.

We can all discuss is brasero is better than nero or k3b some other day.

[COS]

Hey edawg, you're one of the lucky ones like me. I procrastinated on updates and got lucky.

I get my procrastination from my dad. It's in the Gene pool.

Oh, and I burnt a DVD of this forum.....damn i'm good.

[Trystam]

COS

This is hardly an excuse for not patching

I have 2 "test" boxes on U2 and worked my way around the bug, you shouldnt run from the mighty upgrades ... just embrace them ..... "carefully"

[KlinikenLB]

Let's see what tomorrow brings.

Good night from Germany.

[ElmbrookDan]

I onced burned a dvd which was a dvd of me burning a dvd.

[curriertech]

6:07 PM PST, no patch.

keeps waiting

[boneman]

We'll it could be worse. At least with VmWare I only have 1 patch to test tommorow.

[Microsoft|http://www.eweek.com/c/a/Security/Microsoft-Patch-Tuesday-Targets-26-Vulnerabilities/#] released its August Patch Tuesday update today with 11 bulletins that plug 26 security holes across multiple products.

"The patches that

address zero-days issues are the most critical as some of the zero-days

are actively being exploited by attackers,” said Amol Sarwate, manager

of the vulnerabilities lab at Qualys. “It’s important to note that

users should make it a priority to install all of the patches this

month. This is the biggest batch of client-side security patches we’ve

seen from Microsoft."

[Timbo71]

I hope that VMware realizes that they did not pull all the downloads for ESXi. At once you log in, you can download the binaries to "try" ESXi 3.5 build 103909 released on 7/25/2008. People could still be downloading the bad code and unless they purchase support will not know about this issue. Unless they browse the forums of course.

Tim

[bunglebee]

Of course Vmware knew about it on Monday, the world does not operate in one time zone. I had noticication of it from VMware in Australia at 2pm our time, which would have been 9pm Monday night US pacific time. Just so happens we are ~15hrs ahead and had been having problem all day long. It was 9:30am when I first picked it up and spoke with our SE from Vmware Australia. (4pm Monday US PST)

My guess is that they notified their large enterprise clients as first priority.

[bunglebee]

Oops double post, mods, pls remove

[DLeeSFI]

6:30, where's the patch?

[Gonecase]

The patch should be out in some time. It won't require a reboot of the host but the VMs have to be vmotioned or powered off n powered on.

[ElmbrookDan]

That doesnt make sense to me.....

[Gonecase]

Apologies but which part is not making sense

[DLeeSFI]

Yes, if we could use Vmotion or other licensed features we wouldn't have the problem.....I'm not risking a time change when my SQL server with my company's time clock database is on it....