VMware Cloud Community
mattjk
Enthusiast
Enthusiast

BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

The express patches have been posted. This thread is long.

Please post technical experiences here and non-technical feedback here. --JohnTroyer

Hi all,

We've just encountered a serious bug with our ESX cluster - serious enough that I thought I should post about it here as a prior warning for others running ESX 3.5 Update 2.

The VMWare tech support person we spoke to wouldn't 100% confirm whether this was / would be affecting all ESX3.5u2 installs, but he strongly alluded that it was widespread. For others sake I hope I'm wrong and it's limited.

The bug:

Starting this morning, we could not power on nor VMotion any of our Virtual Machines. The VI Client threw the error "A general system error occurred: Internal Error".

Further digging lead us to messages like this one in /var/log/vmware/hostd.log, and the log file for any virtual machine we tried to power on or VMotion:

Aug 12 10:40:10.792: vmx| This product has expired.

Aug 12 10:40:10.792: vmx| Be sure that your host machine's date and time are set correctly.

Aug 12 10:40:10.792: vmx| There is a more recent version available at the VMware Web site: "http://www.vmware.com/info?id=4".

A call to tech support confirmed this as a known problem with a temporary workaround.

The work-around:

Turn off NTP (if you're using it), and then manually set the date of all ESX 3.5u2 hosts back to 10th of August. This can be done either through the VI Client (Host -> Configuration -> Time Configuration) or by typing date -s "08/10/2008" at the Service Console command line on the ESX hosts.

As soon as the date was reset to the 10th - problem solved.

Note that running VMs were operating fine, this only seems to affect initial VM power-on (including from suspended state) and VMotion.

So, it sounds like a serious licensing bug has crept into 3.5u2. Further testing shows that the problem begins as soon as the date hits 12th August - 10th is fine, 11th is fine, 12th and the problem appears.

There wasn't any real reference to similar problems in the forums as far as I could see, but it's quite possible we're seeing this before most of the rest of the world as we're in Australia, and therefore the date here ticked over to the 12th "before" those in Europe, America, etc.

Hope this helps others... took us a couple of hours to get this far - at least we can power on VMs again though!

Cheers,

Matt Kilham

Stratton Car Finance

Message was edited by: JohnTroyer to add new thread links.

Cheers, Matt
Reply
0 Kudos
704 Replies
jpratt_at_norwi
Enthusiast
Enthusiast

yes

see here please

Reply
0 Kudos
dbuchanan
Contributor
Contributor

Yes 82663 = U1

103908 - U2

Dan L. Buchanan | Microsoft Engineer

Barclaycard | Business Technology Group | Infrastucture

Engineering

Telephone (302) 255-8970 | Mobile (302) 507-6297

COMPANY CONFIDENTIAL

Reply
0 Kudos
N3NCY
Contributor
Contributor

Ya know, I spent hours this morning troubleshooting this bug.

A kind soul IMed me this link:

That link led me here only to find out that I am not alone.

THANK YOU VMWARE!

You spam my inbox every freakin day with crap you are selling or to tell me about some VM conference.

Did VMware think enough about me to send me an email telling me about this nice time bomb they built into ESXi that screwed up my day?

Hell no!

Even though you have my email from when I registered and downloaded ESXi, YOU DIDNT USE IT TO TELL ME YOU STINK!

Shhhhhhh.

Be very QUIET and maybe nobody will notice?

Make sure that you (VMware) does not send an email to all the people who registered and downloaded ESXi 3.5 U2.

I REPEAT - DO NOT SEND AN EMAIL

I want all these people to suffer and and hate VMware just like me!

Oh, I get it now, VMware couldn't send email this morning to alert me and the rest of the planet...

They must be running their email server and desktops on top of ESXi 3.5 U2!

Reply
0 Kudos
Phil_White
Enthusiast
Enthusiast

Unless I'm incredibly wrong, this really doesn't affect ESXi at all so your bashing of Vmware is really unnecessary.

Reply
0 Kudos
interix
Enthusiast
Enthusiast

MicroAdmin,

ESX3i is effected by this as well. Update 2 was released for ESX 3.5x and ESXi 3.5.x

Reply
0 Kudos
Phil_White
Enthusiast
Enthusiast

Well then, I am incredibly wrong and the "news" sources should comment on that too as I was under the impression ESXi and ESX were two different things.

Reply
0 Kudos
Kevin_Gao
Hot Shot
Hot Shot

Unfortunately it does. Again - the licensed features stop working only. i.e. DRS, vmotion, powering on a vm

A VM that's already powered on isn't affected. Regardless I think people just want to flame.

Reply
0 Kudos
MaartenK
Contributor
Contributor

Well okay you`ve got a point there.

We are installing a new environment by a customer of ours right now ESX is just installed. I`m so happy the production environment isn`t live yet. First we will test everything twice now. Before going online.

And we wait patiently for the hot fix (hot it is!)

Reply
0 Kudos
Ken_Cline
Champion
Champion

Unless I'm incredibly wrong, this really doesn't affect ESXi at all so your bashing of Vmware is really unnecessary.

You're incredibly wrong...it impacts all versions of 3.5u2 including ESX 3.5 Server, ESXi Embedded, and ESXi Installable.

The bashing was unnecessary, but understandable - everyone's frustration level is VERY high right now

Ken Cline

Technical Director, Virtualization

Wells Landers[/url]

VMware Communities User Moderator

Ken Cline VMware vExpert 2009 VMware Communities User Moderator Blogging at: http://KensVirtualReality.wordpress.com/
Reply
0 Kudos
sam_god
Contributor
Contributor

VMware ESX Team,

In the VI client Configuration -> Time-configuration We should have one new option Synchronize Time for Guests option box with default selected. We know that in VMware Tools inside VM we set similar option so that guest can synchronize time with host, but when we have 100s of VMs in infrastructure its painful to log-on to each VMs then do the needful changes. If VM time sync disabled from ESX Server Side all VM tools should stop sync of time, evenif VM is set for sync (master control for time sync), and then ESX server's time can be rolled back to older date w/o affecting VMs time. This feature will help such issues taken immediate and fast resolution to the problem.

Reply
0 Kudos
Phil_White
Enthusiast
Enthusiast

Well I guess I need to brush up on ESX and ESXi differences some more. From my understanding ESXi is supposed to be on a completely different upgrade structure and ugprades "Like firmware would" so I wouldn't assume they'd both be using the SAME udpates. The differences between the two are incredibly confusing to me.Thanks Ken.

Reply
0 Kudos
Phil_White
Enthusiast
Enthusiast

How very mature of you. I made a mistake and even admitted to it yet you still feel the need to attack me even though you were never attacked in the first place. Enjoy your printer maintenance duty Smiley Happy

Reply
0 Kudos
DSTAVERT
Immortal
Immortal

As an IT professional it is your job to protect the infrastucture and regardless of what VMWare missed it is ultimately your responsibility. Implementing updates without regard for the content is suicide. Compliance isn't about having ALL the patches in place as quickly as you can. It is having the necessary patches in place but ONLY after you are fully satisfied that the patches are necessary. Will the upgrade solve a problem that YOU were experiencing. Did you test it to find out if the patch solved YOUR problem. Even security patches aren't necessary if it isn't something that affects you. As for the "you can't plan for something like this" argument, well two weeks is hardly long enough to have a thorough test and leaving yourself so vulnerable means there are other things in your process that will get you down the road. Time to sit down and re-evaluate your patch process.

I remember several service packs in NT 3.5 and 4 that left you with a BSOD. Really sad when those things happen and you can revert to a snapshot. Back then Microsot took a lot of heat. It was also really tough to recover.

-- David -- VMware Communities Moderator
Reply
0 Kudos
Kevin_Gao
Hot Shot
Hot Shot

OK that's plainly unprofessional and rude. He was just trying to help and he apologized for having the wrong information. Can we stick to the topic of this U2 issue here please?

Reply
0 Kudos
admin
Immortal
Immortal

Dear VMware Customers,

Please find the latest update about the product expiration issue. From this point on, we'll provide an update every two hours.

Problem:

An issue has been discovered by many VMware customers and partners with ESX/ESXi 3.5 Update 2 where Virtual Machines fail to power on or VMotion successfully. This problem began to occur on August 12, 2008 for customers that had upgraded to ESX 3.5 Update 2. The problem is caused by a build timeout that was mistakenly left enabled for the release build.

Affected Products:

- VMware ESX 3.5 Update 2 & ESXi 3.5 Update 2.

- Reports of problems with ESX 3.5 U1 with the following 3.5 Update 2 patches applied: ESX350-200806201-UG

- No other VMware products are affected.

What has been done?:

- Product and Web teams pulled the ESX 3.5 Update 2 bits from the download pages last night so no more customers will be able to download the broken build.

- VMware Engineering teams have isolated the cause of the problem and are working around the clock to deliver updated builds and patches for impacted customers.

- A Knowledgebase article has been published (http://kb.vmware.com/kb/1006716), but traffic to the knowledgebase is causing time outs. A new static page has been published at http://www.vmware.com/support/esx35u2_supportalert.html that customers and partners will be able to view.

- The phone system has been updated to advise customers of the problem

- Vmware partners have been notified of the issue.

Workarounds:

1) Do not install ESX 3.5 U2 if it has been downloaded from VMware's website or elsewhere prior to August 12, 2008.

2) Set the host time to a date prior to August 12, 2008. This workaround has a number of very serious side affects that could impact product environments. Any Virtual Machines that sync time with the ESX host and serve time sensitive applications would be broken. These include, but are not limited to database servers, mail servers, & domain administration systems.

Next Steps:

VMware to notify customers who have downloaded this version and provide an update every two hours.

Resolution:

VMware Engineering has isolated the root cause and is working to produce an express patch for impacted customers today. The target timeframe is 6pm, August 12, 2008 PST.

FAQ:

* What would this express patch do?

More information will be provided in subsequent communication updates.

  • Will VMware still reissue the upgrade media and patch bundles in the timeframe that has been communicated?


    Yes. We still plan to reissue upgrade media by 6pm, August 13 PST (instead of noon, August 13 PST) and all update patch bundles later in the week. We will provide an ETA for the update patch bundles subsequently. NOTE: the "patch bundles" referred to here are for the patches listed above under "Affected Products" and the other bundles released at GA. They are not the same as the express patch which is targeted for 6pm, August 12, 2008 PST as stated above.

  • Why does VMware plan to reissue the upgrade media before the patch bundles? That is a wrong priority call!


    This is not a matter of priority. Since we can get done building and testing the upgrade media before the patch bundles, we want to make that available to customers first instead of reissuing all the binaries later in the week.

  • Can VMware issue a patch that opens the licensing backdoor in the next hour as a critical measure?

There is no licensing backdoor in our code.

  • Does this issue affect VC 2.5 Update 2?

No.

  • What is VMware doing to make sure that the problem won't happen again?

We are making improvements on all fronts. The product team had endeavored to deliver a release with support customers deem important. But we fell short and we are deeply sorry about all the disruption and inconveniences we have caused. We have identified where the holes are and they will be addressed to restore customers' confidence.

The VMware ESX Product Team

Reply
0 Kudos
mrbill007
Contributor
Contributor

DSTADVERT, couldn't have said it better.

I was affected by the U2 problem, its my fault, I patched w/o enough delay time between release and production, howerver, I enjoy riding the bleeding edge ... just not in production. I'll take this lil burp and be happy that it wasn'ta fatal problem, this is easy to fix. Stop complaining and start formatting...

VMotion is your friend, use it well.

Reply
0 Kudos
rhunter
Contributor
Contributor

Reply
0 Kudos
curriertech
Enthusiast
Enthusiast

I'm glad I stumbled into this thread, I was planning to do some maintenance on a couple guest systems tonight, and I'd have had a serious problem if I did! Looks like I'll be doing some work tonight, assuming they get that patch out by 6pm PST like they're hoping.

-Josh.
Reply
0 Kudos
esiebert7625
Immortal
Immortal

His unprofessional and offensive response was removed, please try and act in a civil manner in these forums or we may be forced to lock user accounts and/or this thread.

Eric Siebert

VMware Communities User Moderator

-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=-

Check out my website: VMware-land

Read my virtualization blog: SSV Blog

-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=-

Reply
0 Kudos
Kevin_Gao
Hot Shot
Hot Shot

Thank you Eric. Also thank you VMware for the official update.

Reply
0 Kudos