VMware

This Question is Possibly Answered

1 "correct" answer available (10 pts) 2 "helpful" answers available (6 pts)
1 2 3 ... 47 Previous Next 704 Replies Last post: Aug 25, 2008 11:18 PM by dipaksharma  

BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error posted: Aug 12, 2008 9:10 PM

Click to view mattjk's profile Enthusiast 80 posts since
Dec 19, 2007
The express patches have been posted. This thread is long.
Please post technical experiences here and non-technical feedback here. --JohnTroyer

Hi all,

We've just encountered a serious bug with our ESX cluster - serious enough that I thought I should post about it here as a prior warning for others running ESX 3.5 Update 2.

The VMWare tech support person we spoke to wouldn't 100% confirm whether this was / would be affecting all ESX3.5u2 installs, but he strongly alluded that it was widespread. For others sake I hope I'm wrong and it's limited.

The bug:


Starting this morning, we could not power on nor VMotion any of our Virtual Machines. The VI Client threw the error "A general system error occurred: Internal Error".

Further digging lead us to messages like this one in /var/log/vmware/hostd.log, and the log file for any virtual machine we tried to power on or VMotion:

Aug 12 10:40:10.792: vmx| http://msg.License.product.expired This product has expired.
Aug 12 10:40:10.792: vmx| Be sure that your host machine's date and time are set correctly.
Aug 12 10:40:10.792: vmx| There is a more recent version available at the VMware Web site: "http://www.vmware.com/info?id=4".

A call to tech support confirmed this as a known problem with a temporary workaround.

The work-around:

Turn off NTP (if you're using it), and then manually set the date of all ESX 3.5u2 hosts back to 10th of August. This can be done either through the VI Client (Host -> Configuration -> Time Configuration) or by typing date -s "08/10/2008" at the Service Console command line on the ESX hosts.

As soon as the date was reset to the 10th - problem solved.

Note that running VMs were operating fine, this only seems to affect initial VM power-on (including from suspended state) and VMotion.


So, it sounds like a serious licensing bug has crept into 3.5u2. Further testing shows that the problem begins as soon as the date hits 12th August - 10th is fine, 11th is fine, 12th and the problem appears.

There wasn't any real reference to similar problems in the forums as far as I could see, but it's quite possible we're seeing this before most of the rest of the world as we're in Australia, and therefore the date here ticked over to the 12th "before" those in Europe, America, etc.

Hope this helps others... took us a couple of hours to get this far - at least we can power on VMs again though!


Cheers,
Matt Kilham
Stratton Car Finance

Message was edited by: JohnTroyer to add new thread links.

Click to view McBain's profile Enthusiast 15 posts since
Feb 13, 2006

Hi Matt,

Also in Australia and experiencing the same issue just as you have described. Looking forward to a proper fix from VMware as we have raised a support case as well.

Cheers,

Chris Slater

Click to view rayray_80's profile Novice 9 posts since
Jan 9, 2007
Same happened here.

Except we're unable to change times due to legal obligations. oh well... I'm sure this will be fixed once the US and Europe start calling in too.

Has anyone experienced this problem with a fixed/expiring licence?
Click to view mcowger's profile Virtuoso 2,058 posts since
Aug 22, 2007
Interesting - we run our stuff in GMT, so we should have been hit by this, but we havent....maybe a TZ issue?

--Matt

Click to view ryath's profile Novice 5 posts since
Jun 17, 2008

Thanks mate! I just wasted 4 hours at least trying to solve this issue.

This was time right when my RAID failed too. Rebuilt esx and couldnt figure out why it wouldnt start any vm's. Keep us informed if ESX bring a patch out! This is a major problem!

Click to view Daniel.Grant's profile Lurker 1 posts since
Aug 11, 2008
Thanks. I was just about to open a SR on this exact issue.

I can confirm that I've also seen exactly the same behaviour and that disabling NTP and setting the date backwards a week via the client or so did resolve it for the moment.

Time to set a job to keep resetting the date I guess (from external management host) until VMWare can release a fix.


One thing to note is that it seems to stop collection for all the performance monitoring graphs on the VM servers (but actually being able to turn on VMs is more important to me)

Daniel

Click to view epping's profile Expert 795 posts since
Oct 9, 2003
getting this already in our AAA region, cant wait for rest of the world to wake up, going to be a fun day.
Click to view lholling's profile Expert 291 posts since
Apr 10, 2005
Yep we got it too luckily not all of the servers in our DRS cluster are upgraded yet, so the powered on ones are OK but nothing new on the "broken U2" servers, doh!

My uneducated guess is that the beta for U2 expired on 12th August...

Leonard...
----

Don't forget if the answers help, award points

Click to view armenk's profile Lurker 2 posts since
Aug 11, 2008

I just ran into the same issue.

/me sighs ?:|

Click to view Busybee's profile Novice 6 posts since
Aug 11, 2008

Just got off the phone with tech support and they've been inudated by calls about this very problem. It only affects ESX and ESXi 3.5 Update 2. Setting the time back a day in ESXi should be ok because if memory serves, VMware Tools will only reset time forward and not backwards.


Click to view pgolightly's profile Lurker 1 posts since
Aug 13, 2006

Been bashing my head on this last couple of hours, thought licence issue of some kind.

ESX 3.5 update 2 misbehaving here as described.

Everything was working fine on Friday.

NTP/Date work around working (phew).

Thanks

Click to view Vishy1's profile Enthusiast 27 posts since
Jun 9, 2008
Same here, thanks for the workaround.
Click to view mjlin's profile Novice 19 posts since
Jun 18, 2007
Dear VMware customers,

We are actively working on rootcausing the problem. Once we know the appropriate action to take here, we’ll provide an update.

Apologies for any inconvenience.

The ESX Product Team
1 2 3 ... 47 Previous Next Go to original post

VMware Developer

SDKs, APIs, Videos, Learn and much more in the Developer community.

Learn More

Developer Sample Code

Increase your developer productivity with VMware API sample code.

Learn More

VMworld Sessions & Labs

Online access to the latest VMworld Sessions & Labs and online services.

Learn more

Purchase PSO Credits Online

Purchase credits to redeem training and consulting services online.

Buy Now

Community Hardware Software

View reported configurations or report your own.

Learn More

VMware vSphere

Come witness the next giant leap in virtualization.

Register Today

Communities