mattjk
Enthusiast
Enthusiast

BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

The express patches have been posted. This thread is long.

Please post technical experiences here and non-technical feedback here. --JohnTroyer

Hi all,

We've just encountered a serious bug with our ESX cluster - serious enough that I thought I should post about it here as a prior warning for others running ESX 3.5 Update 2.

The VMWare tech support person we spoke to wouldn't 100% confirm whether this was / would be affecting all ESX3.5u2 installs, but he strongly alluded that it was widespread. For others sake I hope I'm wrong and it's limited.

The bug:

Starting this morning, we could not power on nor VMotion any of our Virtual Machines. The VI Client threw the error "A general system error occurred: Internal Error".

Further digging lead us to messages like this one in /var/log/vmware/hostd.log, and the log file for any virtual machine we tried to power on or VMotion:

Aug 12 10:40:10.792: vmx| This product has expired.

Aug 12 10:40:10.792: vmx| Be sure that your host machine's date and time are set correctly.

Aug 12 10:40:10.792: vmx| There is a more recent version available at the VMware Web site: "http://www.vmware.com/info?id=4".

A call to tech support confirmed this as a known problem with a temporary workaround.

The work-around:

Turn off NTP (if you're using it), and then manually set the date of all ESX 3.5u2 hosts back to 10th of August. This can be done either through the VI Client (Host -> Configuration -> Time Configuration) or by typing date -s "08/10/2008" at the Service Console command line on the ESX hosts.

As soon as the date was reset to the 10th - problem solved.

Note that running VMs were operating fine, this only seems to affect initial VM power-on (including from suspended state) and VMotion.

So, it sounds like a serious licensing bug has crept into 3.5u2. Further testing shows that the problem begins as soon as the date hits 12th August - 10th is fine, 11th is fine, 12th and the problem appears.

There wasn't any real reference to similar problems in the forums as far as I could see, but it's quite possible we're seeing this before most of the rest of the world as we're in Australia, and therefore the date here ticked over to the 12th "before" those in Europe, America, etc.

Hope this helps others... took us a couple of hours to get this far - at least we can power on VMs again though!

Cheers,

Matt Kilham

Stratton Car Finance

Message was edited by: JohnTroyer to add new thread links.

Cheers, Matt
0 Kudos
704 Replies
spex
Expert
Expert

When a vm is powered on it gets its time from the esx server. There is no other possibility. Even without vmtools.

Regards Spex

0 Kudos
joergriether
Hot Shot
Hot Shot

Shame on you vmware, i am such an inerceder for vmware all the years every time saying stuff like exactly this would never happen in a vmware environment and there, boom here we go. You have to fix this NOW!!!!

Joerg

0 Kudos
awbc-au
Contributor
Contributor

hello vmware...

will this patch be immediatly available via the update plugin when you release it, or am I going to have to download some crazy tar file and install it manually?... I am hoping we can just scan for updates and then remediate to get the patch on there as soon as it's released rather than having to do some kind of manual installation....

we also have this problem arise because someone rebooted a server and it never came back up... I am very critical of the error messages which continue to tell me absolutly nothing about the problems... often I get these unknown errors and have to google stuff for hours to get solutions... it would be good if the unknown errors could instead be a little more verbose so we know kind of where the problem is... for example if it said "expired license" I might have found this thread a little quicker and not spent the past 4 hours trying to work out why the hosts wouldn't start up thanks to my unknown error message....

cheers,,,

0 Kudos
hjelmar
Contributor
Contributor

Yup...Same problem here in Denmark :o/ - Hope they wil nuke this bug soon 🐵

"I don’t know why people hire architects and then tell them what to do.”
0 Kudos
mimo17
Contributor
Contributor

Good morning

Switzerland is in the office as well. And we have the same problem.

Be careful with the time change - VM's are affected! This is no workaround for us.

to VMware:

- can we have Trial's for the next 30 days until the issue is fixed?

- can we reset the counter for the initial trial period?

- why do you not proactive inform about this serious issue?

I think you want to be a good product in the enterprise environment, or?

Still waiting that we get any answer to our case.. - Good that the forum is quicker.

Michael

0 Kudos
leeus
Contributor
Contributor

Is there an offical article KB on this yet?

0 Kudos
Lancelot_Wong
Contributor
Contributor

Actually I still believe that ESX should be better than M$ Hypervisor. Although we had purchased the M$ support and M$ would enforce us to change from VMWARE to her hypervisor. (by limiting support level if we use VMWARE at the button.)

Hope that VMWARE will check and test carefully and we didn't want to do patch management for vmware product like M$, to wait for a year for review...

Forgot to let you know that I am from HK. Now waiting for the new release and patches.

0 Kudos
atbnet
Expert
Expert

We are also experiencing this bug. I currently cannot power on or migrate any VMs. We cannot change date/time for legal reasons, so when there off there off until this is fixed.

I notice the ESX 3.5 (Update 2) download is now "temporarily unavilable" from the vmware download area along with the ESXi products update2.

Cant see a patch though. I'll be opening a support request too and asking for an ETA!

Thing is no matter how long you test the updates yourself and apply to a select few servers initially and then to the rest, it could all seem fine until the right time comes around and then boom gone!

Andy

VCP

Andy Barnes
VCP / VCA-DT / MCITP:EA / CCIA
Help, Guides and How Tos... www.VMadmin.co.uk

If you found this information useful please award points using the buttons at the top of the page accordingly.
0 Kudos
RonTom
Contributor
Contributor

KB ID 1006716 seems to be it

0 Kudos
Marcel1967
Enthusiast
Enthusiast

Thanks Matt for posting this serieus issue. We had the same problem here but luckily I was just reading the blog reporting this issue when the problem was reported by a co-worker of mine. Problem 'worked around' by setting the date 2 days back in time. Saved us quite some time in troubleshooting.

0 Kudos
awbc-au
Contributor
Contributor

its just a real shame that the "bug" has nothing to do with the functioning of ESX... it's all because of licensing and is why I am so critical of this particular error... right now we have critical business systems down because of some expiry date that should never have impacted us in the fist place has lapsed... businesses are losing money right now after paying for all their licenses and in some cases the gold support options as well... to find out we simply can't use our servers because of this is probably the biggest slap it could possibly be....

I imagine VMWare will lose some customers because of this error... no one will suspect that it will be the cause.. I never thought that error was licensing until I read this thread... up until then I thought I had some kind of corruption on the server and I was about to format it and reinstall...

is there a way to uninstall the upgrade, reboot and therefore fix the problem?

perhaps vmware should give us a better work around until they release the patch... as I understand it the current workaround is from a member like us... vmware havn't offered anything as a solution?

0 Kudos
beeswarm
Contributor
Contributor

Hi All,

Thanks for posting the reply we too are experience the problem here in aus. Can confirm that setting the date back 2 days we can power VM's on again.

I do find this very sloppy on behalf of VMWare and i think there will be serious fallout from this once an RCA is released.

Cheers

0 Kudos
K-MaC
Expert
Expert

Wow gotta say that I am really glad we didnt do the upgrade. That being said I sure hope that VMware gets a patch out for this and quickly. I imagine a great number of VMware's customers are in the US and are only just starting to wake up. It's going to be a crappy day for many system engineers today I fear.

Goodluck to all those effected by this.

Cheers Kevin
0 Kudos
Aketaton
Contributor
Contributor

Hi all

I'm experiencing the same problem.

I'va also another problem: I've removed an host from VC but I'm not able to connect it back to VC. VI Clent shows a message regarding licensing ("There are not enough licensses to perform this operation").

I've tried to change time setting on both the ESX host and the VC but with no success.

Any idea if this problem is related to this issue?

http://VC and Host are obvoiusly 3.5 Update 2

Thank you.

F.

0 Kudos
sshamilt
Enthusiast
Enthusiast

I'm so thankful that I waited to update!

0 Kudos
ErikBussink
Enthusiast
Enthusiast

gotta say that I am really glad we didnt do the upgrade.

I don't mind installing a new system/cluster with the latest release (Update 2), but I'm always telling my clients to wait at least 2 weeks after an update (release of 3.5u2) before applying it to a production environment. I got burned once already with the Patch 5 or 6/7 for ESX 2.5.4 I believe.

-- Erik Bussink 1st VMware Community account https://communities.vmware.com/people/Erik%20Bussink
0 Kudos
frank_wegner
VMware Employee
VMware Employee

You can see the latest status here: Please check back often, because it will notify you when this issue has been fixed. Until then the best workaround I can think of is:

  • Do nothing

  • Turn DRS off

  • Avoid VMotion

  • Avoid to power off VM's

0 Kudos
SandyB
Enthusiast
Enthusiast

You're a life saver, just spent the best part of an hour trying to get a critical VM running after it was powered off, disabling NTP, changing the date, powered on the VMs and turned NTP back on as we have restrictions on time. everything working fine for now!

Hope VMware get a fix out today, as my mum would say "I'm not angry just very disappointed!!!" :smileyshocked:

0 Kudos
robnik
Contributor
Contributor

We have the same problem in Norway.

Turning back date works ok for us.

Realy hope that vmware will fix this soon.

There is still nothing oficcial about this issue on wmvare.com, thats bad.

0 Kudos

The ESX 3.5 update2 ISOs have been taken offline at least; I am not able to reach the KB article; it appears the server is "somewhat" busy Smiley Sad

Anyone got a definitive answer on putting the date back and VMs that sync back with it while they should not? Was this issue seen only on VMs being started (which makes sense - BIOS time) or also on running VMs with or without timesync set?

Visit my blog at http://www.vmdamentals.com
0 Kudos