BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

[mattjk]

The express patches have been posted. This thread is long.

Please post technical experiences here and non-technical feedback here. --JohnTroyer

Hi all,

We've just encountered a serious bug with our ESX cluster - serious enough that I thought I should post about it here as a prior warning for others running ESX 3.5 Update 2.

The VMWare tech support person we spoke to wouldn't 100% confirm whether this was / would be affecting all ESX3.5u2 installs, but he strongly alluded that it was widespread. For others sake I hope I'm wrong and it's limited.

The bug:

Starting this morning, we could not power on nor VMotion any of our Virtual Machines. The VI Client threw the error "A general system error occurred: Internal Error".

Further digging lead us to messages like this one in /var/log/vmware/hostd.log, and the log file for any virtual machine we tried to power on or VMotion:

Aug 12 10:40:10.792: vmx| This product has expired.

Aug 12 10:40:10.792: vmx| Be sure that your host machine's date and time are set correctly.

Aug 12 10:40:10.792: vmx| There is a more recent version available at the VMware Web site: "http://www.vmware.com/info?id=4".

A call to tech support confirmed this as a known problem with a temporary workaround.

The work-around:

Turn off NTP (if you're using it), and then manually set the date of all ESX 3.5u2 hosts back to 10th of August. This can be done either through the VI Client (Host -> Configuration -> Time Configuration) or by typing date -s "08/10/2008" at the Service Console command line on the ESX hosts.

As soon as the date was reset to the 10th - problem solved.

Note that running VMs were operating fine, this only seems to affect initial VM power-on (including from suspended state) and VMotion.

So, it sounds like a serious licensing bug has crept into 3.5u2. Further testing shows that the problem begins as soon as the date hits 12th August - 10th is fine, 11th is fine, 12th and the problem appears.

There wasn't any real reference to similar problems in the forums as far as I could see, but it's quite possible we're seeing this before most of the rest of the world as we're in Australia, and therefore the date here ticked over to the 12th "before" those in Europe, America, etc.

Hope this helps others... took us a couple of hours to get this far - at least we can power on VMs again though!

Cheers,

Matt Kilham

Stratton Car Finance

Message was edited by: JohnTroyer to add new thread links.

[SanderCL]

vmware-vim-cmd does not exist in the free version of ESXi 3.5 Update 2 installable. From a fresh install of the ISO, after enabling SSH and connecting to your ESXi host and logging in as root:

~ # vmware-vim-cmd

-ash: vmware-vim-cmd: not found

~ # vim-cmd

Commands available under /:

hostsvc/ proxysvc/ vimsvc/ help

internalsvc/ solo/ vmsvc/

This was the reason for my post.

[TJL]

Those who have set up WSUS to update automagically and reboot upon completion are the same ones who did the Boot Camp to get their MCSE.

[ICT-Freak]

I was testing with ESX 3.5u2 not the ESXi version.

[SanderCL]

That makes sense then

Our ESX servers that command worked fine on, it was only the ESXi servers we recently setup to replace VMware Server boxes that we had issues from command line.

[EDTRIANA]

This is sad day for VMware and for all of us who are trying to show the Upper Management that VMware is a mature and robust product, I predict next time I will sit down to request money for more virtualization I will have to be prepare for some seriuos questions...I hope some heads will roll down as a result of this catastrofic event...

Systems Integration Engineer

Rogers Wireless Inc

Montreal, Canada

[TristanT]

Have been loving update 2 for the last week and half - now, not so much...

My concern is how the fix is going to be delivered. If a host reboot is required for patch install and maintenance mode is not available, then we'll need to power down all of our VMs. I'm really, really hoping we can disable NTP, roll back our clocks, go into maint mode, then install the update.

[sam2003]

Martin Croce

Network and Communications Services

McGill University

I thought that a bug like this would have been caught by either Vmware or by Shavlik. Kind of surprised that it got past both levels of testing.

[s1xth]

I cant agree more...I am in the same situation that you are in. I don't care what people are saying, but this is a serious bug, and VMware's turnaround time makes me speechless....a couple days for a patch?! Horrendous!! I would be even more upset if I was paying for ESX, since I am using ESXi I guess it doesnt matter as much....

[alex555550]

That`s why I sometimes love Microsoft. Shure they make also strange thinks, but If you call them they provied you with ah small hotfix installer, 1-2 hours after you`re call an you`re up running.

[rob_nance]

This is sad day for VMware and for all of us who are trying to show the Upper Management that VMware is a mature and robust product, I predict next time I will sit down to request money for more virtualization I will have to be prepare for some seriuos questions...I hope some heads will roll down as a result of this catastrofic event...

Systems Integration Engineer

Rogers Wireless Inc

Montreal, Canada

That is my feeling. My boss is very reluctant of virtualization, and we have just been finally proving to him that it's not just for toying around. I just implemented our first 3 dedicated ESX boxes (We only have about 20 physical machines, so that is a lot for us.) I had to tell him about this, reluctantly, because I am sure this will hit the mainstream outlets before COB. I'm kind of surprised it hasn't yet.

[s1xth]

Now that its on the inquirer its only a matter of time before it hits VM sites and other outlets...

[alex555550]

How can rember the long ping time issue, wasn`t it a year ago? Correct me if, Im wrong but didn´t they need one month to fix it?

[EDTRIANA]

Have you guys notice that up to this minute there is ABSOLUTELY NOTHING ON THE VMware website!!, no ALERTS, NO NOTHING about this bug?, this is really scary how manangement is handeling this hot issue...we as an IT community deserve better than that if they expect us to run and support their product.

my two cents..

Systems Integration Engineer

Rogers Wireless Inc

Montreal, Canada

[Mike_Glenn]

. . . .The only thing that would be worse is if the bug had been introduced by a third party. . . . .

Question: Isn't the License Server indeed written by a third party?

<RANT>

I knew something like this was going to happen, sooner or later. The first time I laid eyes on that blasted License Server, the first thought I had was "Single Point of Failure." Worse, it is a SPF whose as-designed failure mode is to shut everything down!

I wonder what bean-counter thought this one up. VMware's engineering department better get busy, because I guarantee that after this there won't be a single Fortune 1000 firm that will tolerate their silly License Server within their IT infrastructure.

I know I won't.

</RANT>

(And yes; I feel a bit better now.)

[Kevin_Gao]

www.google.com

Click on news

Type in VMware and search

Results are already popping up everywhere.

[PhilipArnason]

You can't make a baby in a single month by putting 9 women together.

Just a nice visual for those who are "speechless" about the time putting a patch in place. VMWare has said new binaries will be available August 13th at noon PST, that's 14 hours from now. QA takes times, whether it's a patch, or a general update. This is regardless of the QA team's failure to notice this bug before Update 2 release.

Philip Arnason

[vmproteau]

There is a support alert on the main VMWare support page. . Not much consulation but it is there.

[goldeneye_007]

Well that is funny, because yesterday I set up my first ESXi installation. I've been using VMware Server and Workstation

for a good while now. I just 'happened' upon this thread this morning. Fortunately i am still in test mode

and I hadn't even powered on the ESX box yet this morning...so for me , I was just lucky.

[RParker]

Unbelivable, you claim to be a enterprise solution. You really SUCK BIGTIME. 36 hrs is not acceptable

Had I written this, I would be crucified. But I have to say in my years with Microsoft Products, I don't remember seeing anything quite like this. What you say about MS:

MS has always been the target for people to yell at when it comes to buggy software but they have never managed to do something like this.

Is exactly right. I agree, this hasn't ever been this bad with MS. At a time when its absolutely crucial for VM Ware to shine, they completely dropped the ball at the goal line just as time runs out... This is going to be detrimental.. I know they will over come.. but it's going to go down as perhaps the biggest faux paus in computing history.

3- Windows ME

2- Windows BOB

1- VM Ware ESX 3.5 Date bomb

I have been supporting and getting behind VM Ware for the last 2 years.. But this is very tough pill to swallow...

[gi-minni]

Please calm down everybody and let the vmware people do the needed work to

resolve this nasty issue. We all know that if the time pressure is high the quality of work dramatically degrade. This

is a severe problem without excuses, but no one is perfect in software development. If we look

in the past VMware has delivered very complex and high quality code. Let us conduct a technical based

discussion and not a flame war against them. We are all sitting in the same boat.

My 2 cent ...