VMware Communities: Message List - BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

dipaksharma — Tue, 26 Aug 2008 06:18:32 GMT

Hi Guys,

I did have a look at the URL and i am attaching the screen shot (image_2) here. I am also attaching another screen shot (image_1) that shows what ESX server i am running.

Can you please advice how can i install the patch with the setup that i have?

Thank in advance.

Regards,
D

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

kstailey — Thu, 21 Aug 2008 13:26:13 GMT

Author : dipaksharma
Profile : http://communities.vmware.com/people/dipaksharma
Message:

http://...

I read on the forum that i need Update manager, but i do not have that
software.

http://...

When you install ESX (or just boot it once it has been installed it puts a
URL (http://blah..) on the console (monitor)

On a Windows system put that URL in a browser and a page will come up with a
link to the Windows-only software.

If you are running a Mac or Linux or Solaris or HP-UX or BSD or VMS or ITS
or TOPS-20 system feel free to go all http://tinyurl.com/323g9h about it.

Thanks,

Ken

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

dipaksharma — Wed, 20 Aug 2008 02:35:46 GMT

Hi All,

I am running ESXi 3.5 Update 2 and would like to apply the patch that was release. I have downloaded the patch from this linkhttp://www.vmware.com/go/esxiu2patch081208

The problem is i do not know how do i install this patch. I read on the forum that i need Update manager, but i do not have that software. Also, please note i am using the free version as i am portioning only one single server.

Please advice how i do install this patch.

Thanks in advance.

Regards,
D

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

mjlin — Sat, 16 Aug 2008 01:44:38 GMT

Dear VMware Customers,

In addition to the express patch and the re-issued ESX/ESXi 3.5 Update 2 release, we now have an alternative installation process for customers who haven't applied either to hosts that were affected by the product expiration issue.

Below are the details we list at http://www.vmware.com/landing_pages/esxexpresspatches.html:

*Known VMware ESX 3.5 Update 2 Express Patch Installation Challenges

The following message is applicable ONLY for customers who had installed the impacted release of ESX 3.5 Update 2 (build number 103908), but not yet applied the express patch.

We are aware that you may encounter the following challenges installing the express patches needed to correct the problem:

Internal change control procedures
No available server to VMotion running VM's onto
Unable to schedule a maintenance window

If you experience one of the challenges listed above, please contact your support provider and indicate you need assistance with the U2 Alternative Install Process (U2 AIP). The support team can assist customers with this alternative installation procedure.

Thank you,

The VMware ESX Product Team

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

mcowger — Fri, 15 Aug 2008 16:32:53 GMT

I have an open SR for this, and will get back to you on VMware's findings.

--Matt

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

larstr — Fri, 15 Aug 2008 11:37:06 GMT

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

Erik Zandboer — Fri, 15 Aug 2008 09:46:15 GMT

It also occurs if you vmotion a VM to a host with a wrong date.
Personally experienced this in my environment, so I can't wait for you to tell me it didnt happen.
--Matt

Hi Matt,

This is where things get interesting! I tested this on 3 (yes three) environments, and was unable to get a VM jump back in time by vmotioning it to a backdated ESX host. I tested this by monitoring the vmotioned VM, and the date simply remained unaffected... I am wondering what might have caused your leap in time to happen. I would like to know, because this kind of unexpected behaviour might sting you later on...

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

macgreen — Fri, 15 Aug 2008 09:04:08 GMT

Clarification needed on patching.

I had updated my VirtualCenter Server from 2.5 U1 to U2 on August 10th from the 3.5 Infrastructure Management Installation download. My ESX 3.5 Servers are at U1 and were not upgraded to U2 (was planning this week - Wed).

My understanding is that code on ONLY ESX 3.5 Servers with U2 pre-12Aug are affected, so VC2.5U2 is OK?

And I should just proceed to install the latest build 110268 'ESX 3.5 U2 Refresh' ?

Appreciate any guidance from the community.

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

lholling — Fri, 15 Aug 2008 08:12:03 GMT

Hi Lee

First thing to do would be to start a new thread as a lot of people have given up on this one because it is too big.

Having said that, have you put the patch onto your 3.5U2 servers so that the licensing bug is fixed? If you have create the new thread and describe exactly what you have got versions and all. Can you power on VMs on the same servers that you cannot VMotion from / to?

How did you restore the patch. Put this information into a new thread or if you are under maintenance and you need it fixed quickly I would speak to support.

Leonard...

----

Don't forget if the answers help, award points

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

ISD Plc — Fri, 15 Aug 2008 08:03:58 GMT

Ok Guys I have a question,

We haven't updated to U2 yet but update manager has downloaded it (25/07) and the fix but and heres the rub.... Should we install the original U2 and the fix or somehow remove the U2 update as I understand that has been fixed itself now?

Any pointers greatfully received.

Harvey Dowler
IT Support Manager
Ideal Shopping Direct PLC
Registered Office: Ideal Home House, Newark Road, Peterborough, PE1 5WG, UK. Registered in England No. 1534758

www.idealshoppingdirect.tv, www.idealworld.tv,www.createandcraft.tv, www.idealvitality.tv, www.createandcraft.biz,
Freeview Channel 22, Satellite Channel 634, Virgin TV 747

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

leeroyrichardson — Fri, 15 Aug 2008 07:10:15 GMT

I have managed to get my VC back onli by applying the 2.5 u2 patch, I have reconnected all my esx servers, but I am still unable to migrate an live vms, anyone any ideas whats causing this.

Lee Richardson

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

Valentini — Thu, 14 Aug 2008 16:41:34 GMT

thanks for the replies regarding update manager and the recent patches. In case anyone else is running into an issue with Update Manager the fix for me was to change the schedule once again I set it to two minutes in the future, daily vs weekly and let it run. Finally it detected and is downloading the updates.

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

JohnTroyer — Thu, 14 Aug 2008 16:40:16 GMT

hey martin -- just left you a voicemail. You can reach me on my mobile 510 520 7832.
Everything ok?

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

mcowger — Thu, 14 Aug 2008 16:27:28 GMT

It also occurs if you vmotion a VM to a host with a wrong date.

Personally experienced this in my environment, so I can't wait for you to tell me it didnt happen.

--Matt

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

Valentini — Thu, 14 Aug 2008 15:50:22 GMT

Great, thanks for the info.

Ed Valenciano
Global Technical Enablement - HP Software
303.886.1544 mobile | Ed.Valenciano@hp.com
3404 E Harmony Road | FTC06 | Fort Collins | CO 80528
www.hp.com/software

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

jason.g.boche@wellsfargo.com — Thu, 14 Aug 2008 15:50:17 GMT

I also used update manager to roll out the ESXu2 time bomb fix. One esx box had to be bounced (lost conn to the vc server) while the other did not. This was in a lab.

Jason Boche, MCSE NT4/2000/2003, MCSA 2000/2003, MCP, VCPx2, CCAx2, A+
Systems Support Analyst 4
Wells Fargo Bank
625 Marquette Ave.
16th floor, N9311-162
Minneapolis, MN 55479
612-667-2473 office 612-910-5281 mobile
jason.g.boche@wellsfargo.com Email/AIM

**Sent from my BlackBerry handheld device**

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

Ivan Drago — Thu, 14 Aug 2008 15:20:18 GMT

The update manager works for applying the latest patch which fixes the time bomb issue with U2 hosts. I used the Update Manager to patch my hosts.

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

dwcrist — Thu, 14 Aug 2008 15:14:12 GMT

Yes, I was able to get it from Update Manager yesterday. However, I did have to cycle the VMware Update Manager service before I could see it include it in a baseline. But it did come down when I forced the download from my client.

-Doug

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

Valentini — Thu, 14 Aug 2008 15:10:24 GMT

I didn't want to read through 46 pages of posts. so if someone can give me a quick answer I would appreciate it.

IS THE PATCH DOWNLOADABLE VIA UPDATE MANAGER? MY UPDATE MANAGER IS NOT SEEING THE PATCH, IS THIS A PROBLEM ON MY END?

vmware says to view the kb articles but the LINKS ARE BROKEN.

Anyone...

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

lholling — Thu, 14 Aug 2008 15:00:22 GMT

Hi Lee Roy

What I would do is take a look at this other thread http://communities.vmware.com/thread/155382. If that does not help, it might be time to call suporrt.

Leonard...

----

Don't forget if the answers help, award points

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

leeroyrichardson — Thu, 14 Aug 2008 14:42:12 GMT

Just downloading the VC centre 2.5 u2 patch. In the meantime here is mt log

Log for VMware VirtualCenter, pid=3332, version=2.5.0, build=build-84767, option=Release, section=2
~~2008-08-14 10:37:02.121 'App' 3256 info~~ Current working directory: C:\WINDOWS\system32
~~2008-08-14 10:37:02.121 'BaseLibs' 3256 info~~ HOSTINFO: Seeing Intel CPU, numCoresPerCPU 4 numThreadsPerCore 1.
~~2008-08-14 10:37:02.121 'BaseLibs' 3256 info~~ HOSTINFO: This machine has 1 physical CPUS, 4 total cores, and 4 logical CPUs.
~~2008-08-14 10:37:02.121 'App' 3256 info~~ Log path: C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Logs
~~2008-08-14 10:37:02.121 'App' 3256 info~~ Initializing SSL
~~2008-08-14 10:37:02.355 'BaseLibs' 3256 info~~ Using system libcrypto, version 90709F
~~2008-08-14 10:37:03.168 'App' 3256 info~~ Vmacore::InitSSL: doVersionCheck = true, handshakeTimeoutUs = 120000000
~~2008-08-14 10:37:03.168 'App' 3256 info~~ Starting VMware VirtualCenter 2.5.0 build-84767
~~2008-08-14 10:37:03.168 'App' 3256 info~~ Log directory: C:\Documents and Settings\Default User\Local Settings\Application Data\VMware\vpx.
~~2008-08-14 10:37:03.168 'App' 3256 info~~ Account name: SYSTEM
~~2008-08-14 10:37:03.168 'App' 3256 info~~ VpxOsLayer Enabled low-frag process heap.

~~2008-08-14 10:37:03.168 'App' 3256 info~~ VpxOsLayer Enabled low-frag crt heap.

~~2008-08-14 10:37:03.168 'App' 3256 info~~ VpxLRO 34 max LROs
~~2008-08-14 10:37:03.168 'App' 3256 info~~ VpxLRO 6 reserved internal LROs
~~2008-08-14 10:37:03.168 'App' 3256 info~~ VpxLRO 6 reserved blocker LROs
~~2008-08-14 10:37:03.168 'App' 3256 info~~ VpxLRO 6 reserved short LROs
~~2008-08-14 10:37:03.168 'App' 3256 info~~ VpxLRO 4 reserved long LROs
~~2008-08-14 10:37:03.168 'App' 3256 info~~ VpxLRO 600-second task lifetime
~~2008-08-14 10:37:03.871 'App' 3256 error~~ Vdb::IsRecoverableErrorCode Unable to recover from 42000:195
~~2008-08-14 10:37:03.871 'App' 3256 error~~ Vdb::IsRecoverableErrorCode Unable to recover from 42000:8180
~~2008-08-14 10:37:05.934 'BaseLibs' 3256 warning~~ VpxdOsLayer Couldn't read registry entry vmdbPort
~~2008-08-14 10:37:05.934 'BaseLibs' 3256 warning~~ VpxdOsLayer Couldn't read registry entry managedIP
~~2008-08-14 10:37:06.074 'App' 3256 error~~ OptionMgr Ignoring unknown entry from DB: AgentUpgrade.maxConcurrentUpgrades
~~2008-08-14 10:37:06.074 'App' 3256 error~~ OptionMgr Ignoring unknown entry from DB: Perf.Stats.FilterLevel
~~2008-08-14 10:37:06.277 'App' 3256 info~~ VpxdMoOptionManager Invoking callbacks for key vpxd.npivWwnGeneration.singleNodeWwn
~~2008-08-14 10:37:06.371 'App' 3256 info~~ VpxdMoOptionManager Invoking callbacks for key vpxd.npivWwnGeneration.portWwnNumber
~~2008-08-14 10:37:15.074 'Locale' 3256 info~~ Locale subsystem initialized from C:\Program Files\VMware\Infrastructure\VirtualCenter Server\locale/ with default locale en.
~~2008-08-14 10:37:15.090 'App' 3256 info~~ Initializing SSL Contexts
~~2008-08-14 10:37:15.090 'App' 3256 info~~ SSL Setting VerifySSLCertificates to FALSE
~~2008-08-14 10:37:19.684 'App' 3256 error~~ VpxdMain Failed to initialize: not well-formed (invalid token)
~~2008-08-14 10:37:19.684 'App' 3256 error~~ Failed to intialize VMware VirtualCenter. Shutting down...
~~2008-08-14 10:37:19.684 'App' 3256 info~~ Forcing shutdown of VMware VirtualCenter now

Any help would be much appreciated

Lee Richardson

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

dbuchanan — Thu, 14 Aug 2008 14:40:17 GMT

All you have to do is build another VC and point to your database add in
the host and then vmotion your virtual VC to a working server, then
shutdown the new VC then start up the Virtual VC to continue on.

Dan L. Buchanan | Microsoft Engineer

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

lholling — Thu, 14 Aug 2008 14:38:08 GMT

If the service does not start have you had a look at the VC log files to find out what the problem is? Also do you have your database server up and running.

Start by taking a look in the following directory C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Logs

Leonard...

----

Don't forget if the answers help, award points

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

dbuchanan — Thu, 14 Aug 2008 14:37:00 GMT

All you have to do is build another VC and point to your database add in the host and then vmotion your virtual VC to a working server, then shutdown the new VC then start up the Virtual VC to continue on.

Dan L. Buchanan | Microsoft Engineer

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

mrbill007 — Thu, 14 Aug 2008 14:35:29 GMT

Another important thing to note:

If you are using VC 2.5U1, and you added any ESX 3.5 U2 hosts to the VC, you have to upgrade VC to 2.5 U2 or the service will fail to start / restart / do anything if you reboot the machine.

Upgrade VC to 2.5U2 - Use the .zip, not the ISO, there is something borked with the ISO also.

VMWARE is aware of the problem, has KBs up related to the ISO problem - I don't recall the exact #, but its out there.

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

leeroyrichardson — Thu, 14 Aug 2008 14:05:14 GMT

I am still in the really bad situation of not being able to kick my VC into lift, the service just refuses to start.

I cannot migrate my live VMs and as such I cannot do any upgrades on my esx servers.........Im in a very worrying position.

Lee Richardson

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

THP — Thu, 14 Aug 2008 13:48:21 GMT

And I wouldn't (& didn't) argue with that - I am merely suggesting that this is the cause of one particular poster's issues, perhaps he has restarted his VMs?

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

Erik Zandboer — Thu, 14 Aug 2008 13:41:45 GMT

One last time. The VMs inside an ESX host whose date is set back do NOT alter their time/date unless you START them (NOT if they are running). So AD has NO WAY he can figure out the date/time of the ESX host. I tested this in several setups. The time inside the VMS in UNAFFECTED. So there is no policy on this world that can trigger on the ESX host date/time settings. It only knows about the local time and date inside the VM.

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

dmanconi — Thu, 14 Aug 2008 13:20:24 GMT

Hi

Looks like the new updated ESX 3.5 update 2 (ESX and ESXi) ISO files are up......

Cheers

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

THP — Thu, 14 Aug 2008 13:16:42 GMT

In AD you can limit user accounts to certain hours or certain machines only - I would suspect that AD is applying a correct set of rules but that by messing with the underlying time of the host/vm this is triggering the warnings. Setting the time back to the correct time or using domain time should solve it - or failing that just take the rules out of AD until VM is fully patched and working.

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

Erik Zandboer — Thu, 14 Aug 2008 13:12:49 GMT

Hi, the problem is, that your DC has NO knowledge of the altered time. It just sticks to its own time, and has no knowledge of the time of the ESX host. None whatsoever. Therefore your VM can never decide you are outside working hours....

What CAN happen is that you have the ESX host dated back for a longer period of time... your DC(s) will then slowly start to loose their correct time (because the VMware Tools are no longer correcting it since the host is days behind)

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

vmfanboy — Thu, 14 Aug 2008 13:00:50 GMT

Hi,

You probably have some rules in your DC that permit logging for some user only on work hours. And since the date was set to sunday... I can't verify this at the moment but I'm sure your can do a rule like that

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

mjlin — Thu, 14 Aug 2008 12:32:01 GMT

Dear VMware Customers,

We have re-issued the entire ESX/ESXi 3.5 Update 2 release (ISOs, upgrade tar and zip files, and patch bundles). They are available for download at http://www.vmware.com/download/.

Thank you,

The VMware ESX Product Team

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

Erik Zandboer — Thu, 14 Aug 2008 08:38:32 GMT

Did you (or the customer) figure out what the REAL problem was? Because if your DC's clock is off, you should get an error about time/date not being in sync. And not an error about "work hours"... Again, assuming the customer did not START a VM on an ESX host with the date set back.

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

eliot — Thu, 14 Aug 2008 07:35:31 GMT

I had a customer who rolled back to sunday's time and users reported being locked out because it was outside 'work hours'

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

Erik Zandboer — Thu, 14 Aug 2008 07:09:59 GMT

That is the problem - they're not. A lot of people think they need to suspend or shut their DCs while rolling back the ESX hosts date because of this issue which is not an issue at all....

BTW, suspending a VM does not even work on an unpatched host...

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

RonWBrown — Thu, 14 Aug 2008 07:04:27 GMT

and the exception you mentioned is what everyone is talking about.

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

Erik Zandboer — Thu, 14 Aug 2008 06:33:41 GMT

Quote:

"Guys please be careful when setting the clock back on ESX if you have virtual AD servers - AD doesn't like going back in time.
Perhaps suspend the DC whilst you do this trick. "

I don't know where everybody gets this from. VMware Tools do NOT set the time back on your VM if the ESX servers clock is set back. I tested this (yes even with a domain controller). The time remians OK inside the VM, even when you vmotion a VM to a host with its time set back. This is by design, and there is no risk involved when you set the clock back on an ESX host.

Exception is when you START a VM on a host while the clock set back on that host. This is another story. Since the VM does not have its own realtime clock, the VMs BIOS has no idea of time. So when you START a VM, its gets its BIOS time from the ESX hosts time.

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

APN.NZ — Thu, 14 Aug 2008 05:34:28 GMT

Hi From New Zealand

I have successfully patched all our ESX 3.5u2 servers with the express patch

Download from here http://kb.vmware.com/1006716

We have a website on one of our web servers which allows directory
browsing, I download the patch and extracted the files to a directory
under the website that allows directory browsing.

1. All of our ESX servers are ESX 3.5u2, so we check which ESX Server
was running vm's that would be least effected by a time change.
2. Opened port 80 out on each of our affected servers. command = esxcfg-firewall -o 80,tcp,out,HTTPclient (did this to patch via website)
3. Disabled HA on all of our clustered ESX's
4. Set DRS on all of our clustered ESX's to manual
5. Checked that all vm's running on ESX1 were syncing time from ntp or domain and made sure none were using time sync from vmware tools.
6. Set the Date back on the server ESX1 the least affected by a date and time changes. Stopped NTP service. command = service ntpd stop then set the date back command = Date -s 08/08/08
7. Vmotioned all powered on vm's from ESX2 to ESX1 (the server with the old date and time)
8. Once all vm's were vmotoined set the date back on ESX1 command = service ntpd start
9. Put ESX2 into maintenance mode
10. Applied the patch to ESX2 by connecting to the web server with the extracted patch files. command = esxupdate -r http://webserver/patch/ESX350-200806812-BG/ESX350-200806812-BG/ update
11. Once patched, tried to exit maintenance mode on ESX2 but got the following error the session is not authenticated
12 .Restarted the vpxa agent on ESX2 command = service vpxa-vmware restart
13. Exited maintenance mode successfully on ESX2 once the vpxa agent had restarted.
14. Vmotioned vm's from ESX1 to the patched ESX2
15. Patched ESX1
in the same manner as mentioned above. (kept on moving vm's off
unpatched servers and onto patched servers till all servers in the
cluster were patched)
16. Enabled HA
17. Set DRS to the setting they were on prior to the patching.

I was able to patch all our ESX 3.5u2 servers in this manner with zero downtime for any of my production VM's

For esx35.i just use the remotecli vihostupdat.pl and vicfg-ntp.pl or use the vmware infrastructure update gui tool

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

mjlin — Thu, 14 Aug 2008 03:59:59 GMT

Dear VMware Customers,

We are experiencing a delay in releasing the new version of ESX/ESXi 3.5 Update 2. Our testing of this release is taking longer than anticipated. We are now targeting to release the update between 2 AM PDT (0700 UTC) August 14 and 8 AM (1500 UTC) August 14 PDT.

We apologize for the delay. VMware engineering teams are working around the clock to deliver this new version. We will continue to keep you informed of developments until this issue is fully resolved.

Please note this update is only relevant to customers who did not install the impacted release of ESX 3.5Update 2 (build number 103908) and ESXi 3.5 Update 2 (build number 103909). If you have installed either of these please visit http://www.vmware.com/landing_pages/esxexpresspatches.html and install the express patch.

Thank you,

The VMware ESX Product Team

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

kimono — Thu, 14 Aug 2008 03:13:36 GMT

Rock and a hard place. Change control for a new release of anything should take much more than two weeks, yes, but we all have to ask ourselves, what if the bomb was Aug 12 2009 ?

I've disabled VMware Update Manager service until we see some sort of Post Review on this.

/kimono/

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

jketron — Wed, 13 Aug 2008 20:14:39 GMT

awesome and thank you for the clarification

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

Anton V Zhbankov — Wed, 13 Aug 2008 20:12:06 GMT

Yes. All the ways.

But now UM will also install fix for time bomb, so at final you'll have working ESX 3.5.0 u2.

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

mjlin — Wed, 13 Aug 2008 20:11:51 GMT

Dear VMware Customers,

We have released the express patches for the product expiration issue. Please go to http://www.vmware.com/go/esxexpresspatches for download and KB articles. Since our last customer email we have completed our verification tests that the express patches we've released are fully compatible with the VMware Update Manager. Please see the KB articles for deployment information regarding Update Manager.

The KB articles are kept up-to-date. Please refer to the KB articles for information and updates.

In our last update, we referred to an initiative by our support and engineering teams to find an option to apply the patch without the necessity of entering maintenance mode and VMotion of VM's to other servers, or VM power-off and re-power-on. Our earlier tests have not found a consistently successful way to address this. We continue to investigate this possibility, as we know that it would reduce the maintenance burden on our customers who may not have a patched server available for VMotion.

We are on target to release updated versions of the ESX/ESXi 3.5 Update 2 patch at 6 PM PST today. This is for customers who have not already upgraded to the previously released version of ESX/ESXi 3.5 Update 2

Thank you,

The VMware ESX Product Team

Problem:

An issue has been discovered by many VMware customers and partners with ESX Update 2 (build number 103909) and ESXi 3.5 Update 2 (build number 103908) where Virtual Machines fail to power on or VMotion successfully. This problem began to occur on August 12, 2008 for customers that had upgraded to ESX 3.5 Update 2. The problem is caused by a build timeout that was mistakenly left enabled for the release build.

The following message is displayed in the vmware.log file for the virtual machine:

This product has expired. Be sure that your host machine's date and time are set correctly.
There is a more recent version available at the VMware web site: http://www.vmware.com/info?id=4.

Module License Power on failed.

Affected Products:

VMware ESX 3.5 Update 2 & ESXi 3.5 Update 2. Thank you, The VMware ESX Product Team

The problem will be seen if ESX350-200806201-UG is applied to a system.

No other VMware products are affected.

Resolution:

VMware Engineering has produced express patches for impacted customers to resolve the issue.

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

jketron — Wed, 13 Aug 2008 20:06:27 GMT

Let me ask a question here,

Can the bug be introduced using Update Manager or just from ISO or a manual install?

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

RamseyMS — Wed, 13 Aug 2008 19:34:20 GMT

Can we take a look at this for a second? Other than this highly unusual situation with update 2 virtualizing 'virtual center' has MANY benefits and minimal risk. VI3 has no critical dependencies on virtual center other than for vmotion. HA will continue to survive without virtual center and you normally have two weeks to replace the licensing server before bad things happen with powering on machines.

The benefits include:

The ability to take snapshots of the VC on a regular basis.
Easier and complete backups of the VC
Making VC highly available

Just saying....

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

aenagy — Wed, 13 Aug 2008 19:21:06 GMT

While I agree with you that it pays to have VC/DB/License Server on a separate server, you might not have read:
http://www.vmware.com/pdf/vi3_vc_in_vm.pdf

in which VMWare does not recommend against running it as a VM but in fact "fully supports" (to use their words) VC as a VM.

Except, of course, in the very unlikely event that the software determines it has 'expired', and then regardless of where your license server is, you're screwed..

Chris Bennett (cgb)

Yeah, I read that too. I made the argument that putting VirtualCenter on a virtual machine creates a circular dependency and therefore an unacceptable risk. Management agreed and paid for a physical machine.

As it is, we were not affected as we are still on ESX Server 3.0.2 Update 1....

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

RamseyMS — Wed, 13 Aug 2008 19:20:26 GMT

Probably the better solution would be to ensure the VM's time is not synchronized to the host before you roll the time back.

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

aenagy — Wed, 13 Aug 2008 19:17:44 GMT

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

eliot — Wed, 13 Aug 2008 18:28:29 GMT

Guys please be careful when setting the clock back on ESX if you have virtual AD servers - AD doesn't like going back in time.

Perhaps suspend the DC whilst you do this trick.

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

Ivan Drago — Wed, 13 Aug 2008 16:11:14 GMT

Letter from VMware CEO Paul Maritz

Last night, we became aware of a code issue with the recently released update to ESX 3.5 and ESXi 3.5 (Update 2).
When the time clock in a server running ESX 3.5 or ESXi 3.5 Update 2 hits 12:00AM on August 12th, 2008, the released code causes the product license to expire. The problem has also occurred with a recent patch to ESX 3.5 or ESXi 3.5 Update 2. When an ESX or ESXi 3.5 server thinks its license has expired, the following can happen:
• Virtual machines that are powered off cannot be turned on;
• Virtual machines that have been suspended fail to leave suspend mode; and,
• Virtual machines cannot be migrated using VMotion.
The issue was caused by a piece of code that was mistakenly left enabled for the final release of Update 2. This piece of code was left over from the pre-release versions of Update 2 and was designed to ensure that customers are running on the supported generally available version of Update 2.
In remedying the situation, we’ve already released an express patch for those customers that have installed/upgraded to ESX or ESXi 3.5 Update 2. Within the next 24 hours, we also expect to issue a full replacement for Update 2, which should be used by customers who want to perform fresh installs of ESX or ESXi.
I am sure you’re wondering how this could happen. We failed in two areas:
• Not disabling the code in the final release of Update 2; and
• Not catching it in our quality assurance process.
We are doing everything in our power to make sure this doesn’t happen again. VMware prides itself on the quality and reliability of our products, and this incident has prompted a thorough self-examination of how we create and deliver products to our customers. We have kicked off a comprehensive, in-depth review of our QA and release processes, and will quickly make the needed changes.
I want to apologize for the disruption and difficulty this issue may have caused to our customers and our partners. Your confidence in VMware is extremely important to us, and we are committed to restoring that confidence fully and quickly.
Thank You,
Paul Maritz
President and CEO
VMware

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

gnomes — Wed, 13 Aug 2008 16:00:29 GMT

I was able to update my free ESXi servers by using the ESXe350-200807812-O-BG \ ESXe350-200807812-I-BG \ remoteInstall.exe in windows command prompt, after reboot of the host I was once again able to start VMs.

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

dwcrist — Wed, 13 Aug 2008 15:18:08 GMT

Well, good things come to those who wait! It finally finished, and I can power on my VM now! This was a new host with no VMs on it yet. In my production cluster I will have to pick one host on which to shut down the VMs. Once it is patched, I can then use Vmotion to work my way through the reset of the cluster.

-Doug

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

dwcrist — Wed, 13 Aug 2008 15:09:12 GMT

It's running now. I had had a putty session open, and that cause it to 'timeout while trying to lock the host'. It now says it has installed the patch, but it seems to be hung at 39% on the remediate task. Could it be waiting for me to do something, like reboot? I've never had to do anything manually before when using UM.

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

Erik Zandboer — Wed, 13 Aug 2008 14:58:37 GMT

Check out my previous post (4 up).. worked without any issues on my two clustered hosts...

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

dwcrist — Wed, 13 Aug 2008 14:52:00 GMT

Just an update with Update Manager:

I was able to download the patch, and once I cycled the Update Manager service, I could see it in the list. I created a baseline, did the scan, and told it to remediate my host (which I had already placed into maintenance mode). It failed -- 'VMware Update Manager had a failure'. I'm looking at the logs now.

Has anyone gotten Update Manager to apply the 6812 patch successfully?

Thanks,

Doug

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

Jaune — Wed, 13 Aug 2008 14:44:37 GMT

Awesome! Thanx guys!!

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

Erik Zandboer — Wed, 13 Aug 2008 14:36:17 GMT

Hi All,

As far as I have read this thread, most people went for the manual update to fix this issue. I just managed to upgrade my environment 100% from the VI client, without having to shut down a single VM.

I first disabled NTP and set the time back on one host, vmotioned all VMs to that host, and started NTP again (watched it sync to the correct time). Then entered the empty one into maintenance mode, performed the upgrade using the update manager. After that, got out of maintenance mode, and vmotioned all VMs back to the now patched host, and repeated the upgrade for the first host. Worked without any issues, while all my VMs use the timesync option form the VMware Tools (and kept their time without problem inside the VMs). Just avoid starting VMs on the host with an altered date (that screws things up because BIOS time is then wrong).

I must say "VMware very well done". After an almost unforgivable mistake, they managed to make it right again in a remarkable short period of time. The issue can even be solved without any VM downtime, just as is always promised by VMware. Awesome guys (and girls)!!

EDIT: I must say that I did disable HA on my cluster, but I left DRS enabled (manual). This is because I did not want to loose my resource pools...

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

Ivan Drago — Wed, 13 Aug 2008 14:41:20 GMT

You can VMotion from a non-patched server. I was able to do this yesterday without any problems.

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

Erik Zandboer — Wed, 13 Aug 2008 14:39:39 GMT

Hi Jaune,

Read my previous post. You CAN use vmotion, but you must set the time back on the target host. Hosts that have been patched can be used as a vmotion target without having to set the date back. So basically, you CAN make this problem go away without any VM downtime!

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

Jaune — Wed, 13 Aug 2008 14:30:37 GMT

Guys,

My environment is still running fine as I haven't shut down any VMs. In order to install the patch, i need to put the host in maintenance mode. SInce VMotion doesn't work either, I have to shut down the VMs.

Does that mean I will have to shut down every VMs? Once a host is patched, can I vmotion from a non patched server or I will still have the Lic issue?

Thanks

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

j-swift — Wed, 13 Aug 2008 14:22:40 GMT

Yep, I did read that. But when it (VC) first came out I worked for a well-known broadcaster, and I eventually recommended they go to VMware (I tested MS Virtual Server too - so, at the time, there wasn't a great deal of difficulty in recommending VMware), we had a tech. architect from VMware with us, to discuss if we were designing it right, and he advised against using a VM to run VC, for the very reason that if your ESX host is unavailable, so is your VC (this was with 2.x). Now, with HA, etc., that might not be the case, but it's very clear here that having your VC on your ESX host isn't a really great idea. I also know there were lots of people asking VMware to support this -- that's not the same as recommending it, which they do not. They support it, but they don't recommend it.

Also, SQL can place a significant load on any box. Best to have a physical SQL box (x2) in a cluster, to keep the load off your ESX boxes. VMware have been saying for years you can run anything on ESX, including Exchange, but I've known 4000 Exchange users on one DL585, and that worked fine, but how many would you get on one ESX box - and frankly, why would pay extra to buy an ESX license to do that? Doesn't really give much extra, except VMotion - and you can do that with clustering.

VMware is great, but best targested to specific work loads.

Just as a further thought, imagine how bad this could really have been. Top-notch licensing code simply wouldn't allow you to turn back the clock. Clever licensing software would tracks the dates, in an encrypted file, and not allow you to simply fiddle things by putting the date back. Imagine also, code that doesn't just stop you from starting new VMs, or using advanced features like VMotion, but code which actually shuts down all your VMs on that expiration date ! It could have been very much worse. I think was most people have been feeling has been a loss of control, that point where you can't really do much and depend upon VMware to allow them to continue their business. People will want a lot more control over their licenses in the future months, to eliminate the possibility of VMware shutting down, worldwide, businesses IT services.

Jonathan.

While I agree with you that it pays to have VC/DB/License Server on a separate server, you might not have read:
http://www.vmware.com/pdf/vi3_vc_in_vm.pdf
in which VMWare does not recommend against running it as a VM but in fact "fully supports" (to use their words) VC as a VM.
Except, of course, in the very unlikely event that the software determines it has 'expired', and then regardless of where your license server is, you're screwed..
Chris Bennett (cgb)

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

APISupport — Wed, 13 Aug 2008 14:02:45 GMT

I confirm to you guys. Same problem and same workaround in Europe.

Great thanks to mattjk to find this workaround.

Sorrry, I didn't see there were 43 pages in this post .... looking for the path right now.

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

cgb — Wed, 13 Aug 2008 13:52:38 GMT

j-swift wrote: Bad idea to have VC/infrastructure only as a VM - you need to buy a physical box to run VC (& licensing). In fairness to VMware they did recommend against running it as a VM.

While I agree with you that it pays to have VC/DB/License Server on a separate server, you might not have read:
http://www.vmware.com/pdf/vi3_vc_in_vm.pdf

in which VMWare does not recommend against running it as a VM but in fact "fully supports" (to use their words) VC as a VM.

Except, of course, in the very unlikely event that the software determines it has 'expired', and then regardless of where your license server is, you're screwed..

Chris Bennett (cgb)

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

ejward — Wed, 13 Aug 2008 13:51:14 GMT

I've gotten so used to using update manager that I frankly don't remember HOW to use esxupdate. It's been so long since I've done it. It was so cumbersome when I was using it that I just didn't patch as much. With Update manager, look where staying on top of patching has gotten me

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

Trystam — Wed, 13 Aug 2008 13:50:12 GMT

Basicly:

You download the ESX350-200806812-BG.zip file up upload it via SFTP to your ESX box onto a directory, in my case /var/updates.

Put the ESX system in maintenance mode.

then:

~~root@talx01 root~~# cd /var/updates/
~~root@talx01 updates~~# ls -ltr
total 106720
-rw-r--r-- 1 root root 109165839 Aug 13 14:48 ESX350-200806812-BG.zip
~~root@talx01 updates~~# unzip ESX350-200806812-BG.zip
Archive: ESX350-200806812-BG.zip
creating: ESX350-200806812-BG/
inflating: ESX350-200806812-BG/VMware-esx-vmx-3.5.0-110181.i386.rpm
inflating: ESX350-200806812-BG/VMware-hostd-esx-3.5.0-110181.i386.rpm
inflating: ESX350-200806812-BG/descriptor.xml
creating: ESX350-200806812-BG/headers/
extracting: ESX350-200806812-BG/headers/VMware-esx-vmx-0-3.5.0-110181.i386.hdr
extracting: ESX350-200806812-BG/headers/VMware-hostd-esx-0-3.5.0-110181.i386.hdr
inflating: ESX350-200806812-BG/headers/header.info
inflating: ESX350-200806812-BG/headers/contents.xml
inflating: ESX350-200806812-BG/headers/contents.xml.sig
inflating: ESX350-200806812-BG/contents.xml
inflating: ESX350-200806812-BG/contents.xml.sig

~~root@talx01 updates~~# cd ESX350-200806812-BG

~~root@talx01 ESX350-200806812-BG~~# esxupdate update

The patch will install, and you will loose connection from your Virtual Center to the box for about 2 to 5 minutes, after that she will show up again in maintenance mode.

Remove machine from maintenance mode and your good to power up everything again.

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

cgb — Wed, 13 Aug 2008 13:41:10 GMT

hughs wrote:
I did, but where did it tell me to do this? The VMware info is really poor. The email refered to the page with the
patches, which is great. But that page then says look at the KB for 'deployment consieration and something'. Fine.
But that then tells me to download and use three different .pdf's. Yawn. How about "Download this huge file. Use the
Infrastructure Update software to apply it. Reboot." (Yes I know it's all more complicated than this).

I felt the same way - even though I'm familiar with using esxupdate to pull down updates from a locally built repository, I think given the impact this problem has had, VMWare could have provided a very clear step-by-step procedure for performing the update so those who needed it resolved urgently could get it applied asap, rather than have to go read VMWare Update Manager documentation or Patch Management Guide.

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

cgb — Wed, 13 Aug 2008 13:15:11 GMT

Bastian.Haas wrote:
To whom it concerns at VMware: Could you please change this useless error message
"A general system error occurred: Internal Error" in a future VIC update? With this message
no one knows what the problem is, regardless whether it's a real licensing error or a timebomb

Very good request - nothing annoys me more than non-intuitive error messages. Even when debugging
in the logs for this problem, the message logged was 'Product expired. Be sure that your host machine's
date and time are set correctly. '. I mean, it was accurate given the cause, but totally unexpected when
I'm building up a validly licensed new server for a customer..

Even more annoying, was the suggestion in hostd.log:
There is a more recent version available at the VMware Web site: "http://www.vmware.com/info?id=4".

and then to go to that URL and all the while VMWare knew about the problem and didn't update that URL
content with a useful link to the knowledge base article (while it was up) and/or to the forum post where it all
began. Even now, the content is still not changed to point people to the issue.

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

vmwaredimetronic — Wed, 13 Aug 2008 12:40:15 GMT

excellent!!!!

I´ve updated 3/5 of my esx servers succesfully throught this steps!!

The other two i have problems with metadata updates and i can not resolve at moment, also this esx servers with the same version that another, not including esxupdate command.

Thanks!!

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

hughs — Wed, 13 Aug 2008 11:58:07 GMT

I did, but where did it tell me to do this? The VMware info is really poor. The email refered to the page with the patches, which is great. But that page then says look at the KB for 'deployment consieration and something'. Fine. But that then tells me to download and use three different .pdf's. Yawn. How about "Download this huge file. Use the Infrastructure Update software to apply it. Reboot." (Yes I know it's all more complicated than this).

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

hughs — Wed, 13 Aug 2008 11:25:18 GMT

I don't know if anyone can help but I'm still having problems. I downloaded the patch file (ESXe350-200807812-O-BG.zip) and used Infrastructure Update to apply it to a server running 3.5u2 ESXi. It said it had finished ok, so I reset the time to today and.... nothing. I'm still getting the same error. Now if I try to add the package I get an error saying 'Package does not contain applicable updates'

Help!

Edit: It works. I've rebooted the host PC (wow, it's just like using Windows) and I can now boot my VM's. Good luck to all those with many many computers that need rebooting.....

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

ICT-Freak — Wed, 13 Aug 2008 11:48:09 GMT

Did you restart the ESX3i host?

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

j-swift — Wed, 13 Aug 2008 11:18:00 GMT

I think some people have been caught out because they have run their VC as a VM and thus, once powered off, it won't now power on if that system has update 2.

Bad idea to have VC/infrastructure only as a VM - you need to buy a physical box to run VC (& licensing). In fairness to VMware they did recommend against running it as a VM.

I'm expecting some Architects will be tweaking their design and recommendations based on what has happened. Bl**dy good job that it happened now, when virtualisation is growing, but still a relatively small part of most data centres (I work across two DCs with around 5500 devices, around 5000 of which are servers). Just imagine what 'could' have happened if most servers were virtualised - in the next 10-15 years this could be the case.

Like most shareware, and trial software has done for decades, a small piece of code providing a warning about expiration, which logs a message to VC, syslog, whatever, as you approach that date, but to disable it instantly, without warning, is where VMware had shot themselves in the foot. Luckily, we can all learn from this and make more informed decisions about the risks of licensing models (which sort to use), and also about business risks of allowing guests to timesync to a host, implications for moving dates backwards on a host, and making sure VC is on a separate physical machine.

Jonathan.

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

leeroyrichardson — Wed, 13 Aug 2008 11:20:05 GMT

I am running both the VC and licensing on a physical box and no matter what I do I cannot get the VC to start. We will be unable to migrate the servers and complete the upgrade process without the VC running.

Lee Richardson

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

Bastian.Haas — Wed, 13 Aug 2008 11:15:11 GMT

To whom it concerns at VMware: Could you please change this useless error message "A general system error occurred: Internal Error" in a future VIC update? With this message no one knows what the problem is, regardless whether it's a real licensing error or a timebomb

Thank you.

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

abrjgl — Wed, 13 Aug 2008 10:45:44 GMT

I reported this problem in this thread http://communities.vmware.com/message/1019825#1019825

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

Trystam — Wed, 13 Aug 2008 10:26:45 GMT

My Virtual Center is U1 and i have no record of that problem in my infrastructure nor i believe that behaviour being reported.

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

Trystam — Wed, 13 Aug 2008 10:24:55 GMT

Lee,

This issue doesnt affect the Virtual Center, so you must be experiencing some other problem too.

You cannot VMotion when you connect your Virtual Client directly to the ESX Box since you are only seeying part of your infrastructure, VMotion is a Virtual Center based/controlled feature.

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

abrjgl — Wed, 13 Aug 2008 10:23:13 GMT

mjlin - have you upgraded to Virtual Center 2.5 Update 2? VC 2.5 Update 1 can have this problem if you have ESX 3.5 update 2 hosts.

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

leeroyrichardson — Wed, 13 Aug 2008 10:21:43 GMT

I am logged onto my esx server ui because my vcenter is down, but I cannot see any options to migrate my current vms???

If I changed the dates on my esx servers would my vcentre come back online???

Lee Richardson

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

APN.NZ — Wed, 13 Aug 2008 09:56:52 GMT

vihostupdate is a remote CLI you use to scan and update ESX Server 3i you could try it with esx3.5 but if esxupdate is not on two of your systems sounds like you have bigger issues!

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

Trystam — Wed, 13 Aug 2008 09:52:52 GMT

by using the esxupdate command you need to have get the host in maintenance mode:

eg:
~~root@talx01 ESX350-200806812-BG~~# esxupdate update
INFO: No repository URL specified, going with file:///var/updates/ESX350-200806812-BG
INFO: Configuring...
ERROR: This bundle requires the host to be in maintenance mode. Since the host is not in maintenance mode, esxupdate cannot proceed. The VMs need to be turned off or migrated to another host first.
~~root@talx01 ESX350-200806812-BG~~#

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

leeroyrichardson — Wed, 13 Aug 2008 09:21:07 GMT

I cannot open my vcentre to run the auto patch, so I have downloaded the patch to a temp file on the affected esx boxes, do I need to take the servers offline before running the patch??

Lee Richardson

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

THP — Wed, 13 Aug 2008 09:16:57 GMT

I'll second this - we were patching (intendeing to go to U1) on the day U2 was released.

Initially before scan for updates we saw U2 mentioned - after the scan it had gone (at the time we thought this was perhaps that it had been pulled). However when we remediated none of the listed patches were called 'Upgrade 2' so we thought we were safe to continue and did so.

On checking version we found we had done Upgrade 2 as well. As it is our dev area we didn't panic but were a little annoyed at the lack of clarity in the tool. Obviously yesterday we became a little more cheesed off about it.

And to allk thjose saying M$ do worse - yes they have done but at least when I am installing a service pack to an OS it clearly lists itself as a service pack so I can understand the risk I am taking on!!

I hope one lesson learned from this is that VM make their patch labelling much clearer.

Sure the 'perfect & smug people' can heap scorn on those of us who were misled but it seems to have hit more than just the novices so perhaps worth addressing?

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

abrjgl — Wed, 13 Aug 2008 09:09:10 GMT

jamieorth and Kevin Gao - As I tried to explain: Of course I configure Update Manager but as I tried to explain I specifically did not tell Update Manager to install ESX350-200806201-UG (part of Update 2). I did however tell Update Manager to install ESX350-200804404-BG and ESX350-200804405-BG. Both of these updates will trigger the install of ESX350-200806201-UG and they should not do that.

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

Chris76FiSi — Wed, 13 Aug 2008 08:14:53 GMT

Installed the critical fix trough VC Update mechanism and now everything works fine again. Thx to VMware for this fix.

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

fredz — Wed, 13 Aug 2008 08:06:24 GMT

Is it safe to install the full "update 2" update now? I have a few systems with "update 1" which need to be upgraded.

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

vmwaredimetronic — Wed, 13 Aug 2008 08:03:53 GMT

Hi guys!

i´ve tried to install this patch in my 5 esx servers, and i have a problem with two of them, in these i have not a esxupdate command, the esx version is the same (3.5.0.103908) in 5 servers and i can not run update manager in them because failed metadata, another command to update it??

Thanks.

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

rollin71 — Wed, 13 Aug 2008 07:58:26 GMT

You can ftp the patch to the tmp directory on all of your affected ESX hosts and then from whatever directory you copied it to you can run
esxupdate update

The version should be ESX Server, 3.5.0, 110181

You might need to open the firewall ports on your ESX host so you can use an ssh ftp client to get into your ESX Host

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

leeroyrichardson — Wed, 13 Aug 2008 07:53:34 GMT

I have tried to logon to the vcentre this morning but I cannot get the service to start, all the other services are loading fine. Is this part of the bug or has something else happened to my vcentre???

Unless I can get the vcentre working I will be unable to run the auto-patch.

Lee Richardson

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

rollin71 — Wed, 13 Aug 2008 07:48:32 GMT

The posted by noon tommorrow being the 13th or should i say later today.

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

trxman — Wed, 13 Aug 2008 07:46:40 GMT

will some new (fixed) install ISO be released for fresh installations?

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

lholling — Wed, 13 Aug 2008 07:43:50 GMT

rollin71 wrote:
Sorry i meant VC but i tried to put the ESX server in evaluation mode so i could power on the VC server. Also the VC server has been off since the problem started late last night. I applied the update to the ESX server that had the VC server vm on it and afterwords i wasn't able to power anything on due to licensing. I then had to generate a single server license just to get the vm's to be able to power on and then i pointed the licensing back at the VC server.

Now for the rest of the ESX hosts.

Fair enough then good luck with the rest of your patching then!

Leonard...

----

Don't forget if the answers help, award points

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

lholling — Wed, 13 Aug 2008 07:33:37 GMT

rollin71 wrote:Everything has gone ok but my VI server is also my licensing server which can't power on after or before the update.

I am working with vmware to generate a host based license file because the server can't be put back into evaluation mode.

I am a little confused if by VI server you mean VC all you need do in this situation is point your VI Client directly at the ESX server and use your root userid and password, suspend all of the VMs that are running on it and apply the patch. Once the patch has applied you will be able to power on your VC and then start the process for your remaining servers.

Leonard...

----

Don't forget if the answers help, award points

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

rollin71 — Wed, 13 Aug 2008 07:39:49 GMT

Sorry i meant VC but i tried to put the ESX server in evaluation mode so i could power on the VC server. Also the VC server has been off since the problem started late last night. I applied the update to the ESX server that had the VC server vm on it and afterwords i wasn't able to power anything on due to licensing. I then had to generate a single server license just to get the vm's to be able to power on and then i pointed the licensing back at the VC server.

Now for the rest of the ESX hosts.

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

lholling — Wed, 13 Aug 2008 07:38:26 GMT

MaartenK wrote:I receive an error when importing the update into Update Manager

Because I`ve got an isolated network without internet connectivity I downloaded the update manually and then imported it into my VC server using a CD.

It says it misses a signature file?

Could someone help pls?

The bottom line is don't use Update Manager copy the file to the server and then apply it using the esxupdate program from the command line.

To copy it to the server use a program like WINSCP or equivalent and then go onto the console of the server and apply the update by going into the directory of the patch and typing "esxupdate update"

Leonard...

----

Don't forget if the answers help, award points

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

MaartenK — Wed, 13 Aug 2008 07:26:26 GMT

I receive an error when importing the update into Update Manager

Because I`ve got an isolated network without internet connectivity I downloaded the update manually and then imported it into my VC server using a CD.

I run the following command but it does work

vmware-updateDownloadCli.exe --update-path Y:\update --config-import esx --vc-user administrator

Output commandline:

C:\Program Files\VMware\Infrastructure\Update Manager>vmware-updateDownloadCli.e
xe --update-path Y:\update --config-import esx --vc-user administrator
Please type in password for user <administrator>: <password>
Connecting to VMware Update Manager Service to import updates
INFO - vcIntegrityProxy, 288 Successfully connected to Integrity.VcIntegrity
INFO - vcIntegrityProxy, 1385 Get configure file information successfully
INFO - vcIntegrityProxy, 1486 Set configure file information successfully
INFO - vcTaskWrapper, 147 Waiting for a task to complete...
INFO - vcTaskWrapper, 208 Update state: running
INFO - vcTaskWrapper, 203 Update Progress: 50%
INFO - vcTaskWrapper, 208 Update state: running
INFO - vcTaskWrapper, 203 Update Progress: 100%
INFO - vcTaskWrapper, 208 Update state: running
INFO - vcTaskWrapper, 208 Update state: running
INFO - vcTaskWrapper, 208 Update state: running
INFO - vcTaskWrapper, 208 Update state: running
INFO - vcTaskWrapper, 208 Update state: error
INFO - vcIntegrityProxy, 1123 Download/Import task failed.
For detailed information, please refer to Virtual Infrastruture Client.
For detailed information, please refer to Virtual Infrastruture Client.

Event in VC:
Failed to import the update signatures and update packages from folder X:foldername. Downloadinh host update metadata failed after trying 1 times

Failed to download host update packages
Failed to download host update metadata
Task: Update Signature

It says it misses a signature file?

Could someone help pls?

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

rollin71 — Wed, 13 Aug 2008 07:19:14 GMT

Everything has gone ok but my VI server is also my licensing server which can't power on after or before the update.

I am working with vmware to generate a host based license file because the server can't be put back into evaluation mode.

Fun

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

mjlin — Wed, 13 Aug 2008 06:34:05 GMT

Dear VMware Customers,

We have released the express patches for the product expiration issue. Please go to http://www.vmware.com/go/esxexpresspatches for download and KB articles.

Since our last update we have completed our verification tests that the express patches we've released are fully compatible with the VMware Update Manager. Please see the KB articles for deployment information regarding Update Manager.

The KB articles are kept up-to-date. Please refer to the KB articles for information and updates.

Thank you,

The VMware ESX Product Team

Re: BIG bug in ESX 3.5 Update 2 - If you're using 3.5u2 read this now! - A general system error occurred: Internal Error

tom howarth — Wed, 13 Aug 2008 06:31:10 GMT

That is Excellent news, I am glad your issues are solved. It makes a change to see the full circle. thank you keeping us informed

Tom Howarth
VMware User Communities Moderator