ESX 3.5U2 Patch experiences?

[admin]

Just posted:

• KB with a description of what happened: http://kb.vmware.com/kb/1006716

• ESX Express Patches: http://www.vmware.com/go/esxexpresspatches

Let's close down the other thread, as it's getting really long and hard to navigate.

• Letter from Paul Maritz: http://communities.vmware.com/thread/162713. Please post non-technical commentary here.

• Please post further technical commentary about the patches in this thread.

Thanks. Rest assured that many people inside VMware are reading your feedback and experiences here.

[wila]

Deployed patch using

esxupdate update

and everything is running fine. NO host reboot required as advertised.

thanks

Wil

[KlinikenLB]

I would really like to do

but VMtion works only to unaffected hosts

[AndyMcM]

You will be able to VMotion onto Pre Update 2 machine no problem, at the end of the day this bug is just down to licensing and what you can and can't do on an unlicensed version of ESX.

In fact I have read but not tried, if you change the date to pre 12th August on an Update 2 Machine you should be able to VMotion from/too it, useful if all your hosts are already Update 2 and need to put one into maintenance mode.

[KlinikenLB]

Why does everybody talk about the host reboot? Is there any advantage that the host doesn't need to boot?

I'm more interested in no VM reboot ...

[Bastian_Haas]

I would really like to do

but VMtion works only to unaffected hosts

You could set the destination host system date before 2008/08/12, then it will work

[KlinikenLB]

Yes, it works for the host. But not for my my VMs

Otherwise I could have done this solution yesterday ...

-

---

Has anybody an idea for my CPU masks?

[AndyMcM]

I am in middle of updating our 11 ESX, 8 done so far and not done a single reboot of the host, am like you more concerned about minimizing shutting down VM's!!!!

Only thought I guess, its good to reboot an OS now and then to flush the pipes.

[AndyMcM]

Just wanted to put in my 2cents worth and update my experiences, think they might be useful to some people.

We have 4 ESX hosts that have around 60 VM's on them a lot of them are production so no chance on getting them shutdown and they are evenly spread over the 4 hosts.

Sooo, what I did was shut down 20 odd test/development ones. leaving the prod ones there.

Next I stopped ntpd on each of the boxes from service console (service ntpd stop) and then changed date using (date -s 080808) , I was then able to VMotion off all the VM's off one of the hosts and put it into maintinance mode. After which I was able to VMotion from the three hosts one at a time and update all the hosts, without any VM reboots!!!

One caviat, make sure that VMware Tools do not sync time with the host, we are fortunate enought that we do not do that and use our AD Server for time sync.

Hope this helps someone!!!

A.

[RickPollock]

Installed the patches via esxudpate update and everything is now back to normal.

[George_B]

Why are people setting their clocks back to do the upgrade?

I have a simple two machine cluster which runs about 90 development VMs and the upgrade has been relatively easy. Fortunately I had all the VMs bar one on one host as I was testing iSCSI jumbo frame performance on the other host. It was just a matter of using esxupdate to install the patch and then slowly VMotion VMs to it so I could then upgrade the other box. This worked even though the host with the VMs was running the faulty version.

As long as you can have at least one host in your cluster which is not running VMs then you should be able to do the upgrade with no VMs being turned off or clock roll back. Alternatively just find a spare server, install the latest ESX version and run it in 60 day trial mode, then migrate VMs to it off one host, upgrade, move back, rinse and repeat.

[sjourney]

UM downloaded patches last night, did scan on hosts this AM, UM still says they are Compliant?? What gives? I know I'm effected. Have not had any problems with UM before...

[supporti]

Hello,

thanks to all for helpful advices. After getting the failed update process killed (stopping update process, shutting down all guests , reboot host) updating worked fine. Now the update process is fixing the problem on the last of 10 hosts. O.K. I had to shut down to live hosts, but they were "only internal".

regards

Thomas

[carpenike]

Confirmed on changing the date on destination host. Disable NTP and change date, VMotion to that host from the other hosts and than enter maintenance mode. Just worked for me.

For some reason I got this file ESXe350-200807812-O-BG.zip when I used the download link. It didn't have the contents.xml.sig file inside of it so esxupdate gave me an error. Just went to the page and downloaded it again and got a different file that worked fine. Thanks VMWare.

And in answer to the dude further back who threw a fit... Get a grip man. It's a bug, but it makes sense that they'd put something like that in their development code to ensure alpha/beta users upgraded at release date... Granted I think it's a bit severe and should have given a week or two of warnings by default... Oh well.

Thanks for the quick resolution!

[Bastian_Haas]

UM downloaded patches last night, did scan on hosts this AM, UM still says they are Compliant?? What gives? I know I'm effected. Have not had any problems with UM before...

Is your baseline dynamic? If not you'll have to change the baseline to include the patch.

[Go_Bucky]

First, let me say I appreciate everyone who has posted their experiences with this patch. Reading through what you've all done gave me a little more confidence.

For the most part, I did the same process as AndyMcM, except I used the VI Client rather than the CLI, and I used Update Manager. So far, no major issues.

A couple of things did come up for me, that I thought others might see:

1. My Update Manager did not have the new patch because I had it set to download only once per week. As mentioned by others, you can force Update Manager to download the new patches by going to Scheduled Tasks, right-click on your VMWare Update Manager Update Download task, and click run.

2. I had production VM's running on all my hosts, so I had to use the time trick send one host back to Tuesday. No problems there...I did this right from the VI Client. But, after doing this, I started seeing an HA error being reported on my cluster and on all hosts. I figured this was due either to the license problem or the time change, and ignored it. After VMotion'ing all VM's to other hosts, I started the remediation task. This took about 45 minutes.

When it was done, I still had the HA errors. I then enabled NTP on the host where it was disabled. That seemed to clear up the error on the Cluster and unpatched hosts, but not the newly patched host. This error also prevented me from VMotion'ing VM's to the newly patched host, which was the plan. After scratching my head for a while, I right-clicked the host, and clicked Reconfigure for VMware HA. This resolved the error. I was then able to VMotion VM's to this host, and I was on my way to patching our additional hosts.

Anyway, I don't know if anyone else will run into the HA error, but if you do, it's easy to resolve. Thanks again to all who are posting their experience...

[GregMon]

Does this patch require u2, or can it be applied from a base 3.5?

Also can it be applied to go from 3.01 to 3.5?

Thanks,

Greg

[Dollar]

We are in the process of patching about 100 hosts using Update Manager, with about 50 completed. Of the 50 hosts, I've only experienced problems with one (VMKernel unloaded and did not come back) requiring a reboot of the host.

We are using a "Swing Host" in each Cluster to start the process.

No other problems noticed with the patch. However, I notice that Update Manager can no longer be used to update any Hosts prior to U2. If I attempt to update a U1 (or earlier) device using Update Manager, the VMKernel unloads and does not load back, which prevents the Host from being rebooted or Update Manager from completing. The host server remains in a "not responding" state until it is manually rebooted, at which point comes back up and is accessible and functional. The version number of the server will be 3.5.0.110181 but if you check Update Manager it still shows "Update 2" as not being installed.

I hope this is clear enough..... it's been a long night.

[KlinikenLB]

My environment:

4 x HP DL585G2, each with 4 x AMD8220 and 64GB RAM, all affected by the timebomb.

My solution steps (all managed with VC, without console):

1.) Actualize my UM baseline with the timebomb patch.

2.) Find two hosts without timecritical VMs / without VMs synching time to host.

3.) Disable NTP on one of this two and set date to last week.

4.) Migrate all VMs to this host so that the other host ist empty.

5.) Enable NTP on this host again.

6.) Scan and remediate the empty host. After this I had build 110181 on this host.

7.) Not neccessary, but I booted this host manually.

8.) Migrate all VMs from the "full" host to the updated host.

9.) Scan and remediate the now empty host ...

... and so on for the rest of my hosts.

Everything seems to work fine now.

No question, it was a big bug. But thanks to VMware: they solved it very well.

[EvareAdmin]

Updated all of our servers using 3.5.0u2 last night about 20 minutes after the patch downloaded to VMware Update Server. We weren't able to use a swing machine so we had to pick a handful of machines that we could do without for ~5 minutes. After the first machine was patched we did rolling upgrades and everything has been working smoothly since then. We're using EQL iSCSI storage and HP DL360G5p for hosts.

Eric

[mdonovan]

My results: 10 hosts, used Update Manager to do the update, VMotioned all vm's off host, applied patch, set date/time to current day, no problems I see.