ESX UpDates in a special order - important?

[jmcclymont]

hello all,

sometimes we get a bulk of new updates and up to now i update the esx boxes in the same order like the patches shown on the patch-website, from the bottom up.

Is that necessary or could i update disorganized?

thanks and greeting from germany

jeff

[Paul_B1]

People have reported strange build #'s when applying patches out of order. I would recommend doing it in order, and using one of the various "auto patch scripts" to ensure consistency. I've used Dominics and it works GREAT!

[jmcclymont]

thanks paul,

where can i find infos about this program by dominic?

is that right, to work from bottom up or should i use the patch numbers for sorting?

is there a logic in the patch numbers?

why these numbers so different without any order?

[Paul_B1]

See this thread for Dominic's script info:

http://www.vmware.com/community/thread.jspa?threadID=77514&tstart=0

I would HIGHLY recommend going with some form of script. Even with the auto-script, it took almost an hour to patch a brand new server with everything. I can't imagine how long it would take if I was actually sitting watching and waiting for the patches to finish before moving to the next one!

If you do it manually start with the oldest patch by date and work towards the newest. They're is not a logic to the numbers that I can see, so you don't want to do it by numerical order. So far patching is a HUGE gripe in my opnion with VI3. Hopefully one of the step releases will address the issues people are having with it.

Message was edited by:

Paul.B

[fdouma]

I talked to my TAM about this in the last week.

Word I got was:

\- There is no best practices published at this time

\- Do them in numeric order

My own experience is:

I do them in brute force numeric order, have been since my initial 3.0.1 install. I use no reboot option, and reboot at the end. I've never had a problem, and my version numbers match.

My opinion:

People are working WAY TOO HARD on this issue. Excepting the actual downloading from VMware, it is trivial to patch an ESX server given that you can (and are supposed to) do it in numerical order.

Frank D.

[dominic7]

I talked to my TAM about this in the last week.

Word I got was:

- There is no best practices published at this time

- Do them in numeric order

My own experience is:

I do them in brute force numeric order, have been

since my initial 3.0.1 install. I use no reboot

option, and reboot at the end. I've never had a

problem, and my version numbers match.

My opinion:

People are working WAY TOO HARD on this issue.

Excepting the actual downloading from VMware, it is

trivial to patch an ESX server given that you can

(and are supposed to) do it in numerical order.

Frank D.

Patching by hand would still be horribly laborious. Installing 24 patches entirely scripted takes about an hour, without a script it would require you to be at the command prompt waiting for one patch to complete to start another one. Let's assume this only add 10% to the total patching time and you've got around 65 minutes where you have to be monitoring the system. Perhaps you've only got two or three hosts, but there are many people who literally have hundreds of ESX hosts that need to be patched and that can add up to a very significant amount of time.

If you look at the patches available

ESX-1541239 Patch | 03/29/07 | Critical Patch

ESX-2257739 Patch | 03/29/07 | Critical Patch

ESX-2559638 Patch | 03/29/07 | Security Patch

ESX-6431040 Patch | 03/29/07 | Security Patch

ESX-9916286 Patch | 03/29/07 | Security Patch

ESX-3199476 Patch | 03/05/07 | Critical Patch

ESX-5031800 Patch | 03/05/07 | Security Patch

ESX-5885387 Patch | 03/05/07 | Security Patch

ESX-6050503 Patch | 03/05/07 | General Patch

ESX-6856573 Patch | 03/05/07 | Security Patch

ESX-9865995 Patch | 03/05/07 | General Patch

ESX-1271657 Patch | 01/31/07 | General Patch

ESX-1917602 Patch | 01/31/07 | General Patch

ESX-2031037 Patch | 01/31/07 | General Patch

ESX-2092658 Patch | 01/31/07 | General Patch

ESX-3996003 Patch | 01/31/07 | General Patch

ESX-5497987 Patch | 01/31/07 | General Patch

ESX-6075798 Patch | 01/31/07 | General Patch

ESX-2066306 Patch | 12/28/06 | Critical Patch

ESX-6921838 Patch | 12/28/06 | General Patch

ESX-8173580 Patch | 12/28/06 | General Patch

ESX-9986131 Patch | 12/28/06 | Security Patch

ESX-1006511 Patch | 11/30/06 | Critical Patch

ESX-1410076 Patch | 11/30/06 | Critical Patch

ESX-2158032 Patch | 11/30/06 | Critical Patch

You will see that if you maintain your host on a regular basis then you wouldn't be able to install patches on a strictly numerical basis. Patch 2158032 that was released on 11/30/2006 is numerically greater than 1541239 which was released on 3/29/2007.

Your version numbers may match because you installed the same patches in the same order, but they may not be right and you may not have the fixes installed that you suspect to be there. Not having a problem does not necessarily mean that the problem doesn't exist, it just means that you haven't run in to it (yet). Potentially you could not patch the host at all ever and not have a problem.

[fdouma]

Oh my.

I don't really wait for anything.......

Assume:

I have a directory that contains all my patches.

I have Vmotioned everything off the box.

Here's my method:

cd to patch directory

for i in ESX-???????;do pushd $i;esxupdate --noreboot update;popd;done

When I get the prompt back, I type "init 6".

I'd be willing to script this, all three lines of it, and run it that way, if you'd like.

Not exactly rocket science, and I think I could do it on 1000's of servers.

Telling me how I may this and might that is meaningless.

I'm rather suprised that my \_lack_ of the mismatch problem others have reported is taken as not a sign of success, but just a lack of failure to this point in time. Eeek. If apparent success isn't to be interpreted as success, uh, just what are we trying to achieve here?

Given I've been dilgent about reading a great deal, if not all, of VMTN's discussions about this specific topic, I'll go with my own experience, backed by my TAM's word.

BTW: I highly suspect the patch ID's are encoded for a particular target area, allowing for numerical order to work. A wag on my part.

[skearney]

I think you are asking for trouble patching in numerical order.

Take a look at ESX-1161870. This provides:

VMware-esx-vmx-3.0.1-42368.

Now look at ESX-1410076. This provides:

VMware-esx-apps-3.0.1-34176

VMware-esx-backuptools-3.0.1-34176

VMware-esx-debug-tools-3.0.1-34176

VMware-esx-srvrmgmt-3.0.1-34176

VMware-esx-vmx-3.0.1-34176

VMware-hostd-esx-3.0.1-34176

Notice that 1410076 provides an earlier vmx package. If you apply these in numerical order, 1410076 will fail because it cannot downgrade a package. It will not simply skip the vmx package. None of the other updated packages in this patch will be applied.

Using the --force option also does not help in this circumstance. The only way I know to apply these patches out of order is to use the '-x' option to exclude the vmx package on 1410076.

[fdouma]

I thought you had an argument winner there, skearney!

(And I'm just as happy to be proved wrong as right, 'cause either way I end up with the right information)

But....

I was all over the rpms and xml's within the various patches yesterday, so we are on the same page. However, I tried this on my test box just now, using the latest patches that I haven't applied yet:

Specifically:

ESX-1541239

Gives you drivers and scripts

ESX-2257739

Gives you scripts and kernel

ESX-2559638

Gives you scripts and info

I had scripts at 32039 when I started.

By going numerically through the above, I got scripts, kernel, and info, with scripts skipped as up to date for the 2nd and 3rd run of esxupdate.

So, it looks like esxupdate/rpm/yum, handles imbedded versioning in the RPM's as it should.

I'm getting kind of torqued at VMware for not jumping on this thread and slapping or validating me on this one. There has been some good work out there to handle in arbitrary/date order, but I still maintain it SHOULD be unecessary if VMware is/would handle patching more gracefully. Only they know for sure, and they ain't telling definitively as far as I can see.

Lastly, a consequence of numerically also appears to be that you will occasionally eliminate the installation of packages you are just going to replace on the next patch. This will shorted your patching cycle.

(It appears that VMware is using the RedHat standard for non-sparse rpm's, so you get a complete package at any version of the rpm)

(eg. At this point the esx-vmkernel has been released at 34176, 37464, 35804, 40087, 42368. If I just install 42368 first, then I'll not waste time doing the other 4 rpms. Since it isn't sparse, there is NO reason to do the others.)

If I'm a betting man, I say VMware's internal change control uses patch ID God knows how, and VMware is using standard rpm and yum techniques to keep us all whole, regardless of order.

[skearney]

That worked because the scripts package is the same version in those packages (I believe, anyway. I'm too lazy to go look right now). If the scripts package in the 2nd or 3rd update had been a lower version, the patch would have failed.

I suspect you're right about being able to install patches that contain just the latest version of the RPMs. But without a statement from VMware, what appears true today is not guaranteed to be true tomorrow.

Your method works because it happens that the higher numbered updates include the most up to date RPMs. But if the two updates I mentioned above were the only ones available, or simply were the only two you decided applied to you, your system would not be up to date by patching chronologically.

[fdouma]

OK, starting to feel like a religous issue, so probably my last post:

0) VMware really needs to settle the discussion

1) esxupdate opens the patch directory, looks at the rpms within it, and operates on each rpm individually. Look at /var/log/vmware/esxupdate.log and you can see it do it.

2) Therefore, I believe you are wrong about generating a failure for a single rpm causing all other rpms to be skipped. I've given a specific real world example where that doesn't happen. Why you think an "already installed" would be handled differently from a "would have to down rev" is speculation at best.

3) If folks wanna install 5 kernel rpms at ever increasing rev levels just to get to the highest rev level, when only the latest need be done - go for it. It's their time.

4) I just installed all patches, numerically, that have ever been issued for 3.0.1, whether they had been applied or not, on my test systems. The command line and esxupdate.log shows it worked just as it should, skipping, installing, or upgrading the specific rpms within those patches as necessary.

That's what I see with my own eyes, on my own systems.

Lastly, check out the install_patches script in ESX-6431040. It builds its patch array with a simple ESX-*, giving it numerical order. Not exactly definitive, but interesting....

Frank D.

[skearney]

Not a religious war and not about being right or wrong. Given the lack of official direction, I think it's good for the community to 'talk it out'.

To point #2, it looks like we may both be right. Using my example with the 2 patches and updating using an HTTP repository location, if I try to deploy 1410076 after 1161870 it will fail. It will also fail using the --force option.

For grins, I created a local repository for update 1410076. As expected, it failed using the normal update command. However, using the --force option in combination with a local repository worked and all the RPMs were installed.

So there appear to different behaviors for different repository locations.

Unfortunately, this also meant it downgraded the VMware-esx-vmx package that was provided with 1161870. So now I have a system that tells me I have applied both 1410076 and 1161870 but the actual RPMs on my system differ from another system that had these updates applied in reverse order.

Given this, and the fact that I had to use the --force option, it seems to me that 'chronologically' is not an appropriate way to apply updates.

I think the consensus here is that chronological order can be used as a "tie-breaker" for updates that have the same release date.

[fdouma]

Hmmm. Interresting on the repository issue.

I'm using a single NFS datastore available to all my ESX servers. I just cd into /vmfs/volumes/blah blah blah blah and esxupdate from there.

I NEVER force anything. My experience has been that the "error" is just as I want it, specifically esxupdate is telling me that I'm either already up to date, or I'm effectively trying to down rev a package. This is just as I expect, so i just ignore it. I'm never, ever, going to force a down rev as part of this whole topic. Both command line logging and esxupdate.log appear to back my play on this.

I'll put my position a different way:

I've become convinced that these failures, for whatever reason, during a run of esxupdate are at the individual package/rpm layer. Not at the Patch ID layer. Thus, esxupdate/yum/rpm will happily upgrade only specific packages from a patch when others have "failed" due to either their being current or at a later rev (causing the down rev error). All within the same patch ID.

Also, I maintain that the rpm's within the Patch ID's are not sparse. Therefore, applying a later versioned rpm of any package gets you completely up to date. (This would imply a REVERSE chronological order if you really wanted to save time patching. Eeek!)

The sum of the two ideas above is that I believe I can blindly esxupdate, in Patch ID numerical order, every patch I've (ever) got, over and over and over as I download new patches into my common directory. The end result will always be the latest rev of all the rpms, and a sane, consistent system. (only caveat is to read the doc with the patch, in case VMware puts some sort of special instruction in there) Remever, I NEVER force, so I never go backwards in rev.

This blind, yet consistent, method I have used on my own test boxes. They appear by all measures to be sane, up-to-date, and tidy.

(Gee whiz I wish VMware would jump in here......)

Frank D.

PS: so much for "last post"......

[Cova]

Ok - I just got off the phone with VMWare support about this very topic.

I've written my own VERY simple script for applying the patches from a web repository. It has no error checking at all, and applies everything in alphabetical order. As I download patches from VMWare and add them to my repository, I rename them by pre-pending a number that increases for every patch I do - this makes sure that alphabetical order = release date order.

But last night while applying about 30 fixes in one big batch, I had 2 patches fail. Both of those fixes contained only a single RPM, and in both cases that same RPM was upgraded to a newer version by a later patch. Now trying to apply either of those patches through esxupdate tells me I need to downgrade.

So I called up VMWare to find out if I'm good where I'm at with the latest RPM releases all installed, but 2 patches missing from the 'esxupdate query' output. They're response was that to be supported they look at the 'esxupdate query' output, not at the RPM release numbers. And so my options to get this server back to a supportable state would be to either -force the missing patches (and then re-apply everything afterwards), or to uninstall patches 1 at a time back to the first failed one, and then re-apply starting from there. IMHO both of those procedures are more likely to break the host than leaving it as-is - so I'm leaving it as-is. My own comparisons to other hosts that have had the patches all applied without errors show me that my host is at the correct level for everything, so I'm going to leave it that way. In a few months when my new storage controller gets here I'm rebuilding that host anyways as all it's internal disks are moving to a SAN shelf.

[fdouma]

I see something similar on my boxes. Specifically:

6075798 contains VMware-esx-tools at rev. 38803

3199476 contains VMware-esx-tools at rev. 41412

Working numerically, as always.....

When I installed 3199476, I got the newer rpm, which makes me happy.

Subsequent attempts to install 6075798 say I have to downrev, which also makes me happy. I of course will not be downreving, as that is now nonsense.

esxupdate query doesn't even list 6075798 which is also fine with me, as I don't have 6075798 anymore, I have 3199476 query reports.

It all makes sense to me. Perhaps you have the same situation?

Frank D.

[Cova]

But the problem you may encounter fdouma is if you need to open a support request with VMWare, and they think the fix 6075798 will solve your problem. If you had applied them in order, both would be listed by query, though obviously only the newer RPM would be installed. At that point, they may require you to force install 6075798, which would then require you to re-install 3199476 again afterwards.

Or what about when the earlier ESX update contains multiple RPM's. Say we were looking at VMware-hostd-esx, which has had 4 revisions across various different releases, sometimes bundled with other RPMs, sometimes not. If you missed 1410076 (which also includes VMware-esx-apps, VMware-esx-backuptools, VMware-esx-srvrmgmt and VMware-esx-vmx) but applied 6921838 (which has only VMware-hostd-esx, a newer revision than 1410076) - when you go to apply 1410076 would it not fail, and then you're missing out on updates for 4 RPMs.

Anyways - whether skipping things and just going by RPM release numbers works or not (it does seem to work) - I would never recommend anyone purposely do something that would cause support problems. VMWare's official response to my service-request about this was to apply all patches in release order. Spend the small amount of extra time applying the stuff in order now, and save headaches later if problems pop up.

[fdouma]

Wait a minute.

I HAD 6075798 applied and in.

I didn't do anything except install the earlier numbered patch afterwards.

esxupdate did its thing, unmolested by me, and what I got is what I got.

Not my fault 6075798 decided to stop showing up. But it makes sense to me. In fact, given that I do not believe the enclosed rpm's are sparse, there is no such thing as having "both" applied. That is the RedHat standard, and I've seen nothing to indicate VMware isn't following it thus far. I now have the latest version of tools, a complete rpm at rev. 41412

I've done the multi rpm thing. Everything works like it should, so far. I get upgrades and installs as I should. However, I've been shown a case where a need for downrev kills the whole patch. I've never experienced that, and I don't expect to. (I've scoured esxupdate.log to validate my experience).

(I patch in numerial order off numerical ESX-??????? and never downrev, regardless of the error message - my whole point)

Lastly, MY official answer from my TAM is to do them in ESX-??????? numerical order, regardless of issue date. Sooooooo,

I AM doing them in order, and I really like the results I've been getting -

they make total sense to me, and my systems show sane with vmware -v, esxupdate query, and via the VC.

Go figure!

[fdouma]

Ooops.

No I didn't.

Never had 6075798.

I'd already applied the newer rpm by the time I hit 6075798.

Sorry for saying that.

So, disregard first part.

Still standing by the rest.

BTW, the reason I'm ALL OVER this topic, on several threads, (and getting the reputation as somewhat of a crank) is I am hoping to keep stirring the pot so VMware finally issues something definitive. I've also been dumbstruck on occasion by the advice that has been handed out to go ahead and force a downrev just 'cause esxupdate says so.

I'm conviced that that is virtually never the right thing to do, given everything I've done on my systems AND given everything I've read on VMTN about patching 3.0.1.

Frank D.

[acr]

It would be nice to get a more structured patch methodology..

The force option isn't recommended as it omits the -q which would check for dependencies.

[Cova]

Ya - I'd like to see something in writing from VMWare as well - especially if it supports the theory that as long as you have the most recent RPM releases for everything that 'esxupdate query' doesn't matter.

If anyone would like to reference the service request I had about it, it's #187040531