esx automatic patching

[dominic7]

I've been out of touch with the ESX community for a while as I worked on a 6 month project, but I've freed up some time and can work on more fun stuff again ( yay ).

Anyway, I've written a script to take care of one of the more tedious parts of ESX v3, patching. I was inspired by this gentleman (http://virtrix.blogspot.com/2007/03/vmware-autopatching-your-esx-host.html), but his process didn't quite fit all my needs. I wanted something that would work via ftp, and all check the integrity of the patches before they were installed. The result is this:

Script:

http://www.vmprofessional.com/material/esx-autopatch.pl

Pre-compiled patchlist.txt

http://www.vmprofessional.com/material/3.0.1/patchlist.txt

http://www.vmprofessional.com/material/3.0.0/patchlist.txt

If you like it, please drop me a line at dominic@vmprofessional.com, if you don't like it email me and let me know \*why* you don't like it.

Cheers,

Dominic

[ANSA]

I do exactyly as you stated. I only install the General patches if they apply to my infrastructure. I also install my patches in the order of release date.

[bigusdadius]

It's my concern that I may download and install a patch that does not apply to my environment and that it may cause an issue with the server. Not that I've seen this behavior with ESX in the past, but I have with other operating systems.

[dominic7]

I just downloaded the script and I'm excited about

using it, but I have few questions first. Do you

install all General patches? Normally, I would only

install the Critical and Security patches, then apply

any General patches that may apply to my environment.

Have you given any thought into differentiating

between the types of patches?

I can add that feature, it shouldn't take long but will require adding a 4th field to the patchlist.txt. The other option is for you to create your own patchlist.txt by deleting anything that you don't want to install from the one I have available.

[bigusdadius]

I can add that feature, it shouldn't take long but

will require adding a 4th field to the patchlist.txt.

The other option is for you to create your own

patchlist.txt by deleting anything that you don't

want to install from the one I have available.

The addition of this feature would be great, but I also have no issue editing the patchlist text file either. Thanks for the quick reply!

[IMSM]

I have a site setup but when I run the script from the console of the VMware server (via PuTTY) I receive this error message:

Could not open file:

./patchlist.txt.

I have my patchlist.txt and patches in a site under esxupdates/3.0.1/ but it seems like the files aren't downloading.

[IMSM]

Ah, I figured out the problem.

uncommented the open-firewall variable in the script and it took off!

[Svante]

First of all, thanks for a nice script!

I have tried it (on 3.0.1) , and had varying success. I went ahead and ran it first on one of our 3 esx servers (more or less fresh installs). It went fine, and every patch was applied.

I then moved on to the two other (identical) servers, and ran into the following situation (from /var/log/esxupdate):

\[07Apr18-1357] DEBUG: system: | tar: Error exit delayed from previous errors

\[07Apr18-1357] ERROR: states: Error (123) executing [ls /boot/grub/grub.conf /boot/grub/device.map /boot/initrdvmnix.img | xargs t

ar -P -czf /var/spool/esxupdate/systemfilesbackup.tar.gz /etc/vmware/*]

tar: /etc/vmware/esx.conf.READLOCK: Cannot stat: No such file or directory

tar: Error exit delayed from previous errors

It seems what happened was that the script was running too fast, and started the next patch when lockfiles were still in place? Maybe a pause between patches would have helped? I any case, I think the script should abort by default when a patch fails, so one has a chance to intervene manually rather then continue to avoid a situation like this.

This happened with two patches on one server (ESX-8173580 & ESX-6856573) and one patch on the other (ESX-2031037). None of these seem to be very critical, but it feels less than good to have different patches applied to different servers (they are clustered)

A couple of questions:

Is it possible to rollback patches so I can get back to the point where the problems started and continue patching? Forcing patches seems like a bad idea since maybe critical components would get downgraded?

Would an upcoming 3.0.2 upgrade on all systems bring them back to "synced" patch level?

What would you guys do - reinstall of the ESX servers? I'm still not yet in production so this is an options I guess..

Thanks in advance,

Svante

[ANSA]

What would you guys do - reinstall of the ESX

servers? I'm still not yet in production so this is

an options I guess..

I would rebuild the server, in my opinion, in light of all the issues that are being discussed in this forum on patching.

[derekn]

I believe you can just patch from where you had the error. I dont think a reinstall is necessary of esx. So i fyou were applying 5 patches and it bombed at 3, you would have to apply 3, 4, and 5 again. Well thats my understanding from vmware support.

[Svante]

derekn,

You mean forcing the first bombed out patch and then forcing every patch after that? Or would the patches after 3 (in this example) get marked as "not installed" again and would install without forcing them? Unfortunately, I was applying all 25 patches at once, and in once case one really early patch failed...

Interestingly enough, I have the same build numbers (vmware -v) on all 3 servers, but "esxupdate query" reports different patches installed...

I'm also considering trying to preserve the config files (if going the reinstall route) to not have to do everything from scratch once I'm up and running. Then again, one never knows what the patch level on those are (if patches ever change them?)

Sorry for all the questions, I'm still a newbie when it comes to ESX.

edit: typo

[derekn]

To my experience 3 would come up as not installed. but 4 & 5 would be installed. I just usually install 3,4,5 over again just because you never know if the 4 & 5 had files that overwrite patch 3 files...

I was curious so i had to do soem testing...

The build number will change after all patches have been applied. I just tested this on my test server here. Applied 8 patches with the physical file from one missing so it would skip it. the build was 39823. after i applied the skipped patch the build was 42368. I then decided to reinstall my esx host server (thank goodness I have a raid config, pulled a spare before my experimentation) I then put a fresh esx and patched it with all patches. The build is 42368. So the build matches, the directory sturcture matches. So I feel it might be ok to just apply your patches that bombed and keep patching beyond that. Maybe I have an isolated scenario here, but its just my testimonial.

[Svante]

Ok, thanks for your help! I'm kind of considering doing a reinstall after all, might be good practise for me anyway, and I would sleep better knowing everything is perfect before actually going into production.

It's weird my build number are the same (42368, as yours) on all three servers even though they are all different patch-wise right now though..

[scinella]

Dominic,

where would i modify the script to instead of using ftp or http to use a vmfs lun that is shared between my servers?

[Svante]

As a followup on my previous posts:

I reinstalled one of the ESX servers. I then modified the autopatch script a bit so that I got to press enter just before end after each patch is applied. I then ran the update again, and noted something interesting: Under certain conditions, possibly when multi-patch patches are applied, control is returned to the perl script before the update is finished (several minutes in fact, in my case). This causes the script to try to apply the next patch while the previous one is still running! This is clearly why the script failed in my case, and caused the unfortunate situation.

I would advice against using the script in it's current form.

[skearney]

I have seen this type of thing with my own scripts. When the next update begins, sometimes there are still locks on files. I put in a 30s sleep between updates which seems to take care of the problem.

[dominic7]

I haven't seen this behavior in any of the installations that I have done, but regardless I'm working on a fix for it. I should have it finished sometime tomorrow.

[jayoungncsu]

Dominic, I can't tell how how fantastic a resource this is and how many kudos you deserve for sharing it. Thank you so much for this.

VMware staff reading this? I can't tell how utterly asinine your patch process is. esxupdate is nice - but it's all downhill - wAAAAYY downhill from there. Please fix your patching methods.

[smsf]

Can I use this script for updating from an NFS depot?

I think it could be done by not uncommenting FTP or HTTP fields but I am not sure.

[dominic7]

I haven't added support for local and nfs installs, but it's on my list of things to do. Maybe I'll get it all done on monday.

[smsf]

Nevermind. I went with the http update method. I now have to edit my update list to include just the critical and security updates.

I downloaded all the general updates too, just in case but might have to go through them at a later time to determine which will apply to me.

Are there any general updates I should be installing? I saw one for DST but it's for Australian.

Message was edited by:

smsf