Build number problem with the new 8/6 patch

[jasonboche]

Has anyone installed the new 8/6 patch on their ESX4.0 hosts yet?

I installed the new 8/6 patches on my ESX and ESXi hosts. Both were installed and rebooted without incident. The problem is the resulting build numbers:

ESX4 now shows a build number of 175625

ESXi4 now shows a build number of 181792

Because of this descrepancy, FT is now incompatible between the two hosts whereas it was working fine before when the build numbers were in alignment prior to the 8/6 patch.

# esxupdate query

-

---

Bulletin ID---- -

---

Installed--- -

---

Summary--

---

ESX400-200906412-BG 2009-07-10T11:07:28 Updates esxupdate

ESX400-200906401-BG 2009-07-10T11:07:28 Updates VMX

ESX400-200906404-BG 2009-07-10T11:07:28 Updates CIM

ESX400-200906411-SG 2009-07-10T11:07:28 Updates udev

ESX400-200906405-SG 2009-07-10T11:07:28 Updates krb5 and pam_krb5

ESX400-200906406-SG 2009-07-10T11:07:28 Updates sudo

ESX400-200906402-BG 2009-07-10T11:07:28 Updates ESX Scripts

ESX400-200906407-SG 2009-07-10T11:07:28 Updates curl

ESX400-200906403-BG 2009-07-10T11:07:28 Updates VMware Tools

ESX400-200906408-BG 2009-07-10T11:07:28 Updates SCSI Driver for QLogic FC

ESX400-200906409-BG 2009-07-10T11:07:28 Updates LSI storelib Library

ESX400-200906413-BG 2009-07-10T11:07:28 Updates vmkernel iSCSI Driver

ESX400-200907401-BG 2009-08-07T16:34:29 Updates vmklinux and vmkernel64

#

rpm -qa at the very end shows:

vmware-hostd-esx-4.0.0-0.5.175625

pam_krb5-2.2.14-10

vmware-esx-lsi-4.0.0-0.5.175625

vmware-esx-tools-4.0.0-0.5.175625

vmware-esx-iscsi-4.0.0-0.5.175625

vmware-esx-esxupdate-4.0.0-0.5.175625

vmware-esx-drivers-vmklinux-vmklinux-4.0.0-0.6.181792

[i]Jason Boche, vExpert[/i]

[boche.net - VMware Virtualization Evangelist|http://www.boche.net/blog/][/i]

[VMware Communities User Moderator|http://www.vmware.com/communities/content/community_terms/][/i]

[Minneapolis Area VMware User Group Leader|http://communities.vmware.com/community/vmug/us-central/minneapolis][/i]

[kschurig]

Same problem in my test environment. After appying the latest patches FT can not be used because of the different build numbers of the 2 hosts (one ESX 4 and one ESX 4i).

[jasonboche]

I sent an email to some VMware contacts asking engineering to look into this ASAP. This isn't good.

Thanks,

Jas

[i]Jason Boche, vExpert[/i]

[boche.net - VMware Virtualization Evangelist|http://www.boche.net/blog/][/i]

[VMware Communities User Moderator|http://www.vmware.com/communities/content/community_terms/][/i]

[Minneapolis Area VMware User Group Leader|http://communities.vmware.com/community/vmug/us-central/minneapolis][/i]

[depping]

I will contact engineering but please submit a SR asap.

Duncan

VMware Communities User Moderator | VCP | VCDX

-

---

Blogging:

Twitter: (*NEW*)

If you find this information useful, please award points for "correct" or "helpful".

[vmroyale]

Hello.

I haven't applied these specific patches in 4.0, but this isn't the first time this has happened. This post back in April saw the same thing happening in 3.5.

Of course, breaking FT changes things quite a bit with vSphere.

[jasonboche]

Thanks depping. I've opened an SR. 1436450911

Jas

[i]Jason Boche, vExpert[/i]

[boche.net - VMware Virtualization Evangelist|http://www.boche.net/blog/][/i]

[VMware Communities User Moderator|http://www.vmware.com/communities/content/community_terms/][/i]

[Minneapolis Area VMware User Group Leader|http://communities.vmware.com/community/vmug/us-central/minneapolis][/i]

[kooltechies]

Just updating the thread to get updates.

Thanks,

Samir

[kaos]

Tracking the thread. Pinged some internal folks.

[RParker]

isn't the first time this has happened. This post back in April.

That post refers to the build being WRONG. I don't think the build number is wrong, what's happening now is that FT needs to sync between ESX of the same build.. which leads me to wonder, what difference does it make the build? That's odd, it would be like ESX looking at the BIOS version of 2 machines and saying they couldn't be in the same cluster until its updated.. but that makes NO difference. CAPABILITY is one thing VERSION is something else..

[RParker]

Has anyone installed the new 8/6 patch on their ESX4.0 hosts yet?

So things brings up new questions.

a) why are you running ESX & ESXi? Why not ALL the same version of ESX?

b) Why would FT care what version of ESX is running? Certainly there should be other methods to view the machines and see that they are identical, because to me you can simply hack the version (a la change config file where the build number is, and FT would be fine).

I understand different build numbers, that's plausible, but why totally different products? That is perplexing..

[jasonboche]

This is in a lab, not DEV/QA/PROD

Build numbers were synchronized and FT between ESX/ESXi worked before patching.

Build numbers are no longer synchronized after patching and FT is broken.

VMware followed up with me this morning confirming the patches were installed correctly and they're escalating this up to engineering.

http://www.ntpro.nl/blog/archives/1090-Fault-Tolerance-Checklist.html :

"Required  ESX/ESXi: Ensure that the primary and secondary ESX/ESXi hosts are running the same build of VMware ESX/ESXi."

Jas

[i]Jason Boche, vExpert[/i]

[boche.net - VMware Virtualization Evangelist|http://www.boche.net/blog/][/i]

[VMware Communities User Moderator|http://www.vmware.com/communities/content/community_terms/][/i]

[Minneapolis Area VMware User Group Leader|http://communities.vmware.com/community/vmug/us-central/minneapolis][/i]

[sloneycho]

Hi Jason -

The requirement from the vSphere Availability Guide is that "All hosts must have the same ESX/ESXi version and patch level." Because ESX and ESXi are patched using different bulletins, it is not possible for both host types to be at the same patch level once patching has begun.

But that's really just a technicality. Compatibility of the underlying software is what is important, especially for FT. VMware engineering is working on better ways for FT to validate compatibility between patched ESX and ESXi hosts.

[jasonboche]

Is it a coincidence that ESX and ESXi were at the same build number (or close enough) for FT to work with the 7/10 patches (3 ESXi and 13 ESX) installed? I hadn't checked the build numbers of ESX and ESXi before installing the 8/6 patches.

Is the resulting ESX build number correct after installing the 8/6 patch? It doesn't look right. http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=101302... says the Build is 181792. To me it looks like the ESXi build number incremented correctly but the ESX build number did not.

[i]Jason Boche, vExpert[/i]

[boche.net - VMware Virtualization Evangelist|http://www.boche.net/blog/][/i]

[VMware Communities User Moderator|http://www.vmware.com/communities/content/community_terms/][/i]

[Minneapolis Area VMware User Group Leader|http://communities.vmware.com/community/vmug/us-central/minneapolis][/i]

[sloneycho]

The build number displayed by vClient for ESX hosts can be a bit misleading. It's origins are from the pre-ESX 3.0 days, when all ESX components were delivered with every patch, and a single build number could represent the software state of the host. These days, there is a build number per ESX/ESXi component, and each component has the potential to be patched independently.

Patching on ESXi replaces the entire firmware image, including the component that reports the build number to vClient.

Patching on ESX replaces only those components affected by the fix. When the component that reports the build number to vClient isn't part of the patch, the reported build number doesn't change.

[kooltechies]

Hi Jason,

There is a detailed KB article describing how to determine the build number on vSphere.

http://kb.vmware.com/kb/1012514

Thanks,

Samir

P.S : If you think that the answer is helpful please consider rewarding points.

[sloneycho]

A couple minor corrections to that KB (these have been submitted to the maintainers):

"vmware -l" reports the update level of the host. It wil report "GA" for all current 4.0 hosts. When update 1 is released and applied to a host in totality, it will report "Update 1".

"vmware -v" reports the build number of the last update applied to the host in totality. Today it reports either "Build-164009" (original release) or "Build-171294" (evaluation release). When update 1 is applied, it will report the update 1 build number.

A note about updates:

You will still be able to apply individual bulletins out of the update release, but the update level of the host is not considered to have advanced until all bulletins in the update release have been applied.

[jasonboche]

VMware has conceded via my SR that the 8/6 patches break FT compatibility between ESX4 and ESXi4. I also got clarification that ESX and ESXi are supported in the same cluster so the ESX vs. ESXi debate is moot.

Jas

[i]Jason Boche, vExpert[/i]

[boche.net - VMware Virtualization Evangelist|http://www.boche.net/blog/][/i]

[VMware Communities User Moderator|http://www.vmware.com/communities/content/community_terms/][/i]

[Minneapolis Area VMware User Group Leader|http://communities.vmware.com/community/vmug/us-central/minneapolis][/i]

[vCalendar|http://www.boche.net/blog/index.php/vcalendar/] Author[/i]

[jasonboche]

VMware KB article 1013637 Do not use both ESX and ESXi hosts in clusters with fault tolerant virtual machines has been released addressing the FT incompatibility issue between ESX and ESXi hosts having the 8/6 patches.

http://kb.vmware.com/kb/1013637

[i]Jason Boche, vExpert[/i]

[boche.net - VMware Virtualization Evangelist|http://www.boche.net/blog/][/i]

[VMware Communities User Moderator|http://www.vmware.com/communities/content/community_terms/][/i]

[Minneapolis Area VMware User Group Leader|http://communities.vmware.com/community/vmug/us-central/minneapolis][/i]

[vCalendar|http://www.boche.net/blog/index.php/vcalendar/] Author[/i]