Unable to upgrade embedded ESXi 3 to U2 on IBM BladeCenter HS21 XM

[SunF1re]

Hi all,

we just purchased an IBM BladeCenter and for a start loaded with 3x HS21 XM Blades with the embedded ESX3i. In general everything seems to work fine but we are unable to upgrade to the latest U2 patch. The Blades are shipped with embedded ESXi 3.5 build 71173.

When we run the VMware Infrastructure Update client does show the updates and when we choose to install the firmware update progresses to about 98% and then displays and error message "Failed to install Firmware update.: Unable to Install". I attached a sceenshot of what is showing in the ESX server event log after the failed upgrade. We also tried the vihostupdate from the remote CLI package with the very same result.

Did anybody run into similiar issues? This for all 3 Blades so I guess this may be a more general problem with the embedded ESX3i running off a USB Flash drive on the IBM's?

Any help is appreciated.

Thanks, Thorsten

[Sealordmwt]

The new recovery CD is the same build as the one the blades shipped with (71173)

[AdamSnow]

I am having the same issue with HP. The preinstalled version works great, but if I use the recovery CD to reinstall to a new USB drive, I am unable to run update manager aginst the ones that i built via the recovery CD. HP is sending me to vmware, and vice versa. We are considering dumping 3i altogether and going to regular esx 3.5

[benny_hauk]

All I can say as of now is that we are working on figuring this issue

and others like it out. Please stay tuned.

And that, dilpreet, is really all I needed to hear. Thanks for keeping us in the loop in as much as you can. Your comments here have been very much appreciated.

[shormell]

Glad to hear that VMWare is going to potentially step up to solve this issue. I'm starting to wonder why we would use ESXi at this point. There is a HD slot in the blade, I'm thinking of getting a drive and just going to traditional ESX. My biggest concern is even if (when?) we get this issue resolved, what is going to happen the next time there is an issue. I don't want to play this game in the future, having to flash the drive and resetup the blade everytime I need to solve a problem. I'm not on business day 8 since opening a ticket with IBM without a solid resolution. At this point, noone has even attempted to look at my logs or solve this issue from a technical standpoint, only attempting to ship me a new USB drive and (multiple) CDs (that for some strange reason is not available to download)

If I don't have an answer to this by Monday, I'm going to consider ESXi vaporware and move on and start getting full ESX installed so I can utilize my hardware purchase that for nearly 4 weeks now has not seen any VMs on it yet!

[benny_hauk]

Great observations, shormell. I'll add though that I don't think I'd feel any better with a single HDD drive (non-mirrored) than I would this USB solution. You might get it up and running faster but when the drive fails and production VMs are down... better to fix things correctly now, before they are in a working production environment instead of doing something hasty and shooting yourself in the foot down the road.

It definately feels like we are experiencing "early-adopter pains" but so far I think it's a good strategy and still glad we went with the embedded (I'm not on day 8 yet either though ).

That said, it ought tobe clarified that your problem is not with "ESXi" but rather with "ESXi, embedded". ESXi seems like the way VMware sees the future (a world without service consoles) and I am fine with riding VMware's wave of enthusiasm. I definately don't want to start down "the road less traveled". If it's the begining of the end for regular ESX then I want to stay on what they are pushing and more importantly supporting and also that which has the largest customer base (the largest voice). Basically I would stick with ESXi even if you use the installable version on an HDD until the manufacturors and VMware work out the kinks in the embedded version. The quicker you learn to live in a world without service consoles the better.

[shormell]

I see your point, but I'm still seeing the USB flash as a single point of failure in the system. I could boot from SAN for this as well, which would provide a bit more redundancy.

I would be down at this point had we gone live on these hosts as it is. Having to reflash to enable/troubleshoot patching is a bit scary for me. Having to wait DAYS to get an updated CD is just insane.

If indeed service console will be going away, then I guess you are right, better to learn it and embrace it now then later. At this point, I am still more confortable with service console, and have not quite gotten everything I use to work with the RCLI. I'm hoping to get into a session at VMWORLD to get some tips on using this.

[alex555550]

ESXI is great an when you use the unsuported way, by flashing manually, you will see it`s only an issue with the Recovery CD`s from the vendors. Seems like they didn´t test what they made.

[rburbridge]

What advantages does having the IBM CIM providers included in the embedded version really provide? I would assume hardware monitoring with Director, but in the case of the HS21XMs, most hardware events are detected through the AMM.

We are evaluating ESXi installable and are looking to move to the embedded chip in the XMs. But this thread is a little concerning. It almost sounds like the best way to go is to buy the mirrored SSDs for the XM and just use the installable version.

[shormell]

The blade I have will only support one internal drive. We arrived at this blade due to the dual quads and 32GB of RAM capability while still in 1 blade slot. I'm not aware of another that can do that (in an IBM bladecenter at least)

I am considering doing a boot from SAN or single internal drive.

I would love to see any breakdown of significant advantages of ESXi vs. installed hypervisor, beyond the lack of internal drive or SAN boot need.

[rburbridge]

Shormell,

I am quite familiar with the HS21XM blades. The mirrored SSD that I refered too actually resides in the single internal drive slot (SAS). Currently they come in a single 15.8 GB, single 32 GB, or mirrored 15.8GB drives (that still fit in the single slot). This solution provides redundancy in the single slot footprint.

I agree that I would be interested in the advantages that the embedded version has over the installable. Aside from the convenience of course.

Rawley

[Dave_Mishchenko]

The difference between the 2 is really only that embedded eliminates the need for booting from a local drive / SAN. Otherwise they are functionally identical.

The custom versions coming from the venders is perhaps more significant as those will include the CIM providers for hardware montioring, but with HP for example you can get their custom install on an Installable ISO for install to a HD.

That said, if you have a way to manage the hardware OOB that may be the way to go for now.

[benny_hauk]

I don't know that this will help anyone but I'll share this in case it does. Once all the files are successfully transfered to the host, the file my HS21-XM host keeps getting hung up on for some reason according to my /var/log/vmware/hostd.log appears to be the cim.tgz file that is in the root of the VMware-image.tar.gz file. See my screenshot for more details.

[shormell]

9 business days after opening a ticket with IBM, I have finally gotten their answer on this problem:

IBM will not be supporting update 1 of ESXi, and they are working to get update 2 image ready for deployment. I was told a few weeks if it passes testing.

This is one of the most ridiculous things I have heard from IBM (and I've heard many insane things in the 5 years we have been working with them) I'm not sure how they can pull support of a product I purchased from them, but that seems to be what they have done.

With this experience, I now believe that ESXi is vaporware, and not able to be sufficiently supported in a production environment. After all this time, IBM (or VMWARE for that matter) has done no log collection, or troubleshooting of the problem with patching the system. IBM has not even had me read a logfile to them. They have ZERO clue about ESXi, and keep trying to ship hardware to solve this issue, that is obviously software related. I’m not surprised by this, as I’m dealing with hardware break fix people, not engineers that know much about the applications their hardware is designed to run. It is clear to me that IBM does not even understand what my issue is, proven by a lack of answer about how I will be able to test Update Manager once they release this "fix". I’m still not even sure I would be able to download this update, or have to wait 2-3 days for them to ship me a CD, as I did the first two times they attempted this.

We are looking back to IBM to see what can be done to resolve this issue without having our organization pay for their lack of ability to support a product they sold me. We will not be doing ESXi anytime in the near future. Until either VMWARE supports this product directly or IBM improves their ability to support it, I think that ESXi creates greater risk then providing benefit to my infrastructure. It is still shocking to me that I would be directed to IBM from VMWARE support, as VMWARE has always been one of the most amazing support groups available in the IT world.

This has not shattered my faith in VMWARE overall, but I will show a bit more pause when adopting new products and features from them in the future. This whole experience has been less then stellar to say the least.

[dilpreet]

I understand your frustration. I have not asked for logs because the

problem is known and it is due to a corrupt partition in your ESXi

filesystem. This is why VUM/Update is failing. The updated IBM CD is

supposed to solve this, still figuring out why it is not.

[shormell]

What I'm most concerned about is the amount of time this is taking to be resolved. We are lucky to have bought these blades with buffer time before needing to put them into production. I've now lost this window, and this is impacting my project, but at least it's not live yet. What if I discovered this issue after going into prod? The solution of having to reflash the host and then reconfigure the settings is far from ideal.

It's also concerning that VMWare has been unable to get an answer to this problem.

I'm just not seeing the benefit of ESXi if it can't be supported like all other VMWare products.

[benny_hauk]

This is unfortunate that it's taken so much leg work on your end. I'll say though that going forward I don't expect it to be much better even if they really get this stuff working great. For instance, in a perfect world where they communicate with each other and with their customers and they properly test, I think the best we can hope for is that when VMware releases an upgrade for the installable version of ESXi that the "IBM embedded version" won't immediately be available for installation but rather will lag behind due to IBMs need to do further testing (which is obviously needed from reading this thread). In other words, once Update 3 is released on VMware's website, IBM embedded users won't be able to get the upgrade for another month or so. So basically the downsides in using the embedded version (even in a "best-case" scenerio) seem like they would have to include:

1) Delayed availability of latest updates

2) No availability of beta releases

For us that's okay since we like to operate more in the peleton than leading ahead of the pack. I'm too busy with other projects to be chomping at the bit over any VMware release. For instance, the Update 2 features sound really benefical (Storage VMotion, et al) but we can live with plain 3.5 for now so we are going ahead with deploying our blades into production as-is. Since we operate on an "n-1" load with DRS enabled, I can take any one blade down at any time to upgrade it to U2 once the working version is released.

Going forward with future blade purchases, I'm leaning more towards the mirrored SSD drives in the HS21-XM blades and running the installable version of ESXi. From what I've read the mirrored SSD drives take the same form factor as a single HDD drive.

Question: Why can't VMware release a "vanilla" embedded image for those that are willing to forego the vendor-specific CIM modules? Sounds like someone above already figured out a hack to install that very thing. I understand some vendors (HP comes to mind) might be severly limited but if hardware management is mostly handled out-of-band then what's the point? An IBM blade ought to be just fine without those CIM modules. VMware ought to make the vanilla version available and support it. If nothing else VMware support would then be able to tell a customer, "image this non-IBM version on your box and see if the problem goes away". If it does then VMware support can legitimately say, "sorry, but the problem lies with the CIM extensions IBM has added. Our product obviously rocked until IBM screwed it up."

(no offense, IBMer's... if you are even listening)

[dilpreet]

After looking at the IBM recovery CD currently shipping, we have an issue where the altbootbank is corrupt. To solve this issue immediately please follow the instructions below, if you still have issues updating please let me know:

1. Go to the unsupported shell (I think folks have details on how to get here)

2. Find out the HBA name

vmkfstools -P /altbootbank/

vfat-0.03 file system spanning 1 partitions.

File system label (if any): Hypervisor2

Mode: private

Capacity 50101248 (48927 file blocks * 1024), 50100224 (48926 blocks) avail

UUID: 76ec2b74-a7965bdd-bf7d-a26c90143d09

Partitions spanned (on "disks"):

vmhba1:0:0:6 <---- HBA name

3. run dosfsck to check the partition

dosfsck -v /dev/disks/vmhba1:0:0:6

Return code zero indicating no recoverable error.

4. repair file systme problem.

dosfsck -a /dev/disks/vmhba1:0:0:6

Sorry for the long lead time in responding.

[shormell]

I do not know how to get to the unsupported shell. Can you provide these instructions as well?

[benny_hauk]

Thanks dilpreet!

That did it, sort of. At least it's gotten past that 'Failure to upgrade' at 98%. The VMware Infrastructure Update says the Update 2 upgrade was successful now but once it finishes and I reboot, ESXi still says it's running the same old version (VMware ESX Server 3i, 3.5.0, build 71173). The console and the VI Client both tell me the same thing. However when I run the VMware Infrastructure Update now, it's saying there is no more updates for that ESX host so some how it thinks Update 2 was successfully installed but the ESX host itself doesn't seem to know anything about the update. Is there anything else to do other than run the VI Update tool from my desktop then reboot the ESX host?

At this point this server blade isn't attached to any VirtualCenter server, it's just a standalone host.

I tried your suggested fix (the dosfsck commands) again once VI Update said it successfully updated the host but it seems to still be fixed now (the update didn't cause the altbootbank issue to return).

By the way, the 'vmkfstool -P /altbootbank/' reported to be on vmhba32:0:0:6. I recall from VCP class that the vmhba32 was some sort of special hba but don't remember any details at all.

I'll update my SR with this latest development.

As a side note, we just ordered additional blades with the 16GB SSD and no HDD or embedded USB for deploying ESXi installable instead of embedded. Going forward I feel a little better about the installable edition even though I'm confedent the embedded version will eventually be a good solution as well. If nothing else we'll have both embedded and installable to be able to pinpoint issues with embedded in the future. Actually if it comes to that we'll probably convert our embedded blades to installable (replace HDD and embedded USB with a single SSD).

thanks again dilpreet.

Thanks again!

[Dave_Mishchenko]

You can follow the 1st 3 steps here - http://www.vm-help.com/esx/esx3i/ESXi_enable_SSH.php.