ESXi update has killed our server

[darren_wardle50]

Hi, I'm wondering if anyone can help me.

I have just updated our DL380 G5 with the latest firmware: ESX3350-200805501-O-SG PATCH

Since the update the server will not boot correctly, during startup we are getting an error saying "lvmdriver failed to load with...", I can't see what else is says as it goes off screen. After that message appears the screen goes blank except for the yellow bar at the bottom where the F2 and F12 options are, pressing F2 doesn't work mostly, occasional it does but I can't get further than the login screen.

The VI client can't connect and I can't even ping it.

We upgraded from the 82664 build because the April patch didn't work.

I thought if an update failed it would use the backup copy but it hasn't and I can't see how to do it manually dispite turning it off and on several times.

I can get into the Teck Support mode however I don't know what do with it, I don't have a backup of the USB pen and its only ESX server we have. As it came with the server we do not have a Support subscription with VMware and HP are closed at the moment.

I found the rest of the message lvmdriver failed to load: Invalid Argument along with vmfs3 failed to load: Invalid Argument.

Darren

[darren_wardle50]

I have found some more info: in /var/log/messages it says Failed to find VMKernel MAC addresses.

[ocremel · Accepted solution]

When you upgrade the system will try to determine if the upgrade was successful but that is heuristics. So there is a manual way to force a rollback.

At the bootloader screen (the first white progress bar), hit shift+r (uppercase R), this allows you to go back to the previous system image.

Still, your failure would indicate that no NIC exists in your machine which is a strange failure to get after upgrade. Let me know if the manual rollback is successful. Thanks.

[darren_wardle50]

Thanks ocremel, that has sorted it, it's now back on Build 82664.

Any ideas why this could of happened and how to go about fixing it? Will re-applying the patch cause the same issue, I used the VI update utlity to do the upgrade.

We've not have much luck with ESXi updates recently, first the April patch wouldn't apply at all and now this one.

I couldn't see that Shift +R command in the manuals.

Darren

[ocremel]

Setup guide, p114.

I am following up on the actual failure.

[sneginha]

Can you provide the lspci output? What NICs are you using? Also some /var/log/vmkernel logs around the time the network driver fails to load would help.

[darren_wardle50]

lspci output:

00:00.00 Bridge: Intel Corporation

00:02.00 Bridge: Intel Corporation

00:03.00 Bridge: Intel Corporation

00:04.00 Bridge: Intel Corporation

00:05.00 Bridge: Intel Corporation

00:06.00 Bridge: Intel Corporation

00:07.00 Bridge: Intel Corporation

00:16.00 Bridge: Intel Corporation

00:16.01 Bridge: Intel Corporation

00:16.02 Bridge: Intel Corporation

00:17.00 Bridge: Intel Corporation

00:19.00 Bridge: Intel Corporation

00:21.00 Bridge: Intel Corporation

00:22.00 Bridge: Intel Corporation

00:28.00 Bridge: Intel Corporation

00:28.01 Bridge: Intel Corporation

00:29.00 Serial bus controller: Intel Corporation

00:29.01 Serial bus controller: Intel Corporation

00:29.02 Serial bus controller: Intel Corporation

00:29.03 Serial bus controller: Intel Corporation

00:29.07 Serial bus controller: Intel Corporation

00:30.00 Bridge: Intel Corporation 82801BA/CA/DB/EB PCI Bridge

00:31.00 Bridge: Intel Corporation

00:31.01 Mass storage controller: Intel Corporation 631xESB/632xESB IDE Controll er

01:03.00 Display controller: ATI Technologies Inc

01:04.00 Generic system peripheral: Compaq Computer Corporation

01:04.02 Generic system peripheral: Compaq Computer Corporation

01:04.04 Serial bus controller: Hewlett-Packard Company

01:04.06 : Hewlett-Packard Company

02:00.00 Bridge: ServerWorks

03:00.00 Network controller: Broadcom Corporation Broadcom NetXtreme II BCM5708 1000Base-T

04:00.00 Bridge: ServerWorks

05:00.00 Network controller: Broadcom Corporation Broadcom NetXtreme II BCM5708 1000Base-T

06:00.00 Mass storage controller: Hewlett-Packard Company Smart Array P400

09:00.00 Bridge: Intel Corporation

09:00.03 Bridge: Intel Corporation

10:00.00 Bridge: Intel Corporation

10:01.00 Bridge: Intel Corporation

10:02.00 Bridge: Intel Corporation

ESX is using the two built in Broadcom's NICs in the DL380 G5.

Unfortunately I don't have a copy of the vmkernel logs and I haven't tried the update since.

[admin]

Hi Darren,

I am just curious if you have even been able to install ESXi patch bundle on this box before.

[darren_wardle50]

When we fist setup ESXi I tried the update fom April I think, but it failed to apply it every time, there was quite a few people with the same problem.

So at the moment it is running build 82664.which it came with, which is not ideal but it is working OK.

[admin]

Is it possible that you get vm-support data for me?

Have you reported this issue to HP as I assumed you got this server 3i box from HP?

[darren_wardle50]

waltert, I have sent you a PM with the link to download the vm-support output.

I have called HP a couple of times, but as we don't have a support contract they won't help, trying to get the company to get one has been impossible. So at the moment we are stuck with a working server but I'm concerned that patching it again will break it completely.

Neither VMware support or HP want anything to do with us. So far both attempts to update our server have failed, the first patch wouldn't even apply and the second one wouldn't boot properly, it's not looking good for future client projects using ESXi and maybe ESX as well.

Darren

PS this is my second attempt to post this, the first one came back with System Error, The specified thread was not found.

[Dave_Mishchenko]

Hello Darren, both posts made it through, but there's a forum issue that is resulting in the error that you're getting. I've deleted the first copy.

Since you've marked your post as answered, you can also click on the correct / helpful answer links to give points to the people that have helped you (up to 1 correct answer and 2 helpful). The points aren't worth cash, but people still like to get them

Dave Mishchenko

VMware Communities User Moderator

[admin]

Hi Darren,

I missed your post. Could you post it again? Thanks.

[admin]

Never mind. Found it.

[darren_wardle50]

Thanks for removing the double post, I didn't see either one after I got the error. I don't remember marking the post as answered but I have assigned points now.

I see that ESX 3.5 update 2 has arrived with some useful updates, I wonder if it will work on the HP USB stick, has anyone tried it yet?

[darren_wardle50]

I recently updated my HP USB pen to Update 2, below is an account of what happened:

First boot looked OK, server was pingable, ssh worked but webpage wouldn't connect and VI client wouldn't connect.

I have created a VM-support file, if anyone from VMware is interested, there are a few errors in the logs.

Second and third reboot ip wasn't pingable.

Shift + R didn't work, complained about current build currently being upgraded.

Reset back to facory didn't revert to previous build like it did last time but did get the ESX server to boot properly.

Setup IP address again, but had an issue with the password, I found it had reset it back to defaults for some reason, set it up again and all looked fine, added VMs back to inventory and resumed them.

Today I tried the patch to fix the bug that appeared yesterday, after applying the update and rebooting the USB pen is now useless to me. I have a attached some screen shots of various screens that appeared, the first shows the error loading lvmdriver, and the second is all that is displayed of the main system status page, the last one I think was Ctrl+F12, which is updating every second or so. Pressing F2 does nothing this time, so I can't do a factory reset.Ctrl + R didn't work again, it complained it was being upgraded. The photos are from a spare DL320 G5 just so I could take them, the behaviour is identical to DL380 G5.

I am currently using the free ESXi installable on a USB pen drive in the HP server with the Update 2 Build and date bug in it.

I called HP but they don't want to know as we don't have a support contract with them and have the same problem with VMware themselves. This 209 quid USB pen is currently useless to me. Three of the four updates I have tried on it has failed, with the fourth needed a factory reset to work again. I also never received a CD with the pen to revert to an older build.

I guess those of us who have bought the USB pen from HP for 209 quid will not be getting any kind of refund even though they now sell the same part no. for 21 quid (same as the unlicensed version) and those who bought it from VMware are getting a refund.

It seems to me that ESXi is not ready for production, the recovery mechanism doesn't work when you most need it and four patches have caused problems for me.

Darren

[John_Dryden]

Having the exact same problem. Mine was even more fun as I've "lost" the config on the USB key 3 times now, ending up with corrupted keys and ESXi unable to boot. Luckily our supplier replaced the key twice. I had trouble updating to the Update2 patch today over a slow wan link (10MB), but then successfully did it over a 100MB WAN link which was intersting and VMWare support had no idea, then again I guess it's not that common for someone to do the update remotely, guess I must be the only person having ESX servers in a Datacenter :-). One of the ESX3i host update worked fine, the other one gave me the same problem you have had. After two attempts of doing a restore (shift+r) I finally got it back to the previous version i.e build 103909. First attempt complained that it couldn't be done during an upgrade??

Downgraded now and got my host back but then tried to upgarde again and got the corrupted message. Ouch.

Managed to roll back to 103909 again

Guess I should mention that during all this I lost my MSA2012i Storage Controller, which probably don't help. All I can think is that because ESX still thinks the VMs are on this second host even though it can't see the iSCSI LUNs...

Not had much luck with HP on this project, duff memory, duff MSA SC, 3 HP ESXi keys corrupted...

Anyway enough moaning, hopefully when I get the MSA backup tomorrow and migrate the VMs off this server it will upgrade, either that or get some hard drives and install ESX3.5

[MHAV]

Hi Darren,

I´m having the same trouble with one of my DL380G5. PSOD after upgrade with a PCI-Error for the bmx oder bnx (wich is the NIC) driver.

Upgrading the BIOS didnt do a thing. Haven´t had time to troubleshoot jet becuase of some other VMware Patch Problems.

By the way i patched some other DL380G5 before and everything work fine.

Regards.

Michael

[ccarrera]

I had the exact same problem happening to me.... I even created a post that I was not able to access the console using the Root password (since that was the only thing I noticed back then), but then I posted that it was "solved" since "somehow, the server got reset and lost all its configuration". Since I am an ESXi newbie, I thought I may have done something wrong or pressed the wrong key or something, but I am extremely careful with what I do when dealing with new technology, so I was suspicious about that theory.

After reading this post, I know I was not seeing things, and something went awfully bad with the update. The new firmware was installed, but after the second reboot, I had a new ESXi server, with no IP address (was picking up 0.0.0.0), no root password and disabled. Luckily, this is just a test server, so this was just a learning experience.

I also have ESXi on a pen drive. I have not seen people with a bare metal installation having this upgrade issue, so I am thinking that it is an issue with USB pen drive installations.

[virg8]

Guess what? I got the same problem yesterday.

Hardware: HP DL380 G5

I tried to upgrade from build 82664 to build 11027 (using VI Update).

Then the message appears after the reboot "LVMDriver failed to load with"...

It was impossible to rollback using the Shift-r (because "can't rollback during update process")

Then I tried to boot from a installation CD of ESXi build 11027. I had two choices: Install or repair. I was afraid to choose repair baucase I don't want to loose all my settings and my VMs (3).

I finally chose "cancel" and the server shut down.

When I rebooted, the server came up successfully but with the old firmware (build 82664).

For us, this server is critical... I can't afford an other update crash as yesterday (and my heart either).

What can I do now to update?