USB Key w. ESXi e - bad experiences

[niggo]

Hi,

we've been using ESX Enterprise w. Service Console for over 2 years - never had any trouble. Our ESX "engines" were based on Dell's PE 6850 with 4 dualcores and 48 GB of RAM. In September '08, we ordered 2 of Dell's new flagship servers (R900) with 4 quadcores and 64 GB of RAM, using ESXi embedded installed on a Kingston USB Key.

This year, March to be precise, one of the new hosts got disconnected from vCenter. We could connect directly via VI client, but the logs showed that vCenter failed in reinstalling the vpx-Agent. The reason was a unresponsive USB Key. The messages-log was full of those messages: "...StorageMonitor: 196: vmhba32:0:0:0 status = 0/7 0x0 0x0 0x0...", (vmhba32 is the R900's USB-Controller).

After issuing a call by Dell I redirected some of the directories which seemed to be required for installing the vpx-Agent to a san storage and copied the non corrupted contents from another ESXi-Host .(e.g. /etc/opt, /opt, /tmp, ...)

This workaround enabled us to VMotion all VMs of the host and reinstall ESXi embedded onto a new USB Key.

I thought this was a unique event, as even Dell's staff told us that this was the first case they heard about a corrupted USB-Key.

Ok, it may have been a little naive - Anyway yesterday the other host crashed with the same error.

Has anyone had similar problems? (except of the HP Keys which were defective out of the box)

We're running 55 VMs on the two R900 machines, which shouldn't be too much (btw it's far away from VMware's limits...)

Thanks in advance!

Nico

[jasonlitka]

I've got 3 servers with ESXi on the USB drives below and have had no issues thus far. They've been running since August 2008. I'm actually very pleased with them and am about to add a 4th box also using one of these drives. That said, given their low price, what I do is get the system configured the way I want and then duplicate the drive to a second. I've never had to use one of the backups, but $6 insurance per box to get back up and running in 5 minutes is worth the effort.

http://www.newegg.com/Product/Product.aspx?Item=N82E16820609294

Jason Litka

[Kevin_Gao]

I'm running HP ESXi (USB) since it came out (June of last year). Only issue I've had was when I patched/upgraded the process caused a bad flash so had to reflash it. Other than that...no issues at all. 😕

Do you have the option of trying a ESXi installable version or are these servers diskless?

[niggo]

Hey litkaj,

I totally agree, a second USB drive for backup purposes is a good invest - at least theoretically.

The problem is, that, in contrast to usual SCSI or SATA drives, there's no monitoring of the drive.

When the usb key has an error, you'll realize it too late for doing a stressless restore. The host first gets a liitle "sluggish", then vCenter will try to reinstall the vpx-Agent, which always failed in our case, and finally you end up having a disconnected host with running VMs. Maybe you can switch the Key while ESXi is running, but I wouldn't want to try it in a production environment.

@Kevin Gao: Our servers are diskless, we're thinking about installing some drives after these experiences.

Nico

[dominic7]

I had a bad experience with the USB keys. I initially ordered about 50 blades with the 'internal' USB keys from a large two-letter company. The keys didn't ship inside the blades however, and many went 'missing'. It turns out that some of them never shipped, and some were being used by employees who worked on the raised floor who thought they were freebies. Also, none of the licensing worked and it took a while to sort it out.

Once we got them installed and running, we found out that our vendor was using USB 1.1 parts, which were pretty slow but shouldn't affect operation as they don't need to be terribly fast. Then we started to have keys that failed, and since there is no monitoring it was difficult to detect when the keys failed. Then the replacement parts would roll in ( which in itself wasn't easy to find someone who knew what part to bring ) and then we had to strong-arm the vendor to giving us the cd to flash the USB key. The vendor then informed us that the usb 1.1 keys were no good and we went to 2.0 keys, and later to a newer version of the 2.0 keys that had a lower failure rate associated with them. Failure rates were never resolved. Even with dual usb ports this is still an issue, if the usb stick dies there is no hot-failover, and the part is non field-servicable since it's inside a blade which needs to be physically removed to access the part which creates additional risk.

In the end the vendor gave us 2 new disks for each system and we've been running installable on them. I can't say that everything has been perfect since then, but we no longer have usb and/or disk related issues.

[isg-oes-info]

Thx, dominic7, replacing the USB Keys with hard drives seems to be the way to go.

I'm awaiting the answer of our technical account manager, hopefully he makes an offer we can't refuse

[niggo]

sorry, the post by "isg-oes-info" was me, too.

I forgot to re-login...

[jasonlitka]

I was actually just planning on replacing them every so often during normal maintenance. Before I install them I do a DoD wipe, write the image, and then read it back to verify, just to make sure that there aren't any existing errors on the drive.

Anyway, I just ordered a bunch of the drives at the link below. They're even smaller than the PicoUSB that I'm using now.

http://www.engadget.com/2009/04/22/eagletec-nano-flash-drive-makes-losing-data-easier-than-ever/

Jason Litka

[niggo]

Replacing the USB-Key maybe a workaround for smaller hosts with less important VMs, but I don't see why I should replace drives periodically just because the solution I payed thousands of € for failes every few Months. Looking back, it would have been less expensive to order standard harddrives running ESX installable.

[niggo]

Hey all,

I've got some news on this topic:

It maybe that neither the USB keys were defective nor the filesystem was corrupt. Dell's Support called me today telling me that there was a known bug prior to ESX3.5i embedded U4 that caused some partitions to run out of space. Hence the VI Agent fails.

I would have expected some other messages in the logs, but maybe that was what happened to us.

niggo