VMware Cloud Community
shebang
Contributor
Contributor

ESXi local storage recommendation

If using local storage for ESXi, should this be raid storage or is a simple harddisk sufficient. What is recommended? P.S. The VM's are on the SAN.

Reply
0 Kudos
55 Replies
AlbertWT
Virtuoso
Virtuoso

Hi,

@truo1van, Yes i don't recommend it as well to install ESX on USB, but in my production environment, I'm using ESXi 4 on Dell Power Edge 2950-III I plug the Corsair Voyager GT Hi-SPeed 16 GB USB2 stick and it runs flawlessly very fast and reliable since I'm plugging the USB in the external port (behind the server).

So far no problem with ESXI 4 on USB as long as you choose the good brand. FYI Dell comes with Kingston 1 GB USB but I'm not using it in production.

Kind Regards,

AWT

/* Please feel free to provide any comments or input you may have. */
Reply
0 Kudos
J1mbo
Virtuoso
Virtuoso

Re HP etc withdrawing support for internal USB - I wonder if the crux of the issue is exactly why these same servers all use registered DIMMS instead of the cheaper unbuffered - namely electrical noise. Mounted externally the device should be immune to that at least.

Please award points to any useful answer.

Reply
0 Kudos
s1xth
VMware Employee
VMware Employee

Guys... I am going to have to disagree about the USB drives. I run all

embedded USB ESXi in production across 25 hosts. Never had a single

problem and you shouldn't. There are very few writes made to the

drive, also once the server is booted the firmware is loaded entirely

into memory and runs from the server...NOT the USB drive!! If you are

having issues you need to look at your hosts. USB is entirely support

by VMware.

Sent from my iPhone

On Feb 4, 2010, at 5:42 AM, AlbertWT <communities-emailer@vmware.com

http://www.virtualizationimpact.com http://www.handsonvirtualization.com Twitter: @jfranconi
Reply
0 Kudos
truo1van
Contributor
Contributor

Gents,

I know ESXi can run on USB, supported by VMware etc.. thats why we run on USB from day on our Prod!.

For me, the ultimate question to ask is: if I had a safer (and I do) choice, albeit conventional less "flashy", more reliable would I take it?.

Its risk management.

USB stick dangling off front or back of host in datacentres is not an option for us. Dont believe large organisations or law enforcement agencies with security mandates/ceritfications.. would allow you to do that.

Corrections: We run Dell 2950 III in Prod and 2950 II + R610 in tes/dev.

When things runs OK, all good, but when they dont.. thats why we need to minimize if cant eliminate the chances.

Btw: lets throw in the mix ==> vShield zones to be used for hosts in DMZ! Imagine that playing up!

For me, need things proven and solid - verified by own experience.

chrs,

VanT

Reply
0 Kudos
s1xth
VMware Employee
VMware Employee

The new Dell r610 servers have embedded SD cards inside the server

cases. Fully meeting all security/hippa requirements.

Heck you can PULL a USB drive from a running server and it still

function.

Sent from my iPhone

On Feb 4, 2010, at 6:01 PM, truo1van <communities-emailer@vmware.com

http://www.virtualizationimpact.com http://www.handsonvirtualization.com Twitter: @jfranconi
Reply
0 Kudos
DSTAVERT
Immortal
Immortal

Think about all the routers and switches that you depend on. Run from flash. Have since day one. I posted a link to a reliability report on USB flash drives. Can't find it right now but it basically said something like if you wrote to it constantly the drives would last many many many years.

We are a very small organization but I replace several dozen spinning hard drives drives every year. Hard Drives ABSOLUTELY will fail. Most of my problems with USB disks are drying them out after they have gone through the washing machine.Over the years I have had very very few failures. Even the ones that have gone through the washing machine.

Added bonus USB contributes very little to the heat load and the power load. Those are good things for a server.

-- David -- VMware Communities Moderator
Reply
0 Kudos
AlbertWT
Virtuoso
Virtuoso

Yes David,

I Agree with you, one of the reason I go down with using USB is that I can now very confident that with better brand such as Corsair Voyager GT it would last very long time around 10 years of non stop read write operation.

Plus I also got miraculous incident where I accidentally put my Corsair USB into washing machine and found out that the data inside of it still intact nothing is lost until now Smiley Happy

Hope this helps.

Kind Regards,

Albert Tedjadiputra

/* Please feel free to provide any comments or input you may have. */
Reply
0 Kudos
chrisaug
Contributor
Contributor

I agree about the thumb drives causing less heat and all, but again, why introduce a single point of failure into a production environment? Sure, the raid controller is a single point of failure, but so is the USB controller. One task, in building our ESX environment, is to limit single points of failure to the least amount possible to increase uptime.

If you have a thumb drive and it fails or gets pulled out, what's the difference if you have a raid array and a drive dies. Nothing except ESX can still write logs and whatever else it needs to the remaining raid array. And secondly, what kind of notification do you have if a thumb drive fails? If what you say is true about how everything gets loaded into RAM after boot, then you won't know the thumb drive failed until the next reboot. Things keep working with a raid array and I can see the amber light when a drive goes. I can replace it and let the array rebuild without any downtime.

Reply
0 Kudos
s1xth
VMware Employee
VMware Employee

I can see your point...but you have to undetstand if you are this

concerned about redunancy than you should have your servers in an HA

cluster. There are many points of failure besides just where esxi runs

from that have higher failure rates. (ex. Switches, controllers, SAN

storage, nic cards, back planes etc). The new cisco Ucs servers have

no hard drives and run all embedded esxi. VMware designed esxi to run

via a flash memory device. All of my r610 servers have zero hard

drives and run embedded esxi from Dell (built in custom sd chip

connected directly to mobo.. All 3rd gen 2950s also have a internal

USB plug on them on the side that you can use).

Like I said, if you are so concerned about HA looking at esxi is least

of your worries. I have been running 3.5 on flash drives before it was

even supported and on all servers running 4.0 using Sandisk flashes or

using the Dell Embedded if they are new boxes.

I trust the flash drives over hard drives any day.. Out of 600 servers

I have 20 drive failures a year.

Just my opinion, just giving you all the information from my exp.

Sent from my iPhone

On Feb 4, 2010, at 9:33 PM, chrisaug <communities-emailer@vmware.com

http://www.virtualizationimpact.com http://www.handsonvirtualization.com Twitter: @jfranconi
Reply
0 Kudos
truo1van
Contributor
Contributor

Gents,

Check this thread out. Exact same symptoms as I had.

http://communities.vmware.com/message/1470996

This is real life ESX Prod experience!

Reply
0 Kudos
J1mbo
Virtuoso
Virtuoso

Think about all the routers and switches that you depend on. Run from flash. Have since day one...

This of course is absolutely true. However they also (traditionally at least) have a lot less going on inside, for example a router typically has one or two unbufferred DIMMS. Those just don't cut it in a high-density ESX box like (say) an R610.

Perhaps what is needed is EMI shielded flash.

Reply
0 Kudos
spectVM
Enthusiast
Enthusiast

The moral of your story should be not to buy cheap USB drives instead of not running ESXi on USB drives.

There has been an influx of bad USB drives as of late as all the good NAND chips are going into SSD drives.

Currently, non-reputable manufacturers are building USB drive out of NAND rejects (not cut out for SSD usage).

Buying from a reputable manufacturers is more important now than ever.

Last, you should try to buy SLC thumb drives (like the Corsair Voyager GT) and not the MLC versions.

SLC chips last 10x longer than the MLC chips and are faster too.

Reply
0 Kudos
oasisinin
Enthusiast
Enthusiast

Interesting post folks

learnt a lot from here and will be installing my ESXi on a USB Drive for sure

Reply
0 Kudos
s1xth
VMware Employee
VMware Employee

'Technically speaking' ESXi 3.5 was never designed to run on flash. It was later (in its development and life cycle) supported by certain vendors to run off flash. It was until vSphere it was FULLY supported and is actually the RECOMMENDED method of installation.

Everyone is entitled to their own opinions and own research/experience but vSphere is 100% safe on USB and FULLY supported.

http://www.virtualizationimpact.com http://www.handsonvirtualization.com Twitter: @jfranconi
Reply
0 Kudos
spectVM
Enthusiast
Enthusiast

I should also add that USB drives have a data retention time limit.

Old good USB drives made from SLC chips had a 10 years data retention limit.

New cheap MLC drives can have as little as 1 year data retention limit.

Sadly, most manufacturers have stopped making SLC thumb drives in favor of the cheaper to make MLC ones.

The largest SLC thumb drives that were ever made topped at 4GB and production stopped around 2007.

Any drive made after 2007 is certain to be MLC.

If you have a cheap drive, it would be a good idea to re-load it from an image once every year or so (if you don't upgrade your ESXi install passed that time).

So, start hunting those old good SLC thumb drives if you want to hold onto a particular install for several years.

Reply
0 Kudos
mcssoftware
Contributor
Contributor

I'm coming in late to this conversation, but I have the same sort of configuration as those above and the VMware response (at least the sales arm) hasn't really made me feel good. I really don't care about the HDD v SD card/Flash reliability issue too much (if I have a quality SD card). To me the pros outweigh any cons by going Flash/SD.

The bigger deal is the support. Are user installed SD cards supported? Right now I'm configuring a new R710 with the internal SD reader module and my support is through VMware and not Dell. I was told (by Vmware sales) that VMware does NOT support SD card installs of ESXi unless it is installed thru the OEM directly. But that seems strange. I have Essentials so I don't get support unless I pay $300 per incident anyway. But I just want to make sure that IF I call VMware they don't hang up on me because I use an SD card to boot ESXi. I really like installing on flash/SD since I think the management is far superior to local HDD installs (as mentioned in previous posts). Unfortunately, now I don't know what to think regarding support... Smiley Sad There is no mention in the HCL of recommended or supported Flash or SD cards which is really horrible exclusion IMO since the ESXi 4 U1 install documentation refers you to the HCL for Flash devices that are supported. Seems like there a lot of confusion about this issue from the support standpoint(?) Can anyone clear this up once and for all? I wish VMware would.

Reply
0 Kudos