garyfritz
Enthusiast
Enthusiast

ESXi & vSphere support on older hardware?

Jump to solution

I do freelance consulting work.  I often have need of a VMware platform for work I'm doing.  Mostly I've been running on Workstation on my laptop.  That works OK for some things, but for other things I definitely need ESXi.

Being a small shop with a small equipment budget, I'm looking for a low-cost solution.  I've found an old HP DL380 G4 for a very resonable price.  It's not a huge box but it would do fine for my needs.

My question:  how long is this platform likely to be supported in ESXi/vSphere?  Any reason to expect it to stop working anytime soon, or should I expect it to work "forever"?

Thanks!

0 Kudos
1 Solution

Accepted Solutions
a_p_
Leadership
Leadership

I have some old DL360 G4 models in my test environment and they definitely don't have VT support. A quick look at the Intel site shows the XEON CPUs without VT-x support (http://ark.intel.com/products/27089/64-bit-Intel-Xeon-Processor-3_60E-GHz-2M-Cache-800-MHz-FSB)

See also the HP Quickspect for details on the DL380 G4 at http://h18000.www1.hp.com/products/quickspecs/12028_div/12028_div.html and the HP compatibility guide at http://h18004.www1.hp.com/products/servers/vmware/supportmatrix/hpvmware.html

André

View solution in original post

0 Kudos
19 Replies
a_p_
Leadership
Leadership

Even though the DL380 G4 is a robust system, I wouldn't recommend it. You may be able to run ESXi 4.x and 5.0 (haven't tried this myself yet), but you definitely cannot run any 64-bit VMs due to the lack of VT support. If you want to have good performance you would also have to look for battery-buffered cache for the RAID controller. If you want to stay with HP, you should rather look for a G5 model which is listed in the HCL. You can get them for an affordable price, either used or - if you are lucky - as a renew model.

André

0 Kudos
garyfritz
Enthusiast
Enthusiast

Hm.  You're certain the G4's don't have VT support?  If so, that would be a deal-killer.  The entire purpose for buying this box is to run VMware, and that will include many/mostly 64bit VMs.

The seller says the DL380 G4 has "2x3.6GHz Intel Xeon CPUs with Hyper threading, 800MHz FSB, 64-bit."  http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100394... says

"All Intel Xeon processors feature:

  • Intel Virtualization Technology
  • Intel Extended Memory 64 Technology
  • Execute Disable Bit"

Thanks for preventing a possible expensive mistake!!

0 Kudos
a_p_
Leadership
Leadership

I have some old DL360 G4 models in my test environment and they definitely don't have VT support. A quick look at the Intel site shows the XEON CPUs without VT-x support (http://ark.intel.com/products/27089/64-bit-Intel-Xeon-Processor-3_60E-GHz-2M-Cache-800-MHz-FSB)

See also the HP Quickspect for details on the DL380 G4 at http://h18000.www1.hp.com/products/quickspecs/12028_div/12028_div.html and the HP compatibility guide at http://h18004.www1.hp.com/products/servers/vmware/supportmatrix/hpvmware.html

André

View solution in original post

0 Kudos
JohnADCO
Expert
Expert

It's a switch in the bios.    If the switch is not included in the bios the host doesn't support even if the processor supports it.

I have run into some Dell hosts that don't support it even though they have procs in them that support it.

0 Kudos
garyfritz
Enthusiast
Enthusiast
I have some old DL360 G4 models in my test environment and they definitely don't have VT support.

Then that's pretty definite.

Darn.  This was an outstandingly good deal.  The guy's selling a dual-Xeon DL380G4 with 6GB RAM and 4x146GB hard drives -- for $340 delivered.  G5's go for more than that with no RAM and no HDs.  *sigh*

But it's not a good deal if it's not going to work for me!!  Thanks all!

0 Kudos
Datto
Expert
Expert

If you're committed to buying a physical ESXi 5.0 box for an in-house lab environment, I would not waste money on any physical server that can't do EPT (Intel) or RVI (AMD). Those CPU features allow you to run multiple nested ESXi 5.0 instances each with their own 64 bit or 32 bit nested VMs. Why buy a server that can only be one ESXi 5.0 instance when you can get one that will run ESXi 5.0, ESXi 4.x, ESX 4.0 and ESX 3.5 instances. Since you're a consultant you may run into clients that are using an old version of ESX and your nested ESX setup would provide much more capability for you to simulate their environment.

So, for the cost of one physical box you can run multiple ESXi 5.0 instances, each with their own VMs. This makes for a great lab environment if you have some shared storage (or can run some of the free shared storage VM appliances right on the physical host).

Also, it will be memory in your server -- not so much CPU horsepower -- that lets the server run more VMs. So you should get as much memory in the server as you can afford. I'd say at least 8GB of system memory as a bare minimum but 16GB or 32GB would be much better.

I've bought several rack type servers on eBay that run ESXi 5.0 (and ESX 4.0) well enough -- I've put 16GB of system memory in each for less than $500 per server (all-inclusive hardware cost, dual quad core CPUs and the 16GB of system memory). So you might think about using eBay if you have the time and patience to wait for a deal to show up.

Datto

0 Kudos
garyfritz
Enthusiast
Enthusiast

While that would certainly be a nifty feature, 1) I don't often have need for ESXi, so this would be used only part-time, 2) I could easily live without multiple ESXi instances, and 3) I suspect any hardware that would support that would be newer, higher-end, and expen$$ive.  I'm looking for a very inexpensive option for something that I use only occasionally.

And yes, I agree RAM == Good.

0 Kudos
Datto
Expert
Expert

>> 3) I suspect any hardware that would support that would

>> be newer, higher-end, and expen$$ive

I just bought and installed some matching quad-core CPUs that have RVI and run multiple instances of nested ESXi 5.0 with 64 bit and 32 bit nested VMs for $21 for each CPU (delivered cost). I also bought some slightly faster matched quad core CPUs with RVI that were $45 each and those run six instances of ESXi 5.0 and other hypervisors, each with their own 64 bit and 32 bit VMs.

It's just a matter of knowing what to buy and where to buy it if you have a limited budget if you're setting up a lab environment versus running a production server.

Eric Gray has a very good website -- vcritical.com -- that describes the nesting of ESX if you're interested. Below is the link to the discussion there:

http://www.vcritical.com/2011/07/vmware-vsphere-can-virtualize-itself/#comments

Also, William Lam's website has details on nested ESXi 5.0 also. Here's the link to that website:

http://www.virtuallyghetto.com/

It's pretty clear that nesting is the way of the future for professionals involved in virtualization and cloud. The flexibility that nesting provides is just too great to pay the same dollars for a server and not get the capability for nesting of 64 bit VMs. Plus, a few of the more advanced features by other major vendors also require EPT or RVI (Microsoft RemoteFX for instance).

I wouldn't waste my money on anything that didn't have EPT/RVI capability because that is the way of the future and it doesn't cost much, if any, more to get it now than it would to buy a decent server without it anyhow. So why not get a server with that capability?

Datto

0 Kudos
garyfritz
Enthusiast
Enthusiast

OK, so would something like a DL380 G5 support this?  So I could use vanilla ESXi unless/until I need nesting, then upgrade the CPUs to support it?

How would I find the appropriate CPU upgrade?  EPT is supported on Nehalem CPUs.  Newegg says the cheapest Nehalem Xeons start at $205 and go up rapidly from there.  I'm guessing your $21 and $45 CPUs wouldn't work in the DL380?

Maybe I should quit looking at DL380s (just because I'm familiar with them) and figure out what boxes would support ESXi inexpensively -- and with the option of upgrading to EPT.    

0 Kudos
Datto
Expert
Expert

You may be better off paying more money and buying a server that has what you need already in it since that will be less trouble for you. In that case you should be picking a server specifically from the VMware Hardware Compatibilty List (HCL) and just stay with whatever is on the HCL for  ESXi 5.0. Here's the link to the VMware HCL where you can select a server by brand name, rack type or tower type and ESX version support:

http://www.vmware.com/resources/compatibility/search.php?source=web&cd=1&ved=0CB8QFjAA&url=http://ww...

Datto

0 Kudos
garyfritz
Enthusiast
Enthusiast

I suspect that brings me back to my "too much $$ to justify for my needs" issue.

I already looked at the HCL link, but I couldn't find any way to tell if something supported the EPT.  Closest I could find was the Sandy Bridge Generation (under Enhanced vMotion Capability Modes, which is probably not at all what I want), and that returned a pretty short list.  I don't know what CPU Generation would qualify.

So I think my options are 1) get something like a DL380 G5 that supports 64-bit ESXi 5.0 but not nesting, or 2) get some unknown box for unknown price that supports nesting.  Unless you can explain how I find qualifying boxes in the HCL I don't know where to go from here.

0 Kudos
Datto
Expert
Expert

The problem you're facing with Intel CPUs -- you're not alone by the way -- is that Intel doesn't offer EPT across the board on the fancier Xeon CPUs. You have to delve into the details of each and every CPU that Intel offers that you might be interested in purchasing in order to determine if the CPU has the feature. Intel does this for what is called "segmenting" the market, meaning, trying to get as much profit out of every possible buyer who doesn't investigate the details of the CPU they're buying.

For AMD CPUs, 13xx, 23xx or 83xx Opteron CPU that have a "C" level stepping should be able to do RVI and should be able to nest 64 bit VMs in nested ESXi 5.0 VMs if the server itself doesn't block this feature (some manufacturers, HP in the server world and Sony in the desktop/laptop world, have been known to deliberately block CPU capabilities in the BIOS on lower end boxes in order to make you buy a fancier box). I have a Dell PE2970, a Dell SC1435, an HP DL385 G2 (not G1) and an HP DL365 G1 that have CPUs in them that I've bought seperately that will do RVI fine and run nested 64 bit VMs under nested ESXi 5.0 just fine. As I remember, only the Dell PE2970 is on the VMware HCL though and is listed as FT capable.

Also, some retail CPUs -- AMD 7750 CPUs and AMD 7850 CPUs for instance -- for use in retail white box computers will also do RVI although AMD doesn't advertise that (because they'd previously figured that if you wanted to use RVI you'd use an Opteron to do it and didn't realize IT people would want/need RVI capability so much now even in plain ol' desktop white box PCs).

On the VMware HCL --

Eliminate any machine that can't do VMware FT (FT is one of the choices on the HCL search listing). That doesn't necessarily guarantee the FT capable machine will do nesting of 64 bit VMs on nested ESXi 5.0 though but if the machine can't do FT then it can't nest 64 bit VMs under a nested ESXi 5.0.

Another method to find an EPT/RVI capable CPU is to look at these two listings below of CPUs from Wikipedea and see which AMD Opteron CPUs in the list have "Rapid Virtualization Indexing" listed as a feature (sometimes also called SVN) -- note that the stepping column can't say B2 and have nesting work properly since there was a CPU bug a few years ago in AMD Opteron and Phenom CPUs and today, ESX blocks the RVI feature if the CPU has that bug (to protect customers from the CPU bug I guess). Some of the B3 stepping CPUs that have RVI/SVN listed as a feature can do nesting of 64 bit VMs under nested ESXi 5.0 VMs but to be sure, you'd need a C stepping for the RVI/SVN capable CPU. For the Intel listing, you'll have to pick out the likely candidates (Nehalem or Westmere CPUs are candidates to start with) and then go to Intel's website and check there also and possibly other places as well to be sure the Intel CPU has EPT capability (Intel makes it a bit more difficult to know for sure ahead of time).

AMD Opteron:

http://en.wikipedia.org/wiki/List_of_AMD_Opteron_microprocessors

Intel Xeon:

http://en.wikipedia.org/wiki/List_of_Intel_Xeon_microprocessors

If you only want Intel and you don't want to spend the time or don't have the time available to delve into the details and just want to roll the dice, pick out an ESXi 5.0 supported server on the VMware HCL and make sure it can do VMware FT. That will give you the best chance (but no guarantee) for getting a server that can nest 64 bit and 32 bit VMs under a nested ESXi 5.0 VM.

Datto

Message was edited by: Datto for grammar.

0 Kudos
rlund
Enthusiast
Enthusiast

I use a dl 360 and 380 g5 in my lab with esxi 5.

Roger lund

Roger Lund Minnesota VMUG leader Blogger VMware and IT Evangelist My Blog: http://itblog.rogerlund.net & http://www.vbrainstorm.com
0 Kudos
Datto
Expert
Expert

Here's a detailed listing of costs from eBay for one of my lab servers -- this was originally posted a while ago on Eric Gray's vcritical.com website. Today, this same Dell SC1435 server described below runs nested 64 bit VMs and nested 32 bit VMs under nested ESXi 5.0 VMs -- the physical ESXi 5.0 server runs ESXi 5.0 from a USB flash drive/stick. The VMs are on a SAN and I don't use local hard drives for anything so I don't know if the local SC1435 drive controllers would work properly under ESXi 5.0. I run a bunch of other VMs straight off the physical ESXi 5.0 also as well as VMs running off the nested ESXi 5.0 VMs:

My total cost for the Dell SC1435 server with parts from eBay for it was $512.70 (detailed costs were 2x Opteron quad 2356 CPUs were $84.26 delivered, the Dell SC1435 server was $222.44 delivered and I inventoried the parts not used from this server, 16GB ECC REG memory was $160 delivered, the QLogic PCI-e HBA was $40 delivered and the flash drive was about $6 delivered). I didn’t use the 2x hard drives delivered with the Dell SC1435 server nor the existing DC CPUs that came with the Dell SC1435 server since I’d separately bought two 3rd generation 2356 Opterons for nesting purposes and intended to run the VMs from SAN.

Datto

0 Kudos
garyfritz
Enthusiast
Enthusiast

You must be a better shopper than I am.  Smiley Happy  I can find SC1435's for $300-400, but not $222.  Maybe yours didn't have any CPUs.  But 2 Operon 2356's for $84!?  Best I can find is about $110 *each*.

And the wiki page says the 2356 is B2/B3, which you said wasn't safe for nesting, I'm sooo confused....  Smiley Happy

I don't know the Dell servers at all so I'm not well set up to grab some unknown boxes and start swapping out parts.  I've worked with the DL380's and similar HP gear quite a bit -- maybe I should look at a DL385 (AMD based) and possibly swap in appropriate CPUs later?

0 Kudos
Datto
Expert
Expert

That $222 Dell SC1435 had 2x dual-core Opterons in it when it was delivered but I'd already known those existing CPUs wouldn't do what I'd needed (they went into my inventory) -- that's why I'd purchased seperately the quad-core 2356 CPUs that had the RVI that I'd needed and didn't have the TLB bug from a few years ago in AMD CPUs. That SC1435 also had 2x hard drives in it but those went into my inventory too since I don't use local hard drives for anything in my lab ESX servers (I used USB sticks for all ESX booting in my lab).

There's a wild fluctuation of prices for 3rd generation Opteron CPUs and decent rack servers on eBay. I recently helped a guy at work (where several thousand servers have been exclusively IBM brand for decades -- excellent servers by the way but a bit pricey on eBay for a lab) buy a couple of low cost Dell SC1435 servers from eBay for his home lab and he got his servers from eBay for even less than what I paid (he bought 2x Dell SC1435 servers for $175 each (delivered price) and just swapped in the better CPUs that are needed for nesting -- I ended up buying another SC1435 too in order to pair it with my other one since that $175 price was so good). It just takes a little time and the items will show up on eBay at a more reasonable price for you than $400. Just make sure you don't buy from anyone on ebay with less than 10 ratings and at least a 98% approval rating. Then be sure to read all of the text in the listing description -- some sellers like to list the flaws or missing parts at the end of the description or bury the flaws into the middle of the text.

For HP servers (Intel and AMD models) you're going to want to make sure you get the appropriate model generation (that's the G number after the model name). The G number will mean all the difference in the world as far as virtualization capability so you have to buy the correct Model Number plus the correct G Number. Otherwise, the server could end up being just a bucket of bolts and useless for virtualization purposes.

Also, you may look through the VMware HCL and see if there's a low-cost model that is on the VMware HCL that would work for you.

For AMD Opteron series, it's the B2 stepping series of 23xx and 83xx series of CPUs that have the TLB bug where ESX blocks use of the advanced CPU features. The C series of stepping is fine for those Opteron CPUs that have RVI capability and the B3 series is a mix of good and bad TLB bug or no TLB bug CPUs. In the case of the 2356 CPUs, the B3 series that I have in place on several boxes is fine and works properly without the TLB bug problems.

Buy the way, you might be able  to use the 83xx series of CPUs in servers that normally would use the 23xx series -- I have several boxes where I've installed 83xx series CPUs instead of 23xx series CPUs and they work fine for me. As far as speed of the CPU, If it's a 23xx or 83xx series Opteron I would stay with at least a 2.3GHz speed as a minimum. Also, you're going to want to match the Hypertransport speed capability of your server with the CPU Hypertransport speed of the CPU (usually 1GHz or 2GHz on older AMD based servers).

As far as CPU paste, if you replace the existing AMD CPUs with different models, scrape off the existing lesser CPU paste and with an old credit card and shine up the surfaces with a clean rag. Then apply only a rice kernel size dab of Arctic Silver 5 brand CPU paste in the middle of the CPU before placing the cleaned CPU heatsink back onto the CPU. That will really help keep the CPU temps down when compared to using the existing CPU paste that comes from the factory. If the existing CPUs in the server are okay and are still seated, just leave the paste as-is.

Datto

Datto
Expert
Expert
For HP servers (Intel and AMD models) you're going to want to make sure  you get the appropriate model generation (that's the G number after the  model name). The G number will mean all the difference in the world as  far as virtualization capability so you have to buy the correct Model  Number plus the correct G Number. Otherwise, the server could end up  being just a bucket of bolts and useless for virtualization purposes.

Example -- the DL380 G4 doesn't even have VT capability much less EPT capability:

http://communities.vmware.com/thread/331888?tstart=0

Datto

0 Kudos
Datto
Expert
Expert

garyfritz wrote:

You must be a better shopper than I am.  Smiley Happy  I can find SC1435's for $300-400, but not $222.  Maybe yours didn't have any CPUs.  But 2 Operon 2356's for $84!?  Best I can find is about $110 *each*.

Two 2.8GHz AMD Opteron 2386 CPUs with RVI and C2 stepping and 1GHz Hypertransport (OS2386YAL4DGI) just sold on eBay for ~$72 delivered -- that's the delivered cost for both of them.

Datto

0 Kudos
garyfritz
Enthusiast
Enthusiast

Awesome.  Thanks for all the helpful pointers, Datto!!

0 Kudos