For those of you interested in creating a low-cost
(relatively speaking) lab for VSphere 4.0 that will also do VMware FT, here's
what I've used for my three white box VMware FT setup.
White Box 1 -- AMD AM2 Opteron 1354 (quad-core single socket
CPU to save money - CPU cost is $85.00 each delivered from Newegg), Asus M2N-LR
motherboard (about $100 delivered Open Box from Newegg, much less on eBay when
available) and you'll need to get the latest M2N-LR BIOS installed but you may
not be able to flash the board using the Opteron 1354 since the board may
require a lesser processor to flash the BIOS -- I had a low-end AM2 3600 CPU
sitting around and used that for flashing, then put in the AMD Opteron 1354),
two PCI-X dual port Intel gigabit cards (I got mine for $16.50 delivered for
each dual port card on eBay -- this combination of add-in NICs and the two
gigabit NICs on the motherboard that work properly with ESX 4.0 give me six
physical gigabit NICs in the white box), 8GB of non-ECC memory (4x 2GB GSkill
PC2-6400 DDR 800 -- the board will take ECC memory if that's your preference
but I had this GSkill memory already), 20G Maxtor IDE drive for $15.00 each as
the ESX 4.0 boot drive. For CPU fan/.heatsync I'm using a Zerotherm NV120
($50.00 from Newegg or about 60% of that used on ebay when available). I don't
put a permanent CD/DVD drive into any ESX host in my lab and just put the
CD/DVD drive temporarily in when I need it. I also never use a permanent floppy
drive and just plug in a USB floppy drive if I need it for say BIOS flashing.
White Box 2 -- identical to White Box 1 above
White Box 3 -- AMD AM2 Opteron 1356 (also a quad-core single
socket CPU to save money and it's within the 400MHz speed difference limit required
by VMware FT between CPU speed in White Box 1 & 2 above), Asus M2N-L
motherboard, combination of PCI-e dual port Intel gigabit NICs in the two PCI-e
16X slots and straight PCI 1000MT gigabit NICs to get to a total of 6 physical
gigabit NICs in the white box, 8GB of ECC memory (this CPU, motherboard and
memory came as a bundle I'd bought on eBay). Note I couldn't get the two
existing Marvell physical gigabit NICs on the M2N-L motherboard to work with
ESX 4.0 and haven't had time to chase down a NIC driver shoehorn process for
the Marvell NICs so I just disabled the Marvell NICs in the system BIOS for now
and use the other PCI-e and PCI gigabit NICs for my FT purposes. The CPU
Heatsync is also a Zerotherm NV120. Boot drive is also a Maxtor 20GB IDE drive.
Regular VMs (non-COS VMs) sit on an Openfiler 2.3 box
providing shared iSCSI storage for the cluster of white boxes. One gigabit NIC
on each box is dedicated to FT Logging-- the likely max number of FT Primary
and Secondary VMs on any single box is likely three to six of these FT Primary
or Secondary VMs before FT logging might get swamped. So far I'm running a
total of five FT protected light-duty VMs in the cluster and they seem to have
no problems and FT does correctly transfer over to the Secondary if the Primary
FT protected VM fails. I'm also running non-FT VMs in the cluster and there are
also two other ESX 4.0 boxes in the same cluster that are not FT capable (they
have AM2 Kuma processors which are VMotionable between the Opteron 1354/1356 CPUs
but ESX 4.0 needs a keystroke hit on the bootup on those Kuma boxes to keep the
bootup from stopping -- something having to do with the Kuma / M2N-E
motherboard combination used in those non-FT capable white boxes and ESX 4.0 --
if I put a standard AM2 Brisbane CPU in those M2N-E boxes they boot normally
for ESX 4.0 (but won't VMotion straight-away with the Opteron 1354 / 1356 CPUs)
so the problem is with using Kuma processors and ESX 4.0). I don't down ESX
boxes much so this isn't a problem for me at the moment.
Note that the M2N-LR boxes will not go into Standby Mode for
unknown reasons (regardless of which NIC is the Standby-capable VMkernel NIC)
but the M2N-L box will sometimes go into Standby Mode so it's likely a BIOS
issue with these Asus motherboards I suspect. The Asus M2N-E / Kuma CPU
motherbaord/CPU combination have no problem going into Standby mode but with
the Kuma processors in the M2N-E motherboards the boot back up process needs a
keystroke hit so that's a problem.
Also note the huge Zerotherm NV120 heasyncs cover the
closest (the first) motherboard slot in every motherboard I use. Some people
have just trimmed the fins of f the NV120 to make room as necessary.
VCenter runs as a VM on another cluster.
These are not a supported configuration so don't go using
this in your company production setup. Hope this helps folks wanting to build a
relatively economical VMware FT lab setup.
On a lark I decided to rig up the AM2 Kuma 7750 / Asus M2N-E white boxes in my personal lab to see if they'd also do VMware FT since those Kuma CPUs do correctly VMotion with the Opteron 1354/1356 CPUs that are VMware FT capable.
The AM2 Kuma 7750 CPUs will do VMware FT. Ha. A surprise -- the Kuma's must have the necessary VMware FT CPU instruction set built in also like the Opteron 1354/1356 CPUs do.
Those two Kuma 7750 CPUs came free with a special hard drive deal from Newegg. As I remember, they're $60 at Newegg when Newegg has them on-sale.
So that gives me two more VMware FT capable white boxes in the personal lab to complement the existing three VMware FT capable Opteron 1354/1356 boxes (total 5 boxes in the ESX personal lab cluster).The lab boxes at the office are all IBM x3650 and x3650 M2 and FT capable from the specs but I have them all tightly scheduled right now so there's no time available to fool around with them (production ESX boxes are also all IBM x3650 and x3650 M2).
What tipped me off to the possibility that the Kuma 7750 CPUs might work with VMware FT is when VC wanted to move the secondary VM of a previously setup FT VM to the Kuma 7750 boxes when I wanted to down one of the Opteron 1354 boxes. At the time I'd thought that was strange, thinking the Kuma 7750 CPUs shouldn't be capable of that. After thinking about it for a while I'd decided to just setup VMware FT fully on the Kuma 7750 CPU boxes and vi-ola it worked.
Just thought I'd pass it along in case anyone else is looking to build some white box VMware FT capable boxes.
If you want to use your lab to create virtual ESX 4.0 hosts and run VMs inside those virtual ESX 4.0 hosts, here are articles explaining how to do this by running ESX 4.0 under VMware Workstation or running ESX 4.0 under ESX 4.0 -- my experience with running a virtualized ESX host under VMware Workstation is that yes, it will work but it's dog slow and I'm not that patient. But for those with little to no money it might be worth looking into if you can meet the minimum hardware requirements for the box that will run your virtualized ESX 4.0 hosts:
thanks for that guide.
Is there a reason why you use OpenFiler instead of for example FreeNAS ? I do not really know OpenFiler. Do you know it has a better performance than FreeNAS ?
thx in advance
OpenFiler is just what I happened to have taken the time to learn. OpenFiler has worked very well for me over the past couple of years and since I've setup so many OpenFiler servers, the setup is now quick for me.
I set up two openfiler boxes recently with DRBD and heartbeat to create a highly availabile SAN. I've done it many times now and there's an excellent guide out there:
I built my whitebox last year with the following spec:
ESX 3.5 U2 running on:
ASUS P5WDG2-WS Pro (2 x PCI-X)
8GB OCZ ReaperX DDR2 1066 (running at 800 as overclocking to 1066 was not stable)
Intel MT 1000 Dual Port (in PCI-X slot)
Adaptec 2130SLP PCI-X 128MB RAID Controller
2 x 15k 36GB in RAID 1
2 x 15k 146GB in RAID 1
HP rm435 DVD-ROM
Now i'm looking to put together two boxes for 4.0 and one for openfiler (two would be a waste just for my home lab) the same as yourself.
I was going to go with a P45 based board and get 16GB per box (OCZ PC6400 4x4GB for each one). Q9550's for the processors and i looked up vm-help and there's pic-e dual & quad port nics that work with 4.0. At least six ports per box (preferably 8). The on-board nics are no use on all the 4.0 compatible P45 boards i've looked at. Your setup is excellent and far less expensive but I think it may be worth the extra cost to get the 16GB.
For my openfiler box i'll use my existing whitebox and will replace the SCSI controller and drives with 8 X 1GB SATA II drives (not decided yet on which ones) and a PCI-E RAID controller.
What do you think?
I was able to get an AMD Phenom X4 CPU in an Asus M2N-LR (BIOS 515) motherboard to correctly VMotion and utilize VMware FT with the other CPUs/servers in the same cluster in my home lab (the cluster has AMD Athlon X2 Kuma 7750 CPUs and AMD Opteron 1354/1356 CPUs -- see above for details).
Just wanted to add my whitebox that supports FT:
This is what I bought:
- Asus Barebone V3-P5G45 iG45, SATA2 RAID, HDMI (zilver)
- Intel Core 2 Quad Q9400 2.66GHz FSB1333 Box
- Intel PCI-e Adapter Pro1000PT Dual Port LAN 1000Mbit Bulk
- 2x OCZ 2×2GB DDR2 SDRAM PC6400 CL5.0 Platinum (Total 8GB)
- Western Digital 640GB SATA300 16MB, WD6400AACS
(Warning, the onboard nic will not work in ESX, I therefore added the dual port nic as an extra)
32bit - I haven't installed ESXi 4.0 on my M2N-LR systems but -- Wild Guess -- If the version you've installed is later than the GA version of ESXi 4.0, you might try installing from the original GA version of ESXi 4.0 -- the one released on May 21st -- and see if that allows you to boot. Also, you might try installing regular Classic ESX 4.0 with the Service Console and see if that goes in okay.
32bit -- one other thing -- on a different motherboard this morning (not the M2N-LR) I had to turn off all power saving features in the BIOS of that other board to get the install to go in correctly. Seems the power savings in that other board was modifying the speed of the CPU and that caused the install of Classic ESX 4.0 to bomb. Once itne install was in place I re-engaged the power savings in the BIOS and all is working okay.