VMware Cloud Community
ATL2VEGAS
Enthusiast
Enthusiast

VSphere ESX 4 3 Host Cluster Shared Storage

Hello,

I would like to know who has had any experience rolling out Vsphere? I would also like to know what type of shared storage are you using in a small 3 host cluster? I am trying to evualate a shared storage solution for a 3 host cluster and wanted to see who's using what solution?

Andre Chambers Contract IT Consultant Medium & Enterprise Business Solutions AndreChambersLV@gmail.com 702-203-9068
0 Kudos
18 Replies
TomHowarth
Leadership
Leadership

Moving to a more approiate forum

If you found this or any other answer useful please consider the use of the Helpful or correct buttons to award points

Tom Howarth VCP / vExpert

VMware Communities User Moderator

Blog: www.planetvm.net

Contributing author for the upcoming book "[VMware vSphere and Virtual Infrastructure Security: Securing ESX and the Virtual Environment|http://my.safaribooksonline.com/9780136083214]”. Currently available on roughcuts

Tom Howarth VCP / VCAP / vExpert
VMware Communities User Moderator
Blog: http://www.planetvm.net
Contributing author on VMware vSphere and Virtual Infrastructure Security: Securing ESX and the Virtual Environment
Contributing author on VCP VMware Certified Professional on VSphere 4 Study Guide: Exam VCP-410
0 Kudos
AndreTheGiant
Immortal
Immortal

For small environment some entry level iSCSI solution could be fine.

For example, in this case I use Dell MD3000i or the Equallogic PS4000.

Andre

Andrew | http://about.me/amauro | http://vinfrastructure.it/ | @Andrea_Mauro
0 Kudos
AntonVZhbankov
Immortal
Immortal

You can create shared storage solution manually in a couple of minutes if you have one free server with enough disks.

Install CentOS/RHEL/SLES and setup IETD (iSCSI) or NFS.


---

VMware vExpert '2009

http://blog.vadmin.ru

EMCCAe, HPE ASE, MCITP: SA+VA, VCP 3/4/5, VMware vExpert XO (14 stars)
VMUG Russia Leader
http://t.me/beerpanda
0 Kudos
Datto
Expert
Expert

You could also look at OpenFiler for your shared storage -- it's a freely downloadable iSCSI Target software (Linux Based with web based GUI configuration screens) that you would install onto a separate physical server, then tell your ESX hosts (the built-in ESX software iSCSI initiator) to use the shared storage on the OpenFiler server for locating VMs. I wouldn't go more than three ESX hosts with mild activity before moving from OpenFiler to a real SAN based solution but to get up and running quickly OpenFiler is adequate. Download OpenFiler at http://www.openfiler.com -- you can search the VMware forums to find others using OpenFiler for ESX shared storage too.

Datto

0 Kudos
SomeJoe7777
Enthusiast
Enthusiast

I just finished a complete brand-new installation of vSphere Essentials Plus on a 3-host cluster with the Dell MD3000i. System is running flawless, currently hosting 15 VMs. The MD3000i runs very well - make sure and use SAS 15K or 10K drives, performance drops off sharply with SATA.

0 Kudos
sidewaysclever
Contributor
Contributor

Hey Joe - looking at an exact same setup (just can't quite get the budget for an equallogic ps4000e Smiley Sad )

any other tips for this setup? did you work from any particular published guidelines/documentation when it came to configuring everything.

One thing that I can't quite get clear answers on - power outages... OK, we've got UPS's that will hold everything up for maybe 45min- what happens after that. With physical servers it was easy, APC software shuts them down - you boot them back up in the right order

I guess you'd want the vms shutting down first (maybe in a particular order) then the hosts (again, maybe in a particular order), then the SAN (not sure if the MD3000i can shutdown gracefully?), then lastly (in our case) the physical domain controllers - how easy is this to do and then of course booting back up, in reverse order...

regards....

0 Kudos
s1xth
VMware Employee
VMware Employee

Joe- I am also planning on using the same setup, except with 2 Hosts not 3. (yet). What is your Disk configuration? 15 Disks? What Raid setup/Lun setup did you use?

Thanks!!!

http://www.virtualizationimpact.com http://www.handsonvirtualization.com Twitter: @jfranconi
0 Kudos
SomeJoe7777
Enthusiast
Enthusiast

Hi sideways,

The primary reference I used for configuration of the MD3000i is on Dell's tech center site here. I followed these instructions to get iSCSI working with round-robin load balancing as well as failover due to NIC, switch, or controller failure. It works very well. Make sure you are using good switches (I use HP ProCurves) and put iSCSI on its own VLAN at minimum.

I have a UPS as well, also with about 45 minutes of run time. As far as I know, to get graceful shutdowns, you would have to work with scripting. I have seen a few people's compilations of scripts that they're using posted out there, but I have not looked at them. I believe for most of them you must use ESX for the hosts so that you can install scripts and/or agents (like the APC PowerChute agent) into the COS. I did not want to use classic ESX, all my hosts are ESXi. Therefore, I do not have a machanism for shutting down my ESXi hosts nor the VMs on an extended power failure. Because I have so few hosts, my monitoring system (which will alert me via text message) will detect the UPS on-battery condition, and then I will remote in and shut down manually if the power outage is extended.

0 Kudos
SomeJoe7777
Enthusiast
Enthusiast

Hi s1xth,

In my MD3000i I have 10x 146GB SAS drives. Drive 0 is a global hot spare. Drives 1 and 2 form a RAID 1 with 1 LUN. On this LUN are VMDKs that are used for SQL database transaction logs.

Drives 3-9 form a RAID 5 with 4 LUNs:

LUN 1 is owned by controller 0, and is used for VMDKs for the C drives of 1/2 of my VMs.

LUN 2 is owned by controller 1, and is used for VMDKs for the C drives of the other 1/2 of the VMs.

LUN 3 is owned by controller 0, and is used for VMDKs for the D drives of 1/2 of the VMs that need a data storage drive (Exchange, 1 SQL database data files).

LUN 4 is owned by controller 1, and is used for VMDKs for the D drives of the other 1/2 of the VMs that need a data storage drive (another SQL database data files, 2 web servers).

In this setup, the C drives' load is balanced across both controllers, as is the data files drives' loads. The SQL transaction logs are offloaded to their own array so that their I/O doesn't block other VM I/O.

I have not had any storage bottlenecks with this setup, although truth be told, my Exchange and SQL server loads are pretty light. Even so, I feel they could handle a lot more than what they handle now. My graphs in the vSphere client consistently show that the entire system is not being taxed at all.

Once you have all the LUNs set up, then configure the ESX/ESXi hosts to connect to each LUN (total of 5 LUNs seen by each host).

0 Kudos
s1xth
VMware Employee
VMware Employee

SomeJoe-

Wow.. thanks for taking the time to type up your setup, this will really come in handy when I start configuring my setup. I am going to have two Dell Powerconnect switches dedicated for the iSCSI traffic instead of VLANs (at this time). Couple small questions:

-What are you using to backup your VM's? Acronis/Symantec? If so how is the performance during backups?

-Was there any reason why didnt fill the 3000i with 15 drives instead of the 10? Any specific reason you went with 146GB instead of 300GB? (i know they are a lot more money and maybe your environment doesnt need that much space but i was jw if you had a specific reason.)

Thanks!!!

http://www.virtualizationimpact.com http://www.handsonvirtualization.com Twitter: @jfranconi
0 Kudos
SomeJoe7777
Enthusiast
Enthusiast

Both the choice of drive size (146GB vs. 300GB) and the number of drives (10 vs. 15) was for cost considerations. We were not able to put the money in for anything larger at the time. I don't need that much storage for these VMs anyway.

One server that is not virtualized is the main file server, it is still on a physical machine. This was done for two reasons: 1) It's datastore is very large - approaching 12 TB. VMware cannot handle this as a VMDK, it would have to be a RDM, and I'm not sure even then if > 2TB is supported. 2) The storage for this is on a 3rd party iSCSI SAN that is not certified for use with VMware.

For backup, I have a 4th server (other than the 3 in the VMware cluster) which is a physical machine that is the network monitor, tape backup, vCenter, and a few other items that require physical hardware that isn't supported by VMs. I'm running Symantec Backup Exec on that machine, it backs up all servers including all VMs to tape. This is an older version of Symantec Backup Exec (v 11d), which does not support VCB, so I have deployed the Backup Exec Agent inside all the VMs.

That is the primary backup solution. Since Backup Exec doesn't do a bare-metal restore real well (I have the IDR option, but I've used it for recovery a few times and have not been happy with the resulting restored machine), I have a secondary backup solution where I have deployed Acronis True Image Echo Server for Windows inside each VM. This does an image backup of the VM's C drive only once a week, saving the image on the physical file server (the 12 TB server I referenced above). Those images in turn make it to tape via Backup Exec.

That's a total of 5 physical servers -- the 3 VMware ESXi machines, 1 physical file server, 1 physical vCenter/Backup server. All are Dell PowerEdge 1950s.

I have looked at replacing the Acronis secondary backup with VMware's Disaster Recovery (DR) option, as that is included with the Essentials Plus package that I bought, but so many people are reporting problems with DR that I have decided not to deploy it at this time.

Be aware that making Backup Exec's remote agent and Acronis True Image co-exist on the same server is painful. You MUST install the Backup Exec Remote Agent first, then install Acronis on top of that. If you ever need to update the Backup Exec Remote Agent, you must remove Acronis, update the BE remote agent, and then reinstall Acronis. Doing it any other way will result in blue screen crashes during a Backup Exec backup.

Backup performance is quite good, Acronis can image a server with a 16GB C drive in approximately 5-10 minutes. Backup Exec's performance is highly dependent on the server it's running on and the specific tape drive you're using. The physical server running Backup Exec is a 2x Xeon 5310 (2 cores per Xeon), with 8 GB of RAM running Windows Server 2003 R2 x64. The tape drive is an HP MSL 2024, a 24-slot autoloader with a single LTO-4 drive. I can get up to 2GB/minute (~35 MB/sec) with this setup, this is primarily limited by the 3rd-party iSCSI SAN. If the storage was on the MD3000i it could be double that.

0 Kudos
s1xth
VMware Employee
VMware Employee

Understood. I was just wondering, as my Dell rep is pushing me towards 300Gb drives, but they are out of my budget.

I have the exact same setup with Acronis inside my VM's, I am not using anything but the Acronis agent. How is your backup performance over the iSCSI link? Any degration in backup speed when you moved from physical to virtual? Do any of the VM's suffer performance drain during the backup operations? I am just curious because this would be the same setup I would be using, I wouldnt be using VCB or Vmware DR. I have heard the same thing about Vmware DR...going to wait on that, my Acronis backups have been working fine.

Thanks again!

http://www.virtualizationimpact.com http://www.handsonvirtualization.com Twitter: @jfranconi
0 Kudos
SomeJoe7777
Enthusiast
Enthusiast

No degradation in backup performance at all when I did P2V. In fact, for some machines, the backup performance went up because they were on physical boxes with underpowered CPUs.

No CPU performance degradation while the backup is going on. However, because my cluster has CPU resources to burn (most VMs are lightly loaded), I have configured all of my VMs to be 2 vCPUs. I cannot say what would happen to CPU performance on the VM during a backup if the VM had only 1 vCPU, although I suspect it wouldn't be too affected.

0 Kudos
sidewaysclever
Contributor
Contributor

Hi Joe - Just like to second what s1xth said, thanks for the detailed answers - great job!

Just wondering about domain controllers - I guess you virtualised them as well? Any problems doing that?

regards...

0 Kudos
SomeJoe7777
Enthusiast
Enthusiast

I made a post regarding my P2Vs on my DCs here.

0 Kudos
ehinkle
Enthusiast
Enthusiast

Your post never included what the final outcome for your DC migration was. Did you just finally use converter to move it?

0 Kudos
SomeJoe7777
Enthusiast
Enthusiast

I thought my post was pretty clear.

I had 2 physical DCs.

  1. I took the first machine, transferred FSMO roles to the second one

  2. Demoted the first one to a standalone server

  3. Removed all hardware-related software (like Dell OpenManage & PowerChute)

  4. Take an image backup of the physical machine with Acronis (so you can back out if the P2V fails)

  5. Perform the P2V with vCenter Converter

  6. Shut down the physical server

  7. Start up the VM

  8. Recofigure it's network & other virtual hardware

  9. Install VMware Tools

  10. Remove all old hardware from device manager (see Microsoft KB article 315539)

  11. Promote it back to a DC.

Now repeat on the other physical DC.

I now have 2 virtual DCs.

There are some people who will say why bother with the demotion/promotion steps? The reason I do that is so that you can back out of the procedure if something goes wrong and go back to where you were WITHOUT going through the pain of an Active Directory restore/USN rollback. Believe me, you do not want to have to go through that procedure when you can avoid it.

0 Kudos
s1xth
VMware Employee
VMware Employee

I currently have 2 domain controllers in my environment, two are virtual and one is physical. I am going to be moving one of the virtuals to an off site DR site, I am going to do the same thing you did, dcpromo the server, image it and move it to the DR site, and dcpromo it again. Prevent any errors in AD, the only other service running on the server is DNS which is active directory intergrated, so I should be able to move that server with out any problems, leaving it a DNS server.

The best way to move or convert AD/Domain Controllers is with dcpromo/setting up a new server.

http://www.virtualizationimpact.com http://www.handsonvirtualization.com Twitter: @jfranconi
0 Kudos