Deployment Strategy for ESX at remote locations

[steveanderson3]

We are in the process of designing our strategy for hardware replacement at all of our stores across the country.

Considerations:

PCI - we have to maintain PCI compliancy

Remote support - we have no good technical support at these store facilities

Budget - We cannot afford shared storage

Monitoring - We have to be able to see if a disk or other hardware fails on an ESX server

Right now we have a tenative plan of placing 2 ESX host servers, HP DL380 G6's at each remote location. These will have ESXi v4 installed (no service console, so no need for PCI requirement of File Integrity monitoring) 3 - 7 Windows VM's per location, with the possibility of adding a Linux based database server. We like ESXi because of the reduced patching. It presents challenges however, monitoring of local disk (this was added with 3.5 u4, now its gone again!), and no support for replication from Veeam or Vizioncore. Ideally we want to replicate between physical hosts for quick recovery.

We have 70 locations, all connected in a hub topology back to our main datacenter, via T1 or DSL depending on store size. It seems that most of the pieces are almost there for what we need, but I want to put this scenario out there for discussion. I know that many other retailers have to be in this same boat.

[AsherN]

I would rethink the affordability of shared storage. It sounds like you will not need SAN-to-SAN replication or SRM capabilities, therefore there are a number of affordable SANs out there. If you want 2 hosts for HA, it sounds like you will in effect buy twice as much disk as you need to populate 2 hosts. It may even be cheaper to buy diskless hosts, boot from USB and use your disk budget to get a small SAN. If you don't want to get into the HA and vCentre license costs, you could recover those VMs manually.

[jamieorth]

Depending on the demand of your VM's in regard to storage you might want to look at the ix4-200r from Iomega. They are on the HCL and would present as NFS mounts to your hosts. 2TB of shared storage for $1799 list.

I use one of its sibling (the x2) for a lab environment at my house.

Also, vRanger (Vizioncore) 3.3 due out next month will support ESXi 3.5

Regards...

Jamie

If you found this information useful, please consider awarding points for "Correct" or "Helpful".

Remember, if it's not one thing, it's your mother...

[steveanderson3]

One of our upper-managements concerns about shared storage is the re-introduction of a single point of failure. If you have a shared storage device fail, you have a significant outage, depending the severity of the failure. Given that these are sometimes closets, not comm rooms, there can be a lot of dirt and heat that some devices don't hold up to. If that devices fails you have a need for a full DR solution, and no real good way to get all of that data back across the WAN. We currently back up only critical files from each server, and if we have a failure, we can re-image and re-copy that backed up data.

If you had two servers with local disk, you could at least fall back to one if the other failed. Our actual storage needs are very light, the biggest thing will be the big partitions required now for Server 2008. I will take a look at that link, I'm curious about reliability and field-support for this device.

[AndreTheGiant]

One of our upper-managements concerns about shared storage is the re-introduction of a single point of failure.

Good point.

But without a shared storage you have to "sync" your local storage periodically.

How much VM do you have on each site?

Andre

**if you found this or any other answer useful please consider allocating points for helpful or correct answers

[steveanderson3]

The replication issue is really the one thing standing in our way. If there was a product that could replicate between two esxi servers at a remote facility, and manage it reliably from our centrail datacenter, we'd be set. We'll have 3 - 7 vm's, depending on site, and they are 20 - 60 gigs in size.

I've seen the replication scripts that some have put together, but given the criticality of the data, and the number of facilities, I'm not comfortable trying to error check and monitor a manual script process.

[AndreTheGiant]

I mean sync between the two ESX.

To have remote VM DR you can use backup program that have differential or incremental backup with compression (and maybe deduplication).

Andre

**if you found this or any other answer useful please consider allocating points for helpful or correct answers

[jamieorth]

You mention that the data is 'critical'. What are the RPO and RTO in the business requirements? You had mentioned that shared storage is looked as a single point of failure. Having your VM's on a single host is a single point of failure. Most, if not all, types of shared storage you would use in this type of environment have varying types of redundancy built in.

Regards...

Jamie

If you found this information useful, please consider awarding points for "Correct" or "Helpful".

Remember, if it's not one thing, it's your mother...

[meistermn]

Maybe the HP DL 370 is better server for this.

11 PCIe ports

24 x SAS

What about this configuration ?

2 x HP DL 370 a 24 x SAS 146 GB + Lefthand/Storamagic/datacore or Flaconstore

[Josh26]

We are in the process of designing our strategy for hardware replacement at all of our stores across the country.

Considerations:

Remote support - we have no good technical support at these store facilities

Monitoring - We have to be able to see if a disk or other hardware fails on an ESX server

HP is your best friend with these two features.

Ensure you have iLo setup and operating. It will provide you all the remote support you could imagine. We have even fully rebuilt ESXi servers using it.

Look at HP's System Insight Manager. Unfortunately it doesn't do monitoring for vSphere yet, but I believe that's only a few weeks away. We have a whole bunch of ESXi 3.5 servers and one management interface shows us the status of all of them. If a hard drive fails anywhere in the country, we get an alert email to our helpdesk.

I know what you're saying about shared storage, the issue has always been that to do it "right" with quality hardware is always significantly more expensive, compared to doing it "cheap". And what noone seems to get is that doing it cheap makes that single point of failure just too likely. I've used several replication products, and all of them seemed to be more effort than they were worth. I've ended up with the view that a client needs to either accept the possibility of downtime, or invest in shared storage. Some of the better units out there like the MSA2012, although a single point of failure, are so reliable it's a fairly moot point. I've never seen a power supply or storage controller failure, and both of these units are redundant.

[steveanderson3]

We actually have a light data requirement, we have up to 7 servers, but most are almost appliance-like in their function, so we don't need a ton of disk. I am looking at this stormagic product, this looks very promising. Does anyone else have personal experience with StorMagic?

[steveanderson3]

We just got done with a demo of the stormagic product. They are going to give us a pre-release version that has esxi 4, and cross-over cable support. This product looks pretty promising, only concern at this point is it being time-tested in a large implementation.

[steveanderson3]

We have decided to move ahead with a simple, 2 host, no shared storage approach. After evaluating Lefthand (hp), storemagic, and falconstor, we've determined that we don't have the money or the manpower to implement these solutions. Moving ahead with a 2 host solution will at least lay a good foundation for the future, thank you to all for the input.

[JasonVmware]

Just throwing out some random thoughts

Backup Exec 12.5 AVVI Agent leverages VCB and allows you to quickly restore VM snapshots from backup. Granted I belive it ball parks around 3K a esx host for the agent, but it does give you the capability of taking live snapshots of your vm's and vm's can be restored to any host or any network during the restore procedure. You could setup another small server with a tape drive and setup a backup server, however at this point your problem occuring the same kind of costs of a small SAN.

Another duplication software that I have worked with in the past is Double Take and they are starting to jump into the VM replication world. I haven' t had a chance to test there VM duplication, but in theory it sounds good. However most duplication software is usally alot of administrative overhead.