jakemag
Contributor
Contributor

Can you use SAN snapshot/replication with VI3?

hi--

We are considering using VI3 enterprise with EqualLogic PS100E iSCSI SANs replicated between two sites (primary site and DR site). I'm trying to figure out if I can use the SAN replication and snapshot features or not! Is it true that I don't get a crash consistent image on SAN replication? Is it also true that VI3 doesn't integrate with SAN snapshots, and therefore they are also not crash consistent?

If so, then what good is SAN snapshot/replication on VI3? Must I use VI3 snapshots (software not SAN hardware) and some third party replication tool (ESXReplicator or DoubleTake?) for it to be reliable?

Or am I just simply missing something here?

tnx -- Jake

0 Kudos
18 Replies
wobbly1
Expert
Expert

Jake,

To use san based snapshots you have to use physical rdms and not vmdk's on a vmfs. SAN snapshots are not neccessarily crash consistent by default. As with any snapshot technology you need to quiese the o/s and/or application before taking the snapshot to ensure crash consistency.

0 Kudos
mreferre
Champion
Champion

>To use san based snapshots you have to use physical rdms and not

>vmdk's on a vmfs.

Wobbly1,

Is this by law ?

Is this written in the manuals ?

Is this a well known best practice (i.e. not written anywhere) ?

Is this your own personal advice ?

I hear very different opinions on this and also it is my understanding that what you can and cannot do is determined by what your storage vendor allows you to (from a certification / supportability standpoint). I have multiple customers running hundreds of of virtual machines in this scenario (one is right now at 800+) for Disaster Recovery purposes and I have never reported back a consistency problem from them in their D/R tests.

Thanks.

Massimo.

Massimo Re Ferre' VMware vCloud Architect twitter.com/mreferre www.it20.info
0 Kudos
jakemag
Contributor
Contributor

Massimo,

How do you accomplish the snapshot/replication? Do you just use the SAN tools to schedule these actions and do nothing on the hosts/guests, or is there some integration to perform the flushing/consistency? I have heard that if you don't tell VI3 to get crash consistent that the disk image might well be inconsistent - let alone the problem of application consistency!

Also - if using VMFS for the storage, do you use one VMFS per guest or allow many guests to share a VMFS? Doesn't that impact the problem of consistency (because it is harder to get multiple guests to be consistent at the same time).

And ... if the problem of asking each guest to become consistent, and then getting the replication to happen "left an as exercise for the reader"? Or is there some tool kit that automates this? Can it even be scripted, or is this a manual task?

What a complex world this is...

tnx -- jake

0 Kudos
mreferre
Champion
Champion

Jake,

I should have mentioned upfront that I am not using snapshots but rather I am using storage server type of replication (i.e. IBM PPRC just to name a technology) across different storage subsystems.

In my opinion there is no big difference between taking a snapshot of a LUN and replicating it (syncronously) for DR. Disaster time is just a big (unplanned) snapshot event on all your LUN's so to speak.

I am using VMFS LUN's and I host multiple vm's on it (if I was to host a single vmdk per vmfs than I would go with RDM's since you have very little felxibility anyway).

I do appreciate though that if you use RDM's you could "natively" talk to the storage server and quiesce your I/O to get a (much closer to) consistent LUN. However not using RDM's would only allow you to get "crash consistent" vmdk's that is identical to restart a Windows server that has either lost power or went blue. Do you think your Windows server won't restart due to this event ? Well I might argue myself that for an old NT 4.0 box this might not be the best manner to close it ..... but we have reached with the latest NTFS file systems a level of reliability for which I would just restart my servers and go confidently to bed.

Certainly if there was a db running on that guest it will go through some recovery procedures. Granted that there are few, as far as I know, applications that are integrated with the quiesce commands that the storage server could pass (most of the time this quiesce will only quiesce the file system I/O and not the application data structures in memory). Which is exactly the same consistency you would get with the "VMware software Snapshots" that you were referring to.

My two cents.

Massimo.

Massimo Re Ferre' VMware vCloud Architect twitter.com/mreferre www.it20.info
0 Kudos
oreeh
Immortal
Immortal

You can use snapshots and synchronous SAN replication.

As Massimo already said - they are normally as "crash consistent" as powercycling a server.

The consistency of the different snapshots (SAN and ESX) is equal.

There's one possible adavantge using ESX snapshots - you could suspend the guest, take the snapshot and resume the guest afterwards.

For replication I see a slight consistency advantage (depending how the SAN actually does the replication).

There are SANs that do it on the software layer (for example Windows Storage Server based systems) and SANs that utilize hardware replication at the FC switch level.

0 Kudos
mreferre
Champion
Champion

(for example Windows Storage Server based systems)

I thought we were talking about enterprise type of things .... not toys ...... Smiley Wink

Just kidding ... I love MS.

Massimo.

Massimo Re Ferre' VMware vCloud Architect twitter.com/mreferre www.it20.info
0 Kudos
christianZ
Champion
Champion

Interesting discussion here:

http://www.vmware.com/community/thread.jspa?threadID=56934

FYI EQL makes only async replication based on schedules.

0 Kudos
jasper9890
Enthusiast
Enthusiast

On the EMC front (remember they own VMware), i hear they are still working on getting their SAN replicator product RecoverPoint to work with ESX hosts. Seems that is your only option if you are trying to replicate a great distance.

0 Kudos
mreferre
Champion
Champion

>Seems that is your only option if you are trying to replicate a great

>distance.

Actually we are also using PPRC in mode to replicate at bigger distances (bigger than campus). I am talking of distances in the range of 100 - 200 Km .... it could be greater than that but this is the actual implementations I have been working on (with VMFS LUN's hosting more than one VMDK).

So far all the recovery tests provided positive results.

Massimo.

Massimo Re Ferre' VMware vCloud Architect twitter.com/mreferre www.it20.info
0 Kudos
epping
Expert
Expert

u gan use asynch mirrorview over big distances now, if u use fc -ip routers.

i use san mirroring and san snaps, never had a problem with a vm not comming up, i do not virtualise oracle, exchange, big sql boxes as i want to make sure the app is in a consistent state, u good do this though with RDM and agents in the vm, i just choose not to.

0 Kudos
TristalTips
Contributor
Contributor

Massimo,

Sorry for hijacking this thread, but it seems to be spot on topic. I need to do asynchronous replication with 2 x IBM DS4700 Controller over FC-IP using the SAN replication option (Enhanced Remote Mirroring). I would be interested to know how you do an actual failover to the other site. For example, I presume you have an active LUN on the live site with you replicate over to a passive LUN on the secondary site, but are the ESX servers on the secondary site aware of the passive LUN, or do you register the VM's on this manually if you have to failover. I'm fairly new to VM and have been chucked in on the deep end so to speak with having to set-up replicated sites! Can't quite get my head around how it would work, because the servers on the secondary site couldn't write to the passive LUN as it is being replicated over.

Thanks,

TristalTips

0 Kudos
mreferre
Champion
Champion

See this:

http://it20.info/blogs/main/archive/2007/02/23/3.aspx

http://it20.info/files/3/documentation/entry2.aspx

In short, I have never used the midrange storage servers (such as the DS4000) but with the high-end DS8000 the passive LUN's are not visible to the servers connected to it. I am pretty sure that the DS4000 behaves similarly. The PPT above outlines what we have done with this account using PPRC. Notice that with VI3 this sould be even easier because you don't need to spread vmx files across all your systems (everything is on the SAN nowdays).

Hope this helps.

Massimo.

Massimo Re Ferre' VMware vCloud Architect twitter.com/mreferre www.it20.info
0 Kudos
chrisy
Enthusiast
Enthusiast

This is not hard information, but perhaps some anecdotes might help.

First, I've had a dozen or so VMs in a VMFS partition on EqualLogic, which has then been snapped, copied etc. The virtual machines were happy and could even be presented back to the original server and booted from the snapshot. It's been done over and over and was fine each time. However it is true that this is a crash-consistent copy and there is a risk of it not always working. I'd judge that the risk is theoretical for non-transactional systems and very low even for them, but everyone has to make their own choice on this.

ESX software snaps are crash consistent, also. There is some support now for filesystem quiescing, but users report problems with databases and suchlike. Anyway it only quiesces the filesystem automatically, the apps still require scripting to quiesce.

If you use a RDM you can use (in Windows 2003, on EqualLogic) Microsoft VSS with the SAN which will give you a perfect snapshot.

So I'd suggest running nontransactional VMs in a normal virtual disk, important databases on a RDM, and be as SAN-powered snap-happy as you like!

As a final note, if you have a database with multiple disks make sure that they are snapped at EXACTLY the same time. Crash-consistent is usually OK but inconsistencies between drives can really mess a database up. EqualLogic will take consistent snaps of multiple volumes; for VMware software snaps you have do do them as quickly as possible and hope for the best, which is probably a poor idea.

There are no clear, yes / no answers in this area, unless you run everything off a RDM which most people feel is too painful.

0 Kudos
mreferre
Champion
Champion

Chrisy's post summarizes the matter very well. I agree 100%.

My only comment is: if you have a database (beefy machine by definition) which needs a DR plan for which you need 100% logical recovery from a transaction perspective ...... would you be running that on VMware (on RDM's etc etc) ? I personally wouldn't anyway (and not only because I need to use RDM's to do that....). Just a thought ......

But this is my own opinion; again, the previous post summarizes the matter in details.

Massimo.

Massimo Re Ferre' VMware vCloud Architect twitter.com/mreferre www.it20.info
0 Kudos

My only comment is: if you have a database (beefy machine by definition)

Database servers were not all created equal. We have a few database servers running in VMs that are running fine while taking about 2-3% of a CPU. Some database servers have small databases and a lot of queries/sec, others have terabytes of mostly dormant data and very few accesses, and a few are the ones you're talking about, holding a lot of data being queried at a high rate.

which needs a DR plan for which you need 100% logical recovery from a transaction perspective ...... would you be running that on VMware (on RDM's etc etc)

Unless I was running a very busy server (and even then, I might try to...) I'd do it because I could then run the VM on less powerful servers and - God forbid! - save some bucks on the DR Center hardware. Smiley Wink Of course, all depends on the SLAs, but generally it's acceptable to have lower performance just after a disaster.

Hey, Massimo, I was missing this... Smiley Wink Of course, we always agree in the end!

Paulo

0 Kudos
AndrewJarvis
Enthusiast
Enthusiast

One word of warning - and it may be specific to HP EVA SAN - we use HP Continuous Access SAN replication - and you have to disable the 'disallowsnapshotlun=1' option otherwise VI3 will refuse to let the server see the LUN once you 'fail over' to what was the replica- as it mistakenly detects it as a snapshot. You thus need to be VERY careful about playing with snapshots as you could present a snapshot of one of the live datastores and confuse the hell out of VI3 !

This setting change came direct from VMware support BTW....

0 Kudos
abaum
Hot Shot
Hot Shot

Have you updated to the latest version of CA? I hear that VM/Hp have qualified CA for vmdk replication just a few weeks ago. We don't use CA on our EVAs so I can't do any testing with it.

adam

0 Kudos
chouse
Enthusiast
Enthusiast

Are you using sync or async with your EVA CA? How many LUNs do you have in a DR group?

We are using CA in sync mode for the DR groups - we have 4 DR groups and 2 of them have 9 LUNs and 2 of them have 4 LUNs. We are seeing high write latency on the arrays during particularly IO intensive times which causes our VMs to become unresponsive and other VMFS operations to slow to a crawl.

It was recommended to us to have one DR group per LUN and change from sync mode to async mode to resolve the latency.

0 Kudos