Using RDM's for shared data storage...

[letoatrads]

Question for the masses as this will be an area of VI3 I haven't delved into as of yet.

I'm looking at running 3 ESX Hosts attached to a EVA 5000 FC SAN.

The question comes in concerning RDM's - As I've never had the occasion to need them. I understand the principal behind them, and seeing as how they are used for MS Clustering and quorum, etc, I assume they can meet my need here.

I will have 3 VM's with Apache, and a single VM with a CMS server. The CMS server will publish to RDM #1 using it as a datastore/output destination. The 3 Apache VM's will be in a load balanced cluster, each serving out the same web pages and each on their own ESX host. Each virtual Apache server will point to RDM #1 as it a data drive and use it to serve up the lovely HTML the CMS server provides.

Anyone see any issues with this setup? Folks doing similiar have any caveats, or tips?

Thanks in advance

[jodygilbert]

Hi there,

did you ever get an answer on this, we would like to attempt something similar using IIS and Windows 2003.

Cheers

[dominic7]

While this should technically work, you're missing one key piece in both posts ( and maybe the detail was just left out ). You need a clustered filesystem on the RDM and drivers for that clustered filesystem on each VM. In the case of windows it would probably be much easier to just create a DFS root and use DFS replication to get data to each VM individually. In the case of apache you would probably want to use rsync and manage a cron job to keep the data refreshing at a regular interval.

Other options are GFS in the linux space, and Polyserve has a clustered filesystem that also works in both linux and windows.

[jodygilbert]

Thanks for the info, I have contacted our HP reseller to find out about PolyServe.

I have also found out that there is a feature in Windows XP and 2003 for read-only disks, I'm trying to find out from Microsoft this would resolve our issue.

[nrempe]

We are trying to take this same approach. Multiple ESX servers, with mulitple hosts accessing a common "data" drive (on an EMC SAN w/ Navisphere and Access Logix) from IIS6. There is much benefit to this - a single location for deployment, easy backup, saved storage space (no need for data redundancy on a SAN) and various other items. Has anyone done this? If so, what was your approach?

We have tried to set up a file server in the middle, but are hitting a "BIOS connection limit" in Windows due to the large number of sites (and associated files) hosted on our virtual cluster.

We have tried presenting a Raw LUN to Vmware, mapping the drive in Windows as read-only, however, the obvious problem is that we don't see modifications to the file system on the windows machines unless the drive is unmounted and remounted (re-scanned).

We would like to stay away from having to push files to each esx server.

Any help would be greatly appreciated

[jodygilbert]

I'm still waiting for Microsoft to confirm is mounting a drive as read only is supported.

I looked into the PolyServe product, but it was too expensive for us.

[formulator]

we have a redundant FTP server solution doing something similar. VMs run on 2 seperate hosts, unbreakable linux 64-bit and share an RDM with OCFS2 filesystem. Stable with 300GB LUN serving millions of files for a PRD Peoplesoft CRM application for several months now.

[nrempe]

Formulator,

Encouraging....What mechanism handles reflecting changes made to the file system to each VM? We tried mapping a RDM to multiple VMs, but could not see changes to the file system unless the drive was unmounted and remounted

A couple other questions:

How do you protect agaginst data corruption? Are the FTP servers mapping the drive read only?

Do you have some sort of clustering software installed on your VMs to achieve this?

What is your VM configuration in regards to what I would assume to be a shared SCSI bus?

How is performance?

[formulator]

It's all in the filesystem.

Oracle OCFS2 makes it possible:

http://oss.oracle.com/projects/ocfs2/

[formulator]

BTW, if you are trying to do this with Windows you won't have much luck. The NTFS filesystem does not support the locking features needed to have 2 boxes write to a filesystem at the same time.

[nrempe]

I actually don't need to write (at least not from the OS)...just need to read. I can mount the drive read only in windows (I understand the risks) and use a utility like mirrorcopy on the SAN to push modifications from the deployment server's storage to the production storage. I'm most interested in how modifications to the file system by, say, "Machine A" are reflected on the same mapped storage in the VMs on "Machine B"

[jodygilbert]

If you mount the drive as read only using the WriteProt (www.joeware.net) utility, then shouldn't Machine B see the change immediatley.

Then you wouldn't have to use mirrorcopy.

[guido90210]

I'm hoping to do a similar thing with ocfs2 - present a shared filesystem from our HP EVA300 SAN to 2 apache servers and allow simultaneous reads/writes etc.

I've yet to test this, but I wondered if anyone could tell me:

1. Should I use physical or virtual RDM?

2. Will VMotion work for these virtual machines?

Thanks,

G.

[jhanekom]

As soon as there is a possibility that more than one physical host needs to access an RDM, you need to use physical mode. Amongst other things, this prevents the use of snapshots, which makes sense, since otherwise the hosts would need to figure out amongst each other which writes had occurred and on what log file they would need to be recorded.

In theory VMotion will work absolutely fine - it certainly does for single VMs accessing RDMs. It usually also works with MSCS nodes, but has apparently proven under testing to not be completely stable and therefore is not supported. So I'd say chances are good it will work, but you'll probably be on your own if things break.

[guido90210]

OK, thanks! I'll be testing it shortly.

[formulator]

Definitely need to use Physical.

As for VMotion, having worked with the OCFS2 filesystem a fair bit I wouldn't do this. It may cause a guest OS to fence itself if disk or network heartbeats are missed. You will see what I mean once you work with it and read up on it. However, it may be possible to change the timeout values to make this possible.

IMO your best bet if you want to move a VM in an OCFS2 cluster (the way I do it now) is to turn off the VM and move it, then turn it back on. For us this isn't a huge deal because we are using an active/passive config on the IP layer but I can see that for some deployments this may not be ideal and you may want to look into tweaking the timeouts.

[guido90210]

Hmm, yeah, I know what you mean re: heartbeat timeouts during VMotion. Ideally, I'd like to be able to VMotion these machines, but it's not absolutely necessary I guess.

We have two 3-node (physical) RAC clusters where I work which use OCFS2 filesystems, and occasionally OCFS2 will fence and reboot a node. It happens so randomly that it's difficult to pin down, but if it happens again I'm going to increase the timeouts (the timeouts are currently set to the default for OCFS 1.2.7). I can imagine that the additional network latency in a VM situation won't improve the OCFS2 timeout situation any 8-) Especially during a VMotion. Gotta get onto that testing...

[cmanucy]

This probably won't do you much good in your setup, but with our iSCSI solution, we can snapshot (on the SAN) at any interval we want... say it was 5 minutes ago for the sake of the argument.

You could then map that snapshotted LUN (RO if you wanted) to the 2nd (or nth) host... should get around a lot of the issues I'm seeing here.

At least it makes sense to me in my head right now.