SLES + OCFS2 shared physical RDM

[NilC]

I intend to setup a 3 node Linux SLES OCFS2 cluster using a shared RDM in physical mode.

I did this once before with two VM's and set SCSI bus sharing to Physical on the 1st VM and then for the 2nd server selected the VMDK mapping file from the 1st. This gave the desired affect of allowing the physical RDM to be shared but meant that vMotion was not possible for either of these two servers.

What I plan to do this time is setup all 3 VM's the same , each has the physical RDM attached without bus sharing set on the SCSI bus. This is only possible, I think, by setting config.vpxd.filter.rdmFilter to false before adding the RDM to each VM. Then anti-affinity rules will be set on to stop the 3 VM's trying to run on the same host. This should allow the sharing of the OCFS2 RDM and also vMotion of the VMs.

I beleive this will work. Can anyone tell me if there are any problems with it? Is it a supported setup?

Thanks,

Neil.

[MichaelW007]

Hi Neil,

I can't say if it will work or not, but it is definitely not a supported configuration. Only two node configurations are supported.

Kind regards,

Michael

[NilC]

Hi Michael,

When you say only two node configurations are supported, do you mean by VMware with RDM sharing or with OCFS2 clusters?

Just curious, do you have any experience with Linux and OCFS2 on VMware?

Cheers,

Neil.

[MichaelW007 · Accepted solution]

More than 2 nodes is not supported by VMware. OCFS2 supports way more than two nodes. It used to be popular with Oracle RAC until everyone switched over to ASM. Although VMware doesn't exactly state what the non-MSCS clustering support or requirements are. Most things are based on the limitations of MSCS clusters in VMware. I'm about to implement a number of clusters for a customer with OCFS2, but each will be limited to two nodes. We were going to just use FT for our high availability requirements, but having the Guest OS as a single point of failure would not have met the application SLA's.

[NilC]

Thanks Michael for the helpful response.

Will you be considering using vMotion on these OCFS2 nodes? Have you tested it?

From what I've seen, OCFS2 clusters do not like the tiny network outage that can occur with a vMotion. It can cause the whole cluster stack to freeze.

Cheers,

Neil.

[MichaelW007]

Hi,

You can't vMotion any VM with a shared SCSI bus, so you wouldn't be able to vMotion the OCFS2 clusters. I have tested this too, so it's enforced in software, not just a best practice or anything. We've experienced the freeze up when doing testing under non-graceful shutdown scenarios. We observed up to 1.5 minutes of downtime during these tests. Given that the SLA we're working to is only 99.8% 1.5 minutes of downtime is acceptable. We also have inter site failover at the application level, so any service impact at a single site won't impact the application services.

[MichaelW007]

Hi NIIC,

Apparently the VMware stance on >2 node clusters has changed. I've seen a recently slideshow where >2 nodes are supported on vSphere 4.1. No limitation on shared nothing clusters. So it might be worthwhile logging a case with support and getting the official word on that. I guess this is so Oracle RAC can work properly, as that is now also supported on vSphere 4.1.

[NilC]

Thanks again Michael, I'll open a support call and see what the official response is.

Cheers,

Neil