Anyone actually using HA distributed FS for VM storage?

[merititops]

We've recently spent a great deal of time testing out different "distributed" filesystems in high-availability configurations for use with VMware, and after many failures, I feel the question has to be asked.

Is anyone actually using Gluster (or any other distributed filesystem) as VMware storage? Either over NFS or some other method? We have endless issues whenever getting to the more complex configurations, ala replication, recovery of nodes, etc. There's lots of how-to's and guides available, but they always seem to be written by people who set them up but didn't actually use them, as they would've run into the same issues we are currently.

So anyone have a "success" story with distributed FS's?

[Flicka]

Our company is also looking at using Gluster on CentOS for POC engagements with clients.  We have not done any testing of Gluster in our lab yet, but plan to, so I am also interested in the answer to your question.  In fact, could you be more specific?  What are the top 3 issues which you are dealing with now (these being the issues which prompted your post)?

As a general observation, $136M US is a lot of $ for Red Hat to throw at a technology which (for whatever reason) doesn't work well with virtual machines.  As you may know, they are making a big push with KVM built into the kernel of Linux, so this appears to me to be an important piece of making RHEL a true virtualization contender.  That said, there are no Red Hat products yet with Gluster baked in, that I know of.  If I'm wrong, perhaps someone else can correct me here...

Thanks!

[merititops]

Gluster may work fine with KVM, but unfortunately we don't use KVM. We've personally run into several issues in our lab with the auto-replication, self-healing, and other 'core' features of Gluster not quite working correctly with VMware. If this is strictly an incompatibility between these features and VMware’s NFS client, fine, but that basically defeats the purpose for us.

If you search long enough you’ll find mail-list posts like this one:

http://gluster.org/pipermail/gluster-users/2011-October/008947.html

Which basically points out that during a self heal, no file IO is allowed, which supposedly will be fixed in the next release. This means if you suffer an HA event between your two Gluster heads, if the 2nd one comes back up, you just lost storage connectivity. Unfortunately, “will be fixed in the next release” seems to be the status quo for problems like this.

I’m not trying to hate on Gluster, I think it’s a great achievement for an Open source project, I just can’t find anyone who has actually (successfully) used it with VMware.

[Flicka]

I see the problem, and that is a killer for .vmdk files.

As I understand it, GlusterFS is a file level FS, not a block level one.  This probably means that it is counting on being able to lock a large file for more than a few milliseconds in order to do the read from the "good" file to the one that needs healing, without having to watch the "good" file get changed in the process.  To be specific, it relies on extended metadata attributes in the underlying FS to keep track of things, and since a .vmdk file only has one set of these, the whole thing has to be locked.  If I had to guess, I would say that v3.3 will use VMware APIs to get visibility into what the vmkernal is doing so that it can temporarily snap each VM on the fly, heal to the snap image, snap again, heal and keep doing that until it gets the whole thing.  Of course, a more elegant way would be to work with VMware to write a glusterFS driver for ESXi, but that may not be possible.

Now that Red Hat $ is behind the effort, I expect v3.3 to show up more quickly than before.  I guess we'll see...

[THLIG]

I am very keen on using Gluster also for VM storage. I am currently using it as an ISO and Template NFS export for my DEV environment. This has no requirement for perfomance or HA. I am using Centos 6.2 and GlusterFS 3.2.5.

With Red Hats RHEV 3.0 now available and wanting to gain market share in the x86 virtualisation space, I personally do not see Red Hat making time for API intergration with VMware products. I might be wrong but I think they will want to make as much headway as possible of using GlusterFS with KVM.