How to make a mirrored volume accessible to VM

[akhaleel]

Hi there,

I'm trying to solve a problem and need some help.

I've got a fairly small setup at home where I've got a single box with many drives. Currently I've got linux running natively and I've created a software mirrored volume that is mounted under linux.

I would like to move this setup to EXSi and need help as to how I could go about doing this. I don't want to have a separate network storage device.

Essentially I would still like to have a mirrored partition that I could then mount inside my VMs. To do this, would I create a separate VM and then somehow make them access the physical drives?

Any help would be appreciated.

Thanks,

AK

[wdroush1]

You have a few options for a cheap home box:

Buy a hardware RAID card (~$100)

Run ghetto RAID (VM level)

Your own VSAs (OpenSolaris, OpenFiler, FreeNAS, etc.)

ESXi for some reason doesn't have native support for software RAID.

[AndreTheGiant]

Another option is use a guest replication mode.

For the right configuration of the VMs see the vSphere 5 storage guide (page 134).

Andre

[wdroush1]

Andrew Mauro wrote:

Another option is use a guest replication mode.

For the right configuration of the VMs see the vSphere 5 storage guide (page 134).

Andre

Wow, ghetto VM level RAID has an official term. I feel odd even thinking VMWare would entertain the idea. :smileysilly:

[akhaleel]

Thank you very much for respoding. Can you give me more pointers to "ghetto" RAID.

I looked at the VSAs you had mentioned. So I could run these under ESXi as a VM and they would still access the local drives?

Thanks

[wdroush1]

akhaleel wrote:

Thank you very much for respoding. Can you give me more pointers to "ghetto" RAID.

I looked at the VSAs you had mentioned. So I could run these under ESXi as a VM and they would still access the local drives?

Thanks

The VM level RAID (as I called it "ghetto RAID" being as I figured it was completely a hack job not supported by anyone to get around this issue) is running your VM with two VMDKs, one on each datastore, that way if you lose a datastore your VM acts like it lost a drive (software RAID only though).

This puts more workload on ESXi, but for home use is pretty tiny.

For a VSA, what you do is set up a VM that has VMDKs the size of your datastores (using as much as you can), you'll add each VMDK to the VSA, then re-present the storage via NFS/iSCSI back to the ESXi box, same issue with the "ghetto RAID" where it's implemented at the software level (so install ESXi on a USB stick or something), but this way you can implement disk replication on operating systems that don't support software RAID.

So you'd have something like this:

Datastore 1 - No room available

Datastore 2 - No room available

iSCSI - 100% free (start allocating VMs here).

[akhaleel]

I'm new to this so please bear with me

I still don't get it so you might have to explain this further.

This is my host

HP DL145 G3

Disk1 (200G)

Disk2 (1.5T)

Disk3 (1.5T)

1- I have installed EXSi on this host and I will have all the vm disk images in Disk1.

2- I want Disk2 and Disk3 to be a mirrored volume. I do have an LSI card that can do this hardware but for now lets assume I want to do this purely in software.

3- I've installed 3 VMs in Disk1 - Sles11, Win2k8 and FreeNAS

4- I want to use FreeNAS to "see" local host disks Disk2 and Disk3 and do software mirroring.

5- I want the iSCSI interface to use the FreeNAS volumes (from Disk2 and Disk3) and present it to Sles11 and Win2k8

My main criteria here is that I don't want to have a separate NAS box, I want everything to be intergrated in this one host. I would also want Disk2 and Disk3 to have a native format ie EXT3, ZFS etc so that I can always mount the bare disks to another machine in case of failure.

I've done Steps 1,2 and 3 and I've also done Step5 but its Step4 that I'm mostly stumped with. The question really is, how do I make FreeNAS see Disk2 and Disk3?

Thanks for all of your help.

AK

[wdroush1]

akhaleel wrote:

I'm new to this so please bear with me

I still don't get it so you might have to explain this further.

This is my host

HP DL145 G3

Disk1 (200G)

Disk2 (1.5T)

Disk3 (1.5T)

1- I have installed EXSi on this host and I will have all the vm disk images in Disk1.

2- I want Disk2 and Disk3 to be a mirrored volume. I do have an LSI card that can do this hardware but for now lets assume I want to do this purely in software.

3- I've installed 3 VMs in Disk1 - Sles11, Win2k8 and FreeNAS

4- I want to use FreeNAS to "see" local host disks Disk2 and Disk3 and do software mirroring.

5- I want the iSCSI interface to use the FreeNAS volumes (from Disk2 and Disk3) and present it to Sles11 and Win2k8

My main criteria here is that I don't want to have a separate NAS box, I want everything to be intergrated in this one host. I would also want Disk2 and Disk3 to have a native format ie EXT3, ZFS etc so that I can always mount the bare disks to another machine in case of failure.

I've done Steps 1,2 and 3 and I've also done Step5 but its Step4 that I'm mostly stumped with. The question really is, how do I make FreeNAS see Disk2 and Disk3?

Thanks for all of your help.

AK

You set up a FreeNAS box with two hard drives, one on Disk 2, one on Disk 3, both will be ~1.5TB (whatever the usable space is), you'll install freeNAS and tell it to mirror those drives.

It won't "see" those drives physically, you'll just be filling them with a single VMDK for the FreeNAS box. Mounting the drives in another machine due to failure will only be viable if you plan on using the Java built VMFS driver (and even then you'll still need to boot it in a hypervisor, probably ESXi at that... however you may be able to theoretically mount a VMDK [possibly dd if=./myVM-flat.vmdk of=/dev/sdb, purely theoretical though], not sure about the format of the flat VMDK file), it wont be raw device access (ESXi being able to do RDM with physical disks would be nice to ridiculous levels for those of us running it at home).

The only way I know of that you can do that would be to use a separate box (I know you don't want to).

[titaniumlegs]

It's important to point out here that you can't present and use the drives directly to multiple guests any more than you can with physical disks in a SCSI/SAS JBOD box and multiple servers/PCs attached.  This is especially true with different operating systems in the VMs.  If they have the same guest OS and a cluster file system, if you can get ESXi to present the same disk or VMDK to the guest, you can make it work, but I don't know of a cluser file system that supports Windows and Linux.

As others have said, if you go down the path of the FreeNAS or another VSA, you can present a big LUN or NFS datastore to ESXi.  But then you have a datastore and need to carve out individual VMDKs for the other guests from that datastore, so they're not sharing 1.5TB - they each get their own private piece.

Peter

[akhaleel]

Thanks for all of your suggestions but I think I've finally managed to solve my problem.

I can have my VMs directly access the underlying disks using RDM. This link gives a great description of how to do this.

http://blog.davidwarburton.net/2010/10/25/rdm-mapping-of-local-sata-storage-for-esxi/

So this is what I'll be doing in case you're interested.

- There will be 3 VMs. FreeNAS, Linux and Win2k8

- Disk1 will house the vdk's for all three VMs above.

- Create RDMs for Disk2 and Disk3 and add them as Harddisks to the FreeNAS VM where they'll be formatted natively in ZFS and I can also do mirroring of the drives in software (however, I could just use a hardware RAID controller to do the mirroring and present a single drive to FreeNAS - this relieves it of the burden of mirroring)

- Create CIFS shares that I can map/mount inside the Linux and Win2k8 VMs.

Now the performance may be a limiting factor but we'll see but I'm not expecting a large amount of disk traffic.

Any suggestions to improve on the above?

AK

[scottyyyc]

"Create RDMs for Disk2 and Disk3 and add them as Harddisks to the FreeNAS VM where they'll be formatted natively in ZFS and I can also do mirroring of the drives in software (however, I could just use a hardware RAID controller to do the mirroring and present a single drive to FreeNAS - this relieves it of the burden of mirroring)

- Create CIFS shares that I can map/mount inside the Linux and Win2k8 VMs."

Unless there's some special reason why you're doing this RDM and software RAID thing, you're massively overcomplicating things.

Why not let the RAID controller do its job and RAID the disks for you, instead of trying to set up RDMs to local disks and do software RAID? If you let the hardware controller RAID it for you, then you're presented with a single, simple VFMS volume. From there, you can simply create additional disks for any of your VMs. Plus, you then add the ability to thin provision, and all of the other benefits of virtualization. Why do you need to RDM to the local ESXi disks and, present them to VMs, and software RAID. And since your letting the hardware RAID controller do its job, you'll see much better performance.

RDMs are usually intended for allowing a VM to directly access a volume or LUN on a SAN. Unless there's some compelling reason, keep your local disks raided by RAID controllers and let VMFS do its job.

[wdroush1]

scottyyyc wrote:

"Create RDMs for Disk2 and Disk3 and add them as Harddisks to the FreeNAS VM where they'll be formatted natively in ZFS and I can also do mirroring of the drives in software (however, I could just use a hardware RAID controller to do the mirroring and present a single drive to FreeNAS - this relieves it of the burden of mirroring)

- Create CIFS shares that I can map/mount inside the Linux and Win2k8 VMs."

Unless there's some special reason why you're doing this RDM and software RAID thing, you're massively overcomplicating things.

Why not let the RAID controller do its job and RAID the disks for you, instead of trying to set up RDMs to local disks and do software RAID? If you let the hardware controller RAID it for you, then you're presented with a single, simple VFMS volume. From there, you can simply create additional disks for any of your VMs. Plus, you then add the ability to thin provision, and all of the other benefits of virtualization. Why do you need to RDM to the local ESXi disks and, present them to VMs, and software RAID. And since your letting the hardware RAID controller do its job, you'll see much better performance.

RDMs are usually intended for allowing a VM to directly access a volume or LUN on a SAN. Unless there's some compelling reason, keep your local disks raided by RAID controllers and let VMFS do its job.

Actually I understand the RDM, it means one less level of abstraction ESXi has to do (actually a few, VMFS + File management of the VMDK), plus he can throw that drive into a Linux box and mount it and run it.

RDM is a better idea for this setup IMHO.

Also, if the concern is transplating drives after a failure, a hardware RAID will just complicate the issue.