jmcclymont
Contributor
Contributor

vm disk mirroring about 2 san/local subsystems - doable?

hello all,

i'm thinking about this:

i have 2 esx 3.0x server with 2 san or local storage subsystems. the ha and drs function is enabled.

i create a new vm and create one disk on the first storage and a second disk on the second storage. this both disk are used for a raid-1 (mirroring) installation in the vm. under windows i use the diskmanager and under linux the lvm.

has anybody experiences or information? theoretical it should be work, but might be it gives issues with the timing when the storage's are under heavy load.

grateful for statements

and greetings from germany

jeff

0 Kudos
20 Replies

Hello,

I would think that would work. My question would be why? Is your SAN not that reliable? Do you loose LUNs?

On the surface it seems like some added complexity for no gain.

0 Kudos
jmcclymont
Contributor
Contributor

hi,

when you create a new vm with disks on a san or local subsystem and this storage fault, then your disk is gone (ok, expensive san subsystem use snapshots etc ..). all what i want is to eliminate a single point of failure.

we use 2 msa500-g2 in raid-6, but when this msa failed the vm-disks are failed also. and i have some work to restore the vm's - from snapshot or some other backup system.

when i can split the disk to different storage's the one can fault and the disk on the other storage is still ok. yeap, might be the vm drop daown, because the other needable files are gone also, but the second storage hold one part of the raid-1 and i can create very fast a new vm and use this still existing disk. now i can create a new disk for the vm and rebuild the faulty first disk.

ok, this is the theory ... and in the reality????

under linux the lvm, should handle the disk ok and the new setup vm should be booted. might be the same under windows (?).

0 Kudos

Nothing wrong with the theory. I have heard of others doing this as well.

You could also look at something like esxpress: http://www.esxpress.com/ or esxreplicator to make copies of your virtual machines.

0 Kudos
Texiwill
Leadership
Leadership

Hello,

I think this is a great idea for your most important VMs and always allow for this in my designs. THe key here is that even very expensive redundant SANs can have some major issues (I have seen 2 disks fail in the same LUN and during a massive rebuild another disk failed. Causing a 48 hour rebuild of the LUNs. Having some VMs locally will at least mitigate the possible disasters).

This will at least provide a nice level of protection that would allow your VMs to continue going as if they were restored from tape, etc. If you tie your standard backup process into this you can recover everything from local storage as if it was currently on tape for you rmost important VMs. Sometimes it is just not worth it for some VMs.

Best regards,

Edward

--
Edward L. Haletky
vExpert XIV: 2009-2022,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos
jmcclymont
Contributor
Contributor

hi edward,

thats it, BUT how behaves the vm's? what's going on with the windows or linux system when i have different timings in the raid concept.

i thing i should do a test with one disk on a normal load storage and the other disk on a heavy load storage. i have the apprehension that this time delays break the raid and the system goes in failure.

what time delay can the system accept for normal working? the best would be the vm would be wait on the slowest part and that's it. in this case the throughput is defined by the slowest storage. in the worst case i get an error when the time between update the first and the second to big and the system switch the disk on failure. in this case the scenario are unusable.

might be anybody can answer how windows or linux works in this case and have a hint about the best configuration.

when i have time i will use vmmark to setup a vm and try a test.

jeff

0 Kudos
Cloneranger
Hot Shot
Hot Shot

When the two storage media are different, performance is the main issue,

As in the mirrored pair ends up slower then the slowest storage media,

The only time I have seen a use for this is as a temporary measure for live data migrations between storage platforms,

Create a mirror and break it when the data is replicated,

0 Kudos

Not a bad idea at all. ESX is all about host failures and zero downtime, but always assuming the SAN remains "up". This is a very affordable option in order to get synchronous replication... You might have a problem if "disk0" fails in the RAID1 setup and you (try to) reboot the VM. On the other hand, you might be able to solve that quickly by adding the 2nd disk to the boot menu. This way normally you boot DISK0, if it fails, just select bootoption2 and the VM boots from DISK1....!!

The only thing I wonder about is... How much CPU load will this take? RAID1 in software is not very complex, but still it will use resources. I would be interested to know how much addition CPU load is drawn if you have for example 20 VMs running in software RAID1 this way... Interesting!

Visit my blog at http://www.vmdamentals.com
0 Kudos
mcwill
Expert
Expert

We tried this with windows mirroring a few months back but quickly gave up on the idea as it appeared to create more problems than it solved.

We could have been unlucky but found that the windows hosted mirroring was unreliable. If both disks were present and running, then everything was fine, however if we simulated a failure on one drive the system would not always degrade sensibly, ie the mirrored logical drive would go offline instead of continuing on one drive.

Also if one of the drives is no longer present then I believe the VM config will need to be edited to remove the failed drive before ESX will allow the VM to restart.

0 Kudos
jmcclymont
Contributor
Contributor

hi all,

i have testet today with linux and it works great. the overhead is very small and when the storage with the disk without the vm configs failed the vm is still running. when the disk with config fails i create in 2 minutes a new one with existing disk and power on the vm.

ok, the spof is still the placement of the vm configs like vmx etc. might be i could do the following:

1) create vm with 2 disk for mirroring on two storages.

2) with the second disk i create an other vm with only one disk (the second form the raid-system) in the place where the second disk reside.

when the primary system fails i can click on the reserved vm and start the second disk.

i have to less expirience with windows to say anything about this. is it possible to create a new vm with the disk and the windows would boot?

jeff

0 Kudos

I am testing this as we speak on windows Smiley Happy

I created a VM of W2K3 server, one 6Gb drive on local SCSI and another 6Gbon iSCSI. The RAID1 set is now syncing. When done, I plan to kill the iSCSI connection and see what happens.

W2K3 is smart enough to alter the boot.ini file, so that the mirror disk is available as a bootdisk (in case the primary disk fails). Post my results here in about an hour Smiley Happy

Visit my blog at http://www.vmdamentals.com
0 Kudos
Texiwill
Leadership
Leadership

Hello,

Linux VMs can do this using the built in software raid tools which are fairly transparent and work like a charm.

Windows has the same functionality using DuoCor System Guardian tools. Which copies files from disk to disk at predetermined times.

Performance will depend on the speed of the processor, disk, etc. How often the mirroring runs, etc.

However, while you can mirror like this (I do not do this actually), you can also do your backups to a local VMFS as well as a remote location and just make it a part of your backup strategy. You could for example run a SAN to local VMFS backup every 12 hours using Service Console based VCB (copies are made within a single ESX server). Then from Local to remote storage using your specific agents every 24 hours.

To use mirroring, service console based VCB, scheduled snapshots within VC, or other backup tools is more a business need and depends solely on how risk averse each individual VM is. I would mirror on a per VM basis.

Best regards,

Edward

Best regards,

Edward

--
Edward L. Haletky
vExpert XIV: 2009-2022,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos

Hi Edward,

Surely these things can be done. However, all these replication options are far from synchronous. In case of a SAN crash you will have to live with data loss. In the setup where you use software RAID, you would not loose a single byte of data, and have zero downtime!

btw Windows also has this software RAID1 option when you convert your disks to dynamic disks. I am running this as a test right now, cannot wait to kill one of the VMDK sources after sync is done >:)

Visit my blog at http://www.vmdamentals.com
0 Kudos

First results are that it does not work flawless. After the sync I killed the iSCSI connection. The VM can still be pinged, but both RDP and the console freeze (unresponsive). When I bring the iSCSI connection back up, windows comes back alive and reports that "a disk in a redundant array is bad", and restarts to sync the array... Now I have brought the iSCSI connection down again, freezing the VM. Hopefully it will detect the problem and stop looking at the broken down disk.... Keep u posted.

Visit my blog at http://www.vmdamentals.com
0 Kudos
Texiwill
Leadership
Leadership

Hello Erik,

Absolutely these things can be done. Technology makes it possible.

But would you want to do this for every VM or only your most important VMs? That is the business need. Can you live with 'some' data loss or not. I think that is always the question. Consider a LUN to LUN snapshot if for some reason you need to restore that, you then have some data loss, all back ups are like that. Mirroring would need a huge data store local to the VM as well as remotely.

Take my fileserver, it is not huge, only 250GBs of used storage. Most of it static but some things change fairly rapidly. I would need at least 300GBs of local storage to mirror this server and contain a valid copy of ESX. So we make backups instead of mirroring this server. Would we mirror it if had infinite resources, you betcha. Smiley Happy It is of course running Linux.

Linux with software raid, no issues. As for windows can the C: drive be made dynamic, I never could get it to be.... Hence why I use System Guardian whenever I need that functionality.

Best regards,

Edward

--
Edward L. Haletky
vExpert XIV: 2009-2022,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos

Hi, the question is "why would you ever do this". The answer off course is, "because you CAN !!!" Smiley Wink

Not sure if this is always that you want, but it could work for perhaps a few VMs, not neccesarily for all VMs.

converting C: to a dynamic disk is no hassle, rightclick, "convert to dynamic disk", next, next, reboot. And you're all set Smiley Happy

Problem I see now, is ESX based (I think). The machine simply freezes, I guess that ESX does not report anthing back through the "failing" SCSI channel. So I might give two RDM's a go in this setup....??!!

Visit my blog at http://www.vmdamentals.com
0 Kudos
Texiwill
Leadership
Leadership

Hello,

Well that answers the dynamic disk question. I have a SAN, I may try this for that instead of iSCSI.... This does have interesting possibilities... Perhaps there is a iSCSI limitation, what about with NFS?

Best regards,

Edward

--
Edward L. Haletky
vExpert XIV: 2009-2022,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos

I am pretty sure the problem lies within the LSIlogic virtual SCSI card. My guess is that it does not handle SCSI commands correctly anymore when I kill the iSCSI target.

This does not change when I use an RDM as the 2nd disk. So keeping all in sync will work, however some manual actions will have to be done when one of both storages fail (like taking down the VM, disconnecting the failing datasource, reboot).

If I get it working better than this (that is, automagically without any manual actions) I will post it in the forum.

Visit my blog at http://www.vmdamentals.com
0 Kudos

Using an RDM \*DOES* work! Just have to be patient. Now trying to use two RDMs. Seems to be the only way to get a VM to continue to respond if one of the datasources fail... I'll check the time it takes for a VM to recover from the blow as well Smiley Happy

Message was edited by:

Erik Zandboer

Visit my blog at http://www.vmdamentals.com
0 Kudos

I have now got the software RAID1 working on normal (not specifically RDM) VMFS disks! In order to have the vSCSI controller handle storage failure correctly you have to add to the vm-name.vmx file the following line:

scsi0.returnBusyOnNoConnectStatus = "FALSE"

(replace scsi0 with the number of your virtual scsi controller)

When a datasource is removed/breaks down (tested with iSCSI), it takes around 10 seconds (W2K3) for the VM to notice the problem and continue to function without interruption utilizing the remaining datasource!

Visit my blog at http://www.vmdamentals.com
0 Kudos