VMware Communities: Message List - Xtravirt Virtual SAN Appliance

Re: Xtravirt Virtual SAN Appliance

Alexinfo — Tue, 20 Jan 2009 11:38:07 GMT

Hi !

Currently, I perform some test on XVS appliance powered by VMware ESX 3.5. The nodes are configured in dual-primary mode, and there is no problem when I have to shutdown properly one of them. Sync Ok !

But when I simulate a network problem (disconnect the NIC) or when I
made a snapshot of my XVS VM, the both node can't resync' and still
detected a Split-brain.

--root@lvs-node1 ~--# cat /proc/drbd
version: 8.2.5 (api:88/proto:86-88)
GIT-hash: 9faf052fdae5ef0c61b4d03890e2d2eab550610c build by root@lvs-node2.xtravirt.com, 2008-04-14 16:57:48

0: cs:StandAlone st:Primary/Unknown ds:UpToDate/Outdated r---*

ns:0 nr:0 dw:20970844 dr:20970844 al:0 bm:2560 lo:0 pe:0 ua:0 ap:0
resync: used:0/31 hits:2618796 misses:2560 starving:0 dirty:0 changed:2560
act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0

--root@lvs-node2 ~--# cat /proc/drbd
version: 8.2.5 (api:88/proto:86-88)
GIT-hash: 9faf052fdae5ef0c61b4d03890e2d2eab550610c build by root@lvs-node2.xtravirt.com, 2008-04-14 16:57:48

0: cs:StandAlone st:Primary/Unknown ds:UpToDate/Outdated r---*

This is my drbd.conf:

~~root@lvs-node1 ~~~# cat /etc/drbd.conf
global {
+ usage-count no;+
}

common {
+ syncer {+
+ rate 100M;+
+ }+

+ handlers {+
+
outdate-peer "/usr/lib/heartbeat/outdate-peer.sh";+
+
split-brain "/usr/lib/drbd/outdate-peer.sh";+
+ }+
}

resource vmfs-0 {
+ protocol C;+

+ startup {+
+ become-primary-on both;+
+ degr-wfc-timeout 120;+
+ }+

+ net {+
+ after-sb-0pri discard-zero-changes;+
+ after-sb-1pri consensus;+
+ after-sb-2pri disconnect;+
+ allow-two-primaries;+
+ }+

+ disk {+
+ on-io-error pass_on;+
# fencing resource-only;
+ }+

+ on lvs-node1.xtravirt.com {+
+
device /dev/drbd0;+
+
disk /dev/sdb;+
+
address 10.1.14.130:7788;+
+ meta-disk internal;+
+ }+

+ on lvs-node2.xtravirt.com {+
+
device /dev/drbd0;+
+
disk /dev/sdb;+
+
address 10.1.14.132:7788;+
+ meta-disk internal;+
+ }+
}

I try all of the basics commands from drbdadm, like
connect all, up all, outdate all, invalidate all but still failed with
exiting code or it say as the device is held open by someone it can't
change state... How can I resolve Split Brain automatically? I want that the both nodes will come back in a dual-primary mode, once the split brain is resolved.

Best Regards!

Re: Xtravirt Virtual SAN Appliance

TechFan — Mon, 22 Dec 2008 09:33:23 GMT

I was just wondering if anyone had any tips on how to upgrade the VMWare Tools inside the XVS?

Re: Xtravirt Virtual SAN Appliance

TechFan — Thu, 11 Dec 2008 15:44:45 GMT

FYI. I got it. I was trying to see if I could manually set the disk state with dstate (if that was what it even meant) and I found this link:

http://www.drbd.org/users-guide/s-manual-split-brain-recovery.html

The key commands are:

drbdadm secondary resource
drbdadm -- --discard-my-data connect resource

In our case resource=all. There is a bit more info as well, so read it if any of the rest of you are stuck.

Following its one extra step fixed it and allowed it to only resync a tiny portion of the 250GB volume, so it only took a short amount of time in comparison. Thanks for your help.

Message was edited by: TechFan
fixed formatting

Re: Xtravirt Virtual SAN Appliance

TechFan — Thu, 11 Dec 2008 15:26:17 GMT

Lol. It is pretty obvious I haven't done this drbd stuff before. Unfortunately, the setup doesn't want to co-operate still.

I started the service on both sides. I then tried to do drbdadm up all on the primary. It complains that it is attached to a disk (Failure: 124) Device is attached to disk (use detach first)....it echos the command it is trying to run. . .then says terminated with exit code 10. So, I then try drbdadm primary all. It fails again with that message in my previous post. So, I tried detach all, then disconnect all. . .then I tried up all again. . .says disk is still attached, so then detach all, then up all again. . .no errors this time. . .now primary all. . .it gives me the exact same error it did in my previous post again. . .

So, stuck. It thinks it is diskless. . .and has a disk at the same time. Not sure how that works. . .thanks for the tips.

Re: Xtravirt Virtual SAN Appliance

mittell — Thu, 11 Dec 2008 13:30:40 GMT

"drbdadm up all" on the primary first.

Tip, if you do "cat /proc/drbd" it'll give you current status, if you see diskless do an drbdadm attach all then drbdadm connect all (NOTE: drbdadm up all does both actions in one command.)

Re: Xtravirt Virtual SAN Appliance

TechFan — Tue, 09 Dec 2008 22:21:42 GMT

I might have to that earlier, but I started the service first. . .it started and now the file is found. I did the service start command and that made progress, but the one I want to be primary won't accept the drbdadm primary all. The secondary takes its commands properly, but the primary complains about dbbd0: Not outdating peer, since I am diskless.<4> State change failed: (-2) Refusing to be primary without at least one UpToDate disk. Command 'drbdsetup /dev/drbd0 primary' terminated with exit code 11.

Re: Xtravirt Virtual SAN Appliance

mittell — Tue, 09 Dec 2008 18:02:42 GMT

Sounds like the DRBD service hasn't auto-started. If you cat /proc/drbd do
you get a file not found error?

Try a "service drbd start" then those other drbdadm commands.

Alex

Re: Xtravirt Virtual SAN Appliance

TechFan — Tue, 09 Dec 2008 17:45:15 GMT

Ok. I had to shutdown both ESX servers tonight. . .and I even shutdown both sides of the XVS setup. After restarting the servers. . .they again will not sync. . .I tried the outdate all. . .didn't work either. I thought I would write you before I go invalidating all. . .

When I try to run any of the following commands, I get an error (No response from the DRBD driver! Is the module loaded?...terminated with exit code 20):

drbdadm primary all
drbdadm secondary all
drbdadm connect all

this one doesn't complain:

drbdadm invalidate all

Any advice. . .ideas?

Re: Xtravirt Virtual SAN Appliance

TechFan — Thu, 04 Dec 2008 05:31:37 GMT

Thanks again for the additional info. I will play with it more, but the recovery time in case of failure is what has kept me from deciding to deploy anything live on my XVS setup so far and caused me to decrease the volume to 250GB instead of 500GB.

O/T: I also noticed the email reply's are very delayed. . .and that your reply I got via email notification wasn't on the forum thread for a while. . .neither was my reply. . .lol

Again, I really appreciate all you have done with this, it is a great tool.

Re: Xtravirt Virtual SAN Appliance

glazgb — Thu, 04 Dec 2008 04:58:13 GMT

mittel

crrashes for any reason - then only an incremental sync is necessary. DRBD maintains meta-data so this process is automatic, all you need to do is when the crashed node comes back up type "drbdadm connect all" and it should begin the resync, when it's complete type "drbdadm primary all" "service heartbeat start" wait 2 mins "service iscsi-target start" and everything should be up and running again. If the initial drbdadm command doesn't work, try a "drbdadm outdate all" (instead of invalidate, which causes full re-sync) on the node then try again. I've tried this out many times with hard powering off a node and DRBD does an incremental resync successfully every time. The "start services" menu option should automate these commands - but it seems there may be a bug in there that is preventing it from happening so I suggest using the command line.

How made this automatical in start XVS ?
in /etc/RC3.D ?

Re: Xtravirt Virtual SAN Appliance

dilidolo — Wed, 03 Dec 2008 20:59:08 GMT

So this is basically IET + Heartbeat + DRBD?

Good for someone without dedicated ISCSI device for testing.

Re: Xtravirt Virtual SAN Appliance

mittell — Wed, 03 Dec 2008 15:36:06 GMT

deleted duplicate post

Message was edited by: mittell

Re: Xtravirt Virtual SAN Appliance

mittell — Wed, 03 Dec 2008 15:44:07 GMT

Ugh I remember why I gave up using these forums now, the software is so bad.

I typed a long reply to this twice now only to get it lost with an HTTP 500 Internal Server Error, and replies don't seem to be showing up to the threads properly, half the time rvsharpes post is the last in the thread when there are two more replies!

Anyway, for the thirds time...

You only need to do a full re-sync on a split-brain, which should only happen if you lose the network connectivity between the two nodes when they are both live. If one of the nodes, or the host it is on, crrashes for any reason - then only an incremental sync is necessary. DRBD maintains meta-data so this process is automatic, all you need to do is when the crashed node comes back up type "drbdadm connect all" and it should begin the resync, when it's complete type "drbdadm primary all" "service heartbeat start" wait 2 mins "service iscsi-target start" and everything should be up and running again. If the initial drbdadm command doesn't work, try a "drbdadm outdate all" (instead of invalidate, which causes full re-sync) on the node then try again. I've tried this out many times with hard powering off a node and DRBD does an incremental resync successfully every time. The "start services" menu option should automate these commands - but it seems there may be a bug in there that is preventing it from happening so I suggest using the command line.

(select all copy - damn forum)

Re: Xtravirt Virtual SAN Appliance

TechFan — Wed, 03 Dec 2008 14:06:23 GMT

Alex,

Thanks for the response. I figured there wasn't anything more happening
on this when I heard xtravirt was bought out. That is unfortunate but
understandable.

So, basically what you are saying is that the only way to recover is to
completely resync. . .so, resync 250GB if that is the size of the space?
Not much different than just deleting a virtual disk on one side and
resyncing. So, if both hosts go down, then it is likely to be a while
before you can get them back up. . .good reason not to use it in
production. We were just about to start using it as one type of
storage. . .NFS, Local, and VSAN mixed as needed.

Thanks again for the great tool.

Re: Xtravirt Virtual SAN Appliance

mittell — Wed, 03 Dec 2008 13:20:20 GMT

Hi Guys,

Afraid I've currently ceased development of the XVS due to time constraints, so I'd suggest that it is only used for dev/testing unless you have a sound knowledge of Linux/DRBD/Heartbeat.

The resolution for split-brain is to go to invalidate one of the nodes, if you google for drbd split-brain recovery you'll find a lot of information.

Basically you just need to select one node and force it to overwrite the other, the command for this would be something like:

On the master:

drbdadm primary all

On the node to be overwritten

drbdadm secondary all
drbdadm invalidate all
drbdadm connect all

On the master:

drbdadm connect all

This should then cause a full re-sync, be careful about doing it the correct way around!

When the sync is done, do the following on the overwritten node.

drbdadm primary all
service heartbeat start
<wait 2 mins>
service iscsi-target start

If anyone needs any help/advice I'm happy to advise on a best-effort basis, either PM me on here or email me at support (at) xtravirt.com.

Cheers,

Alex

Re: Xtravirt Virtual SAN Appliance

rvsharpe — Wed, 03 Dec 2008 05:17:13 GMT

I get the split brain situation with or without a crossover cable. By the way, can you explain the resolution to resolve the split brain situation?

Ramon

Re: Xtravirt Virtual SAN Appliance

glazgb — Tue, 25 Nov 2008 05:19:03 GMT

Hi!
in documentation "XVS Installation Guide.doc" say:

2.6 Recovering from an XVS appliance node shutdown or failure
If one of the nodes is lost for any reason, including ESX host failure, the other node will take over all storage operations and the volume will continue to function. When the node is restored to operation it is necessary to re-sync the volume to resume full failover capability of the SAN.
Power on the XVS node that was disconnected and allow it to boot. Select Option 2 at the Main Menu to enter the Services Menu.

At the Services Menu select Option 1 to start the XVS services.

Assuming network connectivity between the nodes is functional, you should be presented with a resynchronisation progress screen. This will automatically exit when resynchronisation is complete.

How do "automatical resynchronisation" after power up ?

Re: Xtravirt Virtual SAN Appliance

mfoley@prainc.com — Thu, 13 Nov 2008 21:50:36 GMT

I get the split brain situtaion as well and am using a cross-over cable. I don't see how this coul dbe the problem but has anyone else got a split brain when they didn't use the crossover?

I contacted xtravirt for support and they helped me fix th eproblem but I'd like it to be more reliable...and I wouldn't mind paying for commercial support.

Re: Xtravirt Virtual SAN Appliance

TechFan — Wed, 12 Nov 2008 00:41:25 GMT

Is there any way to get these to be thin provisioned disks. . .like replacing the empty disk after initial syncronization? I am finding that one disk is staying thin, but the other is fully expanded.

I actually have only had that resync issue once so far. All my other power downs and shutdowns have come back online. That first one, I just added a new disk and did resyncronization.

What I really want to know is how to force mount a single part, so I can provide access to the data until a planned support window to resyncronize. . .I'll have to write them.

Re: Xtravirt Virtual SAN Appliance

rvsharpe — Tue, 11 Nov 2008 05:46:10 GMT

Techfan,

I have run into the issue of not synching even after a "graceful"
shutdown...i.e. shutting down the Xtravirt services on one of the nodes
for instance. According to the Xtravirt folks, the application should
failover to the other san appliance but I have yet to experience a
successful failover. I ususlly have to run the initial synchronization
on one or sometimes both of the nodes to get synchronization to start
after shutting down one of the nodes either gracefully or by pulling the
plug to simulate a disaster. I have to agree though, synchronization is
slow using this recovery method.

As far as recovering your data, I have successfully been able to access
data once the sync completes.

One good sign though is that I did receive a response from Xtravirt when
I left an email using the support link on their website. The contact
there instructed me to send logs and even offered to look at my servers
remotely. Unfortunately I have not had the chance to continue to trouble
shoot..I eventually moved my VM's back to a Network SAN device for now.
I suggest that you send Xtravirt support an email regarding this issue.

In closing, I removed the application with the intention of reinstalling
shortly. Let me know how you make out.

Ramon

Re: Xtravirt Virtual SAN Appliance

TechFan — Mon, 10 Nov 2008 04:13:25 GMT

Another question I just thought of. . .does this Virtual Appliance have jumbo frame support on the guest OS level? Apparently, ESXi doesn't support it at the kernel level, only for guest OS's directly. Could make a big difference in sync speeds. . .

I should also ask here if anyone has seen a similiar appliance providing syncronized NFS accessed data stores. . .

Re: Xtravirt Virtual SAN Appliance

TechFan — Mon, 10 Nov 2008 00:45:13 GMT

Ok. So that explains why it wasn't taking the IP address I was trying to use. . .it has a 0. We have been testing with the default IP configuration and it was working okay until I updated one of the test ESX server and restarted it. . .and didn't stop the services first. Now they both seem to "sync" forever. . .never finish.

Is there any way to get to the data again or is it all lost? That is a bit scary for real use if we lose complete access because one node goes down uncleanly. . .

I don't seen any updated releases. . .

Ok, I am not sure if this is the only way, but I deleted the VMDK on the 2ndary and created a new one, then set them to do an initial sync. That is going to be a very long restore process if we have a full 400GB in production. . .I hope there is another way to get them back in sync.

Re: Xtravirt Virtual SAN Appliance

rvsharpe — Wed, 10 Sep 2008 07:01:58 GMT

Never mind...I just changed the network to a 192.168.1.x network. Installation was fine after that. As stated previously in this thread, the application seems to have problems when resyncing after rebooting or shutting down and re-starting one of the nodes..neither node will communicate with each other and a reconfiguration is needed, at least in my case..

Re: Xtravirt Virtual SAN Appliance

rvsharpe — Fri, 05 Sep 2008 09:27:19 GMT

Is there any update regarding using Xtravirt with IP addresses with a "0" octet?

rvs

Re: Xtravirt Virtual SAN Appliance

jgal — Fri, 08 Aug 2008 21:12:46 GMT

Jrink you mind posting some of those emails. We are having a problem using cross over cables as well. After the initial setup it takes a few reboots or to wait some time until we can start the first sync

If we try to sync after configure it never connects. I do rember reading something in the logs that says it didnt want to be the primary. We have full connectivity and the only thing that fails is the ntp sync since its on a crossover network between 2 host.

Re: Xtravirt Virtual SAN Appliance

JRink — Thu, 17 Jul 2008 17:30:30 GMT

Alex,

I'm not sure how it could be a network connection issue when the 2 hosts are connected via crossover cable. ??

I will take some time tonight to go over your email replies more thoroughly and get back to you with more information.

Re: Xtravirt Virtual SAN Appliance

mittell — Thu, 17 Jul 2008 13:23:20 GMT

Sorry for the delay replying guys, been on holiday in sunny Wales (har har).

JRink - replied to all your emails, looks like a network connection issue causing the nodes to go offline to prevent split-brain.

SimonG - the node being in standalone happens for one of two reasons, the network connection cannot be established or both nodes think they hold consistent copies of th storage so refuse to connect (again to prevent split brain). How did you shut down the appliance on the second ESX server? Did you shut down the services as per the documentation, or just pull the plug?

If you could email the contents of /var/log/mesages from the XVS nodes to support@xtravirt.com I'll take a look and try and work out what's going on.

Cheers,

Alex

Re: Xtravirt Virtual SAN Appliance

JRink — Mon, 14 Jul 2008 14:06:53 GMT

Not that I'm aware of. I sent their support a couple of emails about 10 days ago with lots of information, but haven't heard back at all.

Re: Xtravirt Virtual SAN Appliance

pauldavidbarker — Mon, 14 Jul 2008 10:28:24 GMT

I also get this problem has then been an update I missed

Re: Xtravirt Virtual SAN Appliance

SimonG-ioko — Sat, 05 Jul 2008 08:40:48 GMT

I also get the same problem, if I reboot the 2nd ESX box (or even just the SAN VM) it won't come back up (pic attached)

Let me know if/what further info you need?

Cheers

Re: Xtravirt Virtual SAN Appliance

JRink — Fri, 04 Jul 2008 07:17:55 GMT

Sent a couple emails tonight per your request...

Re: Xtravirt Virtual SAN Appliance

mittell — Tue, 01 Jul 2008 06:22:12 GMT

Hi JR,

JRink wrote:

I just figured out the problem. The VM does NOT allow you to enter IP addresses that have "0" in an octet.

I have seen this problem before on a UPS unit's SNMP management card. You couldn't configure the IP if it had a "0" octet. Likely a programming issue.

Anyways, I changed my 10.0.0.x network to a 10.1.1.x network and it finally took the values.

I can confirm this is a bug. I'll get it fixed shortly and put out a new release.

JRink wrote:

More issues with this....

It seems like when rebooting either of my ESX hosts, whether sometimes doing clean shutdowns or simulating a server crash by pulling the plug, the 2 LVS nodes do NOT want to communicate anyone and do not even offer up iSCSI. When I try started the LVS nodes, the iSCSI storage never comes online. Furthermore, going into the LVS node menu and selecting Start Node never ends up synchronizing with the other node. The only thing it says is

" 0: cs:standalone st: secondary/unknown ds:consistent/dunkown r----

ns:0 nr:0 dw:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0

resync: use: 0/31 hits:0 misses:0 starving:0 dirty:0 changed:0

act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0

BOTH nodes are showingthe identical thing. Again, the iSCSI storage is not online.... The only way I can seem to fix this is to reset both nodes and reconfigure both nodes (I can't find a way OUT of that screen when it's trying to sync... ctrl-c, etc. nothing worked to get back to the menu...).

ON a side note, aside from THIS issue, I am having problems wiht the sole VM that is running on Node1. It's a Win2003 VM and is constantly crashing and becoming unpingable for no apparant reason.....

Is there ANY support available for this? I know it's free.. but I would like to introduce this to a couple clients, however I can't really do so if I'm having problems even getting this up and running.

Please advise. a bit frustrated with this...

Sorry for the delay spotting your posts, and some issues getting the XVS to work. Standalone mode is a state where the nodes are refusing to speak to eachother for some reason, usually a split-brain scenario. This seems likely because both nodes believe they are in a consistent state so you end up with a dead-lock. The reason the iSCSI storage won't come up is because in secondary mode the storage is read-only. You'll need to run some commands at the console to fix this state.

I've not seen these issues in my dev setup and have been using the XVS to host lots of test Windows VMs for some time without issue, so I'm sure we can fix the problems. Can you send me some details of your setup, the hardware you're running, exactly how you failed the node, etc, to support@xtravirt.com? I'll do my best to try and get you up and running.

Cheers,

Alex

Re: Xtravirt Virtual SAN Appliance

JRink — Tue, 01 Jul 2008 02:51:56 GMT

More issues with this....

" 0: cs:standalone st: secondary/unknown ds:consistent/dunkown r----

ns:0 nr:0 dw:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0

resync: use: 0/31 hits:0 misses:0 starving:0 dirty:0 changed:0

act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0

ON a side note, aside from THIS issue, I am having problems wiht the sole VM that is running on Node1. It's a Win2003 VM and is constantly crashing and becoming unpingable for no apparant reason.....

Please advise. a bit frustrated with this...

Re: Xtravirt Virtual SAN Appliance

JRink — Sat, 28 Jun 2008 00:56:10 GMT

I just figured out the problem. The VM does NOT allow you to enter IP addresses that have "0" in an octet.

I have seen this problem before on a UPS unit's SNMP management card. You couldn't configure the IP if it had a "0" octet. Likely a programming issue.

Anyways, I changed my 10.0.0.x network to a 10.1.1.x network and it finally took the values.

Re: Xtravirt Virtual SAN Appliance

JRink — Sat, 28 Jun 2008 00:21:32 GMT

I can't seem to figure this out. I'm trying to test this but am running into a problem when I try to get my VM a heartbeat IP address. I am trying to use 10.0.0.5 for the IP address but it says, "Invalid Option" when I do this.

I'm working on test boxes, so they aren't configured exactly as I like but... my configure is with a single NIC in each ESX host.

My 1st ESX server has a single vSwitch0 which has ServiceConsole1 at 192.168.1.5/24, ServiceConsole2 at 10.0.0.11/8, VMKernel at 10.0.0.1/8.
My 2nd ESX server has a single vSwitch0 which has ServiceConsole1 ata 192.168.1.6/24, ServiceConsole2 at 10.0.0.12/8, VMKernel at 10.0.0.2/8.

Why is it erroring when I try to set a custom IP address? If I manually enter 192.168.1.x/24 range for the IP, then it accepts it, but I really don't want to use 192.168.1.x/24 for the network. I typically set all my backend iSCSI networks up on 10.0.0.0/8.

Any advice here? Frustrated.
JR

Re: Xtravirt Virtual SAN Appliance

stormlight — Wed, 18 Jun 2008 14:28:20 GMT

Any answer to the above question from anyone who has tested this?

"Also, about the replication feature between local disks on each ESX host... If my production VM are stored in the LUN off the SAN VM, and the SAN VM is continuously replicating the data from one ESX DAS to the other ESX DAS, how does this effect performance? Wouldn't this replication also hinder the performance of my VMs? (this isn't an Extravirt question really, just a general question about this kind of technology)."

Re: Xtravirt Virtual SAN Appliance

mittell — Tue, 27 May 2008 14:13:41 GMT

Yep, that's exactly the sort of thing it's aimed at.

Re: Xtravirt Virtual SAN Appliance

JRink — Fri, 13 Jun 2008 15:36:04 GMT

I'm trying to understanding how this works... Couple questions.

If I have my ESX host with a 72GB RAID 1 volume (hosting ESX OS and a small Datastore1) and a 280GB RAID 5 volume, I'm assuming I would want to setup the SAN VM on my small Datastore1. I'd then create an additional hard disk for the SAN VM of, for example, 250GB. I'd want this 250GB hard disk to be located on my RAID5 volume. But, I'm not sure I understand this correctly when I read your MS Word document (it mentions using the "Store with virtual machine" option when adding the 2nd disk). I think I'm a bit confused as I never setup a VM with indepedent disks either and maybe that's where I'm confused. Trying to grasp this...

Also, about the replication feature between local disks on each ESX host... If my production VM are stored in the LUN off the SAN VM, and the SAN VM is continuously replicating the data from one ESX DAS to the other ESX DAS, how does this effect performance? Wouldn't this replication also hinder the performance of my VMs? (this isn't an Extravirt question really, just a general question about this kind of technology).

JR

Re: Xtravirt Virtual SAN Appliance

tom howarth — Wed, 11 Jun 2008 07:00:08 GMT

Thread moved to a more appropiate forum

Tom Howarth
VMware Communities User Moderator

Re: Xtravirt Virtual SAN Appliance

roc.ker — Tue, 10 Jun 2008 18:34:09 GMT

I send you email about support. I will test it once more. Thank you.

Re: Xtravirt Virtual SAN Appliance

mittell — Tue, 10 Jun 2008 16:22:58 GMT

I've tested this and it worked fine for me, if you send me an email with your config/some screenshots I'll try and figure out what's going wrong.

Regards,

Alex

Re: Xtravirt Virtual SAN Appliance

roc.ker — Tue, 10 Jun 2008 14:59:56 GMT

Did anybody test complete recovery of vSAN? I try import new XVS appliance, add fresh disk, but can´t sync data from working node to new installed??

Thanks

Re: Xtravirt Virtual SAN Appliance

mittell — Tue, 10 Jun 2008 14:32:05 GMT

We do offer commercial support, please see here for more details: http://www.xtravirt.com/index.php?option=com_content&task=view&id=95&Itemid=123 (or email sales@xtravirt.com).

Thanks,

Alex

Re: Xtravirt Virtual SAN Appliance

roc.ker — Tue, 10 Jun 2008 14:26:57 GMT

What about commercial usage. I think, that this is ideal solution for small bussines - two ESX servers with local storage, ESX standard + VC Foundation. HA solution for great price.

Do you offer commercial support for XVS or do you suggest this appliance only for testing purposes?

Re: Xtravirt Virtual SAN Appliance

mittell — Tue, 03 Jun 2008 15:07:33 GMT

stormlight wrote:

if you where to point a 3rd ESX host to the VSAN what ip would you point it to? The primary or the # 1 vsan?

You could point it at XVS node 1 or node 2 iSCSI target IP (the virtual IP) it doesn't really matter, if you introduced a 4th ESX host, you would want to point it at the other node from ESX3 to even the load, although it is not critical.

so what happens when #1 dies #2 picks up Vsan #1s ip? And all is good

Correct, if XVS node 1 dies XVS node2 takes over it's IP and all storage traffic continues to flow via node 2, when XVS node 1 is brought back online it's IP address fails back over after it is brought back into sync and processing is then handled by both nodes again.

Re: Xtravirt Virtual SAN Appliance

stormlight — Tue, 03 Jun 2008 15:03:15 GMT

if you where to point a 3rd ESX host to the VSAN what ip would you point it to? The primary or the # 1 vsan?

so what happens when #1 dies #2 picks up Vsan #1s ip? And all is good

Re: Xtravirt Virtual SAN Appliance

mittell — Tue, 03 Jun 2008 14:50:55 GMT

BrianRTS wrote:
Looks nice. Are there any plans to support more nodes so more servers can utilize the SAN storage?

Not in the short term, doing so requires writing a bespoke network RAID solution, a lot of work!

You can however run as many pairs of XVS appliances as you like, and there is nothing to stop you from pointing a third ESX host at a pair of XVS appliances on two other ESX hosts.

(tried to draw it in ASCII but the forum mangels it :()

Re: Xtravirt Virtual SAN Appliance

mittell — Tue, 03 Jun 2008 14:43:54 GMT

SimonG-ioko wrote:

Thanks Alex,

This appliance is a great idea... I've got a couple of posts on my blog (http://vinf.net) on how my setup works, once I nail this failover thing in my lab setup I'll post it up..

Cool, thanks, and glad you like it.

A couple of other feedback items that might be useful;

Would be good to have some kind of simple status info when it goes back to the main menu screen, even if it's just something basic like

iSCSI status: started/not started

iSCSI address(es): 192.168.x.1, 192.168.x.2 etc.

Failover/Heartbeat status: primary, lost connection etc.

Writes/connections/sec etc.

Yep that's a good idea, I'll work to include that in the next release, thanks for the input.

Cheers,

Alex

Re: Xtravirt Virtual SAN Appliance

BrianRTS — Tue, 03 Jun 2008 14:36:51 GMT

Looks nice. Are there any plans to support more nodes so more servers can utilize the SAN storage?

Re: Xtravirt Virtual SAN Appliance

SimonG-ioko — Tue, 03 Jun 2008 14:19:05 GMT

stormlight wrote:
cant answer all your questions.. i got mine above. However try a cross over cable between the two nics that this sits in. Since its only two node you can get much faster speeds then 100 mb half

Thats a good idea; hadn't thought of that - I've got 2 x HP D530 PCs doing my ESX stuff - they have GigE onboard I could prob run my iSCSI/VMKernel over a crossover on that, I have some spare 100Mb NICs that I could use to run my external VM networks into the external hub until I can find a switch.

Thanks!

Re: Xtravirt Virtual SAN Appliance

SimonG-ioko — Tue, 03 Jun 2008 14:13:06 GMT

Thanks Alex,

This appliance is a great idea... I've got a couple of posts on my blog (http://vinf.net) on how my setup works, once I nail this failover thing in my lab setup I'll post it up..

A couple of other feedback items that might be useful;

Would be good to have some kind of simple status info when it goes back to the main menu screen, even if it's just something basic like

iSCSI status: started/not started

iSCSI address(es): 192.168.x.1, 192.168.x.2 etc.

Failover/Heartbeat status: primary, lost connection etc.

Writes/connections/sec etc.

Great work!

Now I just need to find a Gig switch somewhere... hmmmm

Re: Xtravirt Virtual SAN Appliance

mittell — Tue, 03 Jun 2008 11:41:45 GMT

SimonG-ioko wrote:

I've got this setup on my home lab, vMotion etc. work ok (even over a 100mb/half hub :))

Heh I bet performance will be, erm, interesting.

I can't get my head around the failover though..

my setup

ESX 1 -> pointing at iSCSI IP on 1st SAN node running on itself

ESX 2 -> pointing at iSCSI IP on 2nd SAN node running on itself

if I reboot 2nd SAN node

does its iSCSI IP transfer to the 1st SAN as well to continue access or will ESX 2 loose access to its iSCSI LUN? - I can't really see whats going on from the console..

Yes, the 1st XVS node will assume the Virtual IP of the 2nd SAN node and ESX 2 will start recieving all of it's iSCSI data from the 2st XVS node until the 2nd XVS node is brought back online and is back in sync.

and secondly, when the 2nd SAN node comes back up and I start services it doesn't seem to do the sync and restart the iSCSI services it comes up 'outdated' - am I doing something wrong?

It should auto resync, sometimes if no data has changed the sync is so fast you don't get to see the status. If the iSCSI service started, it'll be in sync, as it won't start the service until both sides are in sync. You can check this from the troubleshooting console with "cat /proc/drbd". PM me if you need more information.

And I think you may need to mention in your docs that you need to open the firewall port for the ESX iSCSI client for to see the LUNs (or at least I did).

Thanks, that's a fair comment, I'll get the guide updated.

Cheers,

Alex

Re: Xtravirt Virtual SAN Appliance

mittell — Tue, 03 Jun 2008 11:38:57 GMT

stormlight wrote:

Thank you, this is great.

Few questions:

Im trying to wrap my head around networking and what not in general with esx and this.

Shoud I add a Virtual Machine Port Group to my iscsi SAN Vswitch and run this virtual applience on that network?

You can, or you can run it on your normal VM vSwitch - it's up to you, it depends how many NICs you have in your ESX host, if you can dedicate one to the iSCSI network great, if not then it should work anyway but GB is strongly recommended. The only requirement is that the two XVS nodes can talk to eachother.

Then point my esx host to the virtual ip that the XVS uses? If thats done when the two replicate back and fouth its using the nics that are binded to my SAN network correct?

Yes, you need an S/C and VMKernel PG with an IP in the same subnet as the XVS appliances so that they can talk to them both. So if you were using a dedicate vSwitch with a NIC bound to it, you'd need an S/C and VMKernel PG on that vSwitch.

If my physical SAN network is getting saturated is their any issues with running this in the vmotion network security wise? No one can get to our vmotion network unless they are on the esx host anyway. So the only contact would be the esx host software iscsi inatior passing data to this virtual san when it replicates.

No issue with this really, the only problem could be contention if you were writing a lot of data to the XVS at the same time as performning a vMotion. I think in reality it'll work fine.

Hope this helps.

Cheers,

Alex

Re: Xtravirt Virtual SAN Appliance

stormlight — Tue, 03 Jun 2008 10:50:06 GMT

cant answer all your questions.. i got mine above. However try a cross over cable between the two nics that this sits in. Since its only two node you can get much faster speeds then 100 mb half

also you may want to listen to the vmware virtulization podcast. The author of this is on it and talks a little about your question

Re: Xtravirt Virtual SAN Appliance

SimonG-ioko — Tue, 03 Jun 2008 09:32:07 GMT

I've got this setup on my home lab, vMotion etc. work ok (even over a 100mb/half hub :))

I can't get my head around the failover though..

my setup

ESX 1 -> pointing at iSCSI IP on 1st SAN node running on itself

ESX 2 -> pointing at iSCSI IP on 2nd SAN node running on itself

if I reboot 2nd SAN node

does its iSCSI IP transfer to the 1st SAN as well to continue access or will ESX 2 loose access to its iSCSI LUN? - I can't really see whats going on from the console..

and secondly, when the 2nd SAN node comes back up and I start services it doesn't seem to do the sync and restart the iSCSI services it comes up 'outdated' - am I doing something wrong?

And I think you may need to mention in your docs that you need to open the firewall port for the ESX iSCSI client for to see the LUNs (or at least I did).

Re: Xtravirt Virtual SAN Appliance

stormlight — Thu, 29 May 2008 16:38:47 GMT

Thank you, this is great.

Few questions:

Im trying to wrap my head around networking and what not in general with esx and this.

Shoud I add a Virtual Machine Port Group to my iscsi SAN Vswitch and run this virtual applience on that network?

Then point my esx host to the virtual ip that the XVS uses? If thats done when the two replicate back and fouth its using the nics that are binded to my SAN network correct?

thanks again

Re: Xtravirt Virtual SAN Appliance

king@it.ibm.com — Mon, 26 May 2008 14:56:15 GMT

Alex,

this might be a good scenario for your virtual appliance (?)

http://it20.info/blogs/main/archive/2008/05/19/121.aspx

Massimo.

Re: Xtravirt Virtual SAN Appliance

mittell — Thu, 22 May 2008 12:53:41 GMT

Update: there sems to be a problem with the download link to the documentation, if you experience problems please use the link below to download the docs.

http://www.xtravirt.com/index.php?option=com_remository&Itemid=75&func=startdown&id=30

Thanks,

Alex

Re: Xtravirt Virtual SAN Appliance

mittell — Wed, 21 May 2008 23:12:37 GMT

Possibly at a later date but not in the immediate future.

Does it support NFS or strictly iSCSI?

iSCSI only.

If I need to add additional disk to XVS machine, than I will go through same process to get the LUN show up or it should automatically show up when I rescan the LUNs?

Currently each pair of XVS nodes supports only one LUN, therefore if you want multiple LUNs you'll need multiple XVS appliances per host.

I like the LUN failover heartbeat feature, but want to know more details how it works and wondering any options or settings to decide which one is Active/Passive or not.

The XVS is active/active, all reads come from the local disks, all writes go to both sides.

What happen if we have multiple disks with different size for XVS appliance server?

XVS01:
disk01: XVS OS
disk02: 50GB LUN size disk creation
disk03: 100GB LUN size disk creation
disk04: 60GB LUN size disk creation

XVS02:
disk01: XVS OS
disk02: 40GB LUN size disk creation
disk03: 200GB LUN size disk creation
disk04: 80GB LUN size disk creation

How does it failover if it has different disk/LUN size?

The disk sizes on each XVS node must be the same size.

Hope this helps.

Cheers,

Alex

Re: Xtravirt Virtual SAN Appliance

azn2kew — Wed, 21 May 2008 18:31:24 GMT

Quick questions, I've just read the manual and seems very straight forward to configure and 90% are similiar with OpenFiler process. I'm curious if you going to have web base management console like OpenFiler to manage volumes and LUNs etc... Does it support NFS or strictly iSCSI? If I need to add additional disk to XVS machine, than I will go through same process to get the LUN show up or it should automatically show up when I rescan the LUNs?

I like the LUN failover heartbeat feature, but want to know more details how it works and wondering any options or settings to decide which one is Active/Passive or not. What happen if we have multiple disks with different size for XVS appliance server?

XVS01:
disk01: XVS OS
disk02: 50GB LUN size disk creation
disk03: 100GB LUN size disk creation
disk04: 60GB LUN size disk creation

XVS02:
disk01: XVS OS
disk02: 40GB LUN size disk creation
disk03: 200GB LUN size disk creation
disk04: 80GB LUN size disk creation

How does it failover if it has different disk/LUN size?

If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!!

Regards,

Stefan Nguyen
iGeek Systems Inc.
VMware, Citrix, Microsoft Consultant

Re: Xtravirt Virtual SAN Appliance

azn2kew — Wed, 21 May 2008 17:53:53 GMT

Wow, I can't wait to download and test it out. I've been using OpenFiler and FreeNAS for my alternatives but performance still in consideration. So how is performance looks like comparing to XVS and other commercial tools such as mentioned or SANMelody. This sounds like SANmelody and Lefthand's Network VSA but I could be wrong. I'll get some feedback after this weekend testings!

If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!!

Regards,

Stefan Nguyen
iGeek Systems Inc.
VMware, Citrix, Microsoft Consultant

Xtravirt Virtual SAN Appliance

mittell — Wed, 21 May 2008 15:04:25 GMT

Hi Guys,

Over at Xtrairt we've just released a new free virtual appliance that I thought people might find interesting. It lets you use the local storage on two ESX hosts to create a virtual SAN. I've attached an overview diagram to explain how it works.

http://www.xtravirt.com/index.php?option=com_remository&Itemid=75&func=fileinfo&id=29

Summary: The Xtravirt Virtual SAN (XVS) appliance for VMware ESX3 Server is a free solution to provide the benefits of shared VMFS storage without the cost of a SAN – this allows the utilisation of otherwise unused local storage in the ESX server to facilitate enterprise level features such as vMotion, DRS and HA normally only available through the use of a shared storage device. All volume data is synchronously replicated between hosts, providing full fail-over capability with data integrity in the event of host, disk or appliance failure. The appliance is menu driven and has been designed to be as easy to configure as possible, and full documentation on the implemenation process is provided.

Questions/feedback welcome in this thread.

Cheers,

Alex