VMware Communities: Message List - NFS Failover issue

Re: NFS Failover issue

andriven — Wed, 16 Sep 2009 04:17:39 GMT

Sure...no problem....

#1 - 3.5 Update 4 as well.

#2 - yes, just the settings recommended by NetApp in TR-3428 (also put in automatically by installing the free NetApp Host Utilities for ESX as well).

#3 - no, no jumbo frames....and surprisingly no real issues around that at all.

#4 - via GUI I'm afraid....happy to review any CLI commands you're using though.

Re: NFS Failover issue

drobilla — Mon, 31 Aug 2009 21:13:05 GMT

Hi andriven,

Great news! I'm still in the process of building the ESX over NFS setup here. It's nice to know that everything on your side is going as planned.

If you don't mind, I'd like to ask a few questions.

a) Which version of ESX are you using? I'm working with ESX 3.5 update 4.

b) Have changed anything in the NFS Advanced Setting?

c) Have you enabled Jumbo Frames?

e) Are you configuring your ESX machines via CLI? If so, then I'd like to share with you how I do things. This way we can compare/share ideas.

Thanks,

David

Re: NFS Failover issue

andriven — Sat, 29 Aug 2009 05:12:18 GMT

Incidentally, I'm not sure if you're still having issues with this...but I can state from personal experience that NFS can be VERY resilient for cluster and/or switch failover (just did a NetApp ONTap upgrade last weekend (which involves cluster failover) to an 8 ESX host HA/DRS cluster connecting to a NetApp 3050 via NFS....zero issues with VM's).

The thread here is a little bit old so I won't go into anything deep technically...but just wanted to put out there that from my experience there's nothing technically/conceptually intractable about using NFS with VMware/NetApp.

Re: NFS Failover issue

drobilla — Thu, 30 Jul 2009 18:00:23 GMT

@michael12345

Hello Micheal,

I'm not sure it would help you, but I figure it can't hurt. Take a look at this post http://virtualgeek.typepad.com/virtual_geek/2009/06/a-multivendor-post-to-help-our-mutual-nfs-customers-using-vmware.html

They actually talk of your issue in the comments.

There's also a few other good posts related with VMware ESX and NFS.

ESX Server, IP Storage, and Jumbo Frames

Setting VMware ESX vSwitch Load Balancing Policy via CLI

ESX Server, NIC Teaming, and VLAN Trunking

Good luck,

David

Re: NFS Failover issue

michael12345 — Fri, 15 May 2009 15:24:14 GMT

No i am still working on this problem and how to over come the short falls of NFS. Some days i wish i had just done Fiber ... the added cost would be worth the less problems at this point.

Re: NFS Failover issue

pgifford — Fri, 15 May 2009 12:36:36 GMT

@JMorz

Did you ever get an answer/resolution for this?

Re: NFS Failover issue

michael12345 — Fri, 15 May 2009 05:40:33 GMT

Some thing to think about in this problem is that the Netapp is not true Active/Active yes it will share Nvram but it will not share aggregate's so if you have VM's talking to Filer 0 and you drop the network path to filer 0 the system will not be able to talk to the aggregate(Lun/NFS Volume) . The Netapp will change aggregate owners only if the second Filer can see a problem.

If a request for a file comes to Filer 1 it will hand off the request to Filer 0. If Filer 0 cant talk to you thats the end of that request it will not send I/O from aggregates it owns using Filer 1.

I would still use fail over even with LACP you should have nics bonded with LACP and then a second set of nics on a second switch bonded with LACP using fail over and spaning tree blocking this way if you have a switch fail you have a second link source. Also not sure how NFS handles dropped packets.

If something has changed on the NetApp please let me know but this is how i understand it to work on my NetApp 3070 thanks.

Re: NFS Failover issue

JMorz — Mon, 09 Mar 2009 10:12:27 GMT

Hi,

In answer to your queries got the following responses back from our storage group:

Do you have the latest Netapp Host utilities installed on your hosts?

Not sure of the answer to this one, will check with my colleague in the storage team with regards this one again

Have you verfied that the cluster failover on the filer is working correctly?

We have not had a filer failover yet As we are active/active and in a switch failure only one link in the team fails the filer does NOT failover. With LACP all the traffic going down the previously active link should be re transmitted down the remaining active link.

Which cfmode are you using?

Negotiated failover enabled (network_interface).

What version os DOT are you running?

7.2.4

Can you please post your VIF configurations?

SANsideA: 2 links, transmit 'IP Load balancing', VIF Type 'lacp' fail 'default'

VIF Status Up Addr_set

up:

e2a: state up, since 22Feb2009 20:09:10 (14+13:25:28)

mediatype: auto-10g_sr-fd-up

flags: enabled

active aggr, aggr port: e2b

input packets 5621928542, input bytes 4610125046907

input lacp packets 45337, output lacp packets 41936

output packets 6658048095, output bytes 7300071603521

up indications 2, broken indications 0

drops (if) 0, drops (link) 0

indication: up at 22Feb2009 20:09:10

consecutive 0, transitions 2

e2b: state up, since 22Feb2009 20:09:10 (14+13:25:28)

mediatype: auto-10g_sr-fd-up

flags: enabled

active aggr, aggr port: e2b

input packets 6394679077, input bytes 4682641181916

input lacp packets 45326, output lacp packets 41936

output packets 6694968077, output bytes 7436061192087

up indications 2, broken indications 0

drops (if) 0, drops (link) 0

indication: up at 22Feb2009 20:09:10

consecutive 0, transitions 2

Many thanks!

John

Re: NFS Failover issue

pgifford — Fri, 06 Mar 2009 21:42:48 GMT

Hi,

Do you have the latest Netapp Host utilities installed on your hosts?

Have you verfied that the cluster failover on the filer is working correctly?

Which cfmode are you using?

What version os DOT are you running?

Can you please post your VIF configurations?

Thanks

Paul

NFS Failover issue

JMorz — Thu, 05 Mar 2009 15:19:56 GMT

Hi folks,

Wondering if anyone on the boards has experienced this particular issue, a quick run-through of our environment:

7 node cluster running VirtualCenter 2.5 Update 3

Hardware 7 x DL585 G2's(BIOS up-to-date)

Storage: NetApp FAS3070c - NFS mounts used for storage

Each host running ESX 3.5 Update 3 (4 critical patches added-on)

150 virtual machines running

5 vSwitches per host (each with 2 pNICs patched to 2 separate physical network switches (2 x Catalyst 6509)

vSwitch Configuration

Load Balancing: Route based on the Originating Virtual Port ID

Network Failover Detection: Link Status only

Notify Switches: Yes

Failback: Yes

1 vSwitch SC

1 vSwitch VMotion (private VLAN)

1 vSwitch VM Network

1 vSwitch NFS (separate VLAN)

1 vSwitch VM Network (redundant)

Have run repeated physical cable checks to ensure the vmnics are patched properly, all check out and running in their proper VLANs)

HA/DRS/VMotion all running fine.

Logged a ticket with VMware to verify our storage configuration was fine (included settings made from Netapp Best Practices guide for ESX) - confirmed running best practices.

VM's generally running fine (no disk errors/reported)

Issue:

When performing a single network switch outage in around a quarter of the VM's lose access to their VMDKS(effectively if you go on the console of the VM it will display a PXE boot message in DOS). Failover of traffic from one switch to the other can take over 15 minutes on average.

Now to replicate the issue both the "active" NFS vswitch vmnic and the 10gb Fibre connection running from the physical switch to the netapp filer from one physical switch need to be unplugged - the issue does not occur if just one is unplugged.

Tried:

Setting Failback to No

Active/Standby for NFS vswitch vmnics

Made no difference.

Have tested in a lab environment using a single ESX host and 30 dummy VM's

See the VM's hang for roughly 2 minutes before returning back to life - in the event logs a series of disk errors (symmpi) will be reported during the hang period.

Our networking group have confirmed Portfast is enabled on the ports and port security is disabled.

Any suggestions gratefully received - forgive the rather lengthy submission!