NFS vs iSCSI <- which distribution to use for NFS

[nitin_m]

Though the title says as if I am asking which one is better, but I have read good things about NFS and therefore will settle for NFS to store VMDKs.

Now, my question -

Is NFS a standard implementation?(Unlike iSCSI - in which some targets donot have RESERVE/SELECT capability still they say they are iSCSI targets)

Will any NFS implementation work?

I have the following list to choose from

a) Fedora Core 9 (using either XFS or EXT4)

b) Nexenta using their NFS target.

Please note Fedora core 4 is on certified list of NFS targets.

The NFS store will be 1TB big and will be shared by 2xDELL2950 (each hosting approx 10VMs)

(I am choosing NFS over iSCSI because have heard good things about it on these forums. If any of you can suggest me TESTED iSCSI software(cheap/free), I would be inclined to it)

[Texiwill]

Hello,

VMware ESX uses NFS v3 over TCP. The TCP option is very important, not all hardware devices support this. Fedora does.

Another option to investigate is Openfiler (which is free), it will do iSCSI as well as NFSv3 over TCP.

For reliability you may want to downgrade to Fedora 8, since Fedora 9 will not be released to 4/22 (everything is beta at the moment), or go to CentOS 5.1 as that is a recompiled version of RedHat AS 5.1 which has a very good implementation of NFS. I really do not think the distribution of Linux will matter much, except I would not use Beta versions.

I use RedHat 5.1 and Fedora 8 w/iSCSI Enterprise Target as well as NFS.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

[nitin_m]

Thank you for your quick response(This forum is really good in getting answers)

You are using the CENTOS/Fedora 8 in production? Which underlying filesystem you use(ext3 is fastest and xfs is dependable - as per me)

Yes - Fedora 9 is beta but it has never crashed on me (our implementation is going to be end of april - by then fedora 9 is released)- You are right in your comments.

I have read a lot about ZFS and am really inclined. Its almost similar in technology what NetApp uses(the FS)

One thing I learnt from these forums is that almost all of admins/members are sitting on wads of money

I am hell bent on saving money but want the best...(dependable as well as high performance solution) hence trying to brainstorm and learning from experiences of other forum users

Edit - Has anybody had experience with e-open? They now offer 1TB of iscsi/nfs free. Their forum is full with people using e-open as iscsi(in flie i/o mode - what is it?)

[cmanucy]

Don't use Fedora for production stuff. It's not designed for that, and it won't 'last' so far as support goes. Cent/RHEL is supported for years (10, I think?) beyond release - RHEL3/CentOS3 are still getting updates (and yes, I have things still running on that due to dependencies). I don't remember exactly, but I think it's something like 18 months... not very long - especially when all of your VMDK's are dependent on it.

Always stick with the 'Enterprise' line - CentOS/RHEL. Personally, I use Cent 4 for this same thing. Actually, I use it to share out a cheap Promise iSCSI box that won't talk to ESX - so we mount it on a CentOS box, then share that via NFS.

This works great - but we only use it for templates. I wouldn't trust something like this for production... development, sure. Things that don't have to be up 24/7, sure. But you're talking about single points of failure, which is something you should avoid when it comes to production.

----

Carter Manucy

[nitin_m]

I believe that an OS is as stable as hardware below it. I intend to run the store on dell 2950 with 6x146gbx15k.

Even for windows, if there were no patch tuesdays, its stable as a rock(If you dont do anything other than the work it is asked to).

A lot of people here on forums are using openfiler for production use. Openfiler now is taken over by some company(in any case openfiler development was irratic) and therefore I am looking at other targets.

I have VirtualIron also(and I quite sure forum users are toying with XEN) - Isnt it (and in that case esx with its shell) hardened redhat?

I think its all in our heads - we are ready to run our VMs on one OS but refuse to pur our data on other OS.

[Texiwill]

Hello,

My choice is to also run RedHat/CentOS vs Fedora. It has more support and tends to consider stability above anything else. NFS is available on any distro of LInux and I have been running some of mine for many many years in production. Not everyone has gobs of money. You do the best with what you can use. But what it boils down to is YOUR knowledge of the OS in question. As the admin you will have to keep it running in an emergency. If you need the support behind you then go with a long term supported product.

You will want at least some level of network redundancy with any system you choose. Try to alleviate any single points of failure. That will require perhaps etherchannel (802.3ad) to the switch from the NFS Server and then multiple paths to the ESX servers through separate pSwitches, or even use etherchannel. I prefer NIC Teaming.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

[NParker1]

Did you ever have any luck running a linux distro as an ESX NFS server?

I've been trying a few in our lab and from what I'm seeing and reading sounds like there really isn't a linux NFS server option that gives production quality performance for ESX data stores.

So far have tried RedHat 5, FreeNas, and just started testing with Openfiler 2.3. On compairing those 3 with NFS set to sync, Openfiler definitely seems to give the best write performance, but still is not very good. Max I've seen so far for sustained writes was about 8 MB/sec and average more like 4 MB/sec -- and that's on gig-E. That's maybe acceptable for our lab, but only barely. Fortunately we have NetApp in prod, and it's working fine.

If I set the NFS share to async, I get great speeds with RedHat or Openfiler as the NFS server. But I don't think async NFS is an acceptable option.

Let us know what you've found.

Thanks and good luck!

Neal Parker