VMware Cloud Community
EdiB
Enthusiast
Enthusiast
Jump to solution

NFS or iSCSI storage for ESX Cluster

Hi

I am implementing a p2v migration for about 100 windows servers.

I am using Netapp filers as a storage option.

Which do you think would be a better solution nfs storage on netapp or iSCSI storage o netapp?

Thnaks

Reply
0 Kudos
1 Solution

Accepted Solutions
ascari
Expert
Expert
Jump to solution

Hi Edi,

I think that for your enviroment is a god idea think to use an FC Storage.

Anyway, iSCSI is better than NFS NAS. Transfer Block Access of data/LUN is better than File accesso per NFS.

It's very important to separate iSCSI traffic LAN from normal vm LAN.

For example, you can't perform RDM device on NFS. In other world you can not configure MSCS vm and you can't use VCB in NFS enviroment.

You can read this doc http:/www.vmware.com/files/pdf/iSCSI_design_deploy.pdf

Remember ... HCL.

Bye Alberto

View solution in original post

Reply
0 Kudos
39 Replies
ascari
Expert
Expert
Jump to solution

Hi Edi,

I think that for your enviroment is a god idea think to use an FC Storage.

Anyway, iSCSI is better than NFS NAS. Transfer Block Access of data/LUN is better than File accesso per NFS.

It's very important to separate iSCSI traffic LAN from normal vm LAN.

For example, you can't perform RDM device on NFS. In other world you can not configure MSCS vm and you can't use VCB in NFS enviroment.

You can read this doc http:/www.vmware.com/files/pdf/iSCSI_design_deploy.pdf

Remember ... HCL.

Bye Alberto

Reply
0 Kudos
virtualdud3
Expert
Expert
Jump to solution

Not really sure the difference between the two is as large as it was a short while ago. The performance between Fibre SAN, iSCSI SAN, and NFS/NAS is growing closer and closer.

VCB is now supported over Fibre SAN, iSCSI SAN, and NAS.

RDM is by definition not an option for NAS/NFS; this may or may not be an issue. If a lot of your physical serves have a great deal of data (disks larger than 300GB-500GB) stored on iSCSI LUNS, then you may wish to consider going with iSCSI so that you can use an RDM. As I said, I am not sure whether this is an issue.

MSCS is not supported on either NAS/NFS or iSCSI so I am not sure that this is really an issue to choose one over the other.



###############

If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!!

############### Under no circumstances are you to award me any points. Thanks!!!
Rodos
Expert
Expert
Jump to solution

If you can do either without taking shortcuts then iSCSI is going to be better over NFS. 100 VMs is not a small installation. Over time you will want to be taking advantage of all sort of current and future VMware functions and they are usually better support on iSCSI than NFS. Support for FC and iSCSI usually come out long before NFS traditionally.

Here is a brief comparison list I did a short while ago on NFS. I have not reviewed it for 3.5 yet and some of the drawbacks will still be the same for iSCSI

Benefits of NFS

  • It works (VMotion, Boot and Snapshots)

  • Can grow the volumes without using extents

  • Lower infrastructure costs but remember best practices

Drawbacks of NFS

  • ESX uses root so if you root-squash on the NFS server you need to change the account. However this is has experimental support only. VMware provides limited support for this feature. Use of this feature may result in unexpected behavior.

  • Lowest support of VMware features (VCB, DMoition)

  • Only supports thin provisioned disks

  • Default limit of 8 mounts but can be overridden

  • Additional network ports and a separate network should be used (6 minimum / vmKernel)

  • Will generate a noticeable CPU hit on the ESX servers

  • Lower throughput (10 – 50% lower) and latency

  • TCP/IP Offload Engine (TOE) network cards are recommended by vendors in the best practices

  • Highly recommended (by VMware) to enforce a Power Off isolation

  • Cant run Microsoft Clusters

Hope that helps.

P.S. If this info help consider awarding points

Rodos {size:10px}{color:gray}Consider the use of the helpful or correct buttons to award points. Blog: http://rodos.haywood.org/{color}{size}
dalepa
Enthusiast
Enthusiast
Jump to solution

NFS on Netapp is the way to go. We have over 950VMs on 35 hosts on Netapp/NFS. Here's : Why VMware over Netapp NFS

We've also done 300+ P2V's using Platespin...

and are wrong.

Virtual Optics

Reply
0 Kudos
EdiB
Enthusiast
Enthusiast
Jump to solution

Dalepa,

I see why I should choose NFS over iSCSI on NetApp now.

One other question why do you recomend Platespin? Weren't you happy with VMware Converter to do the migration.

How many ESX admins you have managing this environment of 35 Hosts and 950 VMs?

Thnaks

Ed

Reply
0 Kudos
dalepa
Enthusiast
Enthusiast
Jump to solution

When we started the project in 2006 Vmware Converter was somewhat limited as I understand it. We have two dedicated admins and 3-4 supporting admins.

Reply
0 Kudos
pdrace
Hot Shot
Hot Shot
Jump to solution

Having done both ISCSI and NFS on NetApp I second the NFS recommendation, fast, easy and space efficient.

Reply
0 Kudos
MikeAvery
Contributor
Contributor
Jump to solution

I see no distinct advantages to using NFS over iSCSI. I'm not sure whether de-duplication works against LUNs or not. If not, then I suppose using NFS becomes more viable if you need de-duplication.

There's no harm in touting a strong working NFS configuration, but I would stop short of claiming a championship for NFS simply because it works. Using either NFS storage of VMDKs or iSCSI RDM LUNs have specific configuration limitations and critical considerations that will ultimately decide success or failure. Most often it is oversight or lack of understanding through lazy research and testing that leads to project or environment failure, not the technology used.

I looked at all options as well and decided in the end to go with NetApp clustered storage over iSCSI. For me, RDM LUNs offered clear advantage over VMDK over NFS in my environment. There were several reasons for my decision just as I'm sure there were several reasons for yours. In the end, the question should be answered with more questions to get a better understanding of requirements and goals.

To the original poster I can only say that the best answer I can give you is to do as much research as you can and clearly identify your storage goals while considering forward looking goals for your organization. Then do LOTs of testing before deciding one way or another. Your approach to storage will be critically important to your success. You can thank yourself in two years for having taken the time to investigate the best solution for your company, or you can wish you had better wisdom in two years when you realize you've made a strategic error. Taking a suggestion here and running with it will just as likely get you into trouble as it will solve an issue.

- Consult VMware & NetApp storage best practices

- Be sure that your storage house is in order first

- Consider network configuration requirements

- Consider data protection and management before making decisions

- Do lots of testing

- Ask the community about specific things you see in testing in your environment

Reply
0 Kudos
dalepa
Enthusiast
Enthusiast
Jump to solution

I see no reason why not to setup both iSCSI and NFS. Moving from one datastore to another is just a migration away...

Also, I see no advantage of using Netapp iSCSI other than the initial NFS license cost.

There are several reasons to use Netapp NFS vs Netapp iSCSI:

  1. NFS is slightly faster than iSCSI on Netapp

  2. NFS VMs are initially thin provisioned.

  3. NFS volumes can grow and shrink on the fly

  4. NFS is much easier to setup and maintain

  5. You can recover single VMs from NFS snapshots easily.

Netapp ASIS works on the volume level, so iSCSI luns can be deduplicated.

I not sure of the benifits (if any) of using RDM and iSCSI. Maybe someone could elaborate on this...

Why VMware over Netapp NFS

Reply
0 Kudos
pdrace
Hot Shot
Hot Shot
Jump to solution

If you are using Netapp you can do both ISCSI and NFS according to the need of a particular vm.

Of course the NFS licenses are pricey if you have to add them.

Reply
0 Kudos
doubleH
Expert
Expert
Jump to solution

The only disadvantage that I can come up with is that it doesn't appear that NFS is supported with VCB. New features tend to support FC right away, then iSCSI, then NFS.

If you found this or any other post helpful please consider the use of the Helpfull/Correct buttons to award points

If you found this or any other post helpful please consider the use of the Helpfull/Correct buttons to award points
Reply
0 Kudos
dalepa
Enthusiast
Enthusiast
Jump to solution

I don't think there is a plan to support VCB of NFS datastores... Instead of VCB, you can use NDMP, or snapmirror to a second filer which we do and keep 21 days of snapsnots online!

Reply
0 Kudos
MikeAvery
Contributor
Contributor
Jump to solution

How are you performing memory-consistent backups of your VMs if not through VCB quiesced hardware snapshots on the NetApp? NetApp snapshots against active VMDKs will produce crash consistent data protection at best. Backing up VMs "like a box" using agents is a step backwards when you have ESX and NetApp in hand so I am curious to know how you approached this issue. This was one of a few compelling reasons I am using iSCSI RDM LUNs.

This has always worked against iSCSI LUNs and is now supported. vcbsnapshot --> netapp snapshot --> vcbshanshot deletion gives memory consistent hardware snapshots on the filer. How are you attaining the the equivalent over NFS without compromise?

Reply
0 Kudos
dalepa
Enthusiast
Enthusiast
Jump to solution

We are just using crash consistent snapshots. There are scripts available for creating memory-consistent, however we are waiting for Netapp's Snapmanager for Vmware tool(to be released anyday now) to implement memory consistent hardware snapshots on the filer. In the last 18 months we have restored several 100 VMs using the crash consistent snapshots without a single failure (and we have 21 days of snapshots to choose from). Basically if your application can recover from a poweroutage, crash consistent snapshots are fine.

Reply
0 Kudos
MikeAvery
Contributor
Contributor
Jump to solution

We have a fair number of applications and databases running, so crach consistency wasn't going to cut it. I released some scripts for automagic, memory consistent snapshots against NetApp RDM Luns using the viperl toolkit. The interested can find it here: http://communities.vmware.com/message/648343 You will want to tailor to your environment.

How did you lay out your storage? volume-per-vm with vmdks per drive within the guest? By function?

Reply
0 Kudos
kbk00
Hot Shot
Hot Shot
Jump to solution

We just had a small filer in for demo and also liked the NFS implementation. It worked very well while passing all our tests with flying colors. We hammered on the box for weeks and it didn't skip a beat. We just bought our first box and will be buy a dozen more over 2008 as we roll them out to different locations. We are an IBM shop and have purchased the IBM N3300 (low end rebranded NetApp). We didn't bother testing iscsi since we specifically were going after NFS and the benefits from that storage method. Crash consistent backups are just fine for us and we never use RDM.

My issue with the appliance is the cost - specifically of the additional licenses. what a surprise Smiley Happy

We are also looking to bring in a bigger appliance for possible migration of our FC environment to NFS, maybe at least for dev and test. I'm in no hurry for that, though, but it's at least on the horizon. I suppose it depends on how well the small systems work out and if they perform in the field as well as they have in the lab.

If you found this posting to be useful, great. I didn't waste your time.
Reply
0 Kudos
RussH
Enthusiast
Enthusiast
Jump to solution

According to the VCB 3.5 release notes - NAS (read: NFS) is now supported!

http://www.vmware.com/pdf/vi3_35/esx_3/r35/vi3_35_25_vcb_release_notes.pdf

Reply
0 Kudos
dalepa
Enthusiast
Enthusiast
Jump to solution

Interesting... Good catch... But I still think that using snapmirror is a superior way to backup VMs unless VCB somehow just transfers the change blocks...

Reply
0 Kudos
dalepa
Enthusiast
Enthusiast
Jump to solution

Interesting... I was also thinking the fas2020 (IBM N3300) would be a great fit for VMware with it's dual controller for active/active configuration in 2U.

Reply
0 Kudos