VMware Communities: Message List - VMFS-3 via FC SAN versus NFS on NAS

Re: VMFS-3 via FC SAN versus NFS on NAS

francescod69 — Tue, 05 Sep 2006 14:37:59 GMT

Data is not very critical. The problem is to consolidate 24 VM on 3 hosts with the most moderate cost.

We have already deployed a big config ESX in SAN with 5 hosts and 24 VM by hosts, with dual path, because it is about a big site and because it needs performances there.
There it is about a small site where the economic aspect is very important.
Before engaging tests SAN vs iSCSI (+ QLA4010 cards) on a config FAS3020c (3TB), I just wish to share the experiment with somebody.
Thank you for your answer

Re: VMFS-3 via FC SAN versus NFS on NAS

francescod69 — Tue, 05 Sep 2006 09:04:41 GMT

What do you think about this comparative SAN and iSCSI on FAS3020 ?
Network Appliance FAS3020 and EMC CX500: Comparison of Usability and Performance

PS : Don't see EMC CX500 graph

Message was edited by:
ken.cline@hp.com to shorten URL

Re: VMFS-3 via FC SAN versus NFS on NAS

epping — Tue, 05 Sep 2006 14:00:40 GMT

i never trust these tests, its the bit that starts "Network Appliance commissioned
VeriTest, a division of Lionbridge
Technologies Inc., to compare the
usability of a variety of features and
functionality that users of both the
NetApp® FAS3020 and the EMC
CLARiiON CX500"

I am sure if EMC asked them to do some tests the results would be the other way round.

on the other question of NFS or FC, i would say if your data is critical go with FC its the most established, i would wait a few years, that said i would not go to VI3 yet either..

Re: VMFS-3 via FC SAN versus NFS on NAS

francescod69 — Tue, 05 Sep 2006 08:51:13 GMT

Have somebody already made a performance comparative with FAS3020c on SAN FC 2Gbps and iSCSI with TOE QLA4010 cards ?
Do the failover, and the VMmotion migrations work in the same way?
And what about NAS configuration with VIF mode ??

Re: VMFS-3 via FC SAN versus NFS on NAS

MechaNikos — Thu, 24 Aug 2006 10:03:06 GMT

IO offload. FC cards are HBA's, they offload much of
the storage processing. This is significant in the
VMWare world; I went (on the same servers) from local
to FC SAN disk, and the system processor (host
system) dropped 20%, just by offloading the IO from
the processor. You can use TOE, but it does not
offload as much as an HBA.

This is the most important factor imho. The point is not whether you can use SAN/NAS/iSCSI etc.. You can. I'm using a simulated iSCSI SAN at home that has been working fine for more than two years and it performs surprisingly well. But when you need *sustained* throughput in a live environment, I think FC is well worth the money.

Not to mention the fact that a carefully designed FC SAN can come down to a more reasonable cost if you bypass the salesmen and do a bit of reading yourself.

Re: VMFS-3 via FC SAN versus NFS on NAS

Mike Fink — Thu, 24 Aug 2006 01:05:13 GMT

I certainly think that NAS access has its place in the VMWare world, but if you have a large data center that already has an investment in FC, I would really suggest using it for your ESX host connections. A few reasons, which I will discuss below:

1GB vs 2GB. Although this is not really an issue for almost all Windows hosts, remember what VMWare does. Your going to be running many (lets guess 10 for right now) Windows machines down that one trunk. So, each machine has about 100Mb/200Mb of effective throughput. This is almost never an issue, execept when you start to pull backups. When you pull backups, your going to be hitting all the VMs hard for IO, you will see a difference between NAS and FC.

IO offload. FC cards are HBA's, they offload much of the storage processing. This is significant in the VMWare world; I went (on the same servers) from local to FC SAN disk, and the system processor (host system) dropped 20%, just by offloading the IO from the processor. You can use TOE, but it does not offload as much as an HBA.

Reliability. FC is tried and true in the VMWare world.

Cost? FC is very expensive, there is no denying that. However, think about what your doing; your jamming 10 servers down 1 FC card. When you amortize that cost over the VMs, the $$$ spent for the FC adapter really becomes a non-issue (figure 2000 for 2 single port adapters, or 200/server for disk connectivity; not bad).

Someone mentioned using SAN snapshots using NetApp; you can do this using NFS/iSCSI/FC. It does not matter. It works a little different for each, but there is not really any difference between any of them (NetApp stores the FC LUNs as a file, when you take a Snap, your actually taking a Snap of the .LUN file. In the VMWare world, this is exactly the same as taking a Snap of the VMDK file).

FC adds complexity; and costs significant $$$. However, if you have the infrastructure already, I would suggest that you connect the servers via FC. If your looking to build a large VMware farm in the future; again FC presents the right option. However, for many people NAS is the way to go.

Its great that we have all the options; they all have their place. However, nothing out today is gong to top FC for speed. Unfortunately, nothing is gong to top it for cost or complexity either.

Re: VMFS-3 via FC SAN versus NFS on NAS

dalepa — Wed, 23 Aug 2006 20:06:49 GMT

good point... That's the only functionality loss that I have seen...

Re: VMFS-3 via FC SAN versus NFS on NAS

dalepa — Wed, 23 Aug 2006 20:04:45 GMT

We ran benchmarks using the dbench ( http://www.freshports.org/benchmarks/dbench ) tool on our 9980 SAN and 3050 NAS. We added 1 NAS drive and 1 SAN drive to same guest running on a IBM 366 host. The NAS performed slightly (~6%) better than the SAN.

We also ran the same dbench NAS test using a Sun v20z host and compared it to our IBM 366 hosts. The v20z(2cpu) dbench benchmark was 70% faster than our IBM 366(4 cpu) box. So the host does make a big difference... (Can't wait to see how the SUN X4600 16-way opteron box performs )

We also ran a test using dd on both the NAS and SAN guest. Writes to the NAS and SAN were about the same, while reads from the NAS were up to 500% faster than the SAN. And again, the v20z was even higher at 70% faster than the IBM 366 for the dd test.

Summary:
1. Our NAS is just as fast or faster than our SAN.
2. Buy FAST hosts with lots of CPUs.

ymmv

Re: VMFS-3 via FC SAN versus NFS on NAS

Ken.Cline — Wed, 23 Aug 2006 15:12:42 GMT

One thing that hasn't been mentioned, that you do LOSE with non-FC storage is the consolidated backup feature of VI-3. VCB requires FC SAN...just wanted to mention since you asked about lost functionality...

Re: VMFS-3 via FC SAN versus NFS on NAS

AMitchell — Wed, 23 Aug 2006 12:30:41 GMT

Running clusters of DB servers on ESXs that boot off
VMFS

They boot off VMFS????

Edit - Just re-read your post. The virtual DB servers boot off VMFS. All clear now....

Re: VMFS-3 via FC SAN versus NFS on NAS

MechaNikos — Wed, 23 Aug 2006 10:48:26 GMT

It seems there are many conflicting opinions here, so let's get down to the facts (disregarding the cost for now):

FC is superior in terms of performance, scalability, redundancy and stability. The real benefit is that the fibre carries only SCSI commands and delivers a rock-solid sustained throughput, something not possible on Ethernet.

iSCSI is more like a disruptive technology, as it comes in between. The idea of encapsulating those SCSI commands into TCP/IP packets is brilliant, as your connectivity options expand dramatically. However, in order to get decent performance you need TOEs to get rid of the TCP/IP overhead that will crush your CPUs. So the cost can go quite high too without escaping the Ethernet arena.

NFS/CIFS and the lot are probably the easiest option, but also the most confusing. Performance is not optimal as you're subject to network traffic and you have many potential failure points on a rather meandering path. On 10Gb Ethernet things are certainly much better, but it's still early to adopt it.

You can mix the different technologies in order to exploit each one's benefits, depending on your needs and applications. But when it comes down to relying on solid performance for ESX, FC is (still) the clear winner.

My approach generally is something like this:

ONLINE storage (FC SCSI) for mission critical data
NEAR-LINE storage (FC SATA or NAS processor with SATA arrays) for file shares and non-critical data
OFFLINE storage (FC TAPE library and additional disk storage for virtual backups)

Running clusters of DB servers on ESXs that boot off VMFS and have direct access to the FC SAN for data/logs, has been my solution of choice.

Re: VMFS-3 via FC SAN versus NFS on NAS

pops106 — Tue, 22 Aug 2006 12:32:34 GMT

Mine goes in on the 11th sept and trust me i will be testing everything.

FC
iSCSI
NFS

I want to know basically which gives me best performance and quickest failover.

It is a FAS3020 Cluster head with 60 disks all FC 144gb.

Re: VMFS-3 via FC SAN versus NFS on NAS

DanielMeyer — Tue, 22 Aug 2006 08:13:40 GMT

Well, just 5 meters and one wall away there is exactly such a system, but i dont have the time to setup and run proper benchmarks right now.

Actually, what i MIGHT be able to do in the next weeks is a test with esx3 and then nfs vs. iscsi. But i dont have any spare fc-hbas...

If someone can provide them.... *hinthint*

Re: VMFS-3 via FC SAN versus NFS on NAS

dalepa — Mon, 21 Aug 2006 15:51:29 GMT

FC and iSCSI is an option with the Netapp, but until someone can show a performance or configuration reason to use FC or ISCSI, we are using NFS (KISS)

FC involves extra Switches, HBA's, fiber cables, etc...

It would be great is someone built a Netapp with all three protocols and published some performace results. (Hint Hint... Netapp sales)

Well, especially with netapp you have more options
than just NFS...

why dont you use iSCSI?
why dont you use FC with the netapps?

Re: VMFS-3 via FC SAN versus NFS on NAS

DanielMeyer — Mon, 21 Aug 2006 07:47:01 GMT

Well, especially with netapp you have more options than just NFS...

- why dont you use FC with the netapps?

Re: VMFS-3 via FC SAN versus NFS on NAS

AMitchell — Sun, 20 Aug 2006 09:43:33 GMT

Can you still use NAS as shared storage for clusters, as you can with SAN?

Re: VMFS-3 via FC SAN versus NFS on NAS

pops106 — Thu, 17 Aug 2006 22:06:19 GMT

We have just purchased a FAS3020 which we are going to run our VM's on but we already have a FC backend in place from the old SAN. We are going to use the FC fabric at our main site as we have not really tested iSCSI or NFS.

But at the same time i have bought a little FAS270 which i am now really considering using NFS over FC.

Guess i have lots of testing on my hands.

Re: VMFS-3 via FC SAN versus NFS on NAS

kitarra — Thu, 17 Aug 2006 18:22:53 GMT

Absolutely.

Which is why I said that it CAN be managed just requires more planning.

Re: VMFS-3 via FC SAN versus NFS on NAS

dalepa — Thu, 17 Aug 2006 18:20:50 GMT

The advantage of SAN storage is that the servers are
connected to a dedicated, redundant (usually) fiber
network to get to the san. While there may be I/O
contention on the SAN itself, contention for
bandwidth is never an issue. This yeilds consistant
i/o results for your VM.

Agreed... However, it's real easy to create a "NFS" only network back to your NAS device(s)... Just dedicate 1-2 gig ports on your ESX server(s) and plug them all into a isolated switch... Virtual SAN!

dp

Re: VMFS-3 via FC SAN versus NFS on NAS

kitarra — Thu, 17 Aug 2006 17:50:21 GMT

Performane can be comperable. However you have to be a lot more carefull with NAS.

The advantage of SAN storage is that the servers are connected to a dedicated, redundant (usually) fiber network to get to the san. While there may be I/O contention on the SAN itself, contention for bandwidth is never an issue. This yeilds consistant i/o results for your VM.

NFS storage, because it is usually over a network can be subject to the usual network follies. Contention for bandwidth, broadcast storms, and traffic issues.

All of the issues are managable but you have to put a lot more effort into managing those issues. And you must be proactive about it. If you are not you could significantly affect the i/o performance of your servers.

If I/O is not an issue than the NFS solution might be perfect for you. To offset I/O problems I would also increase the RAM on individual VMs so that they do not have to page.

Re: VMFS-3 via FC SAN versus NFS on NAS

dalepa — Wed, 16 Aug 2006 02:10:32 GMT

Listen to your storage people.

We are also moving to VI3 using Netapp/NAS. So far performance is not an issue and our NAS is currently even faster than our SAN.

It's still too early to fully understand all the NFS benefits, but we see some MAJOR benefits by running VI3 over NFS and NO issues so far!

Here are some of the Netapp NFS benefits as we see it:

1. Snapshots! Snapshots! Snapshots!
Keep X daily and Y weekly snapshots of all your VMs!!!
5 minute restores of a VM or SnapRestore a WHOLE NFS volume if required.

2. Replication
Snapmirror critical VMs to a DR site and keep multiple snapshots remotely.

3. Capacity on demand
Netapp can grow and shrink NFS volumes dynamically.
No wasted space.

4. ESX Level Backups
Backup VMs at the EST level using Netapp NDMP directly to tape/vtl
NDMP will take a Netapp snapshot by default...
You may still need a guest backup solution for file level backups.

5. COST
NO SAN Switches
NO SAN HBA's
NO SAN Fiber Cables
NO SAN Drivers
NO SAN ZONING
NO SLEEPY SAN GUYS

6. Performance
Highly dependant on your Storage guys

Highly dependant on your VMs needs.
Netapps support 10Gbit if more bandwidth is needed.

7. Other NAS Benefits
- Ability to create multiple NAS volumes on Multiple NAS appliances and VMotion to any NAS volume.
- You can use other NAS solutions (SUN 4U 4500s 25TB!) for staging VMs
- CIFS/NFS copy VMs to/from your ESX NAS storage.
- Snapshots (how did you sleep without them  )

Re: VMFS-3 via FC SAN versus NFS on NAS

jlauro — Tue, 15 Aug 2006 22:22:35 GMT

I've seen some slow FC SANs, so performance can go either way. Generally you might expect FC SAN to be faster, but that is not always the case.

I am not aware of any feature loss.

VMFS-3 via FC SAN versus NFS on NAS

jgonzaleshou — Tue, 15 Aug 2006 15:27:11 GMT

All,

We are currently testing NFS via ethernet in our environment due to a push from our storage team to move to a dedicated Netapp appliance in our environment.

What functionality, if any, are we losing by using NFS instead of VMFS-3?

We are concerned about losing any fuctionality, or perhaps, any performance by not going w/ VMFS-3 on FC SAN backend storage versus NFS. We are well aware that FC SAN should provide higher i/o performance than NFS. So far, we have yet to see any issues but we only have 20-30 vm guests on a ESX3 server connected to NFS storage. Once we put it under load we should have a better idea of how our environment will be able to handle NFS.

Any thoughts, opinions, or 2 cents is greatly appreciated.

Thanks.