NFS or iSCSI storage for ESX Cluster

[EdiB]

Hi

I am implementing a p2v migration for about 100 windows servers.

I am using Netapp filers as a storage option.

Which do you think would be a better solution nfs storage on netapp or iSCSI storage o netapp?

Thnaks

[EdiB]

Dalepa,

I see that you have 950 VMs on NFS(NETAPP).

My questions are as follows.

1:) How many Netapp filers are you using?

2:) How about networking, how is the network layout. you have 950 VMs on 28 hosts how is the bandwidth? you're running everything on gigabit network have you noticed a lot of collisions?

Would you be able to explain how the network layout is in your environment.

I am concerned about my deployment f 100 VM on a single netapp filer and the network bandwidth.

I am going to team 4 NICs together but can the switch handle the traffic or should I put them in different switches.

How about the filer?

I am very curious to know how you guys setup your environment.

Any Docs or diagrams would be very helpful.

Thanks

Ed

[dalepa]

Dalepa,

I see that you have 950 VMs on NFS(NETAPP).

My questions are as follows.

1:) How many Netapp filers are you using?

Location1: 1 3070c(dedicated to vmware) 1R200 (snapmirror backup)

Location2: 3050 (snapmirror backup)

2:) How about networking, how is the network layout. you have 950 VMs on 28 hosts how is the bandwidth? you're running everything on gigabit network have you noticed a lot of collisions?

We have 4 active port/3 passive on the netapp and a dedicated crossover to the R200 for snapmirror. Bandwidth is not an issue...

Would you be able to explain how the network layout is in your environment.

I am concerned about my deployment f 100 VM on a single netapp filer and the network bandwidth.

Unless you are video streaming or doing high IO applications, bandwidth should not be an issue.

I am going to team 4 NICs together but can the switch handle the traffic or should I put them in different switches.

we use 4 ports on each ESX server. 2 for Vmotion/NFS/SC and 2 for the VMs >

How about the filer?

2 vips 1 vip with 3 ports, 1 vip with 3 ports. Also 1 Xover to the R200 for snapmirrors >

I am very curious to know how you guys setup your environment.

We have done several CON calls with very large companies on the subject. If you like, you can have your Netapp Rep schedule a meeting with ours... >

Any Docs or diagrams would be very helpful.

Ask me for the docs via email and I will send you an overall drawing.. >

Thanks

Ed

[EdiB]

Thanks Dalepa.

Can you please email me the docs in the following email address .

Thanks

[PerryWhittle]

i'm sure NFS is a little more tolerant when it comes to handling lost packets than iSCSI

[BenLe]

From a management perspective NFS might be the easier solution and it allows to increase the storage space on the fly.

But be aware that when creating a VMDK file on NFS the file is created in "thin" mode by default (the file will only use the disk space that it really requires). This can easily lead to an over-allocation of the storage!

Additionaly you should avoid many NFS mounts for storage as this can decrease the performance.

No matter what if you use iSCSI or NFS you should dedicate at least one NIC to a VMKernel vSwitch. If possible I would recommend to have 2 NICs dedicated for the communication with the storage system.

[PerryWhittle]

If possible I would recommend to have 2 NICs dedicated for the communication with the storage system.

definitely, that's what i did. You want at least 2 x 1GB connections for your VMKernel connection to the NFS storage. The thin disk technology can work quite well, but just remember when creating VM's.

[RussH]

I've had mixed results with the thin provisoning on NFS - initially, creating a new VM on an NFS store does indeed get provisioned in thin disk format. However, once converted to a template and deployed or a VM is cloned - it reverts to thick disk - thus there's little benefit.

I'm interested in other peoples experiences in this, maybe its something i'm doing wrong or something perculiar to my deployment..??

Cheers

[BigHug]

Russ,

Yes, after cloning or migrating the thin vmdk, it will change to thick one. Pretty annoying. There is no workaround. Make a customer request so that VMware can change the behavior in the future.

[EdiB]

Anybody knows an easy way of shrinking a vmdk disk?

[BenLe]

I think shrinking is not really supported. However your OS inside the VM would need to support a partition-resize as well.

Technically, if you just want to reduce the consumed disk space, you should be able to convert the vmdk into a thin-mode disk. Then you can use the shrink option from inside the VMware Tools to minimize the vmdk file to the size it really consumes. However inside your VM the disk size will remain the same.

From what I remember from previous ESX version it was also possible to reduce the the vmdk size, using vmkfstools. (Not sure, but as I remember it was using the same command as for increasing, but just using a smaller size instead + an option to force the change). But this can lead to a loss of data inside the vmdk.

Regards,

Ben

[moshker]

Here is the command for converting a static or monolithic vmdk to a dynamic or thin vmdk.

vmkfstools -i /vmfs/volumes/Local/vmtest/test01.vmdk -d thin /vmfs/volumes/Local/vmtest/vmdisk01.vmdk

[BigHug]

To shrink vmdk disk, I found that Converter works pretty good. It's a bit slow but very reliable. Have tried other image tools or livecd. Not quite reliable. You can change vmdk to thin. But be careful that it will expand after migration. Hope this helps.

[MikaA]

Hey, Dalepa,

you say you are using NetApp's A-SIS deduplication for space savings. This sounds great on paper but some non-NetApp reps have not only positive things to say about it.. Now that you seem to actually use it, how well does it really work?

Thanks!

[dalepa]

If I were a non-Netapp rep, I would be saying that also. A-SIS not only works, it works extremely well. I'd even bet that the A-SIS option will be the default setting for all data soon.

virtualOptics

[RaniBaki]

Dalepa, we too recently implemented NetApp with NFS for VMware. I'm interested on how your consistent snapshots script works. You probably have VMs residing on different volumes across all your ESX servers. How do you determine what VMs to quiesce in order to perform your NetApp snapshot? Can you share your script with us?

[dalepa]

We currently don't quiesce any VMs... . If your VMs can take a power loss, snapshots without quiescing works great. We have restored more than a 100 VMs from snapshots without a single issue.

Netapp has some scripts and your Rep should be able to find them if you need them...

Also Netapp will soon release Snapmanager for VMware. It will be a GUI that allow your to manage VMware quiesced snapshots

virtualOptics

[RussH]

Were using NFS on Netapp - each volume on the Netapp corresponds to a datastore within ESX. I then wrote a perl script (using the VIPerltoolkit) which runs on the VC server and quiesces (takes a VM level snapshot calling the sync driver within VMTools to flush I/O) each VM in that Datastore (and therefore Netapp volume) and then takes the snapshot on the Filer.

Kludgy and very localised at the moment, hence I can't share it - but you could fashion something similar by hacking together dsbrowse.pl and snapshot.pl from within the VIPerltoolkit examples.

Cheers

[RaniBaki]

Thank you both. Dalepa, have you restored a snapshot of a SQL database server without a queisced VMWare snapshot and without seeing any issues? Any idea when snapshot manager for VMWare will be released?

[dalepa]

I've restored Oracle and MySQL from snapshot with any issues... No Sqlserver that I can remember... But if your application can handle a powerloss, it should be able to handle a snapshot. No release date from VMware yet... It's in beta now...

[rdpbn]

How much if any data was lost? ie. 10 sec? .5 milli second?