Advice on proper network setup of ESXi hosts and NAS

[ilyo]

I am looking for advice or best practices on setting up my ESXi 4.0 hosts with my NAS...here is the basic info:

- VMware certified AberNAS with RAID 5 array of SAS disks (2 x Gig NIC)

- 2 x HP Proliant Hosts each with 6 - 8 Gig NIC's

Currently, NAS NIC's are teamed for load balancing connected to two seperate switches.  HP Hosts are also teamed for load balancing connected to 2 different switches.

Below is a look at the basic setup in vcenter:

Since I have plenty of NIC's on the server (4 used...but 8 in total), I would like to configure this for improved performance.  So my questions:

1. Should I create a new vswitch for the NAS and dedicate traffic between the hosts and NAS?  How?  Do I place them on a dedicated physical switch between hosts, NAS and LAN?

2. Should I create a dedciated NIC for vmotion of VM's between hosts?

3. Any other configurations that benefit NFS and VMware?

Am open to any idea's. Thanks

[weinstein5]

Per best practice NAS traffic should be on a dedicated and if possible isolated network - with 8 physical NICs I would configur as follows:

2 pnics for vMotion and management - where 1 pnic is dedicated to the management traffic and the other is in standby and the one in standby for the management will be the primary for vMotion and the pnic used by the management port will be in standby for the vMotion network

2 Nics for NAS for redundancy - if you have only a single NAS device or IP address you are connecting I would leave the NIC teaming at default and have one of the NICs active while the other is standby if you have multiple NAS devices with different IP addresses I would then have both active with Load Blancing method set to IP Based Hash

the remaining 4 NICs for VM traffic -

If you find this or any other answer useful please consider awarding points -

[ilyo]

Thank you! Exactly what I was lookign for.  Are you able to clarify the following:

- There are 2 hosts currently (am licensed for 3 and may install at later date). Also, I have 2 gigabit physical switches dedicated to NAS/VM Host nic's and are trunked to LAN..so when designing the management/vmotion pnic setup, do I mimic the same setup on both hosts?  Do those pnic's still stay connected to one of the gig switches?

- With the NAS device the NIC's on the NAS are teamed with single IP address and are dedicated to VM infrastructure.  So would I have 2 pnic's from each Host AND the 2 pnic's from the NAS connected to dedicated isoltaed physical switch and static ip'd all as separate subnet?

Thanks again.

[weinstein5 · Accepted solution]

- There are 2 hosts currently (am licensed for 3 and may install at later date). Also, I have 2 gigabit physical switches dedicated to NAS/VM Host nic's and are trunked to LAN..so when designing the management/vmotion pnic setup, do I mimic the same setup on both hosts?  Do those pnic's still stay connected to one of the gig switches?

Yes you would duplicate the setup on both hosts -

- With the NAS device the NIC's on the NAS are teamed with single IP address and are dedicated to VM infrastructure.  So would I have 2 pnic's from each Host AND the 2 pnic's from the NAS connected to dedicated isoltaed physical switch and static ip'd all as separate subnet?

Yes - as I mentioned with a single IP address on the NAS device the pnics on the hosts would be in a active/standby configuration - this has to do more to the fact that ESXi does not really load balance across both NICs but selects a pnic based on the originating and destination IP addresses - so if neither of these change the traffic will only go out a single pnic

If you find this or any other answer useful please consider awarding points

[ilyo]

Ok, I apologize in advance as I know this should be real simple but am having difficulties wrapping my head around this.  My goal is to isolate the nas traffic.  However a couple of things are confusing me:

- vswitch's do NOT communicate between each other (So can't create a separate vswitch for NFS) and there can be only one vmkernal port per vswitch.  As you can see from my layout I already have a VMkernal port dedicated management traffic and IP storage.  So how do I get the VMkernal port to use specific pnics?

- If I plug 2 pnic's from each host and NAS device into a separate switch...each pnic would be static ip on different subnet then regular data traffic.  So what is routing?  What is the gateway?

Sorry...but am having a hell of a time getting my head around separating out the NFS traffic.  VMWare told me to leave all pnic's plugged into current switch's and simply dedicate the NFS traffic to two pnic's using VMKernal port.  But how do I tell the VMKernal port to use specfic NIC's? As you can see from the picture the VMkernal port AND VM traffic is all using 4 team nic's.

Any help appreciated!  I know I am gonna be kicking myself in the ass when I finally "see" this.

Thanks

[J1mbo]

> VMkernal port dedicated management traffic and IP storage.  So how do I get the VMkernal port to use specific pnics?

Bear in mind that anything with physical access to  your NAS LAN can, with a bit of effort, bypass all your application layer  security.  So really it should be on seperate switches with it's own IP subnet and no router (or at least, one with an access policy).

So  essentially add a new vSwitch with two pNICs, create a new VMK on that vSwitch with a new  IP subnet address, move your NAS onto that subnet, and hook the lot up to a pair  of isolated phycial swiches with a multi-port ISL between them.

HTH

[ilyo]

Perfect...that is exactly what I am looking to do. Below is a diagram of what I think you are describing.  I have included example IP address's to ensure I have the idea.

A couple of additional details tho:

- To confirm, isolated network does NOT require any router or defined gateway to route traffic between NAS and hosts?

- Currently NAS has main data subnet static IP address.  As well the NFS datastore is mounted in vSphere.  So, when moving NAS to isolated network, I change nas IP address to new subnet..connect to new network switch.  However what are the steps in vCenter to get everything back up.  What is the process (ex. shutdown VM's...shutdown hosts...reconnect nas to isolated network, etc) to get NFS datastore back up on vsphere and VM's back on.

- Any additional details for setting up the second vswitch? Just create basic vswitch and assign 2 spare pnic's?

Thanks again!

[J1mbo]

You're on the right lines.  Use two switches on the storage side though with each host linked with one port to each, similarly for each NAS, and a link between the two switches.  If the switches support it, create the link between them with 2 ports in a LAG group.

No router needed on the storage side.

Before doing anything on the NAS front, make sure you have good backups!  Then shutdown the VMs and unmount the NAS on both hosts.  Change the NAS IP address and replumb it to the new network.  Then create the new vSwitch and add new VMK's as on the diagram, then add the NFS storage again via it's new IP address but use the exact same datastore name (case sensitive!).  With a bit of luck the VMs can then be powered on again

HTH

[Josh26]

Hi,

I wouldn't say "no configuration" on the storage physical switch is necessarily accurate.

I would ensure portfast is enabled on all ports at a minimum - this will ensure the SAN comes online quicker. You may also be considering jumbo frames on the NAS - this would need to be reflected on the switch too.

[ilyo]

A year later we are going to change our NAS setup to isolate the storage traffic.

Just want to confirm the following based on the last plan shown above:

- Unmount datastore from Hosts...but can leave VM's as is...only thing to note is exact datastore name.

- Change IP of NAS device (New IP subnet with no gateway)

- Have dedicated switch with only the NAS and specific pnics from hosts

- IP pnics on hosts to same new subnet with no gateway.  Attach these pnic's to new vmk

- Add datastore to hosts with same datastore name as before

I then need to have a separate management port on main network to be able to administer vcenter.

Am I missing anything?