Network and best vswitch design with 8 pNIC ?

[clemence]

Hi !

I just got my 2 new Dell R710 servers and a brand new equallogic PS4000 yesterday

I'm about to deploy Vmware vsphere esxi essential + with this 2 hosts but i need your thoughts about my network design :

- 4 pNIC, 1 vswitch with port groups for the VMs

- 2 pNIC, 1 vswitch for ISCSI (with mpio). Traffic will be isolated in a dedicated VLAN (with no routing)

- 2 pNIC, 1 vswitch for vmotion and management (vcenter). Traffic will be isolated in a dedicated VLAN (no routing). But... do i need routing for vcenter (to download the updates) ? I also don't have DNS access on this VLAN. Can i use ip address instead ? What do you think about this design ?

Thanks you,

F.

[idle-jam]

looks pretty good maybe you might want to create a dedicated backup LAN?

[clemence]

Thanks But is the lack of DNS access (and routing) a serious concern for my management (vcenter VM) / vmotion Vlan ?

For the backup, i plan to use and old server with a lot of local storage and install openfiler with iScsi emulation to perform all the backup tasks. Does it sound good ?

[idle-jam]

vmotion and etc does not required DNS. infact most of the time my vmotion lan are just an isolated purely such use only. since you have 4 NIC for VM accessi think it might be too much, maybe taking 1 dedicated as backup lan so that you have a faster backup?

[AndreTheGiant]

About iSCSI, 1 or 2 vSwitch depends by your type of storage... check the suggested configuration.

About management, you will need routing (maybe with a firewall) and DNS resolution.

About vMotion (use a different VLAN for it) no routing and DNS are required.

Andre

[clemence]

So maybe 2 pNIC for Vmotion/Vcenter, a single vswitch but with 2 port groups then :

- 1 vlan for vmotion totally isolated, no routing

- 1 admin vlan for vcenter/management, with routing & Dns ?

[AndreTheGiant]

Yes... and define portgroup rule to make management active on one NIC e vMotion on the other.

Andre

[sketchy00]

Until something better comes along, I think this series (part 1 link listed below) is the gold standard when it comes to network design of pnics, vnics, vswitches and portgroups.

http://kensvirtualreality.wordpress.com/2009/03/29/the-great-vswitch-debate-part-1/

On one of the posts, he describes ideal arrangements based on the number of physical NICs you have.  The only thing that might change some of this is if you use VMWare's "Fault Tolerance" or have have NIC controllers that offer several ports (which you'd want to consider in your design so that you could prevent a particular service going down by one NIC controller)

Re-read the entire series a few times over, and you'll see how good it is.

[clemence]

Yes, i already know these great articles !

Part 6 :

- 2 pnic vm

- 2 pnic iscsi

- 2 pnic vmotion

- 2 pnic vcenter/management

But maybe only 2 pNIC are not enough for 20-30 Vm with port groups ? Also, wasting 2 pNic bothers me too.

[sketchy00]

If you are worried about that, the great part is that you can adjust later on.  I'd suspect that 2pnics will be plenty for the the number of VM's you indicated.

[clemence]

Ok I used the :

- 4 NIC Vm

- 2 NIC console/vmotion

- 2 NIC Iscsi

setup,and it works fine ! Thanks you !

I enabled jumbo frame on iscsi nic and vswitch. Do i have to enable it on VM and console/vmotion vswitch too ?

[azn2kew]

There are so many types of networks/portgroups to think about and chose which one you'll need and required for your environment, best practices always have multiple vNICs/pNICs/Switches/ for redundancies on both virtual and physical level, uses VLAN for all traffics to be isolated as possible and provide routing network if you need to reach out external sources.  The types of isolations and design will different depends on your corporate security policies.

Networks Considerations:

1. Production

2. DMZ

3. iSCSI

4. NFS

5. FT

6. vMotion

7. iLO

8. Backup

9.  Management

I would assume you need at least the following to start with:

1. SC/VMotion ->pNIC1-2

2. iSCSI->pNIC3-4

3. iLO/Backup->pNIC5-6

4. Virtual Machines->pNIC7-8

May want to read this post if you want to know about redundancy paths:  http://communities.vmware.com/thread/254072

[a_p_]

I think your setup will work just fine. Two things I would probably have implemented differently are: 1.) separating Management and VMotion into two separate VLANs and 2.) no use of Jumbo frames. IMO it just complicates the setup, since you have to find out the correct MTU size supported by all components involved (storage, switches, Hypervisor). I did not do any evaluations myself yet, however from everything I read so far, there's no real benefit in using Jumbo frames with a 1 GBit/s network. In a 10 GBit/s network, it might improve performance.

If anybody was able to really improve throughput using Jumbo frames, don't hesitate to convince me that I'm wrong

André

[clemence]

Sorry, i forgot to say that i used VLANs with my setup :smileyconfused:

To be exact, I used :

- 4 NIC for VMs with different port groups

- 2 NIC console/vmotion. 1 VLAN for Vmotion, 1 VLAN for console. One NIC active for Vmotion, the other for console.

- 2 NIC Iscsi, dedicated own VLAN

- (1 VLAN for idrac, ilo, SAN mngt...)

I already activated jumbo frame for iscsi. I use 2 CISCO 2960S by the way and a PS4000X Equallogic SAN.

I also enabled it for VM traffic and Vmotion/Console but i think it's a bad idea... no ?

[a_p_]

I also enabled it for VM traffic and Vmotion/Console but i think it's a bad idea... no ?

I'm not a dedicated network guy, however afaik the Ethernet standard allows only one MTU size in one network and I don't think all systems in e.g. the Management network are configured for an MTU of 9000. Btw. Jumbo Frames are not even defined in the IEEE 802.3 Ethernet standard.

Personally, I think the iSCSI network - if at all - should be the only network using Jumbo frames. However, if it works this way for you and nobody else does see any severe issues with this setup you may leave it this way (not sure how VMware Support will handle this though).

André

[AlbertWT]

As long as the underlying cable, switch (hardware) and the NIC all compatible then it should be fine.

[clemence]

I have another question concerning physical NICs and switch.

I have a VM vswitch with 4 pNIC on my esx (i've got 2 hosts btw). Default settings for this vswitch.

I've got 2 cisco 2960S. Should i plug 2 pNIC on one 2960S and the other 2 NICs on the other Cisco. Or should i plug the 4 NIC on the same switch ? What is the best practice ? Thanks.

[a_p_]

I'd make sure each vSwitch's uplinks are connected to multiple physical switches for redundancy.

André