I have got 2 ESX servers to setup with 4 port NICs each - so I would like to know the best way to load network traffic based on management, storage and VM network. Should I team 2 NICs for iscsi storage then 1 for management and 1 for customer link?
They all should be in different VLAN with a dedicated iscsi switch and separate switch for customers (with dedicated LAN to the data centre)
What is the best design solution for this? The goal is to give the best performance in terms of bandwidth to the customer premise in terms of accessing emails and file services.
The first question to answer before a design would be what license are you using? Standard, Enterprise or Enterprise Plus? If Enterprise Plus do you plan on using vDS (vSphere Distributed Switches) or vSS (vSphere Standard Switches)?
Basically what you could do is create the following. The VLAN(s) of course will be whatever you have associated to the corresponding traffic type. And your VMNet's will be however many different VLAN(s) you want to use for your guests.
Create two standard vSwitches like this...
vSwitch0 - assign vmnic0 and vmnic1 to this vswitch
Console - Portgroup on VLAN106 - VMK0
vMotion - Portgroup on VLAN107 - VMK1
VMNet-1 - Porgroup on VLAN108
VMNet-2 - Porgroup on VLAN109
vSwitch1 - assign vmnic2 and vmnic3 to this vswitch
iSCSI-1 - Portgroup on VLAN128 - VMK2 - vmnic2 - active -- vmnic3 - unused
iSCSI-2 - Portgroup on VLAN129 - VMK3 - vmnic3 - active -- vmnic2 - unused
The iSCSI port setup like above will allow you to configure MPIO for your iSCSI datastores so you can use Round Robin plus provide failover. They will need to be on different VLANs for MPIO to work correctly. Read the following http://www.vmware.com/files/pdf/techpaper/vmware-multipathing-configuration-software-iSCSI-port-bind... for more information on vSS iSCSI MPIO.
If you decide to go with a vDS setup you can check out the following links.
Below is an image that could represent what you are looking at for your vSS setup which goes along with the above vSwitch designs.
Hope this helps you and there are other ways to carve these ports out but this is the first way that came to me.