We are upgrading to hosts with 4 x 10gbe interfaces. I have not specifically seen it stated but would it best to create 2 nics for iscsi and 2 for vmotion/management/network or combine all four, treat them as 40gb and use switch based QoS instead? We would do this anyways with the network/vmotion/management links anyways.
I know some of it might come down to preference but since all traffic will be flowing through the same switch anyways I figured it might make as much sense to simply group the 4 interfaces together. I do not see any real caveats to this.
Our hosts have 3x dual-port 10gb nics. For a total of 10 ports. How I have divided is as follows:
2 ports(each port on a different physical nic in case a card fails) I have 2 DVS uplinks which are used for all the VLANS for the VMs
2 ports(each port on different physical nic) for VMotion. vmotion on 10gb is pretty nice. You can evacuated a host with 100 vms in minutes.
2 ports(each port on different physical nic) for iscsi.
The above are all on Active/Standby policy.
I use 2 copper ports for management traffic.
Since my traffic is pretty segregated at this point i dont have to do nioc or qos.
This is a very common scenario, since a lot of people end up with 2 x dual-port 10GbE NICs. I have personally setup dozens and dozens of such environments. Unless there is a particular variation that is required, I typical architect the network this way.
If on 5.1, then you can LACP into actual aggregated links. If on an earlier version, then you have the standard load-balancing (outbound) algorithms you can use.
ESXi Switch Design (vDS is preferable with Network IO Control enabled)
The biggest reason for segregating the traffic is to reduce broadcast traffic across physical NICs and improve overall response time.
There is nothing wrong with aggregating all 4 together, and like I said, it comes down to what is acceptable in your environment given your requirements. For most users, there is a still a general requirement to segregate storage-level traffic on a separate vSwitch.