Switching gears a bit. Our architecture on the network ESX side is:

-3 vswitches

-2 gigabit nics per vswitch

vswitch0 - vmk0, vswif0

vswitch1 - vswif1 (iscsi service console), vmk1 (iscsi nic), vmk2 (iscsi nic)

vswitch2 - trunk, vlan tagging vm port groups

The two nics for each vswitch is part of a different physical NIC. In other words, for vswitch2, it is an intel and broadcom nic (igb, bnx2). Each nic is set to active/active. The failover and load balancing policy is as such:

Load Balancing: Port ID

Network Failure Detection: Link Status only

Notify Switches: yes

Failback: Yes

No standby or unused adapters.

I didn't set any of these servers up but its pretty consistent across the board this way.