As I understand it, (again that caveat, not a network guy) you can't span a vswitch across switch ports without a violation of spanning tree which disables all but one port UNLESS your switches support cross stack link aggregation. Someone correct me if I am wrong, please.
You are correct. There is some confusion of the terms only.
There is a network standard called 802.3ad (workgroup name) or 802.1AX (real name now, but the workgroup name sticks around.) This explains how two units can bundle several links together and logically treat them as one. On Cisco switches this is called an "Etherchannel", on HP Procurve switches this is called a "Trunk" and on most servers this is called a "NIC team". On ESX this is called "IP hash".
Both sides has to be configured for this, otherwise the same MAC address will appear at the same time on different ports and this would cause various serious problems, including Spanning Tree trouble, but also other things.
The 802.3ad has two modes, static or dynamic configuration of the link aggregation. The static is quite simple, both sides must just be manually configured. When using dynamic a certain protocol called LACP is used to negotiate this. Cisco also has its own propertiary PaGP protocol that does the same thing.
ESX/ESXi hosts only has support for static link aggregation, so this must also be set on the physical switch side.
The 802.3ad/802.1AX does not specify HOW the loads should be balanced across the available links. That is up to the different units to decide and it could very well be different on both sides of this without problems.
Many switches (both physical and also the Vmware vSwitch) uses a load balancing method that includes the IP adresses in the source and destionation to choose an outgoing NIC. This works well and is quite simple to do, but limits a session between any two IPs to ONE single NIC. If link failure should occur it will switch to a new one.
Some switches has the possibiliy to use more advanced algorithms for this, for example including both src/dst IP address and also TCP port numbers. This is more "expensive" to do as the packet has to be inspected deeper and also causes more CPU overhead, but will allow the load between two IP hosts to use several NICs. This is not supported on the standard vSwitch or the Distributed Switch, but on the 3rd party Nexus 1000v.
However, all methods is link aggregation, just differences in how to place the frames onto the NICs.
As for using two different physical switches the standard does not allow this. The aggregated links must start and end between only two units. However, several switch vendors (e.g. Cisco and HP) has made up their own solutions for this, called Stacked Switch Etherchannel or Distributed Trunking for example.
IF you have such switches with this feature and if configured correctly on the switch side you could set up for example four VMNICs with "IP hash" on your ESX/ESXi and connect these to the two physical switches. This would probably be the best combination of fault tolerance and load balancing.
My VMware blog: www.rickardnobel.se
