So how should the switch side be configured to support this? I'm assuming no LACP or any type of port channeling, just trunk ports (Cisco switches). It's my understanding that when a decision is made to move the traffic to another nic that a rarp would be done so the switch knows the traffic has moved to another nic. This could cause a drop packet or two (it does when vmotion does a similar move). Anyone have any experience on LBT they can share?
You're pretty much correct, ESX does however support a non-negotiation LACP configuration.
If you do want additional throughput load balancing then you could use etherchannel on physical side and IP Hash on vSwitch.
Lots of info on here really
http://communities.vmware.com/message/1392817 just for starters...
And theres the official vmware docs too...
It would also be useful to keep 'Link state tracking' turned on aswell on the physical side.
Personally, I try to stick to the default settings of the vSwitch unless something e.g. performance dictates a change necessary.
You'll get the best load balancing & bandwidth utilization by using a Static EtherChannel upstream, with IP Hashing on the vSwitch. There's no such thing as "non negociated LACP". LACP is a negociated protocol (corresponding Cisco command is "mode active" on the Port Channel interface). A static port channel does not require negociation (Cisco command "mode on") and is what you would use to aggregate two adapters connecting to an ESX vSwitch.
Having one large logical "pipe" will handle traffic more efficeintly than MAC or Virtual port hashing that might cause one of the two links to undergo higher utlization than the other.