Am I correct in assuming that you are using EtherChannel? Also, are you using standard or distributed switching?
Just to be thorough...with CDP enabled on the hosts and switch, please note the switch ports to which the vmnics are connected. Then, confirm that the ports configured on the 4506 are all part of the same etherchannel group and have the same VLAN configuration. Additionally, confirm that the nic teaming algorithm on the switch is IP hash if using vStandard. If you are using vDistributed Switches, confirm that the port group is configured for the IP hash.