What does that say about Kia? 
I think we *may* have possibly solved this issue in our production cluster, but I'm still testing. We are still seeing this in our DMZ cluster, however, and we're about to set up monitoring to see what's going on.
The issue we found in our production cluster is that we had one VM uplink port, on only 1 of 3 hosts, that was assigned to native vlan 35, which is our vlan for iSCSI traffic. Once we removed this setting (it should have been a different vlan), then we have now been able to vmotion to all 3 hosts in the cluster without losing more than a packet here and there. What I don't understand is how can one port on one host affect lost networking when doing a vMotion throughout that 3-host cluster? Our issue has not been isolated to the one host with the misconfigured switchport.
Now, the one thing about our DMZ cluster switches is that they are dumbed down. We basically have no port configurations on those switches because it is a very small dmz cluster. Could this cause a problem? Is there a minimum configuration that's required for reliable service?
I will say that I have suspected the Broadcom nics in the past but since we use them pretty heavily and we really haven't seen any issues elsewhere, I could never zero in on that.
wkucardinal wrote:
Still having the issue..
Sometimes it could be useful to really verify that all VMNIC uplinks for all VLAN does work. One way to try this is to create a new portgroup on the vSwitch used by your VMs on the first host, put one test VM on this portgroup, then go into the NIC teaming policy of the new portgroup and select "Override switch failover order".
Then move down all VMNICs except one to unused, so only one VMNIC is active. Then set the portgroup VLAN settings to one of the production VLANs and try to see if you could ping some different expected addresses. If this works, then move the working VMNIC down to unused and move up another to Active. Try again, and do this for all VMNICs. If this works then you have verifed that the VLAN configuration and other settings are correct on the physical switch ports facing this host.
If having multiple VLANs, repeat the process for all other production VLANs. Then repeat the process on the other hosts.
While might take a while, does will verify if everything is correctly configured on the physical switches. When a vMotion takes place the VM gets a new "Port ID" and is assigned a new outgoing VMNIC. If there is a configuration error on one or several physical switch port this could seem random, but perhaps always happens on VLAN x on VMNIC y. Since the Port ID policy you are using in effect will randomly spread the VMs over the VMNICs these problems could be hard to diagnose. (Doing a disconnect of the Virtual Machine vNIC gives the VM a new port-ID which will move it to a new outgoing VMNIC, which might seem to solve the problem.)
Rickard's suggestion was very helpful. We were able to isolate more than one port in each of our trouble environments and correct them using his suggestion. I believe our "dropped network" issues have been resolved after testing. If not, I'll be back.