We have a a vSphere cluster of 6 hosts that has been running quite well for years. We have vmotioned VM's in the past without incident, but since it's not something we need to do on a regular basis, it's been a while. One of the hosts had one of it's OS disks go bad, so I started vmotioning systems off of it so I could put it maintenance mode before replacing the disk. Wasn't strictly necessary (the disk was hot swappable) but hey, better safe than sorry. However, when the VM's finished moving to another host...they stopped responding on the network entirely. Couldn't even ping the gateway. This has never happened before. Move them back to the original host, and they immediately start pinging again.This also happens if I shutdown the VM, THEN migrate it, and power it back up again. Some VM's, after being moved, will have intermittent network outages. 8 pings go fine, then 10 time out, then 8 or 9 are fine. Sometimes with a pattern like that, and sometimes not. Again, move them back to the original host and all is well.
The hosts are connected to a Cisco switch stack with two nics each, configured with cross stack etherchannel. The nic teaming is configured for Ip Hash with Switch notify. This configuration has been in production for years, and vmotion worked fine on many occasions. We have checked the MAC address table on the switch stack and it is being properly updated as soon as the vmotion finishes. Traffic sniffing shows that some packets from a moved VM are getting out to the network, but not all. For example, after moving one VM, wireshark on a system on the same VLAN saw ARP broadcasts from that VM requesting the MAC address of the network gateway. The other odd thing is that if I create a brand NEW VM, I can vmotion it from host to host to host all day long with zero problems. I also had another VM that exhibited issues, but when I switched out it's virtual network "card", the problems also ceased and I was able to vmotion it at will without any issues. Also, no network issues from any VM's on the hosts have been seen. Everything seems to work just great...as long as I don't try to move anything to a new host. 
From everything I've seen, it's almost as if something has "locked" each VM to it's respective host and moving it anywhere else causes the networking to fail. And before you ask, no...port security is not turned on on the switch.
Full disclosure, these hosts are still running ESXi 5.1 (though vCenter was upgraded this summer to 5.5). I haven't gotten around to upgrading, and now I'm in a bit of a pickle because even though I WANT to upgrade, if I can't vmotion the production systems to other hosts, it becomes rather difficult. Plus, even though our support contracts are paid up, I cannot get any support from Vmware on this issue. I was hoping that someone here may have run into a similar problem and could provide some guidance as to things I could investigate.
I *suspect* that this is something switch related. We recently (few weeks ago) had a bunch of network problems that were traced back to a new switch installed on the network doing Ip Device Tracking. I don't know what that could have done to the VMware cluster, but it's the only strange network occurrence we've had lately. Plus, since it's been probably months since we have vmotioned anything, I can't make a correlation between those network issues and this one. The only thing that makes me suspect it is that our monitoring system (which is on the production vmware cluster) suddenly started having trouble reaching one particular VM on a DEV vmware host in another building. It's intermittent, and it has no trouble reaching other VMs on that same host. I feel like the two issues are related, but I can't put my finger on it.
Accepted Solutions
