vSphere vNetwork

Hello,

In my configuration (vsphere 5.0 U3 Ent+) I use a hybrid vswitch and dvswitch configuration on each esxi host. Each hosts' vswitch has 2 vmkernel ports one for management network and one for vmotion. There are 2 1GB pNICs dedicated to these with one being active and one standby in each respective port group. In my dvs I have 4 1GB pNICs for multiple port groups used for VM networks and also 2 more vmkernel ports one a backup management network and one for vmotion.

Last week after a major outage with our storage array required rebooting all the ESXi hosts we ran into a problem. When all the servers were finally back online I was unable to vmotion between 2 of the 9 hosts. Having already had a ticket open with vmware support for the storage issue I asked them to have a look into what's going on with vmotion. After some troubleshooting we were able to get vmotion working on those 2 hosts by disabling vmotion on the vswitch vmkernel ports and enabling it on the vswitch management network ports. I was told that vmware does not support multi-nic vmotion when both vswitches and dvswitches are in use and that I needed to chose either vswitch or dvswitch only to have multiple vmotion nics. I was promised a document to confirm this but have not received it yet. While I rather not make design changes to my environment to troubleshoot an issue, if my configuration is not supported I will change it. Can anyone here confirm this issue with my configuration while I wait to hear back from support?

Thanks

Wanted to bump this. Still having trouble getting vmware support to answer my question and to provide documentation that they promised.

Hello,

I use a hybrid vSwitch network as well. vMotion will work if the following is true:

     The network setup for the vMotion network is correct on each host

     You can vmkping each vMotion network IP across hosts

vMotion is fairly straightforward so if vMotion is Not working then you need to check into your networking where vmkping is your friend. Can you vmkping the vMotion IP on each host from each host? If you cannot communicate between hosts then your network is not correct. I would

     Verify the IP addresses used for vMotion are correct

     Verify the vmkernel adapter used for vMotion is properly assigned to vMotion and not FT or management

     Verify the gateway for the vMotion network is correct

     Verify that that all vMotion network cables are correctly connected to each physical Switch

     If there is a router between the machines on the vMotion network make sure it routes properly

Best regards,
Edward L. Haletky
VMware Communities User Moderator, VMware vExpert 2009, 2010, 2011,2012,2013,2014

Author of the books 'VMWare ESX and ESXi in the Enterprise: Planning Deployment Virtualization Servers', Copyright 2011 Pearson Education. 'VMware vSphere and Virtual Infrastructure Security: Securing the Virtual Environment', Copyright 2009 Pearson Education.

Virtualization and Cloud Security Analyst: The Virtualization Practice, LLC -- vSphere Upgrade Saga -- Virtualization Security Round Table Podcast

Thanks for your reply.

     The network setup for the vMotion network is correct on each host --- Yes, configuration has not changed and is correct

     You can vmkping each vMotion network IP across hosts --- Yes, vmkping works all around between any of the vmk interfaces between all the hosts

     Verify the IP addresses used for vMotion are correct --- IPs are the same and are on the same segment for all hosts for both the vswitch and dvs interfaces

     Verify the vmkernel adapter used for vMotion is properly assigned to vMotion and not FT or management --- verified

     Verify the gateway for the vMotion network is correct --- verified

     Verify that that all vMotion network cables are correctly connected to each physical Switch --- this I'll need to verify but no indications of any of the pNICs being down from vmware side

     If there is a router between the machines on the vMotion network make sure it routes properly - hosts are patched into multiple switches, not sure how the routing is setup, will verify with our network guy

Hello,

If you can vmkping then generally things are all right with the network unless there is a router/FW that blocks layers 4-7... still worth checking out.

One other thing, do not ever do a security scan of a vMotion port as it will shutdown vMotion completely and only a reboot of the node will allow it to work again. So perhaps there was some other issue... Have you rebooted since resetting up vMotion? Is there anything on this network that is NOT an ESXi host, if so that is also a no-no and could cause issues.

Best regards,
Edward L. Haletky
VMware Communities User Moderator, VMware vExpert 2009, 2010, 2011,2012,2013,2014

Author of the books 'VMWare ESX and ESXi in the Enterprise: Planning Deployment Virtualization Servers', Copyright 2011 Pearson Education. 'VMware vSphere and Virtual Infrastructure Security: Securing the Virtual Environment', Copyright 2009 Pearson Education.

Virtualization and Cloud Security Analyst: The Virtualization Practice, LLC -- vSphere Upgrade Saga -- Virtualization Security Round Table Podcast

If there is a router between the machines on the vMotion network make sure it routes properly - hosts are patched into multiple switches, not sure how the routing is setup, will verify with our network guy

The gist of this is that vMotion is supported only between vmkernel ports residing on the same layer 2 broadcast domain (VLAN, subnet) without any layer 3 routing in between. You're fine in that department if it's just the same VLAN strechted on multiple physical switches.

I suppose there is no public official document and it was more of a generic attempt of explaining a sporadic issue.

Just wanted to come back here and thank you guys for your help and let you know that the issue is now resolved. I am still not sure what caused the issue or what actually fixed it. At all times vmkping worked between all hosts. My network guy was able to rule out any network issues/changes/firewalls/etc. I re-checked the configuration on the 4 host cluster multiple times and verified that everything was configured identically and vmotion was still failing, only disabling the vswitch vmotion vmkernel ports and either enabling vmotion on the management network vmkernel ports on the vswitches or only having vmotion on the vmkernel ports on the dvswitch would work. I ended up picking 2 of the hosts and breaking down and rebuilding their network configuration testing with one vmkernel port at a time also testing with multiple pNICs one at a time and gradually building back up to my original configuration as with each step I was unable to reproduce the vmotion failures. Once the 2 hosts were back up and I was able to vmotion freely between them I changed the other 2 hosts configuration back to match these 2 hosts and again everything just worked. It's been almost a week now without any issues.

Meanwhile vmware support never got back to me, last communication was almost a week ago from me requesting an escalation. It's very disappointing to be given an incorrect answer and then to be ignored. Not my typical experience with vmware support.