Odd Network Issues Since Migrating Environment From vSphere 6U3 > 6.5U1

[coolsport00]

Hello all -

Need your assistance if you can spare a few - have been having definitive network issues in 6.5 U1. Seems to be vDS related. Haven't noticed issues when Host/VMs run on vSS. What issues? VMs occasionally not ping'ing. Cisco Voice servers not getting NTP info from 'cluster' member VM (2 VMs per Cluster for those familiar with Cisco - pub/sub servers). Domain Controller VMs having occasional communication issues (this one I'm not convinced on but my Director seems to think there's an issue here). When VMs run on vSS...issues seem to have dissipated. Weird for sure.

Also, when I was beginning the migration from 6U3 > 6.5U1, and disconnected Hosts from old (6.0U3) VCSA & connecting them to new (6.5U1) VCSA, our "services" (web UI's, DC connectivity, etc.) went down even though VM guest OS & OS (Windows) services didn't go down. We had to reboot all VMs. Never had this happen when just connecting Hosts with running VMs to a new vCenter. Has anyone else ever had this happen? And none of this, Host Connection issue or the vDS issue I described earlier has ever happened before. Anyone else seeing "odd" network issues with 6.5U1?...and seemingly mainly when running on vDS? I've been Googling for quite a while and haven't seen anything out there (yet). My org doesn't have VMware Support unfortunately, so I can't pursue this from a VMW Support perspective, sadly. 😕

Thanks in advance.

[msripada]

Hello,

I have few questions there.. When you have an ESXi host with DVS or vDS (where VMs residing) and removed it from the vCSA 6.0 update 3 and adding it to 6.5 U1 VCSA. The DVS is managed by vCenter 6.0 U3. Have you tried to export the configuration of DVS and restored it to VCSA 6.5? Did you create a new DVS on 6.5?

Is the VM network dropping when the host is added to 6.5?

VDS is managed by vCenter, when you remove the ESXi host from vCenter 6.0 and add it to 6.5, it has a different VC ID or managed ID for the VDS.

If you leave the host disconnected from Vcenter 6.0 , will the VM networking work fine? and is it just adding the host to VCSA 6.5 making the VM connection drop?

Thanks,

MS

[coolsport00]

@MS -

Thank you for the response. Not sure why I didn't get notified when you responded, so apologies for my delayed response. When I disconnected Hosts from vC 6U3 and connected them to vC 6.5U1, I first migrated the Hosts & VMs from the vDS in 6U3 vC to a vSS. I did this for all my Hosts & VMs. So, VMs were all running on a local switch to the Host (a vSS). Obviously, when I connected the Hosts to the new vC 6.5U1 and began having issues even then, VMs & Hosts were 1. not upgraded yet... still on vSphere 6U3 and VMware Tools on VMs weren't upgraded obviously; and, 2. Hosts were running (& VMs connected to) the Host's Standard Switch (vSS). I rebooted the VMs & things started running ok.

I then began upgrading my Hosts to 6.5U1. Initial config of the Hosts I would obviously configure a vSS then put VMs on the Host. After, I would then migrate VMs to the vCenter 6.5U1 vDS (vDS ver 6.5). My vCenter (using appliance [VCSA]) was/is new. Not an upgrade. It was a new install. After install and config, I then created the Distributed Switch before I began migrating Hosts & VMs to it. Hope that helps clarify a bit.

Thank you.

[msripada]

Hello coolsport00,

Thanks for the information. I do not know what is the network drivers you are using but there are cpl of issues with ntg3 on 6.5. If you face this issue after upgrading to 6.5 then kindly try to check the driver/firmware

VMware Knowledge Base

Thanks,

MS

[coolsport00]

@MS -

Thank you for the suggestion. Unfortunately (umm..or fortunately ), we are not using that driver. And, that driver is solely for the Broadcom Adapters. We do have Broadcoms, and the 'model' of Broadcom in our Hosts are the ones affected by both these KBs - https://kb.vmware.com/kb/2150728 and https://kb.vmware.com/kb/2150889 - but we use HP 10Gb 522 and 554 SFP 2-port Adapters for both Network & ISCSI traffic.

[msripada]

Interesting. However when you are on standard switch you are not seeing the issue but only when using DV, you are seeing this issue. Are there any changes in the configuration on the DVS which are different?

Thanks,

MS

[coolsport00]

Correct... when on the vSS, we are not seeing the same behavior. Most everything seems to be fine. The only difference we had on the vDS is Load Balance policy in the vDS was 'traffic based on NIC load', whereas in the vSS the policy is 'based on originating virtual port'. That shouldn't matter though. We don't really use the vDS to its full feature-potential. We only used it as a central network configuration for Hosts.

Thanks.

[msripada]

I do not think that policy with traffic based on nic load should impact anything as it does load balancing. Do you see any packet drops or anything for the NICs when using esxtop?

Thanks,

MS