After restart of VM network adapter fails

[insearchof]

VMware 6.0

I have vCenter running with 4 ESXi 6.0 hosts.

On one of my ESXI hosts which is the last one I installed it has 4 network adapters.

I have several VM's running on this Host Windows 2012 R2 servers Windows 2016 Data Center and 4 Linux Ubuntu, CentOS SUSE Enterprise and Photon vv1.0 which is VMware vCenter Appliance

After restarting any of the above most times on some I do not get network connectivity. I must edit the VM and add another Network adapter. Sometimes I need to remove all and add another.

Any Idea on why my VM's do this?

All other ESXI hosts only have two Network adapters and they do not have this problem.

Thank you in advance

Tom

[daphnissov]

Sounds interesting. Can show screenshots of this host's networking configuration with respect to the VM networking (switch, port groups, settings on port groups)? Also, when you make this statement

After restarting any of the above most times on some I do not get network connectivity.

how are you determining there is no network connectivity? As in you cannot, from the outside, contact the guest? From inside the guest it cannot talk out? Both?

Can you also attach vmware.log for one of these VMs after it has been rebooted and no network connectivity observed?

[insearchof]

When the VM comes up I can not ping by ip address or FQDN on the VM itself I can not ping either.

I attached some screen shots of my Network setup on this ESXI Host let me know if you need more info

Where is the VMware.log file kept on the VM?

[insearchof]

I  found the VMWARE.LOG files

I attached two machines that have the same issue.

One is a Windows 2012 R2 Server and the other is the vCenter 6.5 Appliance

[daphnissov · Accepted solution]

vmware.log is kept in the home directory of a VM with the other files.

Your last screenshot is interesting. It observes no ranges on vmnic1 but does for vmnic0. What networking device is connected upstream? Since this is a vSS, can you show CDP information for each vmnic?

As a test, if those VMs have network connectivity now, change the teaming and failover properties of that port group to set vmnic1 as active and vmnic0 as unused. When you save the changes, do those VMs still have network connectivity?

[insearchof]

I posted the VMware logs did you see them?

Not sure how to get you the CDP information can you send me an example of what to look for?

All the VMs are now connected except for one that I was working on today.

I will look into the teaming later

But how do I get ranges added to the vmnic1 ?

[daphnissov]

I saw the logs.

From the web client, navigate to your virtual switch configuration and click the information button next to the uplink.

You don't add ranges in vSphere, they're discovered from the physical network which means you probably have an inconsistent network configuration outside of your vSphere.

Run the test I recommended and report the results. Post CDP information for both vmnics serving as uplinks for the vSS hosting your VM port groups.

[insearchof]

Attaching the CDP of both nics

Did you see the log files?

[daphnissov]

Two things I see from your CDP information:

1. You are using two different switch firmware versions for your uplinks. vmnic0's switch (10.2.8.4 management IP) is a higher version than vmnic1's switch (10.2.8.5).
2. You have labeled the VLANs inconsistently across your switches. vmnic0 has VLAN 1 presented to it and vmnic1 has VLAN 1019.

How are these switches configured aside from this?

If you change the teaming policy on your switch as I suggested to only use vmnic1, then you may lose connectivity. Be prepared to login to the host where vCenter is running and change the teaming policy back if this is the case.

On your Windows VM you're also using 3 vNICs using e1000e adapter types which are not best practice and you should consider changing to VMXNET3.

2017-12-09T22:51:41.612Z| vmx| I120: DICT   ethernet2.pciSlotNumber = "256"

2017-12-09T22:51:41.612Z| vmx| I120: DICT      ethernet2.virtualDev = "e1000e"

2017-12-09T22:51:41.612Z| vmx| I120: DICT     ethernet2.networkName = "Cluster"

2017-12-09T22:51:41.612Z| vmx| I120: DICT     ethernet2.addressType = "vpx"

2017-12-09T22:51:41.612Z| vmx| I120: DICT ethernet2.generatedAddress = "00:50:56:9e:3a:04"

2017-12-09T22:51:41.612Z| vmx| I120: DICT         ethernet2.present = "TRUE"

2017-12-09T22:51:41.612Z| vmx| I120: DICT           config.readOnly = "TRUE"

2017-12-09T22:51:41.612Z| vmx| I120: DICT   ethernet0.pciSlotNumber = "192"

2017-12-09T22:51:41.612Z| vmx| I120: DICT      ethernet0.virtualDev = "e1000e"

2017-12-09T22:51:41.612Z| vmx| I120: DICT     ethernet0.networkName = "VM Network"

2017-12-09T22:51:41.612Z| vmx| I120: DICT     ethernet0.addressType = "vpx"

2017-12-09T22:51:41.612Z| vmx| I120: DICT ethernet0.generatedAddress = "00:50:56:9e:05:91"

2017-12-09T22:51:41.612Z| vmx| I120: DICT         ethernet0.present = "TRUE"

2017-12-09T22:51:41.612Z| vmx| I120: DICT   ethernet1.pciSlotNumber = "224"

2017-12-09T22:51:41.612Z| vmx| I120: DICT      ethernet1.virtualDev = "e1000e"

2017-12-09T22:51:41.612Z| vmx| I120: DICT     ethernet1.networkName = "VM Network"

2017-12-09T22:51:41.612Z| vmx| I120: DICT     ethernet1.addressType = "vpx"

2017-12-09T22:51:41.612Z| vmx| I120: DICT ethernet1.generatedAddress = "00:50:56:9e:69:1c"

2017-12-09T22:51:41.612Z| vmx| I120: DICT         ethernet1.present = "TRUE"

[insearchof]

Remember this server had 4 nics? And I only had three nics connected to my cisco switches.

Come to find out I had the wrong vmnic teamed

vmnic3 was the one

So I changed the team settings and then went to a server that I added the extra network adapter and removed it  and the network remained working.

Next test was to restart the server and that also worked network back

So now to remove the network adapters from all the other servers.

Thanks for all your help

This can be marked as resolved