VMware Cloud Community
HendersonD
Hot Shot
Hot Shot

Management network linked to two separate nic cards will not work

Just received some brand new HPE DL380 Gen10 servers. Each server comes with two separate nics, each with two 10Gbe ports. Let's call them NIC1 and NIC2

I ran a DAC cable from NIC1 on the server to my Juniper switch. I ran a second DAC cable from NIC2 to my Juniper switch. If I have a nic failure then I will not lose access to my management network

Installed ESXi 6.5 U1, and went to setup the management network. Under network adapters I chose the two nics, added an IP address, mask, gateway, and DNS information and thought I was set

I started pinging the management IP address from my workstation and it would not ping. I restarted the host and during startup I received several pings back but once fully started the pings stopped

After a lot of work I determined having two connections to two different nic cards is the issue. If I go to network setup again and only choose one nic, everything pings fine

Any ideas?

15 Replies
a_p_
Leadership
Leadership

Such a configuration shouldn't actually cause any issues.

How did you configure the vSwitch and port group (e.g. teaming, and failover policies) on the virtual side, and how did you configure the physical switch ports?

André

Reply
0 Kudos
ashwin_prakash
VMware Employee
VMware Employee

Hello Henderson,

1. Make sure that you have configured the vSwitch NIC's as Active and Standby and not both in Active State.

2. Use one at a time in Active state and test the connectivity.

Sincerely,
Ashwin Prakash
Skyline Support Moderator
Reply
0 Kudos
HendersonD
Hot Shot
Hot Shot

These are brand new servers, they have not been added to a vCenter yet. Just racked them, cabled them as I described, installed ESXi 6.5 U1, and at the console configured the management network

Reply
0 Kudos
HendersonD
Hot Shot
Hot Shot

Can the active/passive setup you describe be done at the console since these hosts have not been added to vCenter?

Reply
0 Kudos
ashwin_prakash
VMware Employee
VMware Employee

Initially use one Management NIC.

Post configuration, Use the host UI to change and update the configuration.

Sincerely,
Ashwin Prakash
Skyline Support Moderator
Reply
0 Kudos
a_p_
Leadership
Leadership

Please check the vSwitch, and port group configuration from the GUI to ensure that both NICs are in use, and configured properly. In doubt, please provide some screenshots of the current configuration

André

Reply
0 Kudos
HendersonD
Hot Shot
Hot Shot

So with two cables plugged in and two nics chosen in the console, ESXi is not smart enough to make one of them active and the other one passive? If they are both active would that explain what I am seeing?

Reply
0 Kudos
ashwin_prakash
VMware Employee
VMware Employee

It should work with both the NICs in Active, inorder to test if there is a communication issues or configuration issue.

We would have to manually configure and test both NIC individually.

Would like to confirm as you have mentioned you have tested individually, did you happen to check the second NIC individually with the same IP Address and DNS configuration.

Sincerely,
Ashwin Prakash
Skyline Support Moderator
Reply
0 Kudos
RAJ_RAJ
Expert
Expert

Hi ,

If you use both adapter in active  , incase one failed - server will accessible though other link .

Active Stand By - One failed stand adapter will be active  and server will accessible

issue may be with networking side  , Check Both ports on the switch having same configuration .

RAJESH RADHAKRISHNAN VCA -DCV/WM/Cloud,VCP 5 - DCV/DT/CLOUD, ,VCP6-DCV, EMCISA,EMCSA,MCTS,MCPS,BCFA https://ae.linkedin.com/in/rajesh-radhakrishnan-76269335 Mark my post as "helpful" or "correct" if I've helped resolve or answered your query!
Reply
0 Kudos
HendersonD
Hot Shot
Hot Shot

I opened a case with VMWare and after looking at it they think it is a problem with the HPE server. Going to open a ticket with HPE

Reply
0 Kudos
HendersonD
Hot Shot
Hot Shot

Opened a ticket with HPE and it appears that this is a problem with the nics in this server. After talking with support I am guessing it is a drive issue or an issue with the customized HPE ESXi 6.5 U1 software

ujjwal2018
VMware Employee
VMware Employee

Make sure the Physical switch interfaces connected to both the vmnic's have identical configuration  . You can also try and swap the cable on another interface from the problematic vmnic on the physical switch . Most of the time we have configuration issue on physical switch and one of the interface is either missing some vlan or not configured identical

" Please consider marking this answer "correct" or "helpful" if you think your question have been answered correctly."

Regards,

UJ

Reply
0 Kudos
ujjwal2018
VMware Employee
VMware Employee

Any update ?

Regards,

UJ

Reply
0 Kudos
HendersonD
Hot Shot
Hot Shot

Yes, sorry I did not post the solution earlier. These servers have nic cards in them based on the Intel X710 chipset. This nic has two problems:

  1. It does not handle LLDP in a standard fashion. This is the piece that caused my original problem. This article describes this issue
    Where did my host go.... - Virtual Ramblings
  2. The second issue we did not experience since we never put these servers into production with these nics. Apparently there is a Malicious Driver Detection issue that is the fault of the driver. Intel has released a new driver for ESXi 6.7 that supposedly fixes this problem. They have not release a new driver for ESXi 6.5 or 6.0

If you google Intel X710 esxi you will hit on many posts/articles that detail this nics issues. Here is but one:

https://lonesysadmin.net/2018/02/28/intel-x710-nics-are-crap/

We ended up replacing all of them with 10GB cards based on a Broadcom chipset

ujjwal2018
VMware Employee
VMware Employee

Thanks for sharing the fix Smiley Happy

Regards,

UJ

Reply
0 Kudos