VMware Cloud Community
xyzcloud
Contributor
Contributor
Jump to solution

Vcenter 6.5 - Cisco UCS and 10GB NICS, cannot seem to get these operational.

Hi guys, ok some background. I have 2 Cisco UCS servers, both have a 2 port 10GB NIC in them and these are uplinked to a Cisco Nexus 5548. The relevant VLAn exists on the Nexus, the UCS servers can see the NICs, Vsphere sees the NICS, VSphere has an IP address associated with this VLAN for each 10GB interface, yet I can only ping one of the interfaces from the Nexus. Is  there a reference doc that can walk me through the config and checking / troubleshooting this scenario?

Thanks

Dean

Reply
0 Kudos
1 Solution

Accepted Solutions
xyzcloud
Contributor
Contributor
Jump to solution

Problem - solved.

The issue was related to incorrect config on the Cisco CIMC for one of the servers where the 10GB NIC was setup incorrectly. as soon as that was sorted and the server rebooted, it worked!

many thanks guys for taking the time to try and help.


Dean

View solution in original post

Reply
0 Kudos
7 Replies
daphnissov
Immortal
Immortal
Jump to solution

VSphere has an IP address associated with this VLAN for each 10GB interface

What do you mean by this? Can you share (or show) how you are using each 10 GbE interface within ESXi?

Reply
0 Kudos
xyzcloud
Contributor
Contributor
Jump to solution

Thanks for the quick response here. i will draw up and notate the connectivity and post later in the day

Dean

Reply
0 Kudos
xyzcloud
Contributor
Contributor
Jump to solution

If I look at the distributed switch in vSphere I go to the reevant VLAN, look at the kernel ports and can see that vmk1 has an address of 10.0.30.12 and 10.0.30.13 while vmk2 has an address of 10.0.30.14

These addresses are in the correct ip subnet for the VLAN. The VLAN interface on the Nexus 5k has an address of 10.0.30.2:

IP Interface Status for VRF "default"(1)

Interface            IP Address      Interface Status

Vlan30               10.0.30.2       protocol-up/link-up/admin-up      

From the Nexus:

N5k-PDC-01# ping 10.0.30.12

PING 10.0.30.12 (10.0.30.12): 56 data bytes

64 bytes from 10.0.30.12: icmp_seq=0 ttl=63 time=3.399 ms

64 bytes from 10.0.30.12: icmp_seq=1 ttl=63 time=2.357 ms

64 bytes from 10.0.30.12: icmp_seq=2 ttl=63 time=2.475 ms

64 bytes from 10.0.30.12: icmp_seq=3 ttl=63 time=2.478 ms

N5k-PDC-01# ping 10.0.30.14

PING 10.0.30.14 (10.0.30.14): 56 data bytes

36 bytes from 10.0.30.2: Destination Host Unreachable

^C

--- 10.0.30.14 ping statistics ---

1 packets transmitted, 0 packets received, 100.00% packet loss

N5k-PDC-01# sh mac address-table dynamic

Legend:

        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC

        age - seconds since last seen,+ - primary entry using vPC Peer-Link

   VLAN     MAC Address      Type      age     Secure NTFY   Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------+----+------------------

* 10       005d.731c.8c36    dynamic   10         F    F  Eth1/2

* 30       0050.5668.d493    dynamic   40         F    F  Eth1/2

It cannot see the mac address of the other NIC, although Vsphere sees it and I can see the mac addresses from Vsphere

Reply
0 Kudos
alsmk2
Hot Shot
Hot Shot
Jump to solution

I've seen an issue countless times with UCS (as in EVERY time), where the management interface displays the physical MAC address of a nic, rather than vmware virtual MAC (should start with 00:50:56:xxxx). It's easy enough to check from esxi shell (not SSH as it will lose connectivity if you follow all steps below):

# esxcfg-vmknic -l

Make a note of the vmk nameand the IP's you've assigned. If any don't have 00:50:56, you can delete them from cli and recreate them:

# esxcfg-vmknic -d "Management Network" (or the name of the vmk you want to recreate)

# esxcfg-vmknic -a -i <vmk IP address> -n <subnet mask> "Management Network"

Once you've done that, run the first command to check the MAC address has changed and hopefully you'll be good to go. You will also want to check all of your vmk's after this procedure to check they have the correct options checked (management, vmotion, etc), as I find it does assign whatever was enabled on a vmk you recreate to a different vmkernel.

The above is for a standard vswitch... I can't remember whether I've had to change this slightly for vDS from memory - just ensure the host is in maintenance mode when you do for a belt and braces approach. Once it has a correct address, I find the networking behaves in a much more standard way.

[Edit] Looking at the table above, the MAC showing on VLAN 10 would seem to support the above.

Reply
0 Kudos
xyzcloud
Contributor
Contributor
Jump to solution

Hi again, from the esxi shell:

vmk2       182                                     IPv4      10.0.30.13                              255.255.255.0   10.0.30.255     00:50:56:68:db:4c 1500    65535     true    STATIC              vmotion            

This is the troublesome connection.

Thanks, this is what I see from th ESXI shell:

[root@esxi-13:~] esxcfg-vmknic -l

Interface  Port Group/DVPort/Opaque Network        IP Family IP Address                              Netmask         Broadcast       MAC Address       MTU     TSO MSS   Enabled Type                NetStack           

vmk0       0                                       IPv4      10.0.10.13                              255.255.255.0   10.0.10.255     cc:ef:48:b4:4d:cc 1500    65535     true    STATIC              defaultTcpipStack  

vmk0       0                                       IPv6      fe80::ceef:48ff:feb4:4dcc               64                              cc:ef:48:b4:4d:cc 1500    65535     true    STATIC, PREFERRED   defaultTcpipStack  

vmk2       182                                     IPv4      10.0.30.13                              255.255.255.0   10.0.30.255     00:50:56:68:db:4c 1500    65535     true    STATIC              vmotion            

vmk2       182                                     IPv6      fe80::250:56ff:fe68:db4c                64                              00:50:56:68:db:4c 1500    65535     true    STATIC, PREFERRED   vmotion            

So this all looks good. the weird thing here is.... the other Esxi host has the same config (apart form IP of course) and it works

Reply
0 Kudos
daphnissov
Immortal
Immortal
Jump to solution

If I look at the distributed switch in vSphere I go to the reevant VLAN, look at the kernel ports and can see that vmk1 has an address of 10.0.30.12 and 10.0.30.13 while vmk2 has an address of 10.0.30.14

Can you show a screenshot from vSphere's side which illustrates this? What functions do vmk1 and vmk2 provide? These appear to both be on the same subnet so I'm trying to understand your design here.

Reply
0 Kudos
xyzcloud
Contributor
Contributor
Jump to solution

Problem - solved.

The issue was related to incorrect config on the Cisco CIMC for one of the servers where the 10GB NIC was setup incorrectly. as soon as that was sorted and the server rebooted, it worked!

many thanks guys for taking the time to try and help.


Dean

Reply
0 Kudos