Hi guys, ok some background. I have 2 Cisco UCS servers, both have a 2 port 10GB NIC in them and these are uplinked to a Cisco Nexus 5548. The relevant VLAn exists on the Nexus, the UCS servers can see the NICs, Vsphere sees the NICS, VSphere has an IP address associated with this VLAN for each 10GB interface, yet I can only ping one of the interfaces from the Nexus. Is there a reference doc that can walk me through the config and checking / troubleshooting this scenario?
Thanks
Dean
Problem - solved.
The issue was related to incorrect config on the Cisco CIMC for one of the servers where the 10GB NIC was setup incorrectly. as soon as that was sorted and the server rebooted, it worked!
many thanks guys for taking the time to try and help.
Dean
VSphere has an IP address associated with this VLAN for each 10GB interface
What do you mean by this? Can you share (or show) how you are using each 10 GbE interface within ESXi?
Thanks for the quick response here. i will draw up and notate the connectivity and post later in the day
Dean
If I look at the distributed switch in vSphere I go to the reevant VLAN, look at the kernel ports and can see that vmk1 has an address of 10.0.30.12 and 10.0.30.13 while vmk2 has an address of 10.0.30.14
These addresses are in the correct ip subnet for the VLAN. The VLAN interface on the Nexus 5k has an address of 10.0.30.2:
IP Interface Status for VRF "default"(1)
Interface IP Address Interface Status
Vlan30 10.0.30.2 protocol-up/link-up/admin-up
From the Nexus:
N5k-PDC-01# ping 10.0.30.12
PING 10.0.30.12 (10.0.30.12): 56 data bytes
64 bytes from 10.0.30.12: icmp_seq=0 ttl=63 time=3.399 ms
64 bytes from 10.0.30.12: icmp_seq=1 ttl=63 time=2.357 ms
64 bytes from 10.0.30.12: icmp_seq=2 ttl=63 time=2.475 ms
64 bytes from 10.0.30.12: icmp_seq=3 ttl=63 time=2.478 ms
N5k-PDC-01# ping 10.0.30.14
PING 10.0.30.14 (10.0.30.14): 56 data bytes
36 bytes from 10.0.30.2: Destination Host Unreachable
^C
--- 10.0.30.14 ping statistics ---
1 packets transmitted, 0 packets received, 100.00% packet loss
N5k-PDC-01# sh mac address-table dynamic
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
* 10 005d.731c.8c36 dynamic 10 F F Eth1/2
* 30 0050.5668.d493 dynamic 40 F F Eth1/2
It cannot see the mac address of the other NIC, although Vsphere sees it and I can see the mac addresses from Vsphere
I've seen an issue countless times with UCS (as in EVERY time), where the management interface displays the physical MAC address of a nic, rather than vmware virtual MAC (should start with 00:50:56:xxxx). It's easy enough to check from esxi shell (not SSH as it will lose connectivity if you follow all steps below):
# esxcfg-vmknic -l
Make a note of the vmk nameand the IP's you've assigned. If any don't have 00:50:56, you can delete them from cli and recreate them:
# esxcfg-vmknic -d "Management Network" (or the name of the vmk you want to recreate)
# esxcfg-vmknic -a -i <vmk IP address> -n <subnet mask> "Management Network"
Once you've done that, run the first command to check the MAC address has changed and hopefully you'll be good to go. You will also want to check all of your vmk's after this procedure to check they have the correct options checked (management, vmotion, etc), as I find it does assign whatever was enabled on a vmk you recreate to a different vmkernel.
The above is for a standard vswitch... I can't remember whether I've had to change this slightly for vDS from memory - just ensure the host is in maintenance mode when you do for a belt and braces approach. Once it has a correct address, I find the networking behaves in a much more standard way.
[Edit] Looking at the table above, the MAC showing on VLAN 10 would seem to support the above.
Hi again, from the esxi shell:
vmk2 182 IPv4 10.0.30.13 255.255.255.0 10.0.30.255 00:50:56:68:db:4c 1500 65535 true STATIC vmotion
This is the troublesome connection.
Thanks, this is what I see from th ESXI shell:
[root@esxi-13:~] esxcfg-vmknic -l
Interface Port Group/DVPort/Opaque Network IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type NetStack
vmk0 0 IPv4 10.0.10.13 255.255.255.0 10.0.10.255 cc:ef:48:b4:4d:cc 1500 65535 true STATIC defaultTcpipStack
vmk0 0 IPv6 fe80::ceef:48ff:feb4:4dcc 64 cc:ef:48:b4:4d:cc 1500 65535 true STATIC, PREFERRED defaultTcpipStack
vmk2 182 IPv4 10.0.30.13 255.255.255.0 10.0.30.255 00:50:56:68:db:4c 1500 65535 true STATIC vmotion
vmk2 182 IPv6 fe80::250:56ff:fe68:db4c 64 00:50:56:68:db:4c 1500 65535 true STATIC, PREFERRED vmotion
So this all looks good. the weird thing here is.... the other Esxi host has the same config (apart form IP of course) and it works
If I look at the distributed switch in vSphere I go to the reevant VLAN, look at the kernel ports and can see that vmk1 has an address of 10.0.30.12 and 10.0.30.13 while vmk2 has an address of 10.0.30.14
Can you show a screenshot from vSphere's side which illustrates this? What functions do vmk1 and vmk2 provide? These appear to both be on the same subnet so I'm trying to understand your design here.
Problem - solved.
The issue was related to incorrect config on the Cisco CIMC for one of the servers where the 10GB NIC was setup incorrectly. as soon as that was sorted and the server rebooted, it worked!
many thanks guys for taking the time to try and help.
Dean