Any connections made from 10Gb physical ports don't work

[alwalters]

We've recently upgraded our network equipment, so we can now make use of the 10Gb ports on our servers.  Two servers have Cisco VIC 1227 cards, and the other two have Intel x520 (both are dual 10Gb port).  We've connected the ports to the switches using approved twinax cables, and configured the ports appropriately.  In ESXi (5.5 U3), I can see the card and see both ports.  We first tried just a very standard config, with a single port going to a new virtual switch, and then created an adapter in one of the Windows guest VMs.  The adapter shows up fine, but does not get an IP address.  Tried statically configuring one (which would be how we would normally set it up anyway), but it couldn't ping anything else on the network.  The port config should be fine as far as we can tell, in terms of VLAN and whatnot, as if we configure an open 1Gb port on the same server to the same switch everything works perfectly.

I've also tried creating a new vmkernel port - same issue. Even tried enabling one port for passthrough, and adding it into the VM as a PCI device.  Windows recognizes it, gets the driver (via the old 1Gb interface, when I re-enabled it for this), and shows it as the correct hardware.  But again, no connectivity.

Interestingly, the switch shows the ports as being up, packets as coming and going, and no error messages.  But I just cannot seem to connect with these ports in ESXi no matter how I try.  I updated the drivers with the most recent from the VMWare site, and the cards are both listed on the compatibility list for our ESXi version.

I'm stumped as to what to try next, where to look, etc...  Any posts I'm seeing seem to reference the 10Gb ports not showing up at all, but that's not the case here.  Everything I can see looks like everything's perfect; you know, except for the fact they just don't work.  😄

[ITaaP]

Assuming you have separate 1Gb and 10Gb switches? Are the switches connected to each other? Do you have any core switches that the 1Gb and 10Gb switches are uplinked to? Sounds more like a network configuration issue than ESXi. Unless you don't have the correct VLAN set on the port group.

[alwalters]

No, they're not separate - they're switches (specifically Catalyst 2960Xs) that have both 1Gb and 10Gb ports.  All of the network connectivity everywhere else is fine, and if we set up a 1Gb port with the same VLAN as we set on the 10Gb port, it works in ESXi just fine.

Cisco has reviewed it and so far not found any issues with the config.  (Waiting for a second confirmation after more logs were sent, though.)  So figured I'd try here and see if there was anything I was missing from the ESXi end.

[ITaaP]

Could you possibly paste your switch config here?

[alwalters]

Updating here in case it's useful for anyone later.  Heard back from Cisco, and they couldn't find any issues in the config, so we did a remote session to troubleshoot further.  Eventually, we found that the physical ports on the servers apparently default to being trunk ports, and in the switch and in esxi we had everything set up as access ports.  They thought that should still work, but had me test putting the switchports/port-channels to trunk and adding the VLAN info on the vSwitch, and that solved the problem.  They didn't believe it strictly should've been necessary, but...¯\_(ツ)_/¯

Theoretically we could have changed the server ports to be access and kept the existing switch/esxi config, but that would have required full reboots of all of the servers, so we did this for now.  Don't necessarily see a reason to change it, but will ponder.

Anyway, it did turn out to be network related, but it may still be useful info to someone, so...

[ITaaP]

Glad to hear it was resolved. Typically trunked is the way to go if you need to support multiple VLANs. Even if you don't today, you may want to add more VLANs in the future.