Common Network Questions

Version 3

    The purpose of this document is to capture the most common network questions and answers.

    You can edit this document and add your own questions, or you can add a question in the Comments below, or post a question in Re: audio hardware issues in guest o/s.

    There are 5 questions so far

    1. Why is some IP traffic not being forwarded - I'm using IP Hash
    2. Is there a way to force the choice of a pNIC in a virtual switch?
    3. Can I mix pNIC types in a vSwitch; what happens if they have different features?
    4. How can I tell what features my physical NICs (pNICs) have?
    5. What is the best way to allocate my pNICs between the service console, vmotion, production network, DMZ and possibly storage?

     

     

    1. Why is some IP traffic not being forwarded - I'm using IP Hash



    Some IP traffic is not being forwarded by the vSwitch.  Each pNIC goes  to a different physical switch.  Load balancing algorithm is "IP Hash".

    IP Hash hashes the source and destination IP  address of each frame together and takes that modulo the number of pNICs  in the team to choose a pNIC. MAC hash takes the LSB of the source MAC  in each frame modulo the number of NICs to choose a pNIC.

    In this case you may not use IP based hash since it will confuse the  switches which are unable to do link aggregation across multiple  chassis.  If they are connected to the same switch then the switch ports  must be set to 802.3ad static mode with no LACP or PAGP enabled.

     

     

    2. Is there a way to force the choice of pNIC in a virtual switch?



    You can create policies wihta single active pNIC and the rest standby.
    Only the active will be used until a failure occurs and then we will  pull in a standby.  Since you can have different teaming policies  associated with different portgroups you can configure something like  the following:

    portgroupA:
    • pNIC1 active
    • pNIC2 standby
    • pNIC3 standby

     

    portgroupB:
    • pNIC1 standby
    • pNIC2 active
    • pNIC3 standby

     

    portgroupC:
    • pNIC1 standby
    • pNIC2 standby
    • pNIC3 active

    This will result in Vms attached to portgroupA always using pNIC1 unless  it fails, portgroupB always using pNIC2 unless it fails etc.  But in  the even of failure any of the 3 pNICs can be used as a backup for any  of the 3 portgroups.

    3. Can I mix pNIC types in a vSwitch; what happens if they have different features?



    Yes you can mix different pNIC types in a vSwitch - the vmkernel handles  each pNIC differently in a vSwitch and they don't have to be all the  same brand/vendor, or have all the same features (except Jumbo Frames).

    What does this mean?

     

    • This means that the same vmkernel driver is not required for all pNICs in a vSwitch.

    the vmkernel will use TSO on that pNIC, and perform segmentation in software before forwarding packets to others.

     

    • If one pNIC has a feature that another pNIC doesn't, the vmkernel  will emulate that feature on behalf of the pNIC that doesn't have the  feature which will increase the CPU workload by some unspecified  measure.  TSO is an example of one such feature, where the vmkernel will  perform segmentation on behalf of a pNIC that doesn't have a TSO  feature.
    • Jumbo Frames are an exception to this rule - all pNICs must have the Jumbo Frame feature in a vSwitch or it isn't used.
    • The fact that one pNIC has a feature that the others in the vSwitch don't, does not mean that the unique feature is switched off; all pNICs are not reduced to the "lowest common denominator" feature list.  The vmkernel  will in effect "bring up" all pNICs by emulating the features they don't  have, like TSO but not Jumbo Frames.
    NOTE This last point is new for VI3, and is a change from previous ESX2.5 versions


    people/rodos highlighted a quote from the VMware Networking Team blog:

     

    In reality it should not matter what low level capabilities
    your NIC has or which ones are enabled and used by the
    VMKernel. Most capabilities that are not available on your
    particular NIC, the VMKernel will handle in software. As such
    you can have a vSwitch configured with two NICs, one capable of
    TSO and the other not. VMKernel will use the TSO capabilities
    of the NIC that has it, if a failover occurs to the NIC without
    TSO capabilities, the VMKernel will just handle segmentation in
    software before delivering data to the NIC.
    http://blogs.vmware.com/networking/2009/03/deciphering-nic-capabilities-from-log-entries.html

     

     

    4. How can I tell what features my physical NICs (pNICs) have?



    From Kamau Wangahu and Guy Brunsdon over n the VMware Networking Blog

     


    You may have seen messages like those below in your logs and wondered  what neat capabilities are being enabled on your network card (NIC).

    ESXsrv01 vmkernel: 22:04:29:43.583 cpu1:1198)Uplink: 2491: Setting capabilities 0x0 for device vmnic4 
    ESXsrv01 vmkernel: 22:04:29:43.583 cpu1:1198)Uplink: 2491: Setting capabilities 0x0 for device vmnic5 
    ESXsrv01 vmkernel: 22:04:59:11.410 cpu0:1235)Uplink: 2491: Setting capabilities 0x300 for device vmnic2 ESXsrv01 vmkernel: 22:04:59:11.410 cpu0:1235)Uplink: 2491: Setting capabilities 0x300 for device vmnic3 ESXsrv01 vmkernel: 22:04:59:11.410 cpu0:1235)Uplink: 2491: Setting capabilities 0x2b for device vmnic2 
    ESXsrv01 vmkernel: 22:04:59:11.410 cpu0:1235)Uplink: 2491: Setting capabilities 0x2b for device vmnic3
    



    What you need is The Magic Decoder Ring!

    The Magic Decoder Ring is the table below which states the HEX codes we  use to decode the “Setting Capabilities” for the above style log  messages are as follows:

    0x0001 Capable of scatter-gather transmits 
    0x0002 can checksum only TCP/UDP over IPv4 
    0x0008 Can use DMA to high memory locations 
    0x0020 Supports TCP Segmentation offload, TSO 
    0x0100 Can do VLAN tagging in HW 
    0x0200 Can do VLAN untagging in HW
    



    How to map the log messages to the Magic Decoder Ring
    So now off to the fun stuff of decoding our log file output….

    The first two log entries set the capabilities for vmnic4 and vmnic5 to  0x0. This is logged when the last virtual port of a portgroup that the  uplink is assigned to is disconnected. In other words, the hardware  capability is no longer needed, as no traffic will be sent over these  uplinks (vmnic4 and vmnic5).

    The next capability that is set is 0x300 for vmnic2 and vmnic3. From the  table above we can get 0x300 by adding 0x0200 and 0x0100. This means  that we are enabling:

    0x0100 VLAN tagging in hardware. 
    0x0200 VLAN untagging in hardware
    



    The next capability that is set is 0x2b. From this we can tell we are enabling:

    0x0001 scatter-gather transmits 
    0x0002 checksum only TCP/UDP over IPv4 
    0x0008 DMA to high memory 
    0x0020 TCP Segmentation offload, TSO
    



    as

    0x0001 + 0x0002 + 0x0008 + 0x0020 + = 0x002b
    



    And that is all there is to translating the NIC “Setting Capabilities” log entries!

     

    5. What is the best way to allocate my pNICs between the service  console, vmotion, production network, DMZ and possibly storage?



    Details of many combinations of pNICs and VMware requirements can be found at http://www.networkworld.com/community/blog/7119 (Edward Haletky). The list includes how to best layout the follow:

     

    • DMZ w/5 Physical NICs with VMware ESX
    • DMZ w/4 Physical NICs with VMware ESX
    • DMZ w/3 Physical NICs with VMware ESX
    • DMZ w/2 Physical NICs with VMware ESX
    • 8 Physical NICs with VMware ESX
    • 6 Physical NICs with VMware ESX
    • 5 Physical NICs with VMware ESX
    • 4 Physical NICs with VMware ESX
    • 3 Physical NICs with VMware ESX
    • 2 Physical NICs with VMware ESX