What is the default speed/duplex setting for ESX(i) 4.x?

[hennish]

Hi. I've had some questions from customers regarding the default speed/duplex setting for vmnics, and I just can't seem to find a definitive answer to the question: What is the default speed/duplex setting for ESX(i) 4.x?

It seems the recommendation is to set 1 Gb NICs to Auto as long as their corresponding switch ports are set to Auto, but in that case, why does ESX(i) seem to default to 1000/Full for these?

[Maximenu]

Hi

Take a llok to this document:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100408...

[idle-jam]

the above should answer your question, the default would just be what's being detected the best possible from the switch.

[hennish]

Thanks for the quick answer, but I don't really agree. The article only describes how to change the values, and the recommendations for it. There is no information on how ESX picks the defaults, and why.

[jbruelasdgo]

The advantages of utilizing AUTO-NEGOTIATION on  Gigabit-Ethernet Interfaces:

• AUTO-NEGOTIATION  is highly recommended on ESX Gigabit-Ethernet Interface cards and  physical Gig switch ports for the  following reasons:
  • Even though hard coding the speed and  duplex will work and is in the documentation - In some cases there are performance
    issues after an ESX upgrade 3.5 – setting the configuration to  Auto-negotiate seems to resolve  performance issues.
  • It resolves issues with ISCSI, VMOTION,  NETWORK PERFORMANCE, and related network issues.
  • Duplex settings—while Cisco devices  only support full-duplex, the IEEE 802.3z standard does have support for half-duplex Gigabit-Ethernet.
    Because  of this, duplex is negotiated between Gigabit-Ethernet devices.
  • Flow Control—Because of the amount of  traffic that can be generated by Gigabit-Ethernet, there is a PAUSE functionality built into Gigabit-Ethernet.
    The  PAUSE frame is a packet that tells the far-end device to stop the transmission of packets until the sender  is able to handle all the traffic and clear its buffers.
    The PAUSE  frame has a timer included, which tells the far-end device when to start  to send packets again. If that timer  expires without getting another PAUSE frame,
    the far-end device can  then send packets again. Flow- Control  is an optional item and must be negotiated. Devices can be capable of  sending or responding to a PAUSE frame,  and they possibly do not agree to the flow-control request of the  far-end.

[hennish]

Again, thanks for explaining what Auto Negotiation is, but that wasn't my question.

The question is: Which Speed/Duplex setting does ESX(i) by default set for its gigabit vmnics? Is it "1000 Mb, Full Duplex" or "Auto negotiate"?

The reason for asking this is since if the setting is "1000 Mb, Full Duplex" as opposed to "Auto negotiate", I will need to add changing this to "Auto negotiate" in all my customers' installation checklists/scripts, since

http://kb.vmware.com/kb/1004089 recommends the settings to be the same in the host and the pSwitch.

[hennish]

Hi. I know how auto negotiate works. I'm just not sure if ESX(i) has it as its default setting for the vmnics.

Actually I've had several customers that have experienced the exact opposite (that is, the default being "1000 Mb, Full duplex") setting.

And just to avoid further misunderstandings: Yes, I know that there is one "Configured Speed, Duplex" and one "Actual Speed, Duplex" value, and since the "Configured" one is the one that's changeable, that's the one I'm asking about.

So in short, is there anyone who can point me to some documentation or real world testing that shows me that the default ESX(i) setting for "Configured Speed, Duplex" is "Auto negotiate"?

[bulletprooffool]

In ESX 3.5, the display in the VC remained at autonegotiate, I beleieve in ESX4i it detects the speed / duplex at boot time and sets the value to the detected value (if it was at Auto negotiate before) - in version 4.0 u2 a patch was released to remedy an early bug with this that meant that when a user configured the valkue statically after boot, it still reverted back to re-negotiating at boot time:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=101772...

Speed and duplex settings for a NIC might not be retained after ESX/ESXi host reboots

When the speed and duplex settings for a NIC are manually set  and the ESX/ESXi host is rebooted, the values set for the speed and  duplex settings might not be retained. After reboot, the network adapter  might auto-negotiate its speed and duplex settings.

So in short.

the default setting is Autonegoatiate - when the host boots, this is detected and the detected value displayed. Unless you Manually go and specify a different value, it redetects at boot.

[hennish]

Hi. I'm not sure what you mean by

I beleieve in ESX4i it detects the speed / duplex at boot time and sets  the value to the detected value (if it was at Auto negotiate before

Are you talking about the "Configured" setting, or the "Actual" (negotiated) value?

I just checked two default installations of ESX and ESXi 4.0 U1 (build 208167) in VMware Workstation, and they were both by default set to "1000 Full" in the "Configured" setting as the picture below shows.

Gonna try a U2 installation and a 4.1 this afternoon if I get the time. Or, can anybody else check this?

Please, in future replies, specify whether you mean the "Configured" setting, or the "Actual" (negotiated) value. All I'm interested in here is "Configured".

[hennish]

Okay. After some testing this is finally starting to clear up, and it seems this is simply a bug:

I confirmed that the default setting for "Configured speed, duplex" in ESXi 4.1 is indeed "1000 Full" (as seen in the screenshot in my latest post above).

Then I tried setting the "Configured" setting for vmnic1 to "Negotiate", which of course worked fine, but didn't change the "Actual" value, which was still "1000 Full" (see screenshot below)

I tried to verify this using vicfg-nics -l from vMA, but that command only displays the "Actual" values:

[vi-admin@vma ~][192.168.91.131]$ vicfg-nics -lName    PCI     Driver     Link Speed    Duplex MAC Address...
vmnic0  02:01.0 e1000      Up   1000Mbps Full   00:0c:29:b7:e3:29  1500   Intel Corporation PRO/1000 MT...
vmnic1  02:05.0 e1000      Up   1000Mbps Full   00:0c:29:b7:e3:33         Intel Corporation PRO/1000 MT...

Then I finally figured out how to check the setting of "Configured":

~ # ethtool vmnic0

Settings for vmnic0:

        Supported ports: [ TP ]

        Supported link modes:   10baseT/Half 10baseT/Full

                                100baseT/Half 100baseT/Full

                                1000baseT/Full

        Supports auto-negotiation: Yes

        Advertised link modes:  10baseT/Half 10baseT/Full

                                100baseT/Half 100baseT/Full

                                1000baseT/Full

        Advertised auto-negotiation: Yes

        Speed: 1000Mb/s

        Duplex: Full

[...]

~ # ethtool vmnic1

Settings for vmnic1:

        Supported ports: [ TP ]

        Supported link modes:   10baseT/Half 10baseT/Full

                                100baseT/Half 100baseT/Full

                                1000baseT/Full

        Supports auto-negotiation: Yes

        Advertised link modes:  10baseT/Half 10baseT/Full

                                100baseT/Half 100baseT/Full

                                1000baseT/Full

        Advertised auto-negotiation: Yes

        Speed: 1000Mb/s

        Duplex: Full

[...]

As we can see, the value "Advertised auto-negotiation" was always set to "Yes" on both vmnics, and hence I think there is a bug in the vSphere Client, since it displays this as fixed.

What do you think?

[rickardnobel]

Anders Olsson wrote:

As we can see, the value "Advertised auto-negotiation" was always set to "Yes" on both vmnics, and hence I think there is a bug in the vSphere Client, since it displays this as fixed.

What do you think?

Hej Anders,

I have also seen some weird stuff about this since 4.0. My opinion is that it default incorrectly displays some fixed value (100 full / 1000 full) even when it is in Auto mode. On 4.0 I did some testing also, and checked at the switch what the physical port was put into.

Switch port is configured as AUTO.

* When having the default for the VMNIC (visual Fixed) the switchport goes into 1000 full AUTO. (Not expected, but ok if the VMNIC is actually in auto.)

* When having the VMNIC set to AUTO the switchport goes into AUTO (Fully expected.)

* When changing back the VMNIC to 100 Full for example the switchport goes into 100 HALF (Fully expected, since no negotiating works and it should go to half. = duplex mismatch).

So my view is that the original "full" is really auto, but unfortunaly there is no way to see if someone has altered these defaults, because it COULD mean really "full" ...   

[hennish]

I opened a support case with VMware, and after a while I got a network specialist on the line.

He explained that ethtool is not supported and can't be relied on, since it's not a VMware tool.

He also explained that the reason for the default setting of "1000 Full" is for protection against switch port config errors such as setting "100 Full" or worse there. In that scenario that they think it's better that the link gets lost completely than going down in speed, which would be more difficult to find and diagnose.

What I still can't really grasp (or find any reliable information on) is whether "Auto"<->"1000 Full" between pSwitch<->ESX(i) really is a problem free and reliable configuration. Back in the 100 Mbit/s days it would be a horrible choice which ended up in terrible performance.

According to http://www.dell.com/content/topics/global.aspx/power/en/ps1q01_hernan?c=us&cs=555&l=en&s=biz (see table below) it would be a bad choice (Duplex conflict), but I'm not sure that they are correct.

Does anyone have any reliable info on this? I tried reading the IEEE docs, but didn't really find anything good (nor understandable).

[a_p_]

What I still can't really grasp (or find any reliable information on) is whether "Auto"<->"1000 Full" between pSwitch<->ESX(i) really is a problem free and reliable configuration. Back in the 100 Mbit/s days it would be a horrible choice which ended up in terrible performance.

With Auto/Auto on a GigaBit network you shouldn't see any issues.

There's an article in Wikipedia about autonegotiation which you may be worth reading http://en.wikipedia.org/wiki/Autonegotiation

André

[hennish]

Thanks, but as you can see, the question was about "Auto"<->"1000 Full".

I couldn't find any info on that in the article you referred to. Could you quote the parts that answer the question?

[LucasAlbers]

So is this still a bug in the default install behavior as of 4.1 update 1?

[RS_1]

Still the case in 4.1 U1, i checked on a fresh install

[ICT-Freak]

I can confirm this behavior on a fresh install.

But when I perform an upgrade from vSphere 4.0 update 2 to vSphere 4.1 update 1 I don't see the speed/duplex settings being changed.

[palong]

I have installed fresh ESX and ESXi 4.1 and will confirm that the default state of a pnic for both is to set the pnic Configured Speed,Duplex to 1000Mb, Full Duplex - not Auto negotiate.  If you have Cisco switches like I do, the default switchport config is Auto negotiate.  This is quite disconcerting, particularly as it relates to KB 1004089 - because if you take the default config on both the VMware pnic side and the Cisco switchport side you end up with the only one out of three possible configurations that is NOT RECOMMENDED.

Auto < > Auto   :   Recommended

1000Full < > 1000Full   :   Recommended

Auto < > 1000Full    :   NOT RECOMMENDED

per the KB article:  " VMware does not recommend mixing a hard-coded setting with Auto-negotiate."

Ironically this is exactly the state you are in if you take the default config from both Cisco and VMware.  Reading the rest of that article, it seems that VMware seems to prefer Auto over 1000/Full for the pnic settings as indicated by the statement "AUTO-NEGOTIATION is highly recommended on ESX/ESXi Gigabit-Ethernet Interface cards and physical Gig switch ports for the following reasons: ..."  which really does beg the question - why is Auto negotiate NOT the default setting for a pnic when installing ESX or ESXi when Auto IS the default for every switch vendor out there???  Why would VMware intentionally create a mismatched, not recommended configuration requiring remediation on every pnic?

Now, does VMware recommend I go through my 80+ hosts and set all 8 nics to auto to match the current Cisco switchport config fior Auto or shall I go into my Cisco switches and change the config of 8 switchports per host (that's 640 ports if you're counting) to 1000Full to match my VMware hosts which by default install have set their pnics to the lesser-preferred setting (per the above KB) of 1000Full so I can remediate my "not recommended" mixed hard-coded and Auto-negotiate environment?

[rickardnobel]

As long as you do not change the settings I think you will be fine. This is a GUI bug in all 4.x releases. The setting is in reality AUTO, which is why it works at all.

This is finally fixed in 5.0, where the default NIC setting is displayed as AUTO.

[palong]

Rickard –

Thank you for your response.  I assume that when you say “This is finally fixed in 5.0” you mean that this bug is resolved when the HOST is running ESXi 5.0?  Because both my vCenter and Virtual Infrastructure client are already at the 5.0 release and my hosts all are 4.1.0 Update2 build 502767  - yet still show the configured speed/duplex as 1000 MB, Full Duplex.

If it is resolved in ESXi 5.0 that is great, but I must say for a bug that was discovered in the base 4.0 release to not have been fixed though all interim releases up to current 4.1.0 Update 2 seems a bit strange to me.  Do you think there is any harm (except for the momentary loss of link redundancy) in setting each of my pnics to Autonegotiate just so that the GUI actually reflects the proper setting of the host pnic?