No network on RHEL 6.2 using VMXNET3

[rupeshdnair]

Hello All,

I've run into this issue where after provisioning a RHEL 6.2 VM on esx 5.0 and use the vmxnet3 NIC, I cant get any network. However if I uninstall vmware tools or remove the vmxnet3 and add an e1000 NIC all is working. Anyone had this issue? Appreciate if anyone can help.

Thanks in advance,

Rupesh

[kooltechies]

Have you looked into the vmware.log file of the virtual machine when using the vmxnet3 adaptor and check the messages it's throwing. Error log should be able to shed more light on the condition.

[iw123]

Hi,

Which version of esx are you running? are you using the latest tools version?

[rupeshdnair]

No errors in the logs. The only symptom is that, we cannot ping anything from within the VM, not even the gateway.

We are using ESXi 5.0

[iw123]

Does the VMXNET3 adaptor work if you add it without installing the tools? I'm not 100% but thought redhat had a driver for it included?

[kooltechies]

Are you able to see the adaptor inside the VM and are you able to get an ip assigned to that VM. You can try to run ethereal within the VM and check from where the packets are getting dropped.

Also check the rx/tx on the vnic from inside the VM.

[rupeshdnair]

If we uninstall vmware tools we get network connectvity using vmxnet3. Seems like vmware tools installs some extra modules for vmxnet3 which stops the network.

[rupeshdnair]

Are you able to see the adaptor inside the VM and are you able to get an ip assigned to that VM. You can try to run ethereal within the VM and check from where the packets are getting dropped.

Also check the rx/tx on the vnic from inside the VM.

Yes, we are able to see the adaptor inside the VM. We are able to assign a static IP to it as well. We are unable to ping even the gateway, not sure using ethereal will help much there, correct me if I am wrong.

[iw123]

Which build of tools is it?

[kooltechies]

Uninstalling or unusing any features of vmxnet3 is as good as using vmxnet2 or enhanced vmxnet. If you have a support contract you can open a support ticket with vmware to truoubleshoot further.

[rupeshdnair]

VMwareTools-8.6.0-446312

[lsoto1]

The same for me.

"Bringing up interface eth0: "

And the machine hungs for ever.

I've tryed:

- Kernel update, so no vmtools changes to the modules -> fail

- Booting in single user mode and remove/insert the vmxnet3 into memory -> no error. Activating and desactivating manually the nic after that -> Ok.

- Running vmware-tools-config after the update, without clobber the vmxnet3 module -> fail

- Running vmware-tools-config after that with clobbering (This word is correct??) the vmxnet module -> fail

However, besides using single mode, sometimes the nic comes up.

I suspect but not already tryied, that configuring the interface with dhcp on a dhcp enabled vlan seems to work.

I've seen this behaviour only on Red Hat 6.X (6.2 & 6.3) x64 for sure. I suspect that also on Red Hat 6.X i386.

I haven't tried to remove vmtools completely yet.

The platform is a cluster of ESXi 4.1 update 1. VMotion doesn't aleviate the fault.

Modinfo seems to be the same version for last kernels on "stock" driver

filename:       /lib/modules/2.6.32-279.1.1.el6.x86_64/kernel/drivers/net/vmxnet3/vmxnet3.ko
version:        1.1.18.0-k
license:        GPL v2
description:    VMware vmxnet3 virtual NIC driver
author:         VMware, Inc.
srcversion:     36ED770278AA922F0C94288
alias:          pci:v000015ADd000007B0sv*sd*bc*sc*i*
depends:
vermagic:       2.6.32-279.1.1.el6.x86_64 SMP mod_unload modversions

[lsoto1]

The same template, deployed on ESXi 5.0.1 updated last week, and the same behaviour (vm hung at startup configuring the nic).

----

Changing the nic to e1000 at the original machine on ESX. 4.1 upd 1 and console shows at second reboot:

Bringing up interface eth1: e1000: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None

ADDRCONF(NETDEV_UP): eth1: link is not ready

ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready

(hunging)

Reconfigure vmtools don't aleviate.

De-install of vmtools make the vm usable.

Changing the net to vmxnet3 (without the vmtools - 8.3.12 build-559003 ) works. I know that perhaps the tools are old, but we are waiting to ESXi 5 upgrade of the hosts.

I suspect about the vsock.ko vmtools kernel module. I think I've seen a console messaging with something about it.

When you install de vmtools (and thus, the instalation load the module), after that you can see at the dmesg

"vsock module is older than RHEL 6.2 ... applying fixups"

Installing vmtools of ESXi 5 (8.6.5 build 731933) don't solve the problems on all reboots done.

[lsoto1]

OK!!!

Renaming vsock.ko kernel module (seems to be automatically loaded by vmci driver/module) and re-generating the initram seems to solve the problem, and not warnings/error at boot. 4 reboots of os and 4 cold boots without problems.

Vmtools are marked as ok, and vmci is loaded.

[Wokfan]

Thanks a lot for finding this solution. I had exactly the same problems with one of my machines, it got stuck during boot, waiting for the network to initialize.

But I wonder what the effect of disabling vsock will be. Am I soon to find out a new problem?

[Parrothead1809]

Isoto - I am having the same problems and debugged almost identically - I tried the newer version of vmware-tools (even with the "--clobber-kernel-modules=vmxnet3" option) still No luck.

My guest (RHEL 6)  always hangs on "Bringing up interface eth0".

Can you please provide more details on how to fix the probem?

I found "/lib/modules/2.6.32-279.14.1.el6.x86_64/misc/vsock.ko" - What should I rename it to? And can you please provide info on "re-generating the initram "

Thanks!

[lsoto1]

Hi

- rename the file (just in case you want to go back), and so the mkinitrd doesn't find the kernel module

* mv  /lib/modules/$(uname -r)/misc/vsock.ko /lib/modules/$(uname -r)/misc/vsock.ko.org

- rebuild the initrd image for the next boot

On RHEL, if I don't miss it, the program to use is "dracut". I don't remember just now if you need to add the "-f" flag to force the rebuild.

* dracut

- reboot

* shutdown -r -y 0

All done.

Are you checked before that the server is NOT waiting for ever a dhcp leasing? (I think that it eventually do a time-out)

Regards,

[dandelionxu]

Dear Rupesh,

I run into the similar issue today. We have a centos 6.2 vm installed on ESXi 5.1.0 with three vmxnet3 NIC, static IP configured and vm running for a while already with no problem. But today after I added a new vmxnet NIC with no Ip configured, I cannot ping the gateway for eth2 and another vms in the same subnet. Our solution is to remove the NIC for eth2 and add a new vmxnet3 NIC, reconfig IP and bring up the new interface. Base on your experience is there any bug between linux kernel and vmxnet NIC or else? Do you have any other suggestions? Thanks in advance.