VMware Cloud Community
RockT
Contributor
Contributor

ESXi 5.1: vmsvc warning guestinfo RecordRoutingInfo: Unable to collect IPv4 routing table

Hi,

we have two RHEL6.3 VMs which need 15 minutes to boot!

The problem happens when starting the network interfaces in vmsvc.

Host: ESXi 5.1, latest 5.1 tools from repo installed, two vmxnet3 interfaces

Both interfaces are connected to two different portgroups on the same swtich with same VLAN ID (!) and one uplink.

Please help how to find out the cause of this.

Thx Rainer

vmsvct.png

Tags (2)
87 Replies
blharris
Contributor
Contributor

2nd Edit: After renaming files in previous post it works after subsequent reboot.

  • /usr/lib/vmware-tools/plugins32/vmsvc/libtimeSync.so
  • /usr/lib/vmware-tools/plugins64/vmsvc/libtimeSync.so

Edit: Nevermind.  After another reboot the problem came back.

I was able to solve this by doing the following steps.  Now I don't get the message at bootup anymore, nor does eth0 nor eth1 take more than 1/2 second to become online.


  1. Uninstall the quagga package.  I don't think this matters but I am including for completeness.
    1. rpm -e quagga
  2. Uninstall VMware Tools
    1. /etc/vmware-tools/installer.sh uninstall
    2. rm -rf /etc/vmware-tools
  3. Reinstall VMware tools using the CDROM
    1. mount /dev/cdrom /mnt/cdrom
    2. cd /mnt && tar xf cdrom/VMware.....tgz
    3. cd vmware-tools-distrib && ./vmware-install.pl --default
Reply
0 Kudos
InfoStewards
Enthusiast
Enthusiast

Another datapoint.  I installed vmtools (5.1u1) on four RHEL 6.4 VMs today.  All were the same install, fully patched.  The only difference was that two were single cpu (1x1core), two were SMP (2x2core (4 core total)).  I only experienced the problem on the SMP VMs.

The library rename resolved the issue for the two SMP VMs.

Reply
0 Kudos
fpiris
Contributor
Contributor

I’m having messages in syslog like the ones described in this thread every 30 seconds:

May  9 09:28:46 vdebian vmsvc[2640]: [ warning] [guestinfo] RecordRoutingInfo: Unable to collect IPv4 routing table.

May  9 09:28:46 vdebian vmsvc[2640]: [ warning] [guestinfo] RecordRoutingInfo: Unable to collect IPv6 routing table.

May  9 09:28:46 vdebian vmsvc[2640]: [ warning] [guestinfo] Failed to get nic info.

May  9 09:29:16 vdebian vmsvc[2640]: [ warning] [guestinfo] RecordRoutingInfo: Unable to collect IPv4 routing table.

May  9 09:29:16 vdebian vmsvc[2640]: [ warning] [guestinfo] RecordRoutingInfo: Unable to collect IPv6 routing table.

May  9 09:29:16 vdebian vmsvc[2640]: [ warning] [guestinfo] Failed to get nic info.

I’m seeing this on some VM’s with up to date debian wheezy, VMwareTools-9.0.5-1065307.tar.gz, and several different ESXi 5.1U1 (build 1065491) hosts.

VM’s perform ok, they have no connectivity issues at all. Just the syslog spam every 30 seconds (which is a little bit annoying).

I also have other running debian wheezy with the same version of vmware-tools that do not show theese messages on their syslogs.

The thing all the VM’s showing theese messages have in common is network interfaces with 802.1q vlan’s (whose names are like eth0.XXX).

On two of the VM’s which didn’t show those messages (one with a 32bit system and the other with a 64bit one) I did the following test:

  1. added a vlan to an existing eth0 interface (vconfig add eth0 100)
  2. so far, so good
  3. as soon as I brought that eth0.100 interface up I began seeing those messages in syslog
  4. and as soon as I brougt it down, the syslog spam stopped.

So this is, in my case, definitely related to vlan’s.

Anyone else seeing this? Is this a bug in vmware tools? Is there any known workaround to stop that syslog spam?

PS.: I tried removing those two libtimeSync.so libraries, no difference at all.

Reply
0 Kudos
RockT
Contributor
Contributor

Bug is not fixed. :smileycry:

Problem persists with fully patched ESXi 5.1 release build 1117900

and latest OSP vmware tools 9.0.5

My RHEL6 64 VMs are fully patched, too

But here the workaround with renaming libtimeSync.so worked, vm reboots fine.

Thx

Rainer

Reply
0 Kudos
wolf
Enthusiast
Enthusiast

I sorted it out setting Selinux to Permissive/Targeted.

Cenots 6.4 + kernel 2.6.32-358.6.2, esx 5.1 build 1117900, latest tools

no libtimesync rename

I this this is what is creating issues with Selinux

netlink: 12 bytes leftover after parsing attributes.(dmesg)

Reply
0 Kudos
wolf
Enthusiast
Enthusiast

False positive

After some reboots issue came out again... so no Selinux

Reply
0 Kudos
blharris
Contributor
Contributor

Hi Wolf,

Did you follow the workaround with renaming libtimeSync.so ?  Doing the workaround works for everyone else AFAICS.

Bryan

Reply
0 Kudos
fpiris
Contributor
Contributor

blharris wrote:

Did you follow the workaround with renaming libtimeSync.so ?  Doing the workaround works for everyone else AFAICS.

No it doesn’t Smiley Happy (see my previous post).

Although my VM’s work fine, I have the messages metioned by OP every 30 seconds in the syslog; wich is annoying.

I opened a SR on May 13th. I’ve been exchanging some mails with support but, ATM they haven’t figured out it yet.

[NOTE, I had posted this same message from the wrong account. I deleted that message and re-posted it from the right one. Sorry about that]

Reply
0 Kudos
smuthali
Contributor
Contributor

Hello Experts,


We ran in to a similar issue for our Avaya OVA deployments. What we figured out was that there is a race condition between VMware tools service and the Network service during boot up. The simple fix will be to have the VMware tools service start up ahead of the Network Service.

The following bash script goes about implementing this:

-----

VM_TOOLS_CONF=/etc/init/vmware-tools.conf

TOOL_LINK_ARRAY=(\

/etc/rc2.d/S03vmware-tools \

/etc/rc3.d/S03vmware-tools \

/etc/rc5.d/S03vmware-tools \

/etc/rc0.d/K99vmware-tools \

/etc/rc1.d/K99vmware-tools \

/etc/rc6.d/K99vmware-tools \

);

function printHelp() {

    echo "Usage: bootSyncFix.sh -enable|-disable|-status"

    echo

    echo "-enable will enable a workaround for a known VMWare issue during bootup"

    echo "  where the bootup stalls on network startup and a message in the following"

    echo "  format is displayed in a loop:"

    echo "vmsvc[1234]: [warning] [guestinfo] RecordRoutingInfo: Unable to collect IPv4 routing table."

    echo

    echo "  This change will modify the VMWare tools service to start before"

    echo "  the network service."

    echo "-disable will disable the above workaround and will revert the VMWare"

    echo "  tools service to the standard VMWare init startup."

    echo "-status will display the status of the system (whether the boot sync"

    echo "  fix is enabled or disabled).  This mode will not modify the system."

    exit 1

}

# Implement/enable boot sync fix

function enable() {

    echo "Enabling boot sync fix..."

    # Try to move file so long as it hasn't already been moved

    if [ -f /etc/vmware-tools/vmware-tools.conf -a ! -f /etc/init/vmware-tools.conf ]; then

        echo "vmware-tools.conf already relocated"

    else

        # If destination already exists; backup file before overwriting

        if [ -f /etc/vmware-tools/vmware-tools.conf ]; then

            mv -f /etc/vmware-tools/vmware-tools.conf /etc/vmware-tools/vmware-tools.conf.bak

        fi

        mv -f /etc/init/vmware-tools.conf /etc/vmware-tools/

    fi

    for LINK in "${TOOL_LINK_ARRAY[@]}"; do

        ln -s /etc/vmware-tools/services.sh $LINK 2>&1 | grep -v "File exists"

    done

    echo "...done.  Any configuration changes will take place on the next bootup"

}

# Revert back to original standard configuration

function disable() {

    echo "Disabling boot sync fix..."

    # Try to move file so long as it hasn't already been moved

    if [ -f /etc/init/vmware-tools.conf -a ! -f /etc/vmware-tools/vmware-tools.conf ]; then

        echo "vmware-tools.conf already relocated"

    else

        # If destination already exists; backup file before overwriting

        if [ -f /etc/init/vmware-tools.conf ]; then

            mv -f /etc/init/vmware-tools.conf /etc/init/vmware-tools.conf.bak

        fi

        mv -f /etc/vmware-tools/vmware-tools.conf /etc/init/

    fi

    for LINK in "${TOOL_LINK_ARRAY[@]}"; do

        if [ -L $LINK ]; then

            rm -f $LINK

        elif [ -f $LINK ]; then

            # If file exists and is not a symlink, then print warning but don't delete file

            echo "Warning: Not deleting non-symlink $LINK"

        fi

    done

    echo "...done.  Any configuration changes will take place on the next bootup"

}

# Return code:

# 0 -- Enabled

# 1 -- Disabled

# 2 -- Partially enabled

function status() {

    echo "Checking status of boot sync fix..."

    # By default, assume we are partially disabled unless we detect otherwise

    STATUS=2

    if [ -f /etc/rc2.d/S03vmware-tools -a -f /etc/rc3.d/S03vmware-tools -a -f /etc/rc5.d/S03vmware-tools -a -f /etc/rc0.d/K99vmware-tools -a -f /etc/rc1.d/K99vmware-tools -a -f /etc/rc6.d/K99vmware-tools -a -f /etc/vmware-tools/vmware-tools.conf ]; then

        # All files are there; fix is enabled

        STATUS=0

    elif [ ! -f /etc/rc2.d/S03vmware-tools -a ! -f /etc/rc3.d/S03vmware-tools -a ! -f /etc/rc5.d/S03vmware-tools -a ! -f /etc/rc0.d/K99vmware-tools -a ! -f /etc/rc1.d/K99vmware-tools -a ! -f /etc/rc6.d/K99vmware-tools -a ! -f /etc/vmware-tools/vmware-tools.conf ]; then

        # Fix completely not implemented

        STATUS=1

    fi

    case $STATUS in

        0) echo "Boot sync fix is enabled";;

        1) echo "Boot sync fix is NOT enabled";;

        *) echo "Boot sync fix is partially enabled";;

    esac

    return $STATUS

}

### Script start ###

# Check the current user -- needs to be root level

USER_ID=`id -u`

if [ $USER_ID != "0" ] ; then

  echo "This script must be run by a root-level superuser (UID 0)"

  echo

  exit 1

fi

ARG1=$(echo "$1" | tr A-Z a-z)

case $ARG1 in

    -enable) enable;;

    -disable) disable;;

    -status) status;;

    *) printHelp;;

esac

RaymondG
Enthusiast
Enthusiast

I am also having this problem with RHEL 6.2   It is very annoying getting calls in the middle of the night because a VM was rebooted and never came back......just to find out they only needed to wait 15 mins in between reboots.

There has to be a Fix.  Renaming files in over 100 vms doesnt seem like an acceptible solution.

Please respond if anyone finds a better solution.

Thanks

Raymond Golden VCP3, VCP4, MCSA, A+, Net+, SEC+
Reply
0 Kudos
bealdrid
Contributor
Contributor

I can confirm that adding NTP servers to the ESXi host with the affected VM resolved it for me.  Have done several reboots of the problematic VM with no problem since configuring NTP on the host.  This may not be an option for everyone so I suppose it should still be addressed inside of the VMware tools package. 

Reply
0 Kudos
vm2cloud
VMware Employee
VMware Employee

I revently got this error after trying to clone a cloned copy of a CentOS 5.6 VM.

I imported CentOS OVF from scratch, updated tools and hardware and cloned from template in vCenter and no problem.

If I then tried to clone from the cloned copy I got this error. Not sure if this helps. To resolve my issue I re imported the OVF again and cloned from that. Had to do some rework in the base VM but it ended up working.

Reply
0 Kudos
dmc
Contributor
Contributor

The following VMware KB has been published to document this issue. The primary cause is in the Linux iputilis package and this is causing the delay in the boot process. A workaround is available in the KB advising to set rtc.diffFromUTC=0 within the vmx file.


VMware KB: Linux virtual machine's boot is slow and displays the warning: RecordRoutingInfo: Unable ...


Reply
0 Kudos
airt
Contributor
Contributor

I managed to fix the issue by setting the rtc.diffFromUTC=0 option in the vmx file. Then I got the error http://cdn.chriscolotti.us/wp-content/uploads/2013/02/vmsvct.png and managed to fix it as described here
Workaround: vSphere 5.1 Guest Unable to collect IPv4 routing table

by renaming libtimeSync.so to libtimeSync.so.bak (or something else you choose) in

/usr/lib/vmware-tools/plugins32/vmsvc/libtimeSync.so

/usr/lib/vmware-tools/plugins64/vmsvc/libtimeSync.so

Reply
0 Kudos
RockT
Contributor
Contributor

What?

You managed to fix the issue by setting rtc.diffFromUTC=0 and then you got the error - again?

Reply
0 Kudos
kestenb
Contributor
Contributor

I tried the .vmx mod, renaming the .so 's and even the script by smuthali (with a few changes for file locations).  None of those did the trick.

I was seeing

Unable to collect IPv4 routing table

Unable to collect IPv4 routing table

vmsvc[712]: [ warning] [guestinfo] Failed to get nic info.

spat out together every 30 seconds.

The following worked for me, and may shed some light on the root cause:

Every 30 seconds I am getting two log entries written to syslog:

        vmsvc[712]: [ warning] [guestinfo] Failed to get nic info.

        vmsvc[712]: [ warning] [guestinfo] Failed to get vmstats.

This makes the /var/log/syslog file grow unnecessarily and tools like

logcheck end up spamming me about the messages.

Apparently the 'nic info' message is produced if open-vm-tools is compiled

using --without-dnet and the 'vmstats' message is produced if open-vm-tools

is compiled using --without-procps.

I can turn these messages off by adding the following to the

/etc/vmware-tools/tools.conf file:

        [logging]

        vmsvc.level=error

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=639767

the /etc/vmware-tools/tools.conf fix worked for me.

Reply
0 Kudos
BobDingman
Contributor
Contributor

I have a:

* CentOS 5.9+ 64-bit patched VM

* kernel: 2.6.18-348.18.1.el5

* vSphere 5.1U1

* VMware Tools: 9.0.5.21789 (build-1065307)

* VM version: 7

The VM was originally created under ESXi 3.5, cloned to 4.0, 4.1 and eventually to 4.1U3

The VM has two virtual NICs, eth0 and eth1.

eth0 was configured as VMXNET2 and dhcp

eth1 was configured as VMXNET3 and static IP

I tried renaming the libtimeSync.so file in /usr/lib/vmware-tools/plugins64/vmsvc/ to libtimeSync.so.bak as described in earlier posts.

I tried adding entries to the .vmx file as described in VMware KB: Linux virtual machine's boot is slow and displays the warning: RecordRoutingInfo: Unable ...

and VMware KB: Disabling Time Synchronization.

None of this stopped the "Unable to collect IPv4 routing table" messages I would get on startup on eth0

I would not get any error on eth1; however, it appeared not to actually work (with a ping test to another host)

I then deleted both virtual NICs and recreated them as follows:

eth0 was configured as VMXNET3 and dhcp

eth1 was configured as VMXNET3 and static IP

(i.e., as before but with both as VMXNET3)

That fixed the problem with the routing table message. Both NICs now work as expected.  It may have been possible simply to reconfigure eth0 as VMXNET3 without deleting and recreating the NICs. Perhaps VMXNET2 and VMXNET3 do not work properly together under ESXi 5.1.  (This "mixed NIC" configuration was working fine under ESXi 4.1U3 - and that's where I had been last using this VM.)

Reply
0 Kudos
collectrouting
Contributor
Contributor

System is running "CentOS release 6.4" on "ESXi-5.5.0-1331820-standard"

Network traffic works fine on eth0 (dhcp) and eth0.100 (static)

Problems appears to be "vmtools fails to parse VLAN interfaces"

(disable eth0.100 and no more warnings)

#1 They're not just warnings; ESXi server (VM summary) doesn't show the eth0.100 IP address

#2 My system has no "tools.conf" file to edit; how do I "fix" this by hiding warning messages?

# ifconfig|grep eth0

eth0      Link encap:Ethernet  HWaddr 00:0C:29:F6:F1:3D

eth0.100  Link encap:Ethernet  HWaddr 00:0C:29:F6:F1:3D

# vmware-toolbox-cmd -v

9.4.0.25793 (build-1280544)

# find / -print | grep \\/tools\\.conf$

# find / -print | grep tools\\.conf$

/etc/modprobe.d/vmware-tools.conf

/etc/init/vmware-tools.conf

/etc/dracut.conf.d/vmware-tools.conf

# tail /var/log/messages

Oct 21 17:45:41 devrock1 vmsvc[1168]: [ warning] [guestinfo] Failed to get nic info.

Oct 21 17:46:11 devrock1 vmsvc[1168]: [ warning] [guestinfo] RecordRoutingInfo: Unable to collect IPv4 routing table.

Oct 21 17:46:11 devrock1 vmsvc[1168]: [ warning] [guestinfo] RecordRoutingInfo: Unable to collect IPv6 routing table.

Oct 21 17:46:11 devrock1 vmsvc[1168]: [ warning] [guestinfo] Failed to get nic info.

Oct 21 17:46:41 devrock1 vmsvc[1168]: [ warning] [guestinfo] RecordRoutingInfo: Unable to collect IPv4 routing table.

Oct 21 17:46:41 devrock1 vmsvc[1168]: [ warning] [guestinfo] RecordRoutingInfo: Unable to collect IPv6 routing table.

...

Reply
0 Kudos
virtualtone
Contributor
Contributor

In my case, Network Manager was also completely disabled, however, what fixed it for me was to remove the line NM_CONTROLLED="yes" from /etc/sysconfig/network-scripts/ifcfg-eth0

Reply
0 Kudos
aaber
Contributor
Contributor

Try this:

vmsvc on a Linux console displays a warning similar to:

vmsvc [warning] [guestinfo] RecordRoutingInfo: Unable to collect IPv4 routing table

Cause
This issue occurs when the Linux iputils package causes a delay in the boot process. A warning message appears when the guestinfo plug-in tool fails to parse the content from the /proc/net/route file. The guest operating system's clock is ahead of the host on which it is running, causing the arping process to become unresponsive during boot. This results in the delayed boot times.

When a virtual machine is affected this issue occurs every time it boots, but it does not happen on every Linux virtual machine.


Resolution
To resolve this issue, see the RedHat Advisory http://rhn.redhat.com/errata/RHBA-2013-1290.html.

Note: The preceding link was correct as of December 11, 2013. If you find the link is broken, provide feedback and a VMware employee will update the link.

To work around this issue, add rtc.diffFromUTC=0 to the .vmx file of the affected virtual machine. This forces a time sync, which enables the arping process to complete as expected leading to normal boot times.

To add rtc.diffFromUTC=0 to the .vmx file:
1.Power off the virtual machine.
2.Log in to the ESXi/ESX host on which the virtual machine is located with a secure shell session.
3.Change to the virtual machine's .vmx file location (located at /vmfs/volumes/datastore_name/vm_name).
4.Create a backup of the .vmx file.
5.Open and edit the original .vmx file with a text editor.
6.Add this entry to the end of the .vmx file:
rtc.diffFromUTC=0
7.Close and save the .vmx file.
8.Power on the virtual machine.

Reply
0 Kudos