VM's lose their network connection randomly

[macos1984]

Hi everyone:

We recently installed 3 hosts with ESX 4.0 and migrated some of our VI3 VM's to them. We upgraded 3 VM's (one Linux RHEL 5.2 and two Windows 2003) to the virtual hardware version 7, and let the rest on version 4.

After that, the upgraded VM's are missing the network connection and recovering again, without any action from us. That situation doesn't occurs on the version 4 VM's. I have changed the vNics to vmxnet3, and to E1000 and still happens.

The three ESX 4.0 hosts are HP Proliant BL465 g1 in a HP c7000 enclosure, with Virtual Connect Ethernet Modules. We have 13 more servers, on the same enclosure, with ESX 3.5 Update 2 and are working fine.

Hope someone could help me with this situation

Regards

Marcel Alvarez

[AndreTheGiant]

VMware Tools are updated?

Andre

[Scissor]

Smells like an Ethernet MAC address conflict. Perhaps some of your Guests accidentally have the same MAC address?

[Texiwill]

Hello,

Moved to the virtual machine and guest os forum.

Best regards,
Edward L. Haletky VMware Communities User Moderator, VMware vExpert 2009

Virtualization Practice Analyst[/url]
Now Available: 'VMware vSphere(TM) and Virtual Infrastructure Security'[/url]
Also available 'VMWare ESX Server in the Enterprise'[/url]
[url=http://www.astroarch.com/wiki/index.php/Blog_Roll]SearchVMware Pro[/url]|Blue Gears[/url]|Top Virtualization Security Links[/url]|
[url=http://www.astroarch.com/wiki/index.php/Virtualization_Security_Round_Table_Podcast]Virtualization Security Round Table Podcast[/url]

[macos1984]

I create a new VM and configure it to use the old's VM vdisk, and the problem remains. I think it's not a MAC problem, because it's a new virtual hardware, with a new network interface.

[macos1984]

I have installed VMware Tools 4.0.0 build 164009

[Scissor]

Log into the Host's console and check the Host logs to see if there are errors reported? Under ESX the logs are under /var/log/vmkernel (and /var/log/vmkwarning and /var/log/messages). Under ESXi all entries are logged to /var/log/messages . I once found out I had a bad network cable by reviewing those logs.

I create a new VM and configure it to use the old's VM vdisk, and the problem remains. I think it's not a MAC problem, because it's a new virtual hardware, with a new network interface.

Did you verify that your new VM picked up a new MAC Address? Some OSes allow you to manually specify the MAC address used. In Windows you can do this by editing the advanced properties of your Network Adapter from device manager.

I have installed VMware Tools 4.0.0 build 164009

ESX4(i) update 1 was released a few days ago. Are you able to upgrade your Host to Update 1 to see if that resolves the problem?

[macos1984]

I did what you told me, and didn't find any problems regardin the network. On the other hand, the MAC is not fixed on the network interface of the VM, I erase that field on the network configuration.

1. VMware Inc Unknown device 07b0

DEVICE=eth0

ONBOOT=yes

BOOTPROTO=none

#HWADDR=00:50:56:a4:52:c6

NETMASK=255.255.255.192

IPADDR=172.22.30.20

GATEWAY=172.22.30.62

TYPE=Ethernet

USERCTL=no

IPV6INIT=no

PEERDNS=yes

[mmohr]

VMware has already documented this issue in the knowledge base.

http://kb.vmware.com/kb/1015572

[cfarley]

I don't think KB1015572 addresses this issue. I have a few guests that suffer from this as well, and when they lose network conenctivity they do not lose their static ip configuration. A disconnect/reconnect through VC usually lets the guest arp again but the problem typically reappears within a day or so. Changing the vmniv type does not appear to have any long term effect, nor does deleting and readding it...

[mikidutzaa]

We have a similar problem here, did you find anything?

[sgunelius]

Did anyone ever come up with a solution on this issue? I'm experiencing it too on my RHEL 5.4 VMs on a dedicated 3.5, U5 host. As soon as we logon to the VM's console, it comes back to life on it's own. One odd thing I noted is that when I already have a console open when connectivity is lost I can still ping outside from the VM. I noticed that while I had installed the VMware Tools, the VMs showed a tools status of "Not Installed" from vCenter, so I manually ran the vmware-config-tools.pl and then it appeared to load the vmxnet driver and other drivers properly. The status changed to "ToolsOK" after reboot, so I was very hopeful that the network issue would be resolved, but no such luck.

I am using HP's Virtual Connect and am using the virtual MAC, but I've made sure to use unique HP Pre-Defined ranges for each of my blade server enclosures, so we shouldn't have any overlapping MAC addresses. Our network administrator is starting to think the frequency of the outages may be related to something with the ARP cache on our switches. I've never seen an issue like this with my other VMs, then again, these are the first of our RHEL VMs.

Scott

[AndyPower]

Hi Guys I solved this issue by doing the following:

1: Disable the NIC in VMWare Tools

2: Uninstall the NIC in Device Manager

3: Uninstall VMWare Tools

4: Power off VM

5: Remove NIC from VM in VIClient

6: Power on VM and verify that NIC has been removed

7: Power off VM and add NIC

8: Power on VM and install VMWare Tools

This solution worked for me consistently on every VM since I started trying it. Hope it helps!!

[firestartah]

Have you checked and made sure you have sufficient ports on your virtual switch for the vm's?

If you found this or other information useful, please consider awarding points for "Correct" or "Helpful".

Gregg Robertson, VCP3,4 , MCSE, MCSA, MCTS, MCITP

[macos1984]

We have fixed our problem with the update of our VC Modules to the firmware version 3.15. It seems like this version fixes a bug with the LACP protocol.

[sgunelius]

I haven't tried to reproduce this issue since upgrading our virtual connect firmware, but we narrowed the problem down to the RHEL OS since they were the only VMs affected and we were seeing this issue on our RHEL blades too.  Apparently since these VMs had two vNICs, one for a public IP address and the other for heartbeat IP address, the OS seemed to "forget" about the public IP.  After a while the VMs would stop responding on the public IP, but if we connected to the VM through the VI console or pinged the heartbeat IP externally, the public IP would "wake up".  Red Hat recommended enabling NetworkManager and while that introduces some other networking issues as it supercedes some of the other network settings (DNS in /etc/resolv.conf), it resolved the periodic narcolepsy we were experiencing on that public IP address.

Cheers,

Scott