VMware

This Question is Answered

1 "correct" answer available (10 pts)
14 Replies Last post: Nov 6, 2009 7:09 AM by obsidian009  

Guest network connection lost at random posted: Sep 10, 2009 2:07 AM

Click to view kathmann's profile Enthusiast 18 posts since
Jul 29, 2008
Hi,

As of very recently we have been experiencing random network disconnections in our guests, without any visible reason or order, which are driving me up the wall...

Our setup

We have three ESX 3.5 servers, each connected to the LAN via three vSwitches:
- one vSwitch with a Service Console port group and 2 physical NICs
- one vSwitch with a VMkernel port group and 2 physical NICs
- one vSwitch with several VMnet port groups, one for each VLAN, with 6 physical NICs, one of which is designated the standby adapter

All physical NICs connect to two HP Procurve 5406zl core switches (the five active ports to one switch, the standby port to the second one), on which all the linked ports have the VLANs in use set in tagged mode.

An edited part of the switch config (ports A15 through A18 are connected to one ESX server):

vlan 10
name "PROD"
tagged A15-A18
exit
vlan 20
name "TEST"
tagged A15-A18
exit
vlan 30
name "UITW"
tagged A15-A18
exit
spanning-tree
spanning-tree B8 path-cost 50
spanning-tree Trk1 priority 4
spanning-tree config-name "SITE01"
spanning-tree legacy-path-cost
spanning-tree force-version STP-compatible

The vSwitches are set up as follows:
- Promiscuous Mode: Reject
- MAC Address Changes: Accept
- Forged Transmits: Accept
- Traffic shaping: disabled
- Load Balancing: Route based on the originating virtual port ID
- Network Failover Detection: Link status only
- Notify switches: Yes
- Failback: Yes

The guests have two virtual network adapters: one for our production LAN (VLAN 10) and one for iSCSI access to our NAS (VLAN 106).

Our problem

Since very recently VM's randomly lose network connections. Windows does not show the link as disconnected, but still cannot get traffic in or out to other systems, except to guests that are on the same ESX server (which soft of makes sense as this traffic never actually touches the physical adapter). The really weird bits are:

- A single VM on one ESX may suddenly have this problem at any time, while the other VM's on the same ESX still work fine
- A single VM may have this problem on one NIC but not on both, or sometimes on both cards at the same time
- Neither Windows or VMware report any issues/events/etc.

Does anyone have experience with issues like these? Is this a known issue (I could not find any info on this while search through the discussions here)?
Any help would be greatly appreciated!

Mark.

Re: Guest network connection lost at random

1. Sep 10, 2009 2:17 AM in response to: kathmann
Click to view Gerrit.Lehr's profile Master 827 posts since
Nov 9, 2005
Are the VMware Tools up to date and the virtual NIC driver set to vmxnet? I experienced problems like that when using the vlance driver in windows. Are all VMs affected or only a few?

Kind Regards,
Gerrit Lehr

If you found this or other information useful, please consider awarding points for "Correct" or "Helpful".

Re: Guest network connection lost at random

3. Sep 10, 2009 3:17 AM in response to: kathmann
Click to view Gerrit.Lehr's profile Master 827 posts since
Nov 9, 2005

Yeah, that is exactly the same workaround that I used until I figured out the problem.

Maybe some of these suggesntions work:

http://communities.vmware.com/thread/193905

Kind Regards,
Gerrit Lehr

If you found this or other information useful, please consider awarding points for "Correct" or "Helpful".

Re: Guest network connection lost at random

5. Sep 10, 2009 5:58 AM in response to: kathmann
Click to view kjb007's profile Guru 5,486 posts since
Sep 18, 2006

I would also suggest using the Enhanced VMXNet NIC for your virtual machines. The PCnet32 is the flexible NIC, which after the vmware tools, uses the vmxnet driver. The enhanced vmxnet NIC type is completely virtualized, and needs the vmware tools driver to work. I've found it to be more reliable.

Also, one other thing to check is the memory usage on the service console. This can also cause problems with networking. If you log into the service console, run 'free -m' when you are having problems. By default, you will have 272M of memory allocated to the service console, and this can get used up and then the service console will swap to disk. You can raise this to 800M max, which would help you get around this problem, if you are running into memory issues.

-KjB
VMware vExpert

Re: Guest network connection lost at random

7. Sep 10, 2009 6:18 AM in response to: kathmann
Click to view kjb007's profile Guru 5,486 posts since
Sep 18, 2006
One other thing that can do this is problems with your disk. This seems not very intuitive, but I'd run some IOmeter tests on your vm itself, and see if you are getting high latency. I've run into scenarios when high disk latency causes ping failures, when there is excessive I/O trying to go to that disk. Just another thing to check, and you can do so without having to modify any drivers.

-KjB
VMware vExpert

Re: Guest network connection lost at random

9. Sep 10, 2009 1:35 PM in response to: kathmann
Click to view Scissor's profile Master 1,251 posts since
Oct 8, 2007
Could it be a duplicate MAC address problem?

Re: Guest network connection lost at random

10. Sep 10, 2009 1:45 PM in response to: kathmann
Click to view mlubinski's profile Expert 286 posts since
Jun 10, 2008

hi,

yeah, we encoutered the same issue. It looked similiar (customer reported, that his VM does not respond to pings. After logging into VM I could ping some other VMs (via internal IPs), but after few minutes this also stopped working. The solution was to disable/enable (or repair) Network interface.

I didn't have much these issues, so didn't dig into this. But I think this is some kind of VMware bug in there.

Re: Guest network connection lost at random

12. Nov 5, 2009 12:12 PM in response to: kathmann
Click to view obsidian009's profile Novice 15 posts since
Jul 26, 2006

Hi -- did you ever resolve the issue with support? We seem to be having a very similar issue and was curious if you found a fix.

Thx

Re: Guest network connection lost at random

14. Nov 6, 2009 7:09 AM in response to: kathmann
Click to view obsidian009's profile Novice 15 posts since
Jul 26, 2006

Hm...sounds similar to what we're seeing, but not the same exactly. In your case with a faulty pNIC, I presume this was affecting more than one VM on that vSwitch right? We keep having a specific guest lose its connection and all other guests on the same vSwitch are fine. We're doing the same thing to fix it when it happens though...disable/re-enable the NIC and it instantly comes back. We've tried reinstalling vmware tools, removing the vNic and adding a new one with a different MAC, etc.

I was just working with vmware support last night and we left off by trying to use e1000 instead of vmxnet3. It seems stable for now, but we'll see...

thx

VMware Developer

SDKs, APIs, Videos, Learn and much more in the Developer community.

Learn More

Developer Sample Code

Increase your developer productivity with VMware API sample code.

Learn More

VMworld Sessions & Labs

Online access to the latest VMworld Sessions & Labs and online services.

Learn more

Purchase PSO Credits Online

Purchase credits to redeem training and consulting services online.

Buy Now

Community Hardware Software

View reported configurations or report your own.

Learn More

VMware vSphere

Come witness the next giant leap in virtualization.

Register Today

Communities