VMware Cloud Community
ManivelR
Hot Shot
Hot Shot

Freebsd 11.1 P1 OS packet drops with vsphere 6.7 update 1

Hi All,

I have few freebsd 11.1 p1 vms which is running on vSphere 6.7 update 1.

We started facing high packet drops when we ping other vms in the same subnet(40 rto out of 1000 pings)

It's running with vmxnet3 adapter without any VMware tools from beginning. Hardware version is still having v10.

We started facing this issues after vm migration to different cluster where the underlying storage is same.

Please note, all these freebsd vms were migrated to this vSphere 6.7 environment recently from vSphere 5.5 where we have not faced any issues.

From beginning, we have not installed any vmtools on freebsd vms.

To reproduce this issue, we have built a new free bsd vm with 11.1 version where we have not noticed any single packet drops. All looks fine here.

Next plans;-

Vm tools can be installed. It may fix the issue.

Else we can downgrade the vmxnet3 to e1000 to see the behavior.

Any other idea to resolve this issue?

Thanks

Manivel RR

Reply
0 Kudos
7 Replies
ZibiM
Enthusiast
Enthusiast

Hello

First of all please verify whether your tx buffer has the same size as the rx buffer.

If the tx buffer is bigger than rx buffer, then your VMs might not be able to keep up with the transmit.

I saw this couple of times, especially after migrating to the environment with high speed uplinks.

Second thing is having proper vmxnet3 drivers.

Do you have it without vm tools ?

Are there open vm tools for free bsd ?

E1000 is a good workaround.

This driver has really big symmetrical buffers and should be available by default in the OS.

Reply
0 Kudos
ManivelR
Hot Shot
Hot Shot

Thanks for the response.It might be right because  before VM migration to vSphere 6.7,it was running(VM network) with 2 * 1 G physical nics and now its 2 * 10 G nics(VM network).I mean in vsphere 6.7

As said earlier, we have not installed any vmtools and opentools on freebsd since beginning.

Do we need to install specific vmxnet3 drivers? When we install opentools, vmxnet3 driver will automatically come along this?

I will check the tx and rx size and update you.

Thanks much.

Manivel RR

Reply
0 Kudos
ZibiM
Enthusiast
Enthusiast

Yeah

Open VM Tools will contain vmxnet3 driver among the other things that are nice to have on the VM.

Reply
0 Kudos
continuum
Immortal
Immortal

Why are you  so eager to run vmxnet3 with a FreeBSD guest ?

I would suggest to switch to a nic that comes with stable FreeBSD drivers - if available for FreeBSD I would try the e1000e first (thats also a PCIe device)


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
ManivelR
Hot Shot
Hot Shot

Thanks for the response.

We installed openvmtools on two vms(vmxnet3) but no luck.

We downgraded from vmxnet3 to e1000 but no luck.

We have not adjusted the TX and rx values so far.

Now we migrated other set of windows(vmtools up to date) vms to new 6.7 infrastructure where we are seeing the same issue.

Issue noticed:-

We have 4 to 5 vm vlans.

Vlan id 100 to 105

When we ping the same vlan subnet vms (ex 100), there is no issue. Freebsd vm1 has 10.10.100.5 and windows vm has 10.10.100.6 ---- No packet loss.

When we ping the different subnet vms(vlan 100 to 101,etc....), we are seeing packet loss. I mean 5 or 10 percentage packet loss.

This is cross vlan issue and don't know what is the problem behind.

We migrated some vms back to old vSphere 5.5 infra where there is no cross vlan issue and no packet loss reported among different vlans.

We are using ip hash load balancing with static ether channel.

Thank you,

Manivel RR

Reply
0 Kudos
scott28tt
VMware Employee
VMware Employee

So what is the router between the VLANs?


-------------------------------------------------------------------------------------------------------------------------------------------------------------

Although I am a VMware employee I contribute to VMware Communities voluntarily (ie. not in any official capacity)
VMware Training & Certification blog
Reply
0 Kudos
ManivelR
Hot Shot
Hot Shot

It's a Cisco router.

I got some clue. From this first article it seems this is known issue in vsphere 6.7 update 1 express patch 7

Reddit - vmware - ESX 6.7 - intermittently lose network connectivity over Backup NIC

From this below community, it seems this is known issue in vsphere 6.7 update 1 express patch 5.

Esxi 6.7 VM packet drops on standard vSwitch with Route based on IP Hash

My doubt:- I'm using vsphere 6.7 update 1.Is this known issue in the update 1 as well? Any ideas?

The below link and article relates to vsphere 6.7 update 2 release note. As per this below article, it has been fixed in vsphere 6.7 update 2.

https://docs.vmware.com/en/VMware-vSphere/6.7/rn/vsphere-esxi-67u2-release-notes.html

PR 2238134: When you use an IP hash or source MAC hash teaming policy, the packets might drop or go through the wrong uplink

When you use an IP hash or source MAC hash teaming policy, some packets from the packet list might use a different uplink. As a result, some of them might drop or not be sent out through the uplink determined by the teaming policy.

This issue is resolved in this release.

Thank you,

Raj

Reply
0 Kudos