vmxnet3 rxcompletion descriptor is returning len field as 0 for both Standard and JUMBO Rx buffer ring. Is this expected?

[nirmalanand]

Hello experts,

We are running the below ESX version and running FreeBSD 11 as guest with vmxnet3 interface.

[root@localhost:~] uname -a

VMkernel localhost 6.0.0 #1 SMP Release build-3620759 Mar  3 2016 18:41:52 x86_64 x86_64 x86_64 ESXi

[root@localhost:~]

Currently we have only one Rx/Tx queue enabled for the interface.

We do not have SW LRO enabled on the guest for the vmxnet3 interface. On the ESX host, the default setting is used..

With 20 Mbps of traffic, occasionally we find len field in the Rx Completion ring descriptor having a value 0, when the qid points to STANDARD RX or JUMBO RX Buffer ring.

Questions

1) Is this expected? Please let us know.

2) Should we ignore the zero length descriptors if the completion descriptor points to the JUMBO Rx Buffer ring.

Following is a dump of the Rx completion ring entries

rxd_idx = 85, pad1 = 0, eop = 0, sop = 1, qid = 0, rss_type = 0, no_csum = 0, pad2 = 0, rss_hash = 0, len = 64, error = 0, vlan = 0, vtag = 0, csum = 0, csum_ok = 0, udp = 0, tcp = 0, ipcsum_ok = 0, ipv6 = 0, ipv4 = 0,

  fragment = 0, fcs = 0, type = 3, gen = 0}

{rxd_idx = 359, pad1 = 0, eop = 0, sop = 0, qid = 8, rss_type = 0, no_csum = 0, pad2 = 0, rss_hash = 0, len = 0, error = 0, vlan = 0, vtag = 0, csum = 0, csum_ok = 0, udp = 0, tcp = 0, ipcsum_ok = 0, ipv6 = 0, ipv4 = 0,

  fragment = 0, fcs = 0, type = 3, gen = 0}

..

..

{rxd_idx = 359, pad1 = 0, eop = 0, sop = 0, qid = 8, rss_type = 0, no_csum = 0, pad2 = 0, rss_hash = 0, len = 0, error = 0, vlan = 0, vtag = 0, csum = 0, csum_ok = 0, udp = 0, tcp = 0, ipcsum_ok = 0, ipv6 = 0, ipv4 = 0,  fragment = 0, fcs = 0, type = 3, gen = 0}

{rxd_idx = 366, pad1 = 0, eop = 1, sop = 0, qid = 8, rss_type = 2, no_csum = 0, pad2 = 0, rss_hash = 1091877671, len = 38, error = 0, vlan = 0, vtag = 0, csum = 0, csum_ok = 1, udp = 0, tcp = 1, ipcsum_ok = 1, ipv6 = 0,

  ipv4 = 1, fragment = 0, fcs = 0, type = 3, gen = 0}

Regards,

Nirmal

[pwilk]

Hi Nirmal,

This is typical behavior. In my opinion no action needs to be taken. You can safely ignore the zero length descriptors as well.

Cheers,

Paul Wilk

[nirmalanand]

Thanks Paul for the response.

One follow up question,

We do not have LRO turned on for the vmxnet3 interface on the guest. But why would the back-end activate the Rx buffers in the JUMBO buffer ring? (We have a valid memory allocated though, for all the JUMBO buffer ring entries)

Also, we do not have JUMBO frames enabled on the guest vmxnet3 interface.

Questioin: Should we skip Rx Completion Descriptor entries if the qid points to the JUMBO buffer ring, when LRO or JUMBO frames are not enabled?

Let us know.

Thanks,

Nirmal

[pwilk]

Hi Nirmal,

RxCD should be skipped in this situation only. If you enable LRO or JUMBO frames then you need to action RxCD.

Cheers,

Paul Wilk

[nirmalanand]

Hi Paul

Couple more questions to understand the scenario further:

1) In cases where JUMBO frames and SW LRO are disabled, we observe non-zero length RxCD pointing to JUMBO buffer ring descriptors. Is this expected? When does this happen? Do we need to drop the complete packet or just the non-zero JUMBO buffer ring entries only?

2)Is there any register or setting we can view the current status of SW LRO setting for a VMXNET3 vNIC?

Thanks,

Nirmal

[pwilk · Accepted solution]

1) Well, it's kind of expected however you should still consider dripping at least the non-zero JMB packets according to best practices

2) There is no single panel or "pane of glass" type of command which you can run to view everything at once. I suggest reading through this document VMware Knowledge Base

[nirmalanand]

Thanks Paul for the prompt response. Appreciate it.

Regards,

Nirmal