High pNic error rate detected. Check the host's vSAN performance view for details

[sabarirjpm]

Received this alert in vCenter and the kb aritcle https://kb.vmware.com/s/article/83627 says that it is related to the error conditions between vmnic and the physical port. Can someone help me to identify which port has issues for me to debug further. I see this alert in two of the seven esxi hosts.

[Kamikadz3]

HI,

can you show NIC stats?

# /usr/lib/vmware/vm-support/bin/nicinfo.sh | more 

[buffalix]

NIC:  vmnic7

   vmnic7 0000:03:00.1 ixgben Up Up 10000 Full a0:36:9f:d0:ad:4e 1500 Intel(R) Ethernet Controller 10 Gigabit X540-AT2

   NICInfo:

      Advertised Auto Negotiation: true

      Advertised Link Modes: Auto, 100BaseT/Full, 1000BaseT/Full, 10000BaseT/Full

      Auto Negotiation: true

      Cable Type: Twisted Pair

      Current Message Level: 0

      Driver Info: 

         NICDriverInfo:

            Bus Info: 0000:03:00:1

            Driver: ixgben

            Firmware Version: 4.04.0 0x8000063a, 19.5.12

            Version: 1.13.1.0

      Link Detected: true

      Link Status: Up 

      Name: vmnic7

      PHY Address: 0

      Pause Autonegotiate: false

      Pause RX: true

      Pause TX: false

      Supported Ports: TP

      Supports Auto Negotiation: true

      Supports Pause: true

      Supports Wakeon: false

      Transceiver: 

      Virtual Address: 00:50:56:5a:69:6f

      Wakeon: None

   DCB not supported for vmnic7

   Module Parameters:

      Name     Type          Value  Description

      -------  ------------  -----  -----------

      DRSS     array of int         DefQueue RSS state: 0 = disable, 1 = enable (default = 0; 4 queues if DRSS is enabled)

      DevRSS   array of int         Device RSS state: 0 = disable, 1 = enable (default = 0; 16 queues but all virtualization features disabled if DevRSS is enabled)

      QPair    array of int         Pair Rx & Tx Queue Interrupt: 0 = disable, 1 = enable (default)

      RSS      array of int         NetQueue RSS state: 0 = disable, 1 = enable (default = 1; 4 queues if RSS is enabled)

      RxDesc   array of int         Maximum number of RX descriptors (128..8192)

      RxITR    array of int         Default RX interrupt interval: 0 = disable, 1 = dynamic throttling, 2-1000 in microseconds (default = 50)

      TxDesc   array of int         Maximum number of TX descriptors (128..8192)

      TxITR    array of int         Default TX interrupt interval: 0 = disable, 1 = dynamic throttling, 2-1000 in microseconds (default = 100)

      VMDQ     array of int         Number of Virtual Machine Device Queues: 0/1 = disable, 2-16 enable (default = 😎

      max_vfs  array of int         Maximum number of VFs to be enabled (0..63)

   NIC statistics for vmnic7:

      Packets received: 2127380899

      Packets sent: 828680550

      Bytes received: 1995888077630

      Bytes sent: 2476332356869

      Receive packets dropped: 0

      Transmit packets dropped: 0

      Multicast packets received: 119311

      Broadcast packets received: 123250

      Multicast packets sent: 4236

      Broadcast packets sent: 1168

      Total receive errors: 0

      Receive length errors: 0

      Receive over errors: 0

      Receive CRC errors: 0

      Receive frame errors: 0

      Receive FIFO errors: 0

      Receive missed errors: 0

      Total transmit errors: 0

      Transmit aborted errors: 0

      Transmit carrier errors: 0

      Transmit FIFO errors: 0

      Transmit heartbeat errors: 0

      Transmit window errors: 0

   NIC Private statistics:

      txXon=0

      txXoff=0

      rxXon=5704475

      rxXoff=109262929

      RXCTRL=00000001

      TXCTRL=81000055

      MRQC=00f3000a

      MTQC=0000000a

      LINKS=70000080

      VFRE[0]=ffffffff

      VFTE[0]=ffffffff

      MPSAR[0]=00000000:00000000

      MPSAR[1]=00000000:00000100

      MPSAR[2]=00000000:00000000

      MPSAR[3]=00000000:00000000

      PFVML2FLT[0]=1f800000

      PFVML2FLT[1]=01800000

      PFVML2FLT[2]=01800000

      PFVML2FLT[3]=01800000

      rxq0 (hwq0): ringSize=1024 next2fill=200 next2proc=203 RDH=203 RDT=200 RXDCTL=42000000 dropped=0 rxPkts=273488075 rxBytes=235041434835 rxIrqCnt=136885487 rxITR=50

      rxq1 (hwq4): ringSize=1024 next2fill=0 next2proc=0 RDH=0 RDT=0 RXDCTL=40000000 dropped=0 rxPkts=0 rxBytes=0 rxIrqCnt=0 rxITR=50

      rxq2 (hwq8): ringSize=1024 next2fill=0 next2proc=0 RDH=0 RDT=0 RXDCTL=40000000 dropped=0 rxPkts=0 rxBytes=0 rxIrqCnt=0 rxITR=50

      rxq3 (hwq12): ringSize=1024 next2fill=0 next2proc=0 RDH=0 RDT=0 RXDCTL=40000000 dropped=0 rxPkts=0 rxBytes=0 rxIrqCnt=0 rxITR=50

      rxq4 (hwq16): ringSize=1024 next2fill=0 next2proc=0 RDH=0 RDT=0 RXDCTL=40000000 dropped=0 rxPkts=0 rxBytes=0 rxIrqCnt=0 rxITR=50

      rxq5 (hwq20): ringSize=1024 next2fill=0 next2proc=0 RDH=0 RDT=0 RXDCTL=40000000 dropped=0 rxPkts=0 rxBytes=0 rxIrqCnt=0 rxITR=50

      rxq6 (hwq24): ringSize=1024 next2fill=0 next2proc=0 RDH=0 RDT=0 RXDCTL=40000000 dropped=0 rxPkts=0 rxBytes=0 rxIrqCnt=0 rxITR=50

      rxq7 (hwq28): ringSize=1024 next2fill=0 next2proc=0 RDH=0 RDT=0 RXDCTL=40000000 dropped=0 rxPkts=0 rxBytes=0 rxIrqCnt=0 rxITR=50

      rxq8 (hwq32): ringSize=1024 next2fill=606 next2proc=609 RDH=609 RDT=606 RXDCTL=42000000 dropped=0 rxPkts=266416365 rxBytes=306388871305 rxIrqCnt=313558340 rxITR=50

      rxq9 (hwq33): ringSize=1024 next2fill=118 next2proc=121 RDH=121 RDT=118 RXDCTL=42000000 dropped=0 rxPkts=723633732 rxBytes=580171705404 rxIrqCnt=152989009 rxITR=50

      rxq10 (hwq34): ringSize=1024 next2fill=950 next2proc=953 RDH=953 RDT=950 RXDCTL=42000000 dropped=0 rxPkts=545999413 rxBytes=486492717607 rxIrqCnt=114469728 rxITR=50

      rxq11 (hwq35): ringSize=1024 next2fill=782 next2proc=785 RDH=785 RDT=782 RXDCTL=42000000 dropped=0 rxPkts=317843482 rxBytes=387793437375 rxIrqCnt=65303207 rxITR=50

      txq0 (hwq0): ringSize=1024 next2fill=883 next2proc=883 TDH=883 TDT=883 TXDCTL=02000000 dropped=0 txPkts=135965080 txBytes=241331409000 txIrqCnt=0 txITR=50

      txq1 (hwq4): ringSize=1024 next2fill=0 next2proc=0 TDH=0 TDT=0 TXDCTL=02000000 dropped=0 txPkts=0 txBytes=0 txIrqCnt=0 txITR=50

      txq2 (hwq8): ringSize=1024 next2fill=0 next2proc=0 TDH=0 TDT=0 TXDCTL=02000000 dropped=0 txPkts=0 txBytes=0 txIrqCnt=0 txITR=50

      txq3 (hwq12): ringSize=1024 next2fill=0 next2proc=0 TDH=0 TDT=0 TXDCTL=02000000 dropped=0 txPkts=0 txBytes=0 txIrqCnt=0 txITR=50

      txq4 (hwq16): ringSize=1024 next2fill=0 next2proc=0 TDH=0 TDT=0 TXDCTL=02000000 dropped=0 txPkts=0 txBytes=0 txIrqCnt=0 txITR=50

      txq5 (hwq20): ringSize=1024 next2fill=0 next2proc=0 TDH=0 TDT=0 TXDCTL=02000000 dropped=0 txPkts=0 txBytes=0 txIrqCnt=0 txITR=50

      txq6 (hwq24): ringSize=1024 next2fill=0 next2proc=0 TDH=0 TDT=0 TXDCTL=02000000 dropped=0 txPkts=0 txBytes=0 txIrqCnt=0 txITR=50

      txq7 (hwq28): ringSize=1024 next2fill=0 next2proc=0 TDH=0 TDT=0 TXDCTL=02000000 dropped=0 txPkts=0 txBytes=0 txIrqCnt=0 txITR=50

      txq8 (hwq32): ringSize=1024 next2fill=850 next2proc=850 TDH=850 TDT=850 TXDCTL=02000000 dropped=0 txPkts=692715591 txBytes=2235001112687 txIrqCnt=0 txITR=50

      txq9 (hwq33): ringSize=1024 next2fill=0 next2proc=0 TDH=0 TDT=0 TXDCTL=02000000 dropped=0 txPkts=0 txBytes=0 txIrqCnt=0 txITR=50

      txq10 (hwq34): ringSize=1024 next2fill=0 next2proc=0 TDH=0 TDT=0 TXDCTL=02000000 dropped=0 txPkts=0 txBytes=0 txIrqCnt=0 txITR=50

      txq11 (hwq35): ringSize=1024 next2fill=0 next2proc=0 TDH=0 TDT=0 TXDCTL=02000000 dropped=0 txPkts=0 txBytes=0 txIrqCnt=0 txITR=50

   Current Ring Size:

   RingInfo:

      RX: 1024

      RX Mini: 0

      RX Jumbo: 0

      TX: 1024

   Preset Maximum Ring Size:

   RingInfo:

      Max RX: 8192

      Max RX Mini: 0

      Max RX Jumbo: 0

[Maxilib62]

Hi,

Do you resolve your problem ? I have same error ...

[Kamikadz3]

HI,

from provided stats, there is  no evidence of any problems - no drops, no errors at all.

Maybe that is silly question - are you sure you collected starts form correct ESX? Did VC reported problems on vmnic7?

[ckirim]

hello Dear,

I have exactly the same problem with the Intel network adapter. As you know, Qlogic has already released more newest driver for this particular problem, but we need a similar one for the Intel card. We already opened tickets to both sides (VMware- HPE ). But after all investigations, the only response was to "ignore" the warning messages. We don't have an increasing CRC error on the switch side, we don't have any packet drop or loss, and we don't have any performance issues as well. we need similar from Intel like the below ;

Note: After further investigation, HPE has confirmed that the VMware alarms are produced due to an accounting issue with the qedentv driver that is incorrectly recording certain packets as RX drops. This is not an actual network issue, it is an accounting issue with statistics which generates the VMware alarm.  ( HPE Customer Advisory https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=a00126305en_us)

[SwanseaUni]

We have had this twice and both times it was a faulty SFP in the esx nodes, on our cisco switches we logged in and ran 

sh int | in up|CRC

this will list all the ports and their CRC errors find the port with the high CRC error rate and change the SFP's fibre cables etc and then clear the counters to see if the CRC errors return

[flube]

anyone?