Intermittent NFS APDs on VMware ESXi 5.5.0 build-2718055

[mrpink37]

About a year ago I upgraded my ESXi 5.5 U1 installation to VMware ESXi 5.5.0 build-1881737 to fix problems with Intermittent NFS APDs as per VMware KB:    Intermittent NFS APDs on VMware ESXi 5.5 U1

The system has been running ever since without any problems. Last weekend I upgraded to latest ESXi 5.5.0 build-2718055 and today the Intermittent NFS APDs errors were back again. However, this time, the vms continued to work when on ESXi 5.5 U1 they would immediately crash.

In their KB article VMware states that the issue was fixed as of build 1881737. I am curious why am I still seeing this issue on later builds?

I can provide detailed logs on request.

/edit: I just analyzed our firewall logs from the same period and pfSense started to drop fragmented packets shortly befor the Intermittent NFS APDs started. Not sure whether ESXi or the NFS server is at fault here. For now I will try to mitigate the issue by configuring pfSense so that it clears invalid DF bits instead of dropping the packets.

[DanielOprea]

Hello,

Please lunch the command: esxcli software vib list    and look if the kb is installed. Or watch it on the GUI of the vSphere Client.

[mrpink37]

Hello,

thanks for the reply. It does not seem to be installed! I thought that the latest patch level would always include critical patch history. Can I install just this patch without reverting back the security patches etc. that are part of the current build?

Name                           Version                                Vendor  Acceptance Level  Install Date

-----------------------------  -------------------------------------  ------  ----------------  ------------

ata-pata-amd                   0.3.10-3vmw.550.0.0.1331820            VMware  VMwareCertified   2013-10-04 

ata-pata-atiixp                0.4.6-4vmw.550.0.0.1331820             VMware  VMwareCertified   2013-10-04 

ata-pata-cmd64x                0.2.5-3vmw.550.0.0.1331820             VMware  VMwareCertified   2013-10-04 

ata-pata-hpt3x2n               0.3.4-3vmw.550.0.0.1331820             VMware  VMwareCertified   2013-10-04 

ata-pata-pdc2027x              1.0-3vmw.550.0.0.1331820               VMware  VMwareCertified   2013-10-04 

ata-pata-serverworks           0.4.3-3vmw.550.0.0.1331820             VMware  VMwareCertified   2013-10-04 

ata-pata-sil680                0.4.8-3vmw.550.0.0.1331820             VMware  VMwareCertified   2013-10-04 

ata-pata-via                   0.3.3-2vmw.550.0.0.1331820             VMware  VMwareCertified   2013-10-04 

block-cciss                    3.6.14-10vmw.550.0.0.1331820           VMware  VMwareCertified   2013-10-04 

ehci-ehci-hcd                  1.0-3vmw.550.0.0.1331820               VMware  VMwareCertified   2013-10-04 

elxnet                         10.0.100.0v-1vmw.550.0.0.1331820       VMware  VMwareCertified   2013-10-04 

esx-base                       5.5.0-2.62.2718055                     VMware  VMwareCertified   2015-06-28 

esx-dvfilter-generic-fastpath  5.5.0-0.0.1331820                      VMware  VMwareCertified   2013-10-04 

esx-tboot                      5.5.0-2.33.2068190                     VMware  VMwareCertified   2015-02-25 

esx-xlibs                      5.5.0-0.0.1331820                      VMware  VMwareCertified   2013-10-04 

esx-xserver                    5.5.0-0.0.1331820                      VMware  VMwareCertified   2013-10-04 

ima-qla4xxx                    2.01.31-1vmw.550.0.0.1331820           VMware  VMwareCertified   2013-10-04 

ipmi-ipmi-devintf              39.1-4vmw.550.0.0.1331820              VMware  VMwareCertified   2013-10-04 

ipmi-ipmi-msghandler           39.1-4vmw.550.0.0.1331820              VMware  VMwareCertified   2013-10-04 

ipmi-ipmi-si-drv               39.1-4vmw.550.0.0.1331820              VMware  VMwareCertified   2013-10-04 

lpfc                           10.0.100.1-1vmw.550.0.0.1331820        VMware  VMwareCertified   2013-10-04 

lsi-mr3                        0.255.03.01-2vmw.550.1.16.1746018      VMware  VMwareCertified   2014-10-03 

lsi-msgpt3                     00.255.03.03-1vmw.550.1.15.1623387     VMware  VMwareCertified   2014-10-03 

misc-cnic-register             1.72.1.v50.1i-1vmw.550.0.0.1331820     VMware  VMwareCertified   2013-10-04 

misc-drivers                   5.5.0-2.62.2718055                     VMware  VMwareCertified   2015-06-28 

mtip32xx-native                3.3.4-1vmw.550.1.15.1623387            VMware  VMwareCertified   2014-10-03 

net-be2net                     4.6.100.0v-1vmw.550.0.0.1331820        VMware  VMwareCertified   2013-10-04 

net-bnx2                       2.2.3d.v55.2-1vmw.550.0.0.1331820      VMware  VMwareCertified   2013-10-04 

net-bnx2x                      1.72.56.v55.2-1vmw.550.0.0.1331820     VMware  VMwareCertified   2013-10-04 

net-cnic                       1.72.52.v55.1-1vmw.550.0.0.1331820     VMware  VMwareCertified   2013-10-04 

net-e1000                      8.0.3.1-3vmw.550.0.0.1331820           VMware  VMwareCertified   2013-10-04 

net-e1000e                     1.1.2-4vmw.550.1.15.1623387            VMware  VMwareCertified   2014-10-03 

net-enic                       1.4.2.15a-1vmw.550.0.0.1331820         VMware  VMwareCertified   2013-10-04 

net-forcedeth                  0.61-2vmw.550.0.0.1331820              VMware  VMwareCertified   2013-10-04 

net-igb                        5.0.5.1.1-1vmw.550.2.54.2403361        VMware  VMwareCertified   2015-06-28 

net-ixgbe                      3.7.13.7.14iov-12vmw.550.2.62.2718055  VMware  VMwareCertified   2015-06-28 

net-mlx4-core                  1.9.7.0-1vmw.550.0.0.1331820           VMware  VMwareCertified   2013-10-04 

net-mlx4-en                    1.9.7.0-1vmw.550.0.0.1331820           VMware  VMwareCertified   2013-10-04 

net-nx-nic                     5.0.621-1vmw.550.0.0.1331820           VMware  VMwareCertified   2013-10-04 

net-r8168                      8.013.00-3vmw.510.0.0.799733           VMware  VMwareCertified   2013-10-04 

net-r8169                      6.011.00-2vmw.510.0.0.799733           VMware  VMwareCertified   2013-10-04 

net-s2io                       2.1.4.13427-3vmw.510.0.0.799733        VMware  VMwareCertified   2013-10-04 

net-sky2                       1.20-2vmw.510.0.0.799733               VMware  VMwareCertified   2013-10-04 

net-tg3                        3.123c.v55.5-1vmw.550.2.33.2068190     VMware  VMwareCertified   2015-02-25 

net-vmxnet3                    1.1.3.0-3vmw.550.2.39.2143827          VMware  VMwareCertified   2015-06-28 

ohci-usb-ohci                  1.0-3vmw.550.0.0.1331820               VMware  VMwareCertified   2013-10-04 

qlnativefc                     1.0.12.0-1vmw.550.0.0.1331820          VMware  VMwareCertified   2013-10-04 

rste                           2.0.2.0088-4vmw.550.1.15.1623387       VMware  VMwareCertified   2014-10-03 

sata-ahci                      3.0-21vmw.550.2.54.2403361             VMware  VMwareCertified   2015-06-28 

sata-ata-piix                  2.12-10vmw.550.2.33.2068190            VMware  VMwareCertified   2015-02-25 

sata-sata-nv                   3.5-4vmw.550.0.0.1331820               VMware  VMwareCertified   2013-10-04 

sata-sata-promise              2.12-3vmw.550.0.0.1331820              VMware  VMwareCertified   2013-10-04 

sata-sata-sil24                1.1-1vmw.550.0.0.1331820               VMware  VMwareCertified   2013-10-04 

sata-sata-sil                  2.3-4vmw.550.0.0.1331820               VMware  VMwareCertified   2013-10-04 

sata-sata-svw                  2.3-3vmw.550.0.0.1331820               VMware  VMwareCertified   2013-10-04 

scsi-aacraid                   1.1.5.1-9vmw.550.0.0.1331820           VMware  VMwareCertified   2013-10-04 

scsi-adp94xx                   1.0.8.12-6vmw.550.0.0.1331820          VMware  VMwareCertified   2013-10-04 

scsi-aic79xx                   3.1-5vmw.550.0.0.1331820               VMware  VMwareCertified   2013-10-04 

scsi-bnx2fc                    1.72.53.v55.1-1vmw.550.0.0.1331820     VMware  VMwareCertified   2013-10-04 

scsi-bnx2i                     2.72.11.v55.4-1vmw.550.0.0.1331820     VMware  VMwareCertified   2013-10-04 

scsi-fnic                      1.5.0.4-1vmw.550.0.0.1331820           VMware  VMwareCertified   2013-10-04 

scsi-hpsa                      5.5.0-44vmw.550.0.0.1331820            VMware  VMwareCertified   2013-10-04 

scsi-ips                       7.12.05-4vmw.550.0.0.1331820           VMware  VMwareCertified   2013-10-04 

scsi-lpfc820                   8.2.3.1-129vmw.550.0.0.1331820         VMware  VMwareCertified   2013-10-04 

scsi-megaraid-mbox             2.20.5.1-6vmw.550.0.0.1331820          VMware  VMwareCertified   2013-10-04 

scsi-megaraid-sas              5.34-9vmw.550.2.33.2068190             VMware  VMwareCertified   2015-02-25 

scsi-megaraid2                 2.00.4-9vmw.550.0.0.1331820            VMware  VMwareCertified   2013-10-04 

scsi-mpt2sas                   14.00.00.00-3vmw.550.1.15.1623387      VMware  VMwareCertified   2014-10-03 

scsi-mptsas                    4.23.01.00-9vmw.550.0.0.1331820        VMware  VMwareCertified   2013-10-04 

scsi-mptspi                    4.23.01.00-9vmw.550.0.0.1331820        VMware  VMwareCertified   2013-10-04 

scsi-qla2xxx                   902.k1.1-9vmw.550.0.0.1331820          VMware  VMwareCertified   2013-10-04 

scsi-qla4xxx                   5.01.03.2-6vmw.550.0.0.1331820         VMware  VMwareCertified   2013-10-04 

uhci-usb-uhci                  1.0-3vmw.550.0.0.1331820               VMware  VMwareCertified   2013-10-04 

xhci-xhci                      1.0-2vmw.550.2.39.2143827              VMware  VMwareCertified   2015-06-28 

tools-light                    5.5.0-2.62.2718055                     VMware  VMwareCertified   2015-06-28 

[DanielOprea]

Hy,

Yes you can do from the console of ESXi, or also through the vSphere Update Manager.

From command line:

Search and Download the patch: https://my.vmware.com/group/vmware/patch#search

Copy the patch in your ESXi, for example in the datastorage local or in another datastorage.

Enter Maintenence Mode

vim-cmd hostsvc/maintenance_mode_enter

Navigate to the directory on the datastore:

cd /vmfs/volumes/Datastore/DirectoryName

Install the Patch:

esxcli software vib install -d "/vmfs/volumes/datastore1/patch-directory/ESXi550-201406001.zip"

Verify that the VIBs are installed on your ESXi host:

esxcli software vib list

After the patch has been installed, reboot the ESX host:

reboot

Exit maintenance mode:

vim-cmd hostsvc/maintenance_mode_exit

[mrpink37]

If I follow this procedure it will downgrade my build level to ESX build number 1881737. At least this is what it tells me after the reboot.

~ # esxcli software sources profile list -d /vmfs/volumes/esxi_coreos0/ESXi550-201406001.zip

Name                             Vendor        Acceptance Level

-------------------------------  ------------  ----------------

ESXi-5.5.0-20140604001-no-tools  VMware, Inc.  PartnerSupported

ESXi-5.5.0-20140604001-standard  VMware, Inc.  PartnerSupported

~ # vmware --version

VMware ESXi 5.5.0 build-1881737

So what I am looking for I guess is a way to just get this patch properly installed (VMware KB: VMware ESXi 5.5, Patch ESXi-5.5.0-20140604001-standard) without reverting back to the old build number.

[DanielOprea]

OK, but after downgrade, you're back has passed the VUM to upgrade the last level ??

You have performed the procedure by command line when has downgrade the version, but why not use the VUM to patch it? Because using VUM should not be downgraded.

Please try it with the VUM!

[mrpink37]

I am running a single ESXi host, no vCenter server and no VUM. That's why.

But after giving this a second thought - I am pretty sure that the patch must be included in the esx-base 5.5.0-2.62.2718055 package. Otherwise this doesn't make any sense. So the reason for it not appearing in the esxcli "software vib list" output above is likely that it is included in the higher build level anyways.

[DanielOprea]

Hello,

The patch is included in the latest version of ESXi image 5.5U2 but in the ISO.

If the ESXi only, if you had to date with the latest ISO ESXi you have gotten the patch failure NFS and also all other patches until leaving the last level.

And in the case of the VUM upgrade it had been gotten so this patch like everyone else.

Regards

Daniel

[mrpink37]

Hi Daniel,

so basically what you are saying is that in order for the patch to be included I must upgrade my host with the *.iso rather than using the "esxcli software vib update" command? Or use VUM with a vCenter service that would automatically upgrade my ESXi host? Can you point me to any official VMware resource where this is documented?

Thanks