Slow transfer from NAS to ESXI local datastore

[phuz]

I currently have my VMs on a NAS, but will be migrating them to local drives on the ESXI server itself.  I am trying to copy one of them over, through command line (cp /vmfs/volumes/NAS/(file) /vmfs/volumes/datastore1) and it's transferring at about 17-22mbps.  It's a non-crowded 1gbit network, so why is my transfer so slow?  If it was just this one single VM, I would write it off, but I have 20 more VMs to migrate and would like to avoid these delays.

[phuz]

If I run an internet speed test from within a VM on that host, I get speeds of ~200mbps, so the NIC is not the issue. 

I tried using WinSCP to copy a large file from my laptop to the ESXi host and I got the same slow results (~16mbps). 

I'm baffled.

[phuz · Accepted solution]

In case anyone else runs into this on ESXi 6.5, it has to do with a bad version of the AHCI driver.

See here for fix as it worked for me.  I'm not transferring to local datastore at ~500mbps.

NXHut - IT and Windows - News: Fix slow disk performance (vmw_ahci driver) in ESXi 6.5

[darkzealot]

Thank you very much. I confirm that this issue was bad AHCI driver and your link provided the correct fix (upgrade to update 01-build 5969303).

In case anybody would like the esxcli upgrade command using the vib (update-from-esxi6.5-6.5_update01.zip) along with results here you go:

[root@esxi6:/vmfs/volumes/5ae86f12-1421a779-f946-50465da1f8a3/patches] esxcli software vib update --depot /vmfs/volumes/datastore1/patches/update-from-esxi6.5-6.5_update01.

zip

Installation Result

   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.

   Reboot Required: true

   VIBs Installed: VMW_bootbank_i40en_1.3.1-5vmw.650.1.26.5969303, VMW_bootbank_igbn_0.1.0.0-14vmw.650.1.26.5969303, VMW_bootbank_ixgben_1.4.1-2vmw.650.1.26.5969303, VMW_bootbank_misc-drivers_6.5.0-1.26.5969303, VMW_bootbank_ntg3_4.1.2.0-1vmw.650.1.26.5969303, VMW_bootbank_nvme_1.2.0.32-4vmw.650.1.26.5969303, VMW_bootbank_pvscsi_0.1-1vmw.650.1.26.5969303, VMW_bootbank_qlnativefc_2.1.50.0-1vmw.650.1.26.5969303, VMW_bootbank_sata-ahci_3.0-26vmw.650.1.26.5969303, VMW_bootbank_usbcore-usb_1.0-3vmw.650.1.26.5969303, VMW_bootbank_vmkata_0.1-1vmw.650.1.26.5969303, VMW_bootbank_vmkusb_0.1-1vmw.650.1.26.5969303, VMW_bootbank_vmw-ahci_1.0.0-39vmw.650.1.26.5969303, VMware_bootbank_esx-base_6.5.0-1.26.5969303, VMware_bootbank_esx-tboot_6.5.0-1.26.5969303, VMware_bootbank_esx-ui_1.21.0-5724747, VMware_bootbank_esx-xserver_6.5.0-0.23.5969300, VMware_bootbank_lsu-hp-hpsa-plugin_2.0.0-5vmw.650.1.26.5969303, VMware_bootbank_lsu-lsi-lsi-mr3-plugin_1.0.0-10vmw.650.1.26.5969303, VMware_bootbank_lsu-lsi-lsi-msgpt3-plugin_1.0.0-7vmw.650.1.26.5969303, VMware_bootbank_lsu-lsi-megaraid-sas-plugin_1.0.0-8vmw.650.1.26.5969303, VMware_bootbank_lsu-lsi-mpt2sas-plugin_2.0.0-6vmw.650.1.26.5969303, VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.10-1.26.5969303, VMware_bootbank_vsan_6.5.0-1.26.5912915, VMware_bootbank_vsanhealth_6.5.0-1.26.5912974

   VIBs Removed: VMW_bootbank_i40en_1.1.0-1vmw.650.0.0.4564106, VMW_bootbank_igbn_0.1.0.0-12vmw.650.0.0.4564106, VMW_bootbank_ixgben_1.0.0.0-9vmw.650.0.14.5146846, VMW_bootbank_misc-drivers_6.5.0-0.14.5146846, VMW_bootbank_ntg3_4.1.0.0-1vmw.650.0.0.4564106, VMW_bootbank_nvme_1.2.0.32-2vmw.650.0.0.4564106, VMW_bootbank_pvscsi_0.1-1vmw.650.0.0.4564106, VMW_bootbank_qlnativefc_2.1.30.0-11vmw.650.0.0.4564106, VMW_bootbank_sata-ahci_3.0-22vmw.650.0.0.4564106, VMW_bootbank_usbcore-usb_1.0-3vmw.650.0.0.4564106, VMW_bootbank_vmkata_0.1-1vmw.650.0.0.4564106, VMW_bootbank_vmkusb_0.1-1vmw.650.0.14.5146846, VMW_bootbank_vmw-ahci_1.0.0-34vmw.650.0.14.5146846, VMware_bootbank_esx-base_6.5.0-0.19.5310538, VMware_bootbank_esx-tboot_6.5.0-0.0.4564106, VMware_bootbank_esx-ui_1.18.0-5270848, VMware_bootbank_esx-xserver_6.5.0-0.0.4564106, VMware_bootbank_lsu-hp-hpsa-plugin_2.0.0-3vmw.650.0.0.4564106, VMware_bootbank_lsu-lsi-lsi-mr3-plugin_1.0.0-7vmw.650.0.0.4564106, VMware_bootbank_lsu-lsi-lsi-msgpt3-plugin_1.0.0-6vmw.650.0.0.4564106, VMware_bootbank_lsu-lsi-megaraid-sas-plugin_1.0.0-7vmw.650.0.0.4564106, VMware_bootbank_lsu-lsi-mpt2sas-plugin_2.0.0-5vmw.650.0.0.4564106, VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.10-0.0.4564106, VMware_bootbank_vsan_6.5.0-0.19.5310540, VMware_bootbank_vsanhealth_6.5.0-0.19.5310541

   VIBs Skipped: VMW_bootbank_ata-libata-92_3.00.9.2-16vmw.650.0.0.4564106, VMW_bootbank_ata-pata-amd_0.3.10-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-atiixp_0.4.6-4vmw.650.0.0.4564106, VMW_bootbank_ata-pata-cmd64x_0.2.5-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-hpt3x2n_0.3.4-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-pdc2027x_1.0-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-serverworks_0.4.3-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-sil680_0.4.8-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-via_0.3.3-2vmw.650.0.0.4564106, VMW_bootbank_block-cciss_3.6.14-10vmw.650.0.0.4564106, VMW_bootbank_char-random_1.0-3vmw.650.0.0.4564106, VMW_bootbank_ehci-ehci-hcd_1.0-4vmw.650.0.14.5146846, VMW_bootbank_elxnet_11.1.91.0-1vmw.650.0.0.4564106, VMW_bootbank_hid-hid_1.0-3vmw.650.0.0.4564106, VMW_bootbank_ima-qla4xxx_2.02.18-1vmw.650.0.0.4564106, VMW_bootbank_ipmi-ipmi-devintf_39.1-4vmw.650.0.0.4564106, VMW_bootbank_ipmi-ipmi-msghandler_39.1-4vmw.650.0.0.4564106, VMW_bootbank_ipmi-ipmi-si-drv_39.1-4vmw.650.0.0.4564106, VMW_bootbank_lpfc_11.1.0.6-1vmw.650.0.0.4564106, VMW_bootbank_lsi-mr3_6.910.18.00-1vmw.650.0.0.4564106, VMW_bootbank_lsi-msgpt2_20.00.01.00-3vmw.650.0.0.4564106, VMW_bootbank_lsi-msgpt3_12.00.02.00-11vmw.650.0.0.4564106, VMW_bootbank_misc-cnic-register_1.78.75.v60.7-1vmw.650.0.0.4564106, VMW_bootbank_mtip32xx-native_3.9.5-1vmw.650.0.0.4564106, VMW_bootbank_ne1000_0.8.0-16vmw.650.1.26.5969303, VMW_bootbank_nenic_1.0.0.2-1vmw.650.0.0.4564106, VMW_bootbank_net-bnx2_2.2.4f.v60.10-2vmw.650.0.0.4564106, VMW_bootbank_net-bnx2x_1.78.80.v60.12-1vmw.650.0.0.4564106, VMW_bootbank_net-cdc-ether_1.0-3vmw.650.0.0.4564106, VMW_bootbank_net-cnic_1.78.76.v60.13-2vmw.650.0.0.4564106, VMW_bootbank_net-e1000_8.0.3.1-5vmw.650.0.0.4564106, VMW_bootbank_net-e1000e_3.2.2.1-2vmw.650.0.0.4564106, VMW_bootbank_net-enic_2.1.2.38-2vmw.650.0.0.4564106, VMW_bootbank_net-fcoe_1.0.29.9.3-7vmw.650.0.0.4564106, VMW_bootbank_net-forcedeth_0.61-2vmw.650.0.0.4564106, VMW_bootbank_net-igb_5.0.5.1.1-5vmw.650.0.0.4564106, VMW_bootbank_net-ixgbe_3.7.13.7.14iov-20vmw.650.0.0.4564106, VMW_bootbank_net-libfcoe-92_1.0.24.9.4-8vmw.650.0.0.4564106, VMW_bootbank_net-mlx4-core_1.9.7.0-1vmw.650.0.0.4564106, VMW_bootbank_net-mlx4-en_1.9.7.0-1vmw.650.0.0.4564106, VMW_bootbank_net-nx-nic_5.0.621-5vmw.650.0.0.4564106, VMW_bootbank_net-tg3_3.131d.v60.4-2vmw.650.0.0.4564106, VMW_bootbank_net-usbnet_1.0-3vmw.650.0.0.4564106, VMW_bootbank_net-vmxnet3_1.1.3.0-3vmw.650.0.0.4564106, VMW_bootbank_nhpsa_2.0.6-3vmw.650.0.0.4564106, VMW_bootbank_nmlx4-core_3.16.0.0-1vmw.650.0.0.4564106, VMW_bootbank_nmlx4-en_3.16.0.0-1vmw.650.0.0.4564106, VMW_bootbank_nmlx4-rdma_3.16.0.0-1vmw.650.0.0.4564106, VMW_bootbank_nmlx5-core_4.16.0.0-1vmw.650.0.0.4564106, VMW_bootbank_nvmxnet3_2.0.0.22-1vmw.650.0.0.4564106, VMW_bootbank_ohci-usb-ohci_1.0-3vmw.650.0.0.4564106, VMW_bootbank_qedentv_2.0.3.29-1vmw.650.0.0.4564106, VMW_bootbank_qfle3_1.0.2.7-1vmw.650.0.0.4564106, VMW_bootbank_qflge_1.1.0.3-1vmw.650.0.0.4564106, VMW_bootbank_sata-ata-piix_2.12-10vmw.650.0.0.4564106, VMW_bootbank_sata-sata-nv_3.5-4vmw.650.0.0.4564106, VMW_bootbank_sata-sata-promise_2.12-3vmw.650.0.0.4564106, VMW_bootbank_sata-sata-sil24_1.1-1vmw.650.0.0.4564106, VMW_bootbank_sata-sata-sil_2.3-4vmw.650.0.0.4564106, VMW_bootbank_sata-sata-svw_2.3-3vmw.650.0.0.4564106, VMW_bootbank_scsi-aacraid_1.1.5.1-9vmw.650.0.0.4564106, VMW_bootbank_scsi-adp94xx_1.0.8.12-6vmw.650.0.0.4564106, VMW_bootbank_scsi-aic79xx_3.1-5vmw.650.0.0.4564106, VMW_bootbank_scsi-bnx2fc_1.78.78.v60.8-1vmw.650.0.0.4564106, VMW_bootbank_scsi-bnx2i_2.78.76.v60.8-1vmw.650.0.0.4564106, VMW_bootbank_scsi-fnic_1.5.0.45-3vmw.650.0.0.4564106, VMW_bootbank_scsi-hpsa_6.0.0.84-1vmw.650.0.0.4564106, VMW_bootbank_scsi-ips_7.12.05-4vmw.650.0.0.4564106, VMW_bootbank_scsi-iscsi-linux-92_1.0.0.2-3vmw.650.0.0.4564106, VMW_bootbank_scsi-libfc-92_1.0.40.9.3-5vmw.650.0.0.4564106, VMW_bootbank_scsi-megaraid-mbox_2.20.5.1-6vmw.650.0.0.4564106, VMW_bootbank_scsi-megaraid-sas_6.603.55.00-2vmw.650.0.0.4564106, VMW_bootbank_scsi-megaraid2_2.00.4-9vmw.650.0.0.4564106, VMW_bootbank_scsi-mpt2sas_19.00.00.00-1vmw.650.0.0.4564106, VMW_bootbank_scsi-mptsas_4.23.01.00-10vmw.650.0.0.4564106, VMW_bootbank_scsi-mptspi_4.23.01.00-10vmw.650.0.0.4564106, VMW_bootbank_scsi-qla4xxx_5.01.03.2-7vmw.650.0.0.4564106, VMW_bootbank_shim-iscsi-linux-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-iscsi-linux-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libata-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libata-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libfc-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libfc-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libfcoe-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libfcoe-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-vmklinux-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-vmklinux-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-vmklinux-9-2-3-0_6.5.0-0.0.4564106, VMW_bootbank_uhci-usb-uhci_1.0-3vmw.650.0.0.4564106, VMW_bootbank_usb-storage-usb-storage_1.0-3vmw.650.0.0.4564106, VMW_bootbank_vmkplexer-vmkplexer_6.5.0-0.0.4564106, VMW_bootbank_xhci-xhci_1.0-3vmw.650.0.0.4564106, VMware_bootbank_cpu-microcode_6.5.0-0.0.4564106, VMware_bootbank_emulex-esx-elxnetcli_11.1.28.0-0.0.4564106, VMware_bootbank_esx-dvfilter-generic-fastpath_6.5.0-0.0.4564106, VMware_bootbank_native-misc-drivers_6.5.0-0.0.4564106, VMware_bootbank_rste_2.0.2.0088-4vmw.650.0.0.4564106, VMware_locker_tools-light_6.5.0-0.23.5969300

[root@esxi6:/vmfs/volumes/5ae86f12-1421a779-f946-50465da1f8a3/patches]