Weird transfer speed cap using admin GUI/CLI - No cap on VM

[Neutro]

Hey guys,

I'm having a weird transfer speed cap problem on my ESXI 6.7.0 Update 3 (Build 16713306) server, which is similar to https://communities.vmware.com/t5/ESXi-Discussions/Slow-network-performance-for-ESXi-6-7-Host/m-p/18...
When i try to download or upload an ISO file through the ESXi admin web GUI from my Windows 10 desktop PC, the transfer speed is capped around 140mbps on a 1gbps link.

If i activate the SSH shell and use WinSCP to transfer the ISO file to the datastore using SFTP protocol the transfer speed is also capped around 140mbps.

If i do the same using SCP protocol instead of SFTP, the transfer speed drops to around 5 Mbps.

If i try to export a VM using the web GUI, the transfer speed is also capped around 140mbps.

However, if i try using WinSCP to connect to one of the VM running Centos 8 on the ESXi i get full download and upload speed, uncapped.

I've tried running iperf3 server on the ESXi using the SSH cli and using my Windows10 as client and the result shows full uncapped network speed for each direction.

I've also tried benchmarking the SSD drive from the SSH cli using dd and it shows no disk speed cap, bursting over 110MB/s on esxtop.

There are no hardware bottlenecks, both CPU and memory have only little usage when doing these tests. 

When using wget from SSH cli to download an ISO file from a ftp server the download speed is capped also around 140mbps.

Downloading the exact same file from the Centos 8 VM goes at 850 mbps, almost full link speed capacity. 

Things i've tried with no results:

- rebooting the server,
- using different web browsers to upload/download files (firefox, chrome, iexplorer, all using default settings),
- turning off esxi firewall,
- changing the vswitch MTU to 9000,
- turning off esxi swap,
- switching to root account instead of my user account (which also has root access),
- looking at logs to see if there were related errors showing,
- searching on this forum and google to see if there were solutions for a similar problem,
- Praying to the VMware gods and begging for a miracle.

Could it be linked to the fact that my server is running a free ESXi licence, which makes vStorage API unavailable?

Based on all these facts it looks like there is either some throttle mechanism which caps the transfert speed using the management interface, either a protocol problem with it.

I'm out of ideas and spend countless hours on this problem.

Any help would be greatly appreciated. 

Thanks!

[Neutro]

Update: i've tried installing esxi 6.7 latest version ESXi-6.7.0-20201103001-standard (Build 17098360)  on my local PC on Workstation 15 evaluation and i have the same problem: transfer is capped around 240mbps (30MB/s) while my SSD is able to transfer up to 600MB/s and my local PC is a hardware beast.

Using eval version on esxi as well so no licence limitation and still the same problem.

Also Workstation 15 is compatible with esxi 6.7 so it's not a hardware compatibility problem as well.

Also have the same problem on another server running latest v7.

So it's definitively a bug with ESXI which can be easily reproduced.

@VMware guys could you please reproduce it too to confirm and transmit to the concerned people if possible? TIA 🙂

[Neutro]

I've done some more test with latest esxi 7.0 with latest update today.

With the web GUI the transfer speed is still capped.

With SCP the transfer speed is OK.

So maybe the GUI is capped whatsoever, but my real server might have a SCP speed problem due to its NIC?

Anyway i've found a workaround: running a virtualized esxi on my local computer, then use vcenter converter standalone to copy the VM from my server to my local computer. It allows the VM compatibility to stay the same and no transfer speed cap. 

[Meph79]

I have noticed the same speed cap on several installations.. I think its a cap from vmware to not exhaust the resources for running VM:s for example.. It would be good to get a statement about this from @VMware . Maybe also a parameter that could be changed to remove the cap... It is very annoying when time counts... 

This is my latest example:

Export:
Start: 17/5 10:03
End: 22/5 17:54
Size: 3,02 TB
Time hours: 128 hours
Speed: 6,87 MiB/s (57,6Mbit/s) (No VMs running on host)
VMware ESXi 7.0.3 build-20842708


Doing import now and speed is capped at 230 Mbit/s
Disk array: Maximum Latency Average: 0,07ms
Total disk usage Average: 65MB/s 
No VM:s running...