VMware Cloud Community
Exetus
Contributor
Contributor
‎03-17-2021 11:29 AM

Poor Network Performance in VMs

Hello All,

I have been trying to get a new ESXi Server spun up and I am experiencing very aggravating network performance issues on my VMs.  I have spent the last several days researching some of these issues and trying different fixes / work-arounds; however, I have come up empty.

This issue is causing download rates on my VMs to be much lower than what I would expect.  Into this network I have a Business-class cable connection running at 200Mbps Down / 25Mbps Up; however, I am experiencing on these VMs rates of only 1-4Mbps Down and 15-20Mbps Up.

After all the testing I have done, I still cannot come up with any answers as to what may be going on or how I could resolve these issues.  It should be noted that beside the testing listed below, I also tested using VMs running other operating systems (RHEL, Windows 10, Windows Server 2019, Debian, and FreeBSD); however, the results where exactly the same (major performance issues).

If anyone would be able to shed some light as to what may be going on here, that would be fantastic.

TL;DR Version:

The performance issues I am experiencing pertain the any VM running on this ESXi Server and the network performance when downloading data via HTTP(S).  Please review my testing below...

Detailed Version:

After a great deal of research I thought I had narrowed down the issue to Disk IO; however, after some extensive testing (running virtual machines are various datastore - local, nfs, and iSCSI), the performance impact between these three datastores turned out to be negligible.   I then investigated if this was a network performance issue; however, that too provided no answers and anything running outside the hypervisor is performing as expected.

What I know so far...

  • This is not an issue with the NAS nor VMware's connection to the NAS (either iSCSI or NFS)
    • See testing details
  • This is not an issue with the underlying network - physical devices in the same locations on the network are experiencing no issues.
  • I can confirm this is not an issue with the source of where I am trying to download from.  Issue is persistent no matter were where what files are downloaded.  Issue does not present on any physical devices on the network.

My ESXi Server:

  • ESXi 7.0
  • Dell EMC PowerEdge R630
  • Dual Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz
  • 256GB ECC RAM @ 2400Mhz
  • Intel(R) 2P X520/2P I350 rNDC
    • eth0 (10GbE SFP+) Connected to Primary Network - Servicing Multiple VLANs
    • eth1 (10GbE SFP+) Connected to Storage Network to Access NFS & iSCSI on NAS
    • eth2 (1GbE Copper) Connected to Management Network
    • eth3 (1GbE Copper) Disconnected
  • PERC H730 Mini SAS RAID Controller
    • One Virtual Disk (RAID1):
      • Two Seagate 300GB 10K RPM SAS Drives
  • iDRAC Enterprise - Dedicated NIC

Hosts / VMs Tested:

  1. Control
  2. Host-Based Storage (Storage on VMware Server Internal Disks Datastore)
  3. NFS-Based Storage (Storage on NFS Mounted Datastore)
  4. iSCSI Storage-Based (Storage on VMFS Formatted iSCSI Datastore)

Basics of my Testing:

  • All systems tested were running the latest Ubuntu LTS release.
    • Ubuntu was updated prior to testing.
    • VMware Tools was installed on on Virtual Machines.
  • All Virtual Machines were configured the same
    • 10 CPUs across 2 Sockets
    • 4 GB of RAM
    • VMX Network Adapters
    • 20GB Thick-Provisioned HDD (standard SCSI Controller)
  • VMware Switches and Virtual Interfaces are setup for Unlimited Bandwidth
  • CPU Shares for each of the VMs is set to Normal
  • Memory Shares for each of the VMs is set to Normal
  • There are not firewall rules or routing issues that would be impacting these virtual machines.  They are sitting along-side working physical devices.

Specific Testing Criteria and Methodology:

  • Installation Time:  Manual Timing (Stop Watch) of Installation time of Ubuntu on VM - ISO Stored on Internal Datastore
  • 100mb File Download:  100 Mb file download test (thinkbroadband.com/download)
  • Speed Test (Browser):  Using speakeasy.net/speedtest.  Chose Chicago Sever / DC for every test (geographically closest).
  • Speed Test (CLI):  Using speedtest-cli.  Allowed testing to determine best Server / DC - was consistent throughout all testing.
  • Disk IO Test:  Ran "dd if=/dev/zero of=test-disk-io.out bs=1G count=1 oflag=dsync"
  • iPerf Test (Local):  iPerf Connection Between Tested VM and another VM on the same network (VMX Adapters).
  • iPerf Test (Non-Virtual):  iPerf Connection Between Tested VM and another Non-VM on the same network (1Gbps Copper Network)
  • iPerf Test (NAS - Mgmt):  iPerf Connection Between Tested VM and NAS on Management Network (1Gbps Copper Network)
  • iPerf Test (NAS - Storage):  iPerf Connection Between Tested VM and NAS on Storage Network (10Gbps SFP+ Network)

Here are the results from my testing per VM - items in RED are problematic:

Host-Based Storage:

  • Installation Time:  4 Minutes, 06 Seconds
  • 100mb File Download:  30 Minutes
  • Speed Test (Browser):  4.2Mbps/19.6Mbps
  • Speed Test (CLI):  5.5Mbps/19.9Mbps
  • Disk IO Test:  661Mbps Write
  • iPerf Test (Local):  39Gbps Throughput
  • iPerf Test (Non-Virtual):  1.1Gbps Throughput
  • iPerf Test (NAS - Mgmt):  1.1Gbps Throughput
  • iPerf Test (NAS - Storage):  10.1Gbps Throughput

NFS-Based Storage:

  • Installation Time:  4 Minutes, 41 Seconds
  • 100mb File Download:  30 Minutes
  • Speed Test (Browser):  4.0Mbps/19.7Mbps
  • Speed Test (CLI):  3.6Mbps/18.3Mbps
  • Disk IO Test:  240Mbps Write
  • iPerf Test (Local):  39Gbps Throughput
  • iPerf Test (Non-Virtual):  1.1Gbps Throughput
  • iPerf Test (NAS - Mgmt):  1.1Gbps Throughput
  • iPerf Test (NAS - Storage):  10.1Gbps Throughput

iSCSI-Based Storage:

  • Installation Time:  3 Minutes, 55 Seconds
  • 100mb File Download:  30 Minutes
  • Speed Test (Browser):  3.9Mbps/18.6Mbps
  • Speed Test (CLI):  5.0Mbps/19.3Mbps
  • Disk IO Test:  584Mbps Write
  • iPerf Test (Local):  39Gbps Throughput
  • iPerf Test (Non-Virtual):  1.1Gbps Throughput
  • iPerf Test (NAS - Mgmt):  1.1Gbps Throughput
  • iPerf Test (NAS - Storage):  10.1Gbps Throughput

Control (Physical Ubuntu Device):

  • Installation Time:  n/a
  • 100mb File Download:  6 Minutes
  • Speed Test (Browser):  110Mbps/19.9Mbps
  • Speed Test (CLI):  145Mbps/20.1Mbps
  • Disk IO Test:  793Mbps Write
  • iPerf Test (Local):  n/a
  • iPerf Test (Non-Virtual):  1.1Gbps Throughput
  • iPerf Test (NAS - Mgmt):  1.1Gbps Throughput
  • iPerf Test (NAS - Storage):  10.1Gbps Throughput
Labels (4)
Tags (1)
0 Kudos
1 Reply
Exetus
Contributor
Contributor
‎03-19-2021 12:21 PM

So I blew everything away and reinstalled ESXi.  Everything is working brilliantly now.

I honestly have no idea what caused this in the first place; my configurations with the second installation are exactly the same as they were before... no changes at all.

 

I would be brilliantly impressed if anyone could explain this one.  As for me, this all comes down to a case of FM - "F**king Magic"  Though I have worked in Computers - specifically cyber security - for over a decade, I still find these kinds of situations hilarious.  Very close to "Did you try turning it off and back on again?" 😑🌴

0 Kudos