I'm seeing some really poor network performance on a specific VM and I'm wondering how to best troubleshoot this.
- Is it the NAS?
- Is it the Host?
- Is it the physical network?
- Is it the virtual network?
- Is it the VM itself?
- Is it the destination VM?
- Something else?
While transferring some data to an AWS EC2 Instance, I noticed it was taking an incredible amount of time to transfer a small amount of data. Even after several minutes the box had only managed to transfer 400MB which to me (on a 25/25 connection) wasn't normal. I decided to run through some very simple tests on the VM in question then on the destination EC2 instance. (Just so we're clear: the destination machine is not hosted here, its an Amazon EC2 VM. So all this data is going out over the Internet to the destination machine.)
I found a 30MB file on Softlayer's site that I've been using to build a very simple baseline:
wget --output-document=/dev/null http://speedtest.sea01.softlayer.com/speedtest/speedtest/random4000x4000.jpg
I ran through these tests from (yes I have wget for Windows)
- a physical desktop (Windows 7): 3M/s
- the NAS (via ssh): 3M/s
- the CentOS VM in question: 600-700K/s
- a Windows 7 VM on the same host as the CentOS: 3M/s
- the destination EC2 instance: 3M/s
Note: I realize this only tests downloads & not uploads. I expected to see maybe high download speeds & a poor upoload speed but when the download itself was low, I figured there may be some sort of misconfiguration somewhere. I ran the GUI upload test (http://speedtest.sea01.softlayer.com/speedtest/) on the physical and virtual Windows 7 systems, and the results were roughly 18Mbps up and 25Mbps down.
With 110GB to transfer, I'm a little worried it'll take ages at this rate! Help!
I don't know what information might be useful at this point so I'll give you what I have.
Local VM Info
- CentOS release 5.8 (Final)
- Linux version 2.6.18-308.13.1.el5 (mockbuild@builder10.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-52)) #1 SMP Tue Aug 21 17:10:18 EDT 2012
- 1 vCPU / 2GB RAM
Remote VM Info
- Linux version 2.6.18-xenU-ec2-v1.2 (root@domU-12-31-39-06-0D-E1) (gcc version 4.1.2 20070626 (Red Hat 4.1.2-13)) #2 SMP Wed Aug 19 12:57:15 UTC 2009
- Similar CPU & Memory configuation
Host Info
- Dell PowerEdge T710 Dual Quad Xeon E5504 12GB
- 8x500GB SATA 7200RPM 32MB Cache 2xRAID5
- Storage via iSCSI LUNs
- ESXi 4.1.0 800380
Storage Info
- Synology DS411+ii
- 4x3TB SATA 7200RPM 64MB Cache RAID5
Local Network Connectivity
- FiOS 25/25
- Gig connectivity all around
- No firewalls between host and NAS
- No complex network configuration on switches/routers (other than VLAN tagging)
- No complex VMWare network configuration (other than VLAN tagging)
