v-machines run very fast, but networking so slow. Advice Appreciated...

[sledgezfx]

Hi to all

I have recently installed VMware ESXi 3.5 U3, 123629

Before I get into details about the problem let me give you the specs you will undoubtedly want to know

Motherboard: Supermicro X7DB8

Processors: 2x LGA771 intel dual core XEON 3.2 GHz

Memory: 4GB

Storage: Adaptec Ultra320 SCSI RAID Card 2020ZCR with 6 32Gb Drives in a RAID 5 Configuration

Networking: 4 x 1000Mb FULL DUPLEX Network connections (one 80003ES2LAN Ethernet Controller built on motherboard AND one 8254NXX Add in card installed)

Everything about the installation was easy and fast. I currently have 3 virtual machines installed.

the VMs boot up very quickly. just about everything about the performance of ESXi is outstanding. In the console windows pop-up the moment you click them in the start menu as if they are physical boxes.

BUT

the one caveat is the networking performance....

I've been searching these forums looking for answers but haven't found anything that solves the problem.

Here's What I've tried, based on tips in this forum....

-

---

I've enabled HyperThreading in the VI Client and enabled HyperThreading in the BIOS

I've enabled HyperThreading in the VI Client and disabled HyperThreading in the BIOS

I've disabled HyperThreading in the VI Client and enabled HyperThreading in the BIOS

I've disabled HyperThreading in the VI Client and disabled HyperThreading in the BIOS

and I've gone through VMwareWolf's guide as well...

when copying to/from a simple windows share I can only get about 5Mb a sec.....which is terrible if you consider my networking hardware...

I get slightly better performance(about 6Mb/sec) if I enable HT in the BIOS and disable it in VIClient(this is what its currently set to).

During a copy operation I have watched my hardware usage and the cpu peaks at around 10%, so I don't think its processor related. Machines boot and load programs instantaneously so I doubt its hard drive performance either.

Before moving to ESXi I was running VMware server 2.0 on windows server 2003 R2 and network performance was very fast.....so I'm certain this is a driver/configuration issue

Any help/Suggestions is very appreciated

Thanks

Sledgezfx

[nick_couchman]

A couple of points of clarification:

- When you're copying to/from the VMs, what's on the other end of that connection? Another VM? Your laptop/desktop machine?

- You say networking performance was fast under Server 2.0 - are you doing those speed tests exactly the same way? If no, what's changed?

- Do you have VMware Tools installed?

- What network card type are you using for your VMs?

[sledgezfx]

- I've done both scenarios VM to VM and Vm to my workstation...speed is the same regardless

- yes the setup is the same EXCEPT for vmware ESXi

- VMware tools is installed

- according to the VI Client the nics are configured as follows

- e1000 driver

- they show connected

- shown as being 1000 full duplex

- according to one of the V-Machines windows shows that I have a VMware Accelerated AMD PCNet Adapter installed

I greatly appreciate your quick response, If there is any more information you need please don't hesitate to ask

Thanks

Sledgezfx

[TravisT]

Are the VM's and your workstation in different subnets or VLAN's? I have recently run in to this issue. If I did a tracert to the VM from my workstation I was getting a lot of latency (5-10ms lag times). Also, I noticed that my default gateway was one extra hop to where it needed to go. So, I created a static route to the VLAN that the VM's were on pointing to the VLAN interface on my end as the gateway. Re-ran the tracert and the latency dropped to <1ms. I was doing file transfers of about 500 KB/s before the change and after the change I was getting about 50 MB/s. I added the static route to our DHCP server and now everyone is noticing huge performance increases.

Hope that helps,

Travis Trout

[sledgezfx]

Thank you Travis

I was excited reading your post. What you said made sense and seemed like it could be the problem that's been eluding me.

unfortunately, my network is small and everything is on the same subnet. I currently already have <1 ms ping times and still slow file transfers. I looked around in the routing table to hopefully find something out of place but the network is so small and simple there's not much that could go wrong.

thanks

sledgezfx

[TravisT]

Well, one thing that is very interesting is that you have slow performance from VM to VM. When network traffic goes from VM to another VM it never actually leaves the vSwitch assuming the VM's are on the same host. If they are on different hosts then the traffic does go through your physical network infrastructure. There is a driver you can install, vmxnet enhanced that will bump network performance. However, you have to set the OS type to either Enterprise or Datacenter for the option to be available for network type in the nic properties.

I just find it very odd though that your're not even getting fast transfers from VM to VM. Do you have the VM network on it's on vSwitch, seperate from the VKernel/Management vSwitch or is it all on one vswitch? I would keep them seperate.

2 nics (load balanced) = VKernel/Management

2 nics (load balanced) = VM network

-- Travis

[TravisT]

Also, just out of curiousity, have you tried lately doing a transfer test from a physical pc to another physical pc? You might try also from workstation to server and server to workstation, all physical tests. If you get the same results, you may be having some issues with your network infrastructure. Could you also describe exactly how your network is layed out?

-- Travis

[spex]

Did you check the following article.

http://www.tuxyturvy.com/blog/index.php?/archives/37-Troubleshooting-VMware-ESX-network-performance....

We also had performance problems with shared interrupts.

Whats your OS and did you try another nic within your esx server?

Regards

Spex

[sledgezfx]

I did just now and got 40mb per second...

We have one 24 port cisco catalyst 3750 gigabit switch running on our network my workstation and the physical box running vmware esxi are connected to it....its a very basic network

[sledgezfx]

thanks spex

I read the article, but im not sure how i check this.....

the article is for ESX and im running ESXi how would you check interrupts on ESXi

Thanks for your help

sledgezfx

[TravisT]

where's the guest files located?

[sledgezfx]

on the SCSI drives, in the server.....

a full list of my ESXi server hardware is at the beginning of this post

Thanks

sledgezfx

[TravisT]

Do your NIC's show that they are running at 1000 Full in the Network Configuration for the hosts? Also, check to see if you have NIC Team setup to load balance. Also, how many vSwitches do you have setup?

[sledgezfx]

I'm attaching some images so you can have a better look at my configuration

[RenaudL]

What about the virtual nics of your VMs?

Networking performance is drastically affected by the type of interface you choose: PCNet (without the VMware Tools) yields poor throughput while "Enhanced Vmxnet" is our fastest / most efficient network adapter.

[sledgezfx]

Thank you RenaudL for your response

I have posted another configuration screenshot, this one is of the VM itself. A picture is worth a thousand words, so this will save time and effort

[RenaudL]

Weird... You should probably check for packet drops or unusual latency... I know SMB is very latency-sensitive.

[sledgezfx]

Yeah....this is very confusing

here is another screenshot of my workstation pinging one of the VMs

any more ideas?

I'm starting to worry that I'm going to have to try Hyper-V *cringe*

[MatiasG · Accepted solution]

Hi, I'm running into a similar problem regarding slow speeds but I've seen it happen when I copy things locally, what caught my attention is that you get slow speeds when copying from one VM to another VM on the same host, as someone pointed that shouldn't leave the vSwitch. Pls try to copy the same file from one folder to another on the same drive and tell me what speed are you getting...

Thanks!

[spex]

Sorry my answer was to fast. As I know ESXi handles USB different as ESX with Console - so there should be no interrupt sharing problem.

You could check that using the unsopported access to the console and view /proc/vmware/interrupts

Despite that I alway disables devices in the bios that are not needed on the esx host (usb, parallel, serial,..)

I would propose the use netio for further benchmarking, since you do not have to work with disk io and you do not have to use a upper layer protocoll like smb.

http://www.ars.de/ars/ars.nsf/docs/netio

Its a commandline tool, that easy to use.

Regards

Spex