VMware Cloud Community
heikki_m
Contributor
Contributor

ESXi 3.5 management network very slow

Hello,

I'm having problems with ESXi (3.5 U2 latest, both embedded and installable) on three different hosts. Hardware is HP DL380 G5. Both NICs on every server are connected to 1000FDX ports without any duplex issues. ESXi network configuration is the default: both vmnic0 and vmnic1 are used for VM Network and Management Network. Switches show no errors on the ports.

VM Network is not showing any performance problems. I'm getting steady 30-40MB/s to and from guest machines.

Accessing the management network (copying to datastore, converter access, downloading VI client etc.) is painfully slow. Ranging between 100kB/s to 3MB/s - usually around 1MB/s.. needless to say that this is very frustrating when for example converting existing virtual machines to the ESXi hosts.

Any idea where to start looking for a solution?

Tags (3)
Reply
0 Kudos
107 Replies
englund
Contributor
Contributor

So, are you only moving 2.7G at 50MB/sec? What command are you using to copy the files?

Yes, 2.7GB at 50MB/sec, with datastore browser. And after some console tests I can also get higher throughput if I start multiple simultaneous copies, like:

/vmfs/volumes # cp iSCSI1_DRBD1/SuseEnt-s001.vmdk iSCSI1_raid0/ &

/vmfs/volumes # cp iSCSI1_DRBD1/SuseEnt-s002.vmdk iSCSI1_raid0/ &

/vmfs/volumes # cp iSCSI1_DRBD1/SuseEnt-s003.vmdk iSCSI1_raid0/ &

So there definiteley seems to be a cap and it seems to be per process or datastream...sounds reasonable, right?

So now we only need to make a script that use multiple dd instances to copy separate parts of a large file, shouldn't be too hard Smiley Happy

Reply
0 Kudos
DSTAVERT
Immortal
Immortal

The whole idea behind ESXi was that it come pre-installed on a diskless server via an internal USB stick. NO local datastore. The datastore was to be some form of consolidated external storage, iSCSI, FC or NFS. Moving, copying, backing up etc would be performed by the facilities on those external storage devices. You really can't take advantage of all the really cool features of ESXi unless you do use external storage.

-- David -- VMware Communities Moderator
Reply
0 Kudos
DSTAVERT
Immortal
Immortal

englund wrote:

So now we only need to make a script that use multiple dd instances to copy separate parts of a large file, shouldn't be too hard !

Have a look at lamw's script. http://communities.vmware.com/docs/DOC-8760

-- David -- VMware Communities Moderator
Reply
0 Kudos
englund
Contributor
Contributor

Have a look at lamw's script. http://communities.vmware.com/docs/DOC-8760

Oh...wow. He should get a medal. I get constant full throttle when making backups with his script, >100MB/s. Thanks for the tip!

Reply
0 Kudos
koit
Contributor
Contributor

I have wasted several days on this issue a while back.

I discovered two things when I tested moving files between two ESXi servers:

1. You MUST have ha battery backup up RAID controller (BBU)

2. Installing a Virtual Center (eval) and using datastore (GUI) to copy files is many times faster than using it locally on the ESXi servers

Regards

Bernt

Reply
0 Kudos
Schorschi
Expert
Expert

COS resources are limited and controlled by design. Although your COS NIC throughput seems slower than expected. Have you checked to see how the BIOS has stacked the shared interrupts? A nasty conflict for dueling interrupt vectors can do really unkewl things to your performance.

Reply
0 Kudos
BerndStoltefuss
Contributor
Contributor

Hi,

i faced the same slow speed when copying from esxi to esxi. I was unable to get anything better then 10MB/s using "wget -m"(client, esxi server1) and proftpd(server, esx server2) on a 1GB connection.

I am now getting 42MB/s after doing some changes on the vswitch (i have a dedicated vswitch for the server-server connection).

Open vswitch properties and change:

  1. Number of ports: 8

  2. Activate Traffic shaping on both entries (vswitch and vmkernel):

* set the limits to some very high number (i just added 2 zeros, so it's 10240000 now)

Bernd

Reply
0 Kudos
congocongo
Contributor
Contributor

Hello,

I find your results of great interest. I also have a DL360 G5 with the same very low performance as you had, actually i have two with the very same configuration.

Anyways; can you instruct me how to find out if i have the cache at 256 or 512, and if i have this magical battery added? -i cant seem to find these details in the ProLiant BIOS.

Please help Smiley Happy

BR. Congo, denmark

Reply
0 Kudos