VMware Cloud Community
heikki_m
Contributor
Contributor

ESXi 3.5 management network very slow

Hello,

I'm having problems with ESXi (3.5 U2 latest, both embedded and installable) on three different hosts. Hardware is HP DL380 G5. Both NICs on every server are connected to 1000FDX ports without any duplex issues. ESXi network configuration is the default: both vmnic0 and vmnic1 are used for VM Network and Management Network. Switches show no errors on the ports.

VM Network is not showing any performance problems. I'm getting steady 30-40MB/s to and from guest machines.

Accessing the management network (copying to datastore, converter access, downloading VI client etc.) is painfully slow. Ranging between 100kB/s to 3MB/s - usually around 1MB/s.. needless to say that this is very frustrating when for example converting existing virtual machines to the ESXi hosts.

Any idea where to start looking for a solution?

Tags (3)
Reply
0 Kudos
107 Replies
admin
Immortal
Immortal

Reply
0 Kudos
admin
Immortal
Immortal

Reply
0 Kudos
KBuchanan
Enthusiast
Enthusiast

Well - I guess he "left us hanging" with that post!

Kevin

Reply
0 Kudos
englund
Contributor
Contributor

I've done some tests on this issue too.

My setup is simple:

  • One diskless ESXi 4 (but 3.5 U4 gives the same results too)

  • Storage on a iSCSI target (using software initiator)

  • Storage on a NFS server

  • No traffic shaping enabled

  • All gigabit network

I have a VM with one disk on each storage.

The VM can transfer data from one disk to the other with >70MB/s, which is expected.

BUT, a file copy from one datastore to the other via datastore browser (or cp/dd (bs=4096) from the console) is always capped very neatly at 10-12MB/s.

I'm really going crazy on this!!

Reply
0 Kudos
KBuchanan
Enthusiast
Enthusiast

Agreed. The is strong evidence that supports the theory that the mgt I/F has been capped.

I believe the mgt I/F should always be available or reserved for mgt functions - however, VMWare should allow us to create another interface path that can be 'saturated' for file copy/backup operations.

I can't begin to understand why there is such an bend in their thought process, unless this is to help support the promotion of other products that can function 'better'. Except for SAN based file operations, I'm not aware of any product that increases the effective throughput on the mgt interface.

Kevin Buchanan

Chief Information Officer

Lexington Memorial Hospital

336-238-4286

kbuchanan@lexmem.org

Reply
0 Kudos
dragin33
Contributor
Contributor

In my experience what I originally thought was a cap turned out to be a hardware limitation of our aging hardware. I was able to increase the speed by enabling write-back caching on the storage.

Reply
0 Kudos
englund
Contributor
Contributor

In my experience what I originally thought was a cap turned out to be a hardware limitation of our aging hardware. I was able to increase the speed by enabling write-back caching on the storage.

And this made you achieve considerably higher rate than 10MB/s?

Reply
0 Kudos
englund
Contributor
Contributor

In my experience what I originally thought was a cap turned out to be a hardware limitation of our aging hardware. I was able to increase the speed by enabling write-back caching on the storage.

And this made you achieve considerably higher rate than 10MB/s?

Reply
0 Kudos
englund
Contributor
Contributor

Agreed. The is strong evidence that supports the theory that the mgt I/F has been capped.

I believe the mgt I/F should always be available or reserved for mgt functions - however, VMWare should allow us to create another interface path that can be 'saturated' for file copy/backup operations.

I can't begin to understand why there is such an bend in their thought process, unless this is to help support the promotion of other products that can function 'better'. Except for SAN based file operations, I'm not aware of any product that increases the effective throughput on the mgt interface.

If VMWare would just join in and confirm if it is capped or not I would accept the answer either way. It's the "not knowing" that keeps me from sleeping at night.

I was fiddling around with the network settings in ESXi 4 a couple of days ago and I'm pretty sure that I once was able to copy a file at about 60MB/s, but havn't been able to repeat it so I'm starting to doubt my memory. Maybe I was just wishing very hard. Now my system has gone productive so I can't go berserk in the configs anymore Smiley Sad

Reply
0 Kudos
KBuchanan
Enthusiast
Enthusiast

Write back cache did NOT make any significant improvement with us. I even used a HP P800 with 2Mb cache...no difference.

Kevin Buchanan

Chief Information Officer

Lexington Memorial Hospital

336-238-4286

kbuchanan@lexmem.org

Reply
0 Kudos
KBuchanan
Enthusiast
Enthusiast

Did they change the console to be read only in 4.0?? In 3.5 u3, they have been consistently saying it was a mistake/oversight and they was going to change it back to a read only console unless you purchased the license.

Kevin Buchanan

Chief Information Officer

Lexington Memorial Hospital

336-238-4286

kbuchanan@lexmem.org

Reply
0 Kudos
englund
Contributor
Contributor

Did they change the console to be read only in 4.0?? In 3.5 u3, they have been consistently saying it was a mistake/oversight and they was going to change it back to a read only console unless you purchased the license.

Hmz, read only how? I just enabled ssh and I'm allowed to do anything once I'm logged in. Anything except doing quick copies that is Smiley Happy

Reply
0 Kudos
dragin33
Contributor
Contributor

It did indeed increase the speed by more than double.

Reply
0 Kudos
KBuchanan
Enthusiast
Enthusiast

Maybe they changed their mind...they put it in the release notes.

Kevin Buchanan

Chief Information Officer

Lexington Memorial Hospital

336-238-4286

kbuchanan@lexmem.org

Reply
0 Kudos
DSTAVERT
Immortal
Immortal

I don't know what controller you are using but make sure you have a functional battery backed cache or you risk data loss or corruption.

-- David -- VMware Communities Moderator
Reply
0 Kudos
englund
Contributor
Contributor

I was fiddling around with the network settings in ESXi 4 a couple of days ago and I'm pretty sure that I once was able to copy a file at about 60MB/s, but havn't been able to repeat it so I'm starting to doubt my memory. Maybe I was just wishing very hard. Now my system has gone productive so I can't go berserk in the configs anymore Smiley Sad

I managed to repeat it!

I can get >50MB/s in a copy operation, but unfortunately that doesn't make me understand more of the issue.

It depends on what file I copy. I have one vmdk that looks like this on the disk:

-rw------- 1 nobody nogroup 757596160 2009-06-18 16:05 SuseEnt-s001.vmdk

-rw------- 1 nobody nogroup 391184384 2009-06-18 16:05 SuseEnt-s002.vmdk

-rw------- 1 nobody nogroup 290127872 2009-06-18 16:05 SuseEnt-s003.vmdk

-rw------- 1 nobody nogroup 443678720 2009-06-18 16:05 SuseEnt-s004.vmdk

-rw------- 1 nobody nogroup 335413248 2009-06-18 16:05 SuseEnt-s005.vmdk

-rw------- 1 nobody nogroup 161480704 2009-06-18 16:05 SuseEnt-s006.vmdk

-rw------- 1 nobody nogroup 334692352 2009-06-18 16:05 SuseEnt-s007.vmdk

-rw------- 1 nobody nogroup 82640896 2009-06-18 16:05 SuseEnt-s008.vmdk

-rw------- 1 nobody nogroup 871 2009-06-18 16:05 SuseEnt.vmdk

In datastore browser it shows as just SuseEnt.vmdk 2.7GB.

If I copy this file I can see that data fly over the network at >50MB/s. If I copy any other file not made up of parts I get only ~10MB/s.

This apply only in datastore browser. If I copy from console with cp I still get 10MB/s. Is datastore browser opening up multiple simultaneous streams for the separate parts?

What's up with this?!?

Reply
0 Kudos
KBuchanan
Enthusiast
Enthusiast

So, are you only moving 2.7G at 50MB/sec? What command are you using to copy the files?

Kevin Buchanan

Chief Information Officer

Lexington Memorial Hospital

336-238-4286

kbuchanan@lexmem.org

Reply
0 Kudos
DSTAVERT
Immortal
Immortal

It isn't the unsupported or ssh that is read only it is the RCL that is read only.

-- David -- VMware Communities Moderator
Reply
0 Kudos
KBuchanan
Enthusiast
Enthusiast

Thanks for the clarification.

Kevin Buchanan

Chief Information Officer

Lexington Memorial Hospital

336-238-4286

kbuchanan@lexmem.org

Reply
0 Kudos
boehmenkircher
Contributor
Contributor

Sehr geehrter Absender,

ich bin bis 03.07.2009 nicht im Haus.

In dringenden Fällen wenden Sie sich bitte an

Herrn Münkle (ulrich.muenkle@albwerk.de Tel.: 07331/209-504) oder

Herrn Pfisterer (norbert.pfisterer@albwerk.de Tel.: 07331/209-508).

Ab 06.07.2009 stehe ich Ihnen gerne wieder zur Verfügung.

Reply
0 Kudos