Hard disk performance.. really?

[enkoopa]

Hate to be honest here, but I'm having an extremely difficult time wrapping my head around the piss-poor hard disk performance of ESXi.

Here's a basic benchmark of my desktop sytem. Sure, nice fast new hardware, two SATA disks in RAID0. Nice to know my desktop is up to snuff. 120MB/sec average read.

Alright, let's take a look at a DL380G4 running Windows Server 2003. Sure, it's some older hardware. 5 drives in RAID5. At least enterprise hardware looks cool and has lots of blinking lights. 80MB/sec average read, no dropoff throughout the disk.

Okay, let's run this performance test inside a VM. ESXi, running on a DL380G4 as per above, with 5 drives in RAID5 again. Obviously we'll see some slight overhead from ESXi. And yes, I'm running the benchmark inside the VM - I want to see how the VM is going to perform! Okay here we go...

?! 20MB/sec?! Is this a joke? Re-run it. Nope. No joke. Wow.

Four times slower then a native OS on the exact same hardware. 5 drives, RAID5, 256MB of cache.

Am I missing something here or does nobody need to copy files anymore? Don't get me wrong, I have been loving ESXi. Snapshots save my behind in development, and let my experiment with zero concern for messing up. I'd take a 10-20% performance hit. But... this seems a little extreme? Is ESXi crippled versus the full blown ESX? Am I missing something here?

Thanks,

Greg

[enkoopa]

Good ol VMWare Communities.

Desktop

http://img175.imageshack.us/my.php?image=desktopmb2.png

DL380G4

http://img175.imageshack.us/my.php?image=dl380g4ko4.png

VM

http://img175.imageshack.us/my.php?image=vmwarenx1.png

[jasonlitka]

Which SCSI adapter are you using for your VM?

Jason Litka

[enkoopa]

I believe it is a Smart Controller 6i?

[mcowger]

No - for the VM itself? LSI Logic or Buslogic?

--Matt

[enkoopa]

LSI Logic.

[rkelleyrtp]

Just to throw my $0.02 in here...

I installed ESX-3i on a server in my lab and ran some disk benchmarks. The server has 2xQuad-Core CPUs, 16GB RAM, and 2x1TB Seagate HDDs using the onboard SATA controller. I installed both W2K3-R2 and WXP and ran the ATTO benchmark tool. To my dismay, the disk I/O was less than 5MB/sec. On the same server, I installed ESX 3.5 (full-blown) and ran the same disk benchmarks (same O/S installs with ATTO). This time, the disk I/O was +50MB/sec.

So, it appears (at least on my systems), ESX-3i is much slower at disk I/O than the full-blown ESX-3.5. Granted, these are un-scientific tests, but the certainly confirm the slow disk I/O I was experiencing on this server with ESX-3i.

[enkoopa]

I wonder if ESXi is crippled for disk performance...

[RParker]

> I wonder if ESXi is crippled for disk performance...

Why would they cripple it? That makes no sense.

Where do you READ anywhere, on ANY virtualization product that you will get native speed in a Virtual Machine? You won't find it.. why? Becasue VM's were NEVEr meant to be native speed competitors or speed demons. VM give you 1 thing, consolidation and use resources on a machine that are idle, that's it.

You want performance, you need to keep your physical host. ESX isn't faster or as fast as native access that's not written anywhere nor is it promised. It's fater than COMPARABLE Virtualization products.. but not fast, at all.

If you gives you the ability to run many machines on 1 box, that's pretty much the basics of Virtual machines. You should know that Virtual ALSO means virtual hardware. You can't get direct access to the controller, the disks or the memory or CPU, so it's going through a virtual translation layer.. No native access speeds are possible.

Then there is the "Is your machine on the Hardware compatibility list"? If it's not.. there is a reason why VM Ware PRE-Certifies machines for their product... that's why you should check the HCL before you attempt an install.. you should read the documentation and understand what you are getting into..

[rkelleyrtp]

Where do you READ anywhere, on ANY virtualization product that you will get native speed in a Virtual Machine? You won't find it.. why? Becasue VM's were NEVEr meant to be native speed competitors or speed demons. VM give you 1 thing, consolidation and use resources on a machine that are idle, that's it.

uhhh - I have read many places how VMWare and Xen achieve near-native speeds with their VM products. To make such a claim as you did is very odd. I agree the reason behind virtualization is consolidation. But, if you have a Quad-core CPU with 8GB of RAM, you should be able to run 4 instances of single-core/2GB VMs at near native speed.

.... It's fater than COMPARABLE Virtualization products.. but not fast, at all.

I find this statement a little odd as well. ESX 3.5 seems to run my VMs fast - almost as fast as if they were running natively on the same hardware.

At any rate, the point to this entire thread was to see if anyone else was having similar performance issues between ESX3i and ESX 3.5. In my case, I am seeing the same performance issues as the original poster. I am getting 1/10th the disk I/O speed from ESXi as opposed to ESX 3.5. This is just a little more than "somewhat slower". I am just curious as to why there is such a large difference between the products.

[Dave_Mishchenko]

ESX can perform at a very high level. VMware published a study to see if they could push a single host over 100,000 IOPS.

What sort of controller are you using? It would be worthwhile to log a support request with VMware regarding this (especially since you can reproduce it).

[enkoopa]

a) My servers are on the HCL. HP DL 380 and 580's. G3/G4.

b) I don't expect 100% performance of a native install. But I sure don't expect it to be 1/4 the speed. Maybe years ago we would accept this.

c) Why would ESXi be crippled? I don't know. To get you to cough up money for the fullblown product? To compete with other vendors products, yet still make ESX desirable?

I'm not sure any datacenter manager would say "Hey, man, VMWare is going to simplify life a lot.. other then essentially downgrading our hardware to the same performance as 5 years ago, it sure will be easy to manage!"

[jasonlitka]

There'd be no point to crippling ESXi as paying the $6,000 per machine allows you to use ESXi or ESX; what you're really paying for is the VC agent license. Furthermore, now that ESXi is free you're going to see more and more people who want to invest in VC but not actually reinstall their production systems when they upgrade.

As to disk performance in a VM, I just tested on a CentOS 5 VM under ESXi 3.5.0 build 110180 on my test server (which only has a single 146GB 10K RPM SAS disk for VM storage) and I got write speeds of about 56MB/s and read speeds of about 75MB/s with a block size of 64k (while three other VMs were running). It sounds to me like the problem is either with Windows or with your disk controller.

root@server ~# dd if=/dev/zero of=/root/speedtest bs=64k count=10k

10240+0 records in

10240+0 records out

671088640 bytes (671 MB) copied, 11.8888 seconds, 56.4 MB/s

root@server ~# dd if=/root/speedtest of=/dev/null bs=64k count=10k

10240+0 records in

10240+0 records out

671088640 bytes (671 MB) copied, 9.00004 seconds, 74.6 MB/s

Jason Litka

[benpott]

I just checked perf test (ATTO) on a test 2008 VM I have running on test ESX3i U2 (with the time bomb fix installed) and got reads in the 140MB/sec range and writes in the 100-115MB/sec range. This is a Dell 2950 quadcore running on 4 73GB 15k SAS drives in a RAID 1+0 set.

I didn't run this test on our 3.5 box because its running alot more spindles so it wouln't be much of a comparision.

EDIT: litkaj mentioned his ESX3i box was running other VMs. This machine is running 2 windows 2003 VMs, 1 XP VM, and a single 2008 VM. No decline in disk performance noticed.

[enkoopa]

I'll run ATTO before I leave and report the results in the morning.

[rkelleyrtp]

I just checked perf test (ATTO) on a test 2008 VM I have running on test ESX3i U2 (with the time bomb fix installed) and got reads in the 140mb/sec range and writes in the 100-115mb/sec range. This is a Dell 2950 quadcore running on 4 73GB 15k SAS drives in a RAID 1+0 set.

Unless you mistyped "mb" for "MB", your VM is only getting 15MB/sec thruput for reads and 10MB/sec writes. That still seems very low given your hardware platform.

[benpott]

MB not mb. Edited to correct. ATTO defaults to MB btw.

[alex555550]

Hy,

have you updated the BIOS, BMC, Raid Controller? Has the Raid Controller the write and read cache turned on?

[enkoopa]

alex, good suggestion. just last night before I left I ran the Proliant Firmware update CD.

Ran ATTO on both machines. They are getting comparable speeds now thank god. The native OS gets peak-ier speeds around the cluster size of the RAID array. The VM is less affected - it ramps up as it approaches the 128kb mark, then stays more or less consistant. ~60MB writes, ~80MB reads once you do 128kb blocks or higher.

[enkoopa]

It would appear I've spoken a bit too soon. Just realized it wasn't fair to update all the firmware on the VM host I've finished updating the server with the native install as well.

Seeing consistant ~85MB writes across almost all block sizes. Seeing ~100-130MB reads across the top 75% block sizes.

I'd still consider the VM to at least be functional now with it's 60MB writes!