Disk I/O causes large cpu load

[steven6282]

I'm trying to figure out why any time there is disk I/O going on in a guest OS, it's causing a large cpu load in ESXi.  Shouldn't the hardware raid card be handling most of the reading / writing to disk?  I could understand some cpu usage but it's kicking up to over 3000 Mhz cpu usage for a simple 8 KB/s disk I/O operation in the guest OS.  Seems a bit much.

I've tried two different hardware raid cards to make sure it wasn't the hardware raid.  I've tried a Dell Perc 5/i and a LSI 3Ware 9650SE-4LPML.

Both resulted in the same behavior.

[mcowger]

Sounds like there is something other than the IO causing the CPU load - where is the CPU load coming from?  Which VM?

[steven6282]

Of the 4 VMs I have loaded right now for testing, I have seen the occurence happen durion disk I/O in Ubuntu Server 10.10 64bit, Windows Server 2008 64bit, and Windows 7 64bit.

I'm fairly certian it is caused by disk i/o.  I've watched resource monitors in Windows 7 / Windows 2008 for instance and CPU usage remains 1% or less.  Disk I/O is the only thing that ramps up a little.  In Ubuntu I duplicated it by simply doing dd if=/dev/zero of=1G bs=1M count=1024

[mcowger]

What does esxtop say during this?  Which world is being charges for the IO?  For the CPU?

[steven6282]

Did not know about the esxtop command, useful.

Windows 7 seems to be the biggest cultprit for this.  When it happens it shows a 100 to 120% cpu load on the Windows 7 vm.  But in the Windows 7  guest the peformance manager shows 1% to 2% cpu usage (I've got it set to two cores in this setup).  The 100 to 120% accounts for the 3000 Mhz + cpu load, but I don't understand why esxtop would show that much usage and when Windows 7 performance manager doesn't think it's using it.  Unless ESXi is running a background thread on the VM process but outside of the guest OS.