Disk I/O load on all VM's are hitting 100%

[mail_vijayar]

Hi All,

Need a urgent help here, we recently moved one of our customer virtual on ESXi 3.5 (free version) on Sun X2200 local storage (RAID 1 - SAS disks).

We installed couple of VM on it (8 VM's) but all the VM have very slow perfomance, Exchange server running on it as VM is always struck with 100% disk I/O (checking form perfomance monitor), the ESX server is 2 Quad core with 32 GB of RAM, we have allocated 3 GB RAM for all the VM and set CPu affinity and high shares for the EXCHANGE VM but the perfomance in still dead slow on all the VM's (all are Windows server).

Is there anything i can check to improve the perfomance, i tried increasing the heap size on teh ESX server to 128 with no luck.

Regards,

Vijay

[drewdown]

We are seeing the simliar behavior with a bladecenter S, internal storage 300GB 15K RPM SAS disks running RAID 5 and roughly 16 VM's.

Not 100% I/O but enought to cause performance issues and complaints. The box its running on is a LS41 blade with 32GB of RAM and 4 Quad core proc's.

[mail_vijayar]

Are you running this on ESXi 3.5 UP4?

Regards,

Vijay

[Dave_Mishchenko]

How many physical disks do you have in the array? If you can console access you can run esxtop (or resxtop in the RCLI) to get a disk statistics. Start (r)esxtop and press D to get disk stats. Do you have a battery backed write cache on the controller. That can make a huge difference in performance.

[Lightbulb]

Exchange is an I/O hog. Was this ESXi system funtioning at one time but now is not? If so what changed?

If you have justmigrated all the systems onto the host it may just be a matter of the needs of the VMs excedding the capacity of the storage system on the host.

[mail_vijayar]

This is a Raid 1 array of 2 1TB disks, belwo is the uotput from ESX top:

7:42:53pm up 8 days 12:05, 186 worlds; CPU load average: 0.18, 0.17, 0.16

ADAPTR CID TID LID WID NCHNS NTGTS NLUNS NVMS AQLEN LQLEN WQLEN ACTV QUED %USD LOAD CMDS/s READS/s WRITES/s MBREAD/s M

vmhba0 - - - - 1 1 1 2 128 0 0 - - - - 0.00 0.00 0.00 0.00

vmhba2 - - - - 1 1 1 70 127 0 0 - - - - 8.32 1.78 6.54 0.01

How do we read these results?

Regards,

Vijay

[Dave_Mishchenko]

Once you start esxtop, press d to switch to disk stats. 2 SATA (I presume) would seem be insufficient for your load. What was the I/O load on the old server and what sort of disk setup did it have? Does the controller have a battery backed write cache?

[mail_vijayar]

These are SAS disks and have battery backed write cache as well, the earlier physical server was connected to a raidweb (RAID 5).

the output of esxtop is below, please let me know if we need to check anything here:

8:03:17pm up 8 days 12:25, 178 worlds; CPU load average: 0.11, 0.13, 0.11

ADAPTR CID TID LID WID NCHNS NTGTS NLUNS NVMS AQLEN LQLEN WQLEN ACTV QUED %USD LOAD CMDS/s READS/s WRITES/s MBREAD/s M

vmhba0 - - - - 1 1 1 2 128 0 0 - - - - 0.00 0.00 0.00 0.00

vmhba2 - - - - 1 1 1 70 127 0 0 - - - - 9.51 0.00 9.51 0.00

Regards,

Vijay

[Jackobli]

These are SAS disks and have battery backed write cache as well

You wrote about 2 x 1 TB... so they may have SAS interfaces, but there are no bigger real SAS (10/15k RPM) disks than 400 GB.

For loaded exchange, I would suggest more than RAID1.

[Dave_Mishchenko]

Keep an eye on qued, but as mentioned you'll likely need more drives to meet the IO load.

[mail_vijayar]

Hi Dave,

Adding more drive is ruled out for now, one more correction was that its a SATA disk and not SAS.

[NWhiley]

A 2 disk SATA array is not going to have the IOPS capabilities to deal with exchange, let alone 7 other servers.

Are you sure this is right?

I'm guessing with 2 disks that they are in RAID1 ?

[NWhiley]

This thread has a good discussion on IOPS

http://communities.vmware.com//message/1232830#1232830

and MattG posted this brilliant link to an IOPS calculator.

http://www.google.com/url?sa=t&source=web&ct=res&cd=7&url=http%3A%2F%2Fwww.datacore.com%2Fforum%2Fat...