VMware Cloud Community
hoevring
Contributor
Contributor

Problem with Disk(ms) ??

Hello,

We have bought ourselves a brand new HP Proliant DL 380 G5

Specs:

2x 3.0 Intel Xeon dual processor
14 GB of memory
6x 146 GB 10K SAS disks (Raid 5 without hotspare) with Controller: P400i

VMware ESXi 4.0

----------------------------------------------

We had an earlier VMware ESXi 3.5 as we migrede from over on our new HP server.

Initially, we ran 4.1 on HP server and we experienced no problems with performance on the server. Our problems began after 1 week with all 10 virtual servers running on the hp machine.

We are currently using about 10 GB momory active and 50% of Processor.

We saw that the system started running slowly, such as installing an XP machine
drove unusually slowly, it could take up to 1.5 hours before the installation was done.

Response time (MS) on the disc is over 70 MS sometimes more than 100 MS, we have tried to downgrade from 4.1 to 4.0 as we found out that our raid controller (P400i) was not supported in 4.1.

The changing VMware version, has nothing changes in disk(ms).

We have an IBM X346 running parallel beside HP Proliant DL 380 G5

Once we've moved all virtual servers onto IBM X346, we have under disk(10 ms), why we are very confused.. Because the IBM X346 is a old lady.

-------------------------------------------------- ------------

We've tried to activate writecache, but this doesnt help anything..

We also have tryed Smart Start, reinstalled the server, updated all firmwares ..

We have now reached that it can only be the controller, but WE just want to be sure it's the controller before we go out and buy another, and in case it is, which one can give us the performance we want?

At the last:

We have some peaks about every 3 hours in DISK(KBps) ~ 5-7.000 but the the normaly is under 2.000 KBps.

We hope maybe you can help us...

Best regards

Soeren

Reply
0 Kudos
15 Replies
kjb007
Immortal
Immortal

Are you running the same type of disk in your IBM system?  Same number of disks?  Latency that high is typically your disk system can't keep up with the I/O.

-KjB

vExpert/VCP/VCAP vmwise.com / @vmwise -KjB
Reply
0 Kudos
hoevring
Contributor
Contributor

Hi KjB, thanks for your reply!

No! on my IBM X346 iam running 6x SCSI 146 GB 10K

Disk system, you mean my controller?

//Soeren

Reply
0 Kudos
kjb007
Immortal
Immortal

By disk system, I mean to include your controller and the disks behind it.  Are you also using RAID5 on the older system?

Using the same speed disk should yield you roughly the same IOPS.  A hardware controller should also not be limiting you.

Have you used (r)esxtop to see how your storage is performing?

-KjB

vExpert/VCP/VCAP vmwise.com / @vmwise -KjB
Reply
0 Kudos
hoevring
Contributor
Contributor

Ok, thats also brand new disks i have.

Yes, it is also Raid5 on my older system..

SAS is better than SCSI? Or is i wrong here? SCSI does only have about 320 mb/s, but sas has over 2.0 mb/s??

No, i have not tryed the storage performing, can i try it with VMware on?

Reply
0 Kudos
kjb007
Immortal
Immortal

You should be getting better throughput, but your IOPS will be similar to before, as that depends on the disk.

You can definitely use esxtop when the vm's are running, in fact, that's when you would want to use it.  If you experience the same slowdown, you can esxtop.  Conversely,  you can connect  your vsphere client to your esx host, and see the stats from the host performance tab as well.

-KjB

vExpert/VCP/VCAP vmwise.com / @vmwise -KjB
Reply
0 Kudos
hoevring
Contributor
Contributor

OK, thanks a lot for your time, but iam missing something!

Like you say i can follow the performance statistics from both - console and vpshere client.

But then i got the result, thats the next thing to do?

Iam just want to solve this issue, so i can move on..

Reply
0 Kudos
kjb007
Immortal
Immortal

First thing to figure out is if it's a guest problem, a vmkernel problem, or a device problem.  More than likely, it will be the first or the last.  If it's a device problem, then you've ruled out the vm.  Next is to rule out the storage.

-KjB

vExpert/VCP/VCAP vmwise.com / @vmwise -KjB
Reply
0 Kudos
hoevring
Contributor
Contributor

Hi again,

I have attached the picture with our esxtop with d

For me i looks okay? But i dont know really..?

Reply
0 Kudos
dcoz
Hot Shot
Hot Shot

Can you show the ESXTOP stats for gavg/cmd, kavg/cmd, and davg/cmd?

These will show the latency from the disks?

Thanks

Dougie

Reply
0 Kudos
hoevring
Contributor
Contributor

Yes sure!  here it is:

Reply
0 Kudos
dcoz
Hot Shot
Hot Shot

By looking at the counter DAVG/cmd It looks like your disks are struggling to keep up with the I/O being generated. The counter is telling you in milliseconds how long the request took to come back from the disks.

There is more writes than reads from the screenshot, so I would  look at this blog post about understanding the IO overhead of using RAID  5 etc.

http://www.yellow-bricks.com/2009/12/23/iops/

you can also break down which VMs are causing the most I/O by choosing v in esxtop

Hope this helps

Dougie

Reply
0 Kudos
hoevring
Contributor
Contributor

Thanks for your time, iam happy that are some people that can help.

1. Iam not happy what you're saying... I also read the page you attached, but i have 7 DISKS,

Iam confused, why can the IBM X346 without any problems run alle our servers with 3-5 MS?? Out HP is a brand new server, with faster disks...?

I can reinstall VMware, how big do you think the block shoul be in every disks? and should the writecache be enabled?

I can see which 3-4 servers that are using a big deal of the I/O

I thinking about the SCSI controller on the virtuel machines we have.. They are all selected with:  LSI Logic Parallel - Maybe it is the driver?

I cant see the next step in front of me... I thought it was the raid-controller.

Do you have any further "thinking-points"? Maybe the next step..

//Soeren

Reply
0 Kudos
hoevring
Contributor
Contributor

Reply
0 Kudos
hoevring
Contributor
Contributor

More and more problems with the P400:

http://communities.vmware.com/message/790045

Reply
0 Kudos
kjb007
Immortal
Immortal

Hmm.  That definitely looks like an issue with your controller at this point.  Either that, or the way it is handling the RAID.  Your server should have an onboard RAID controller that you can verify against?  Or is that what you are connecting to?

-KjB

vExpert/VCP/VCAP vmwise.com / @vmwise -KjB
Reply
0 Kudos