Hi All,
I've got 2 x ESXi 5.1.0 hosts (733933) in my environment connected to a iSCSI starwind datastore which has SSDs in a RAID array. It's very fast in that we've never had performance issues.
The datastore in question contains several virtual machines. The virtual machines are as fast as you'd expect, except for one. It's painfully slow, and I can't nail down why (IOMeter test below).
I've tried these techniques to figure it out:
I can't see anything that may cause this issue, anyone got any ideas?
The performance test:
Just a thought we had something like this a long time ago... I was sure it was storage it ended up being memory and CPU limit on the vm. We would see really bad performance on the vm storage wise but the real issue was the vm had limit causing storage I/O wait time. I would look into the virtual machine settings and look at reservation / limit. Remember that even reservation does not override limit.
If you don't have a limit try setting a reservation for all CPU / Memory then test again. If it fails again there is something really wrong with the I/O stack on your virtual machine OS. (You should also try to vmotion it to another datastore)
If you know how to use ESXTOP you may quickly find some information about your issue using the m, c or v buttons.
Let me know what else you find.
So why do you think it is a storage performance issue? How is the machine configured? Have you looked at CPU ready time or if there is abnormal use of the vmkernel swap file?
Where did this problem VM originate from - was it always a VM or has it been P2V'd? What hardware version is it?
Have you checked the disk alignment to ensure that it's not misaligned IO causing this performance degradation?
You can use this script to check disk alignment on Windows servers;
http://www.jonmunday.net/2013/03/20/are-all-your-guest-os-partition-correctly-aligned/
Perhaps running vcscistats will give you some insight into the IO characteristics.
Does performance stay the same regardless of which host you run it on?
What happens if you add a new disk to this VM at check performance of this disk - how does that compare?
Is performance the same if you create a new VM and attach these VMDK's?
This is an interesting one, and I'm looking forward to understanding what's causing this.
Cheers,
Jon
Just a thought we had something like this a long time ago... I was sure it was storage it ended up being memory and CPU limit on the vm. We would see really bad performance on the vm storage wise but the real issue was the vm had limit causing storage I/O wait time. I would look into the virtual machine settings and look at reservation / limit. Remember that even reservation does not override limit.
If you don't have a limit try setting a reservation for all CPU / Memory then test again. If it fails again there is something really wrong with the I/O stack on your virtual machine OS. (You should also try to vmotion it to another datastore)
If you know how to use ESXTOP you may quickly find some information about your issue using the m, c or v buttons.
Let me know what else you find.
Hi!
The important things that hasn`t been asked yet:
To be honest I`m pretty agree with Gortee this looks like something in the VM was misconfigured.
BTW, maybe it is a good idea to contact SW support directly regarding this issue?
Gortee had the correct answer. The CPU reservation. The second I added a CPU reservation it really sped up.
Did you set both CPU and Memory?
For the time being I have only added a CPU reservation.