Very poor performance on one virtual machine

[breakaway9000]

Hi All,

I've got 2 x ESXi 5.1.0 hosts (733933) in my environment connected to a iSCSI starwind datastore which has SSDs in a RAID array. It's very fast in that we've never had performance issues.

The datastore in question contains several virtual machines. The virtual machines are as fast as you'd expect, except for one. It's painfully slow, and I can't nail down why (IOMeter test below).

I've tried these techniques to figure it out:

1. Run IOmeter multiple times of day to ensure it isn't some process on the problem VM that's causing issues
2. Monitored disk queue length of the problem VM to make sure it isn't too busy
3. Ensured all the virtual hardware parameters of the problem VM (disk controller type, CPU, RAM)
4. Installed a fresh VM on this datastore to eliminate the datastore as a cause of the performance issue
5. Checked the shares allocation of the problem VM to make sure it isn't getting pushed to the side under heavy load

I can't see anything that may cause this issue, anyone got any ideas?

The performance test:

[weinstein5]

So why do you think it is a storage performance issue? How is the machine configured? Have you looked at CPU ready time or if there is abnormal use of the vmkernel swap file?

[jrmunday]

Where did this problem VM originate from - was it always a VM or has it been P2V'd? What hardware version is it?

Have you checked the disk alignment to ensure that it's not misaligned IO causing this performance degradation?

You can use this script to check disk alignment on Windows servers;

http://www.jonmunday.net/2013/03/20/are-all-your-guest-os-partition-correctly-aligned/

Perhaps running vcscistats will give you some insight into the IO characteristics.

Does performance stay the same regardless of which host you run it on?

What happens if you add a new disk to this VM at check performance of this disk - how does that compare?

Is performance the same if you create a new VM and attach these VMDK's?

This is an interesting one, and I'm looking forward to understanding what's causing this.

Cheers,

Jon

[Gortee · Accepted solution]

Just a thought we had something like this a long time ago... I was sure it was storage it ended up being memory and CPU limit on the vm.  We would see really bad performance on the vm storage wise but the real issue was the vm had limit causing storage I/O wait time.   I would look into the virtual machine settings and look at reservation / limit.   Remember that even reservation does not override limit.  

If you don't have a limit try setting a reservation for all CPU / Memory then test again.   If it fails again there is something really wrong with the I/O stack on your virtual machine OS.  (You should also try to vmotion it to another datastore) 

If you know how to use ESXTOP you may quickly find some information about your issue using the m, c or v buttons.

Let me know what else you find.

[AnatolyVilchins]

Hi!

The important things that hasn`t been asked yet:

• What StarWind device type are you running (DD, CDP, HA, etc)?
• What RAID type are you running?
• Have you tried to run IOmeter from VM? Can you share the results with us?
• Have you benchmarked the iSCSI storage from hypervisor box?

To be honest I`m pretty agree with Gortee this looks like something in the VM was misconfigured.

BTW, maybe it is a good idea to contact SW support directly regarding this issue?

[breakaway9000]

Gortee had the correct answer. The CPU reservation. The second I added a CPU reservation it really sped up.    

[EXPRESS]

Did you set both CPU and Memory?

[breakaway9000]

For the time being I have only added a CPU reservation.