High Write Latency >300ms

[ezequielcarson]

Hello Folks,

i have a diskgroup with 7 SSD 1TB  M550 and 1 SSD CACHE Micron  PCIEx P420M . All of them are connected directly to a LSI 3008 working in a pass-thru mode.

im having wr latencies above 200 or 300 ms for every VM but i dont have any latency on the disk

  GID VMNAME           VDEVNAME NVDISK   CMDS/s  READS/s WRITES/s MBREAD/s MBWRTN/s LAT/rd LAT/wr

1247418 96945_ESTRELLA_         -      2   15.75     0.98    14.77     0.06     0.24   9.85 339.33

339 ms of wr latency for 15 IOPS.

And now this is the esxtop for every disk.

DEVICE                                PATH/WORLD/PARTITION DQLEN WQLEN ACTV QUED %USD  LOAD   CMDS/s  READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/cmd GAVG/cmd QAVG/cmd

mpx.vmhba32:C0:T0:L0                           -               1     -    0    0    0  0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00

naa.500a07510d32798d                           -              32     -    0    0    0  0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00

naa.500a07510d327bc7                           -              32     -   22    0   68  0.69  1571.66   976.56   595.09    15.26     9.30    14.34     0.00    14.35     0.00

naa.500a07510d327bcb                           -              32     -    0    0    0  0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00

naa.500a07510d327bce                           -              32     -    0    0    0  0.00   488.28   488.28     0.00     7.63     0.00     0.79     0.00     0.79     0.00

naa.500a07510d328e7a                           -              32     -    0    0    0  0.00  1091.00   488.28   602.72     7.63     9.42    20.85     0.00    20.85     0.00

naa.500a07510d32955d                           -              32     -    0    0    0  0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00

naa.500a07510d32956b                           -              32     -    1    0    3  0.03     7.63     0.00     7.63     0.00     0.00     2.15     0.01     2.15     0.00

naa.500a07510d32989b                           -              32     -    0    0    0  0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00

t10.ATA_____Micron_M600_MTFDDAK1T0MB           -              31     -    0    0    0  0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00

t10.ATA_____Micron_P420m2DMTFDGAR1T4           -             255     -    1    0    0  0.00 11528.02 11329.65   198.36    58.54    10.67     0.10     0.01     0.11     0.00

t10.ATA_____ST9250610NS_____________           -              31     -    0    0    0  0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00

im lost here. The VM does not have any limite on the IOPS.

Any clue?

Txs

Ezequiel

[admin]

Hi Ezequiel,

The controller you have is not on the HCL, neither is all-Flash VSAN supported. The VSAN team is currently testing this controller as we speak. If you provide the driver and firmware version of the controller and create an SR (along with support bundles) with VMware support, the team can take a look and get back to you.

Thanks.

Kiran

[DavidPasek]

Can you check and share what is queue depth of your LSI controller?

Read this blog post for details

http://www.yellow-bricks.com/2014/04/17/disk-controller-features-and-queue-depth/

[ezequielcarson]

Sure,

The QD for the LSI 3008 is 600

Checking the epping’s block I see that SATA interface has 32 DQLEN.

Im using the Micron SSD but they are sata interface finally, so I assume this is why we have 32.. am I correct?

Ezeq.

[ezequielcarson]

Hi,

The controller is not on the HCL but we have the 9341 from LSI in the HCL which is using the same chip (3008 ROC)

Txs

Ezeq.

[DavidPasek]

DQLEN is device queue length

AQLEN is adapter queue length

Both Adapter and Disk queue length are important.

You are saying that your AQLEN is 600 which should be good enough.

However you are saying that your SSD disk has SATA interface which means only 32 DQLEN.

SATA interface is good for notebooks but not for servers.

If you need SSD's for VSAN or any other enterprise purpose go with SAS interface which gives you much better performance.

Duncan's blog post with more details is here

http://www.yellow-bricks.com/2014/06/09/queue-depth-matters/

[DavidPasek]

I've double checked your initial post and it seems to me that Micron_P420 has DQLEN 255. That's very good and it looks like SAS interface.

t10.ATA_____Micron_P420m2DMTFDGAR1T4           -             255

You have submitted following performance data from ESXTOP

GID VMNAME           VDEVNAME NVDISK   CMDS/s  READS/s WRITES/s MBREAD/s MBWRTN/s LAT/rd LAT/wr

1247418 96945_ESTRELLA_         -      2   15.75     0.98    14.77     0.06     0.24   9.85 339.33

It means that your are sending to disk only 16 IOPS where almost all of them are write IOs and you are getting write latency 339 ms.

I agree that latency is very high however I would like to know how do you generate disk workload because 16 IOPS is almost nothing. In other words it is very light disk workload.

I would try to stress disk subsystem with tool like IOmeter with several workers (threads or oustanding IOs) and check how many IOPS with which latency you can get.

[ezequielcarson]

Thank you.

Micron has released a DC version of the M500 but they are still SATA interface.

For example Intel DC s3700 are High Endurance Enteprise Disk but they are still SATA. So I would keep having my 32 even If I have an enterprise SATA disk.

Isnot there any way to modify the 32 DQLEN to 128 without creating a RAID-0 per disk?

[ezequielcarson]

David,

I made these tests when the server arrived and i was able to get 40k IOPS using 4k random. The latency was below 10ms

The strange thing here is that I only see this latency on the VM but I don’t see the latency on any DISK or even the SSD

The SSD CACHE is always below 1 or 2 ms or wr latency

The SSD SATA disks are between 1ms and 30 ms (clearly the latency reache 30 ms when the Queue usage is close to 70% or 80%)

But 300 ms is a crazy thing

This is server is an Exchange Server working as CAS Role

Ezeq.

[DavidPasek]

So you don't have any performance problem reporting from application users, right?

You see high latency on VM when there is low disk activity (15 IOPS), right?

You don't see high latency on back end (Cache ane data SSD's), right?

Then I think higher latency is done on disk scheduler on VMkernel level.

VMkernel disk scheduler has same algorithm as enterprise shared storage arrays. It gives better latency to disk intensive workloads and high latency to VMs which are not disk intensive.

I would try to run IOmeter with just let's say 4 workers and with 1 outstanding IOs from Exchange VM to generate more IOs and  check if response time is better. I would bet it helps.

David.

[ezequielcarson]

Yes, you got it!

Users are seeing that this VM is slow for writing operations, even windows takes too much time to reboot and you will see spikes of 500ms or 600ms

I will performe the iometer and share the results

Txs

Ezeq