Guest Disk Performance Limit?

[Cryptz]

Product

VMware vSphere ESXi 5.1

Hello,

We are in the testing phase of a deployment. Currently I have a 24 drive ssd array presented to an esxi 5.1 host as an iscsi target. Locally the 24 drive array can perform about 5000GBps read and 3500GBps write with caching disabled. When i say locally i mean a benchmark ran on the san itself, not involving the esxi host.

If i directly connect the esxi 5.1 host with an intel x520 10g adapter to the san I get about 500MBps read and 450MBps write from a windows 2012 guest vm. The vm is using the lsi sas scsi driver.

If i add a second direct connected intel x520 and configure round robin with iops set to 1 i get the same exact result.

if i change the windows 2012 guest scsi controller to paravirtualization I get about 980MBps read and 820 MBps write in the windows 2012 guest (this is with both intel x520s using MPIO)

Even at 980MBps read and 820MBps write i am still way below the performance of the raid array. Since i was able to double read performance by changing the vmware scsi adapter in the guest i am wondering if there are any settings restricting the paravirtualization scsi controller in the guest.

I have since tried mellanox 40GBe/56infiband cards. I saw no performance improvement when using both 56gb srv or 40gbe. I can get a very good aggregate amount of disk throughput on the host itself (for example 4 vms can get about 1GBps throughput at a time). My goal is to "uncapp" a single virtual machine so that it can exceed 1GBps. I have never found anything that states the vms are capped in any way at 1GBps. I have tried messing with disk share values but it did not seem to change anything. There appears to be some per vm limit. Has anyone been able to exceed 1 GBps transfer per vm? I am talking GB not Gb.

[marcelo_soares]

Interesting scenario. Can you try doing the test on 2 VMs simultaneously to check the results (LSI and paravirtual)?

[Cryptz]

it scales, both vms get the same results at the same time. I can basically max the san if i benchmark enough vms at a time on the host, its just for our purposes we were hoping for better numbers on a single specific guest.

[marcelo_soares]

How about RDMs? Have you tried Virtual/Physical RDMs?

[Cryptz]

i havent, but it isnt really what we would want for the environment.

[AGrueninger]

When I tested our new storage I was able to get up to 1.6 GB/s and I mean GB/s and not Gb/s.

I got this values with Infiniband DDR and SRP protocol on a Solaris 11 based SRP target. The host had 128 GB RAM. The backend was a zpool of 60 3 TB SAS disks in a RAIDZ2 layout with 5 vdevs and 12 disks per vdev, sync disabled. Not recommended for normal use but good to get high values for benchmarks. The benchmark was IOmeter and I used only writes with 128 kb blocks. The size of the testfile was substantially lower than the cache of the storage host. The values observed were near the theoretical maximum of a DDR connection (80% of 20 Gb/s). And this was exactly the aim of this test.

When the same pool was used with NFS over IPOIB we got between 600 and 700 GB/s with the same test.

I guess you used IP based communication over NFS or ISCSI. And I guess you run the same benchmark on your storage.

Take your IB cards, configure them to us IB and not ETHERNET and install the newest Mellanox drivers. They support SRP again with driver version 1.8.1.

Your storage must be able to create luns and share them as SRP targets.

But check it out carefully. It depends on your real workload. SRP ist very fast for sequential access. We got similar results with NFS and random access. And if you have a lot of esx hosts using your storage the I/O is mostly random.

Could you describe your storage? Seems to be a very interesting beast.

Greetings

Andreas

[Cryptz]

It is an Ubuntu 13 host running zfs. The zfs pool is shared via a scst srp target. There are 24 ssds in the pool, no l2arch, 128gb of arc. I am linked up at 56Gbps infiniband currently. I can hit link speed aggregated across all vms at the same time, but cannot seem to surpass 1GB from a single guest, even if I only have 1 guest running in the entire environment. It may just be my benchmark (crystaldisk), but I have benchmarked local storage without issue in the past so I am not sure why the benchmark would be the issue. Also the LSI VMware driver seemed hard limited at 500MBps in my tests, switching the storage drive to paravirtual netted almost 2x the performance. That led me to conclude there was some limiting factor in the hypervisor. Honestly in real world applications you wouldn’t want a single guest hogging all resources, I simply cannot find a documented explanation of what I am seeing.

[maloner]

Hi,

Did you ever get to the bottom of this 1Gbps limit?

I'm seening same issue with Local Storage on a couple machines I have tested with - they each have RAID arrays with 10k or 15k disks and 6Gbps interfaces.

Installing OS straight on host does not have same limitation and as per your findings, I can get 1Gbps per VM.

There seems to be some sort of software  limit, did you ever find a documented or undocumented way to lift this?

It's causing me issues when trying to run backups where the job saturates the 1Gbps and then the Application has nothing left...

Thanks...

[Alistar]

Hi there,

this is indeed a very interesting scenario. As for paravirtualized driver, this is recommended by VMware to be used with performance intensive workload for network-based storage while increasing IOPS and lowering CPU utilization.

The one thing that comes to my mind is queue depth - can you try hopping on your ESXi host via SSH and run ESXTOP as root, checking both the adapter setting with the "d" and "u" keys and network with "n" key while running the benchmarks on your VMs and post the screenshots here? Maybe you can observe yourself if the Queue is being filled or not causing a stall in throughput.

Does your array support VAAI and do you have Storage IO control enabled?