vSAN1

We have recently deployed VSAN at two new datacenter locations, the first will become our primary DC and the second will be our DR site.

Both sites is using new HPE hardware as of May 2016.The primary site contains four identical VSAN hosts that were spec'd very generously:

     dual Xeon E5-2699 v3 (18core/36thread)

     512GB RAM

     4x 800GB SSD, 12G SAS (model HP MO0800JEFPB)

     20x 1.2TB HDD, 10K 10G SAS (model HP EG1200JEHMC)

     4x 10GBe NIC

VSAN config on each host is four disk groups of 800GB SSD + 5 HDD.

VSAN traffic on each host is handled by 2x 10GBe NICs working together in an LACP LAG group. The NICs connect to the facility switching infrastructure which the service provider has set up for our organization using a specific VLAN for VSAN traffic. We have no visibility into the switching infrastructure so I cannot provide data from the switches at this point in time.

When running the built-in VSAN Storage Performance Test using workload type "Basic sanity test, focus on flash cache layer) I'm seeing poor results in the Throughput MB/s and Max Latency. Results below are averaged across all tested components (10 components are displayed per host)

Maybe I am not fully understanding this test, or the test is not truly representative of real-world performance, but 8.7MB/s seems extremely slow for the hardware and network we have in place. Also max latency of 304ms seems excessively high. Monitoring tools we use start throwing alerts when storage latency exceeds 50-75ms so these results aren't looking all that good.

Can anyone help me understand why this is happening? Again, not familiar with how this test stacks up against real-world performance, so if others have had similar poor results while using the built-in benchmark I would be interested to hear about those scenarios as well.

Currently running vCenter/ESXi 6.0 vanilla, I've recommended to management that we upgrade the environment to 6.0 Update 2 to expose the new VSAN Performance Service which should provide better real-world data but for now we would like to better understand the results of the built-in benchmark test.

Good morning, I would completely ignore that test.  In our environment it showed similar results, but real world and other tools (IOMeter) showed much different results.  We have Enterprise Plus licensing and have access to the Insights feature and that also showed much better results.  Thank you, Zach.

Thanks for the recommendation. We are going to ignore the built-in benchmark results as well.

For an alternate test I ran a 60 minute HCIBench pass against our primary vSAN cluster using StorageReview's 8K block size, 70% read & 30% write configuration file. 16 test VMs were deployed (4 per host) using the defaults of 10 virtual disks @ 10GB each.

Datastore: vSAN-Datastore

VMs = 16

IOPS = 200859.37 IO/s

THROUGHPUT = 1569.22 MB/s

LATENCY = 1.5810 ms

R_LATENCY = 1.2957 ms

W_LATENCY = 2.2464 ms

CPU USAGE = NaN%

RAM USAGE = NaN%

=============================

Now we're talkin!!

If you are running vSAN 6.2 (which I assume) there is actually a bug which keeps the dedupe scanner running even in hybrid vSAN cluster, this causes a massive increase in VM write latency. If you haven't already done so, I strongly recommend you to disable this. I've heard that this will be fixed in ESXi 6.0U2. I wrote a blog post about it (http://www.perthorn.com/vsan-vm-write-latency-part-1/) and there is now also a KB about it Performance Degradation of Hybrid disk groups on VSAN 6.2 Deployments (2146267) | VMware KB

Thank you very much for bringing that to my attention.

I am in the middle of bringing our environment to 6.0U2 (from 6.0 vanilla). Our DR site has been successfully updated and primary datacenter will be updated over the weekend.

It sounds like this fix is not required in 6.0U2 but I don't see a definitive answer to this in the KB description. Does anyone know if this still applies to 6.0U2?

Hi,

Sorry, I actually meant 6.0 U3. But reading the KB again they have actually released the fix for this in the 6.0 patch 3 (version 4192238) which was released on August 4th:

VMware ESXi 6.0, Patch Release ESXi600-201608001 (2145663) | VMware KB

So you should be fine as long as you apply the latest updates.

Per