vSAN performance

[YahontC]

I have such infrastructure:

hardware:

4 identical hosts:

AIC SB302-LB (3U 16-Bay Storage Server, not certified for vSphere 6.2)

one Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz

RAM – 128 GB

NVMe-flash  - two HGST Ultrastar SN100 Series NVMe SSD HUSPR3216ADP301 1,6TB PCIe (certified for all-flash Virtual SAN 6.2)

HDD - eight HGST Ultrastar 7K6000 HUS726020AL5214 2TB 7200 rpm SAS 12Gbitps (not certified for Virtual SAN 6.2, certified for vSphere 6.5)

boot SSD 60GB

storage controller - LSI Logic Fusion-MPT 12GSAS SAS3008 PCI-Express (certified for vSphere 6.2, not certified for Virtual SAN 6.2)

2 port 1GbE

2 port IB 40Gbps – HCA Mellanox ConnectX-3 (IPoIB)

IB-switch Mallanox SB7790

soft: VMware vSphere 6.2

vCenter Server Appliance 6.0.0.20100 (with last patch)

ESXi 6.0.0.4600944 (with last patch)

Mallanox ConnectX-3 driver: MLNX-OFED-ESX-2.4.0.0-10EM-600.0.0.2494585

4 host HA+vSAN cluster

vCenter Server Appliance - VM hosted on local boot SSD from 1 host

Virtual SAN engages all cluster hosts drives (exept boot SSD): 8 identical disk groups – 2 DG for 1 host; each DG includes - 1 NVMe-flash for cache and 4 HDD for capacity.  57,64TB - row capacity.

My 1st test:

I've deployed 1 win 7 х64 VM on vSAN: 1 system drive (Virtual SAN Default Storage Policy) and 1 drive for test - 2TB (custome storage policy - 8 strip, 100% thick, raid-1).

So, my test-disk was simmetricaly deployed on 2 hosts - evenly distributed across all disks (2 DG) in the host. VM run on 1 of this 2 hosts. I tried to make a VM deployment to achive max performance.

I've tested vSAN phisical net performance by iperf, it's OK, about 25 Gbps for Mellanox ConnectX-3. So, net is not a bottleneck.

I've run iometr 1.1 on VM: 4 Workers, 64 Outstand IO, full random. And made 4 tests: 1M_sequential_write, 1M_sequential_read, 4K_rand_write, 4K_rand_read.

Results:

access	IOPS	Throughput	Latency
1M_sequential_write		897	299
1M_sequential_read		1197	224
4K_rand_write	4087		62
4K_rand_read	65 000		5

I think that results are miserable. I see (in vSAN monitoring) that whole test-disk is in read-cache in NVMe. But 65K IOPS on rand read is too small for NVMe. 3200GB Ultrastar SN100 Series NVMe PCIe SSD | HGST.

Other results are bad too: for the drives (2x 8hdd stripe) also for flash.

Question: do i make wrong installation (i think that all is correct)? is the problem in non compartible drives and controllers? or it's a normal results for vSAN?

host reboot duration is about 30-40 min (VSAN: Initialising SSD: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Please wait...). I've read in some topics that 10 min is normal for vSAN host reboot, but 40min is too long. Why?

[admin]

Greetings!

Question: do i make wrong installation (i think that all is correct)? is the problem in non compartible drives and controllers? or it's a normal results for vSAN?

host reboot duration is about 30-40 min (VSAN: Initialising SSD: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Please wait...). I've read in some topics that 10 min is normal for vSAN host reboot, but 40min is too long. Why?

Installation of vSAN looks correct as per the screenshots but incompatible Storage Controller and drivers looks the cause of all the problems you mentioned in your question. It is not normal that there is an incompatible storage controller driver being part vSAN deployment. Storage controller/adapter driver and firmware plays an important role in vSAN performance. Many of the issues reported by customers for vSAN Performance relate to unsupported hardware, drivers or firmware.

VMware says that in case of failures to precisely adhere to the guidelines provided in the VMware Compatibility Guide may result in erratic performance, unusual error conditions and/or potential data loss. Also, VMware support may not be in a position to assist you if the components used to build a vSAN environment are not on the VCG.

Hope this answers your question.

_________________________

Was your question answered correctly? If so, please remember to mark your question as "Correct" or "Helpful" when you get the correct answer. This helps others searching for a similar issue.

Cheers!

-Shivam

[YahontC]

Thanks!

Do you think my performance results are bad? What results should i recive on similar devices from VCG?

[YahontC]

Is there any opportunity to find and install correct drivers (from vmware, manufacturer or custom) for my non compatible drives and controllers?

[admin]

I believe you have already checked vSAN VCG. Unfortunately, there is no tweak to make Storage Controller and Driver compatible with vSAN.

We always look at VMware Compatibility Guide - vsan‌‌ > "Build Your Own based on Certified Components." in case of building vSAN with custom hardware. If It is not there on vSAN VCG then Storage Controller and Driver are not supported with vSAN.

Regarding Performance Results, I will test the performance of my vSAN cluster and provide the results here shortly.

[YahontC]

Yes, i've checked:

AIC SB302-LB (3U 16-Bay Storage Server, not certified for vSphere 6.2)

NVMe-flash  - two HGST Ultrastar SN100 Series NVMe SSD HUSPR3216ADP301 1,6TB PCIe (certified for all-flash Virtual SAN 6.2)

HDD - eight HGST Ultrastar 7K6000 HUS726020AL5214 2TB 7200 rpm SAS 12Gbitps (not certified for Virtual SAN 6.2, certified for vSphere 6.5)

boot SSD 60GB

storage controller - LSI Logic Fusion-MPT 12GSAS SAS3008 PCI-Express (certified for vSphere 6.2, not certified for Virtual SAN 6.2)

:smileycry:

I am looking forward to your performance results.

[YahontC]

esxcli software vib list | grep lsi

lsi-msgpt3                     06.255.12.00-8vmw.600.1.17.3029758    VMware  VMwareCertified   2016-11-09

esxcli software vib list | grep nvme

nvme                           1.0e.0.35-1vmw.600.2.34.3620759       VMware  VMwareCertified   2016-11-09

for my LSI SAS3008 used driver: lsi-msgpt3                     06.255.12.00-8vmw.600.1.17.3029758    VMware  VMwareCertified

should i install:  lsi-msgpt3 version 14.00.00.00-1OEM Partner Async, native (VMware Compatibility Guide - I/O Device Search) - to resolve my problem?

for my HGST Ultrastar SN100 Series NVMe SSD HUSPR3216ADP301 - there is no drivers from manufacturer. But there are 1.0e.0.35-1vmw.600.2.34.3620759       VMware  VMwareCertified in my host.

So I have native vmware drivers for my controllers, why vSAN healf shows Warning?