Hello,
I've made some testing with HCI Bench tool for vSAN, and the results were as follow.
Question is, why there is so big difference between 6.7 and 7.0 in performance ????
vSphere 6.7:
vSphere 7.0
Hello Marcin,
Running an unsupported driver for a vSAN controller is absolutely a problem - whether this is the only problem is what needs to be determined.
Can you please downgrade/re-install the supported vSAN 7.0 driver that matches the firmware in use for this controller (lsi_mr3 version 7.712.50.00-1vmw.700) - firmware doesn't need to be changed, the one applied currently is fine.
VMware Compatibility Guide - I/O Device Search
I would advise as next step after changing the driver would be to run another test and then start looking at the vSAN performance data to narrow this down further as to where we are getting the latency e.g. at the disk or the VM level (e.g. if there is a network issue).
If you want to assess/rule-out the inter-node networking and whether this is an issue, you can compare HCIBench tests when running with FTT=1 vs FTT=0 (though this requires some extra steps such as making sure the VMs being deployed as FTT=0 are running on the same node as where their data-components reside - otherwise the data is still traversing the inter-node network just in a different way).
Bob
Would be useful to know more details about your environment:
- Hardware? (CPU / Mem / Flash / Spinning / etc)
- Network setup?
- Policy used?
- Hardware? (CPU / Mem / Flash / Spinning / etc)
CPU: Intel(R) Xeon(R) Gold 6226 CPU @ 2.70GHz
Mem Per host: 512GB
All flash setup: 1x 480GB for cache and 1x 3,84TB for capacity
1 Disk Group
The setup is for 2 node, all flash vSAN.
- Network setup?
vSAN kernel has dedicated 2 x 25Gb pNIC on VSS
- Policy used?
vSAN Default Storage Policy
The setup were the same for testing on vSphere 6.7 and after upgrading to 7.0 it's performance went down.
Did you check the firmware/driver combination for vSAN 7.0?
What do you mean by checking firmware/driver combination for vSAN 7.0 ?
As i Can notice the drive is supported on ESXi 7.0.
Are you using firmware and drivers across all components which are on the vSphere 7 HCL?
Well I've made an upgrade from vSphere 6.7 to 7.0 via dell Customized Iso so I assume that all drivers are fully compatible with vSphere 7.0
I can also run try to patch both servers with Update Manager if that would help?
Hosts are fully updated.
Any ideas?
Hello Marcin,
I think what Duncan was alluding to is that perhaps on the vSAN controllers you have a driver+firmware combination on 6.7 that is supported for the controller and then apply update which updates only the driver and this is not supported/compatible with the firmware which is not changed by updating ESXi.
What controller model and driver+firmware was installed on each builds tests?
Is this reproducible/consistent or did you just do one run on each build version?
If this is reproducible I would advise you open a Support Request with our vSAN performance team for deeper insight.
Bob
Hello Bob,
According to your questions:
1. Is this reproducible/consistent or did you just do one run on each build version?
I did couple of test with the same parameters for vSphere 6.7 and vSphere 7.0, and the results was the same.
For vSphere 6.7, it was around 110k I/O per Second
For vSphere 7.0 it was around 45k I/O per Secound
As i mentioned on the first post.
2. What controller model and driver+firmware was installed on each builds tests?
I have no screen form vSphere 6.7, all i did was upgrade via Customized Dell ISO to vSphere 7.0
Host Specification for vSphere 7.0
SATA AHCI Controller
Ethernet Controller
Could that be a potential problem?
Hello Marcin,
Running an unsupported driver for a vSAN controller is absolutely a problem - whether this is the only problem is what needs to be determined.
Can you please downgrade/re-install the supported vSAN 7.0 driver that matches the firmware in use for this controller (lsi_mr3 version 7.712.50.00-1vmw.700) - firmware doesn't need to be changed, the one applied currently is fine.
VMware Compatibility Guide - I/O Device Search
I would advise as next step after changing the driver would be to run another test and then start looking at the vSAN performance data to narrow this down further as to where we are getting the latency e.g. at the disk or the VM level (e.g. if there is a network issue).
If you want to assess/rule-out the inter-node networking and whether this is an issue, you can compare HCIBench tests when running with FTT=1 vs FTT=0 (though this requires some extra steps such as making sure the VMs being deployed as FTT=0 are running on the same node as where their data-components reside - otherwise the data is still traversing the inter-node network just in a different way).
Bob
Hello Bob,
I've upgraded my driver to lsi_mr3 version 7.712.50.00-1vmw.700
Skyline Health is now "all green"
Its defnitely not a network issue as nothing has been changed.
And I made some test, and the result is below:
Its a bit better (not 30k) but still not close to 100k IOPS
Bob,
Everything is good. I forgot to Clear Cahce option on HCIBench.
The drivers were the solutions. Thank you