VMware Cloud Community
Marcin4
Enthusiast
Enthusiast
Jump to solution

HCI Bench - vSphere 6.7 vs vSphere 7.0

Hello,

I've made some testing with HCI Bench tool for vSAN, and the results were as follow.

Question is, why there is so big difference between 6.7 and 7.0 in performance ????

vSphere 6.7:

6.7.png

vSphere 7.0

7.png

Best Regards
Marcin Gwóźdź
VCP-NV 6, VCIX-DCV 7, VCIX-DTM 7.
linkedin.com/in/marcin-gwóźdź-80b84b122
Tags (2)
Reply
0 Kudos
1 Solution

Accepted Solutions
TheBobkin
Champion
Champion
Jump to solution

Hello Marcin,

Running an unsupported driver for a vSAN controller is absolutely a problem - whether this is the only problem is what needs to be determined.

Can you please downgrade/re-install the supported vSAN 7.0 driver that matches the firmware in use for this controller (lsi_mr3 version 7.712.50.00-1vmw.700) - firmware doesn't need to be changed, the one applied currently is fine.

VMware Compatibility Guide - I/O Device Search

I would advise as next step after changing the driver would be to run another test and then start looking at the vSAN performance data to narrow this down further as to where we are getting the latency e.g. at the disk or the VM level (e.g. if there is a network issue).

If you want to assess/rule-out the inter-node networking and whether this is an issue, you can compare HCIBench tests when running with FTT=1 vs FTT=0 (though this requires some extra steps such as making sure the VMs being deployed as FTT=0 are running on the same node as where their data-components reside - otherwise the data is still traversing the inter-node network just in a different way).

Bob

View solution in original post

Reply
0 Kudos
12 Replies
depping
Leadership
Leadership
Jump to solution

Would be useful to know more details about your environment:

- Hardware? (CPU / Mem / Flash / Spinning / etc)

- Network setup?

- Policy used?

Reply
0 Kudos
Marcin4
Enthusiast
Enthusiast
Jump to solution

- Hardware? (CPU / Mem / Flash / Spinning / etc)

CPU: Intel(R) Xeon(R) Gold 6226 CPU @ 2.70GHz

Mem Per host: 512GB

All flash setup: 1x 480GB for cache and 1x 3,84TB for capacity

1 Disk Group

The setup is for 2 node, all flash vSAN.

- Network setup?

vSAN kernel has dedicated 2 x 25Gb pNIC on VSS

- Policy used?

vSAN Default Storage Policy

The setup were the same for testing on vSphere 6.7 and after upgrading to 7.0 it's performance went down.

Best Regards
Marcin Gwóźdź
VCP-NV 6, VCIX-DCV 7, VCIX-DTM 7.
linkedin.com/in/marcin-gwóźdź-80b84b122
Reply
0 Kudos
depping
Leadership
Leadership
Jump to solution

Did you check the firmware/driver combination for vSAN 7.0?

Reply
0 Kudos
Marcin4
Enthusiast
Enthusiast
Jump to solution

What do you mean by checking firmware/driver combination for vSAN 7.0 ?

As i Can notice the drive is supported on ESXi 7.0.

ssdvm.png

Best Regards
Marcin Gwóźdź
VCP-NV 6, VCIX-DCV 7, VCIX-DTM 7.
linkedin.com/in/marcin-gwóźdź-80b84b122
Reply
0 Kudos
depping
Leadership
Leadership
Jump to solution

Are you using firmware and drivers across all components which are on the vSphere 7 HCL?

Reply
0 Kudos
Marcin4
Enthusiast
Enthusiast
Jump to solution

Well I've made an upgrade from vSphere 6.7 to 7.0 via dell Customized Iso so I assume that all drivers are fully compatible with vSphere 7.0

I can also run try to patch both servers with Update Manager if that would help?

Best Regards
Marcin Gwóźdź
VCP-NV 6, VCIX-DCV 7, VCIX-DTM 7.
linkedin.com/in/marcin-gwóźdź-80b84b122
Reply
0 Kudos
Marcin4
Enthusiast
Enthusiast
Jump to solution

Hosts are fully updated.

Any ideas?

Best Regards
Marcin Gwóźdź
VCP-NV 6, VCIX-DCV 7, VCIX-DTM 7.
linkedin.com/in/marcin-gwóźdź-80b84b122
Reply
0 Kudos
TheBobkin
Champion
Champion
Jump to solution

Hello Marcin,

I think what Duncan was alluding to is that perhaps on the vSAN controllers you have a driver+firmware combination on 6.7 that is supported for the controller and then apply update which updates only the driver and this is not supported/compatible with the firmware which is not changed by updating ESXi.

What controller model and driver+firmware was installed on each builds tests?

Is this reproducible/consistent or did you just do one run on each build version?

If this is reproducible I would advise you open a Support Request with our vSAN performance team for deeper insight.

Bob

Reply
0 Kudos
Marcin4
Enthusiast
Enthusiast
Jump to solution

Hello Bob,

According to your questions:

1. Is this reproducible/consistent or did you just do one run on each build version?

I did couple of test with the same parameters for vSphere 6.7 and vSphere 7.0, and the results was the same.

     For vSphere 6.7, it was around 110k I/O per Second

     For vSphere 7.0 it was around 45k I/O per Secound

     As i mentioned on the first post.

2. What controller model and driver+firmware was installed on each builds tests?

I have no screen form vSphere 6.7, all i did was upgrade via Customized Dell ISO to vSphere 7.0

Host Specification for vSphere 7.0

SATA AHCI Controller

2020-07-15 13_06_36-vSphere - 10.101.0.11 - Hardware Compatibility i jeszcze 2 strony — Osobisty — M.png

Ethernet Controller

2020-07-15 13_10_00-vSphere - 10.101.0.11 - Hardware Compatibility i jeszcze 3 strony — Osobisty — M.png

Could that be a potential problem?

Best Regards
Marcin Gwóźdź
VCP-NV 6, VCIX-DCV 7, VCIX-DTM 7.
linkedin.com/in/marcin-gwóźdź-80b84b122
Reply
0 Kudos
TheBobkin
Champion
Champion
Jump to solution

Hello Marcin,

Running an unsupported driver for a vSAN controller is absolutely a problem - whether this is the only problem is what needs to be determined.

Can you please downgrade/re-install the supported vSAN 7.0 driver that matches the firmware in use for this controller (lsi_mr3 version 7.712.50.00-1vmw.700) - firmware doesn't need to be changed, the one applied currently is fine.

VMware Compatibility Guide - I/O Device Search

I would advise as next step after changing the driver would be to run another test and then start looking at the vSAN performance data to narrow this down further as to where we are getting the latency e.g. at the disk or the VM level (e.g. if there is a network issue).

If you want to assess/rule-out the inter-node networking and whether this is an issue, you can compare HCIBench tests when running with FTT=1 vs FTT=0 (though this requires some extra steps such as making sure the VMs being deployed as FTT=0 are running on the same node as where their data-components reside - otherwise the data is still traversing the inter-node network just in a different way).

Bob

Reply
0 Kudos
Marcin4
Enthusiast
Enthusiast
Jump to solution

Hello Bob,

I've upgraded my driver to lsi_mr3 version 7.712.50.00-1vmw.700

Skyline Health is now "all green"

Its defnitely not a network issue as nothing has been changed.

And I made some test, and the result is below:

Its a bit better (not 30k) but still not close to 100k IOPS

2020-07-16 11_23_43-Window.png

Best Regards
Marcin Gwóźdź
VCP-NV 6, VCIX-DCV 7, VCIX-DTM 7.
linkedin.com/in/marcin-gwóźdź-80b84b122
Reply
0 Kudos
Marcin4
Enthusiast
Enthusiast
Jump to solution

Bob,

Everything is good. I forgot to Clear Cahce option on HCIBench.

The drivers were the solutions. Thank you

2020-07-16 13_48_34-.png

Best Regards
Marcin Gwóźdź
VCP-NV 6, VCIX-DCV 7, VCIX-DTM 7.
linkedin.com/in/marcin-gwóźdź-80b84b122
Reply
0 Kudos