12 Replies Latest reply on Jul 16, 2020 4:49 AM by Marcin4

    HCI Bench - vSphere 6.7 vs vSphere 7.0

    Marcin4 Enthusiast

      Hello,

       

       

      I've made some testing with HCI Bench tool for vSAN, and the results were as follow.

       

      Question is, why there is so big difference between 6.7 and 7.0 in performance ????

       

       

      vSphere 6.7:

      6.7.png

       

      vSphere 7.0

       

      7.png

        • 1. Re: HCI Bench - vSphere 6.7 vs vSphere 7.0
          depping Champion
          User ModeratorsVMware Employees

          Would be useful to know more details about your environment:

           

          - Hardware? (CPU / Mem / Flash / Spinning / etc)

          - Network setup?

          - Policy used?

          • 2. Re: HCI Bench - vSphere 6.7 vs vSphere 7.0
            Marcin4 Enthusiast

            - Hardware? (CPU / Mem / Flash / Spinning / etc)

             

            CPU: Intel(R) Xeon(R) Gold 6226 CPU @ 2.70GHz

            Mem Per host: 512GB

            All flash setup: 1x 480GB for cache and 1x 3,84TB for capacity

            1 Disk Group

             

            The setup is for 2 node, all flash vSAN.

             

            - Network setup?

            vSAN kernel has dedicated 2 x 25Gb pNIC on VSS

             

            - Policy used?

            vSAN Default Storage Policy

             

            The setup were the same for testing on vSphere 6.7 and after upgrading to 7.0 it's performance went down.

            • 3. Re: HCI Bench - vSphere 6.7 vs vSphere 7.0
              depping Champion
              User ModeratorsVMware Employees

              Did you check the firmware/driver combination for vSAN 7.0?

              • 4. Re: HCI Bench - vSphere 6.7 vs vSphere 7.0
                Marcin4 Enthusiast

                What do you mean by checking firmware/driver combination for vSAN 7.0 ?

                 

                As i Can notice the drive is supported on ESXi 7.0.

                 

                ssdvm.png

                • 5. Re: HCI Bench - vSphere 6.7 vs vSphere 7.0
                  depping Champion
                  VMware EmployeesUser Moderators

                  Are you using firmware and drivers across all components which are on the vSphere 7 HCL?

                  • 6. Re: HCI Bench - vSphere 6.7 vs vSphere 7.0
                    Marcin4 Enthusiast

                    Well I've made an upgrade from vSphere 6.7 to 7.0 via dell Customized Iso so I assume that all drivers are fully compatible with vSphere 7.0

                     

                    I can also run try to patch both servers with Update Manager if that would help?

                    • 7. Re: HCI Bench - vSphere 6.7 vs vSphere 7.0
                      Marcin4 Enthusiast

                      Hosts are fully updated.

                       

                      Any ideas?

                      • 8. Re: HCI Bench - vSphere 6.7 vs vSphere 7.0
                        TheBobkin Virtuoso
                        VMware EmployeesvExpert

                        Hello Marcin,

                         

                        I think what Duncan was alluding to is that perhaps on the vSAN controllers you have a driver+firmware combination on 6.7 that is supported for the controller and then apply update which updates only the driver and this is not supported/compatible with the firmware which is not changed by updating ESXi.

                        What controller model and driver+firmware was installed on each builds tests?

                         

                        Is this reproducible/consistent or did you just do one run on each build version?

                        If this is reproducible I would advise you open a Support Request with our vSAN performance team for deeper insight.

                         

                        Bob

                        • 9. Re: HCI Bench - vSphere 6.7 vs vSphere 7.0
                          Marcin4 Enthusiast

                          Hello Bob,

                           

                          According to your questions:

                           

                          1. Is this reproducible/consistent or did you just do one run on each build version?

                           

                          I did couple of test with the same parameters for vSphere 6.7 and vSphere 7.0, and the results was the same.

                               For vSphere 6.7, it was around 110k I/O per Second

                               For vSphere 7.0 it was around 45k I/O per Secound

                               As i mentioned on the first post.

                           

                          2. What controller model and driver+firmware was installed on each builds tests?

                           

                          I have no screen form vSphere 6.7, all i did was upgrade via Customized Dell ISO to vSphere 7.0

                          Host Specification for vSphere 7.0

                           

                          SATA AHCI Controller

                          2020-07-15 13_06_36-vSphere - 10.101.0.11 - Hardware Compatibility i jeszcze 2 strony — Osobisty — M.png

                           

                           

                          Ethernet Controller

                          2020-07-15 13_10_00-vSphere - 10.101.0.11 - Hardware Compatibility i jeszcze 3 strony — Osobisty — M.png

                           

                          Could that be a potential problem?

                          • 10. Re: HCI Bench - vSphere 6.7 vs vSphere 7.0
                            TheBobkin Virtuoso
                            VMware EmployeesvExpert

                            Hello Marcin,

                             

                            Running an unsupported driver for a vSAN controller is absolutely a problem - whether this is the only problem is what needs to be determined.

                            Can you please downgrade/re-install the supported vSAN 7.0 driver that matches the firmware in use for this controller (lsi_mr3 version 7.712.50.00-1vmw.700) - firmware doesn't need to be changed, the one applied currently is fine.

                            VMware Compatibility Guide - I/O Device Search

                             

                            I would advise as next step after changing the driver would be to run another test and then start looking at the vSAN performance data to narrow this down further as to where we are getting the latency e.g. at the disk or the VM level (e.g. if there is a network issue).

                            If you want to assess/rule-out the inter-node networking and whether this is an issue, you can compare HCIBench tests when running with FTT=1 vs FTT=0 (though this requires some extra steps such as making sure the VMs being deployed as FTT=0 are running on the same node as where their data-components reside - otherwise the data is still traversing the inter-node network just in a different way).

                             

                            Bob

                            • 11. Re: HCI Bench - vSphere 6.7 vs vSphere 7.0
                              Marcin4 Enthusiast

                              Hello Bob,

                               

                              I've upgraded my driver to lsi_mr3 version 7.712.50.00-1vmw.700

                              Skyline Health is now "all green"

                               

                              Its defnitely not a network issue as nothing has been changed.

                               

                              And I made some test, and the result is below:

                              Its a bit better (not 30k) but still not close to 100k IOPS

                              2020-07-16 11_23_43-Window.png

                              • 12. Re: HCI Bench - vSphere 6.7 vs vSphere 7.0
                                Marcin4 Enthusiast

                                Bob,

                                 

                                Everything is good. I forgot to Clear Cahce option on HCIBench.

                                The drivers were the solutions. Thank you

                                 

                                2020-07-16 13_48_34-.png