1 25 26 27 28 29 Previous Next 574 Replies Latest reply on Jun 1, 2018 5:44 AM by larstr Go to original post
      • 390. Re: New  !! Open unofficial storage performance thread
        JaFF Novice

        Hi,

         

        I am currently out of the office.

        If you require assistance, please call our helpdesk on 1300101112.

        Alternatively, email service@anittel.com.au

         

        Regards,

         

        James Ackerly

        • 391. Re: New  !! Open unofficial storage performance thread
          Ingens Novice

          Hello everyone,

          I'd like to ask you about some storage performance issues we are facing. We are using Oracle (old SUN) Unified Storage 7110 and have 5 ESXi 4.1 hosts connected to one LUN (which got all the space assigned).
          Past few weeks we had HUGE performance problems and finally figured that it was due to using over 80% of the storage available. Apparently a drop in performance while using RAIDZ system is to be expected when using over 80% of its total capacity. So we tried to lower the storage usage below the 80% mark and achieved a performance increase. However, it was still way lower than expected and thus, we were not yet satisfied. Talking with Oracle support they pointed us to upgrade to the newest firmware release which solved some bugs related to performance issues that we were most likely facing.
          But here comes the weird part, after upgrading the firmware we have noticed a HUGE performance increase on our benchmarks but the VMs, while faster, weren't still working as smoothly as few months ago.
          The only explanation I can come up with is that we are not properly understanding the benchmarks results or that there is something else that we are missing. So here we are asking you guys about our performance results:
          SERVER TYPE: HP Proliant DL120G6
          CPU TYPE / NUMBER: Xeon X3430
          HOST TYPE: ESXi 4.1
          STORAGE TYPE / DISK NUMBER / RAID LEVEL: SUN Unified Storage 7110 / 14 / RAID6+1 (RAIDZ)
          |*TEST NAME*|*Avg Resp. Time ms*|*Avg IOs/sec*|*Avg MB/sec*|*% cpu load*|
          |*Max Throughput-100%Read*|11.39|5212|162|19%|
          |*RealLife-60%Rand-65%Read*|1.82|2313|18|12%|
          |*Max Throughput-50%Read*|2.16|1496|46|10%|
          |*Random-8k-70%Read*|1.13|2354|18|12%|
          • 392. Re: New  !! Open unofficial storage performance thread
            davidbewernick Novice

            Hi Inges,

             

            did the number ov VMs increased in the last weeks?

            You storage might be ok, but having just one big LUN is mostly not really a good idea. 1 LUN means jsut 1 SCSI stream which can quickly become a bottleneck. Are you seeing any disk wait times? Best way to check them is with ESXTOP / RESXTOP

             

            http://www.yellow-bricks.com/esxtop/

            http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1008205

             

            ------------------------------------

            This table lists the relevant columns and a brief description of these values:

            ColumnDescription
            CMDS/sThis is the number of IOPS (Input/Output Operations Per Second) being sent to or coming from the device or virtual machine being monitored
            DAVG/cmdThis is the average response time in milliseconds per command being sent to the device
            KAVG/cmdThis is the amount of time the command spends in the VMkernel
            GAVG/cmdThis is the response time as it is perceived by the guest operating system. This number is calculated with the formula: DAVG + KAVG = GAVG
            These columns are for both reads and writes, whereas xAVG/rd is for reads and xAVG/wr is for writes. The combined value of these columns is the best way to monitor performance, but high read or write response time it may indicate that the read or write cache is disabled on the array. All arrays perform differently, howeverDAVG/cmd, KAVG/cmd, and GAVG/cmd should not exceed more than 10 milliseconds (ms) for sustained periods of time.
            ------------------------------------

             

             

            Also ESX4.1 is in some points not as good as 5 when it comes to SCSI handling. If you copy a lot of files to the system so that you have to allocate new blocks, you will always have a short scsi reservation from on host and the rest has to wait.

             

            So try to get the Latency Statistics

            • Kernel Average / command (KAVG/cmd)
            • Device Average / command (DAVG/cmd)
            • Guest Average / command (GAVG/cmd)

             

            and Queuing Information

            • Adapter Queue Length (AQLEN)
            • LUN Queue Length (LQLEN)
            • VMKernel (QUED)
            • Active Queue (ACTV)
            • %Used (%USD = ACTV/LQLEN)

             

            for further investigation.

             

            Regards,

            David

            • 393. Re: New  !! Open unofficial storage performance thread
              davidbewernick Novice

              Oh, and can you give some more details about the 7110?

              FC, iSCSI, 1,4 or 8gb?

              Dedublication on?

              Snapshots used?

               

              And don´t forget to check if you might have a LUN misalignment issue...

              • 394. Re: New  !! Open unofficial storage performance thread
                _VR_ Novice

                A CALL FOR HELP

                 

                 

                I've spent a week trying to troubleshoot an issue with a new Equallogic PS4100X. A case has been opened with Dell a week ago. After multiple escalations it has gotten absolutely nowhere. I wanted to see if anyone would be able to add some insight.

                 

                IOMeter test result:

                 

                SERVER TYPE: Windows 2008 R2
                HOST TYPE: DL380 G7, 72GB RAM; 2x XEON E5649 2.53 GHz 6-Core
                SAN Type: Equallogic PS4100X / Disks: 600GB 10k SAS / RAID LEVEL: Raid50 / 22 Disks / iSCSI
                ##################################################################################
                TEST NAME--Av. Resp. Time ms--Av. IOs/sek---Av. MB/sek----
                ##################################################################################
                Max Throughput-100%Read.......______18___..........___3217__........___101____
                RealLife-60%Rand-65%Read..._____13___.........._____3438__........_____27____
                Max Throughput-50%Read.........______19___..........____3199__........___100____
                Random-8k-70%Read................_____13___.........._____3463__........_____27____

                 

                DESCRIPTION OF PROBLEM:

                 

                The PS4100X has a system bottleneck that limits throughput to 100MB/s. When a single host is connected with a single path, eth0 and eth1 on the PS4100x can max out at 1Gbit/s. When there are multiple hosts or multiple paths connected (tested 2 - 8 concurrent paths, 2-6 host nics), the throughput of eth0 and eth1 drop to half of the speed (500Mbit/s). The combined throughput of both ethernet adapters can never exceed 1Gbit/s. Unit has been upgraded to v5.2.1 (latest) firmware.

                 

                SEE TEST RESULTS HERE:

                 

                1. Shows eth1 being maxed out in single path, then the connection switches to multipath
                2. Shows eth0 being maxed out in single path, then the connection switches to multipath
                3. Shows two concurrent tests from two separate test hosts

                 

                RULLING OUT NETWORK ISSUES:

                 

                I'm able to replicate the above problem in the following configurations:
                Test host connected to PS4100X via Cisco 6509
                Test host connected to PS4100X directly via cross over cable (two active iscsi paths setup manually)
                Test host connected to PS4100X via dedicated unmanaged netgear switch
                I can further prove that the Cisco 6509 is functioning properly because I'm able to show speeds of 180MB/s+ speeds to the production PS6000XV and the production PS4000E.

                 

                RULLING OUT HOST ISSUES:

                 

                Tested from a host running Windows 2008 R2 and another host running Windows 2003. Both test hosts encounter the issue described above. Both hosts show speeds of 180MB/s+ when running tests against the two Equallogics in production.

                 

                DEALING WITH DELL-EQUALLOGIC SUPPORT HELL:

                 

                The analyst I'm currently dealing with says the PS4100x is working as expected. He refuses to do any further troubleshooting because some of the blades on the Cisco 6509 have QOS and VOIP. The blade the SAN and test hosts are connected to have no QOS or VOIP configured.
                • 395. Re: New  !! Open unofficial storage performance thread
                  davidbewernick Novice

                  Hi _VR_,

                  I had to deal with a EqualLogic Model with 4x1Gb per controller a while ago. When I did this setup it was important to do a right network setup and RoundRobin in ESX.

                   

                  So can you confirm the following:

                  - seperated vSwitches for ever NIC used for iSCSI?

                  - Jumbo Frames enabled everywhere?

                  - RoundRobin used for path selection?

                  - Testes run on seperated volumes? (-> iSCSI reservations etc...)

                  - Disabled: TCP and IP Offload engines on NICs

                   

                  As far as I know, the Controllers are active/passive. So you said there is a limitation of 100MB in the controller head. When you use 1Gbit/s this ends in about 80-100 MB/s you can use. So I´m not really getting the problem here?

                   

                  Did you see http://www.equallogic.com/WorkArea/DownloadAsset.aspx?id=8453 ?

                   

                  Regards,

                  David

                  • 396. Re: New  !! Open unofficial storage performance thread
                    _VR_ Novice

                    Thanks for the reply

                     

                    - seperated vSwitches for ever NIC used for iSCSI?

                    yes, i have two vSwitches. each one has 1 iSCSI nic. i also ran a test from a physical host (non-esx) with 4 iSCSI nics. same results

                    - Jumbo Frames enabled everywhere?

                    i tried turning jumbo frames on. max throughput test runs 5% faster while the reallife test runs 5% slower

                    - RoundRobin used for path selection?

                    same results in round robin and least queue depth

                    - Testes run on seperated volumes? (-> iSCSI reservations etc...)

                    seperate volumes & separate hosts concurently

                    - Disabled: TCP and IP Offload engines on NICs

                    disabling / enabling offload made no difference

                     

                     

                    As far as I know, the Controllers are active/passive. So you said there is a limitation of 100MB in the controller head. When you use 1Gbit/s this ends in about 80-100 MB/s you can use. So I´m not really getting the problem here?

                    The PS4100X has 2 active nics per controller. The expected throughput is 200MB/s (2000/8=250MB/s Theoretical). I see each NIC pushing 100MB/s one at a time. When they're both active the throughput per nic drops to 50MB/s.

                    • 397. Re: New  !! Open unofficial storage performance thread
                      s1xth Expert
                      VMware Employees

                      First....might want to open a separate thread on this, for continued troubleshooting, we try to keep this thread dedicated for storage performance posts.

                       

                      What type of switches are you using with your EQL setup?

                       

                      What type of NICs on the ESX hosts? Broadcom or Intel?

                       

                      Thanks,

                       

                      Jonathan

                       

                      Sent from my iPad.

                      • 399. Re: New  !! Open unofficial storage performance thread
                        Tomek24VMWARE Enthusiast

                        My result on Broadcom 5709 NIC without jumboFrame - NOT SOFTWARE iscsi only dependend HBA

                         

                        SERVER TYPE: VM Windows 2008 R2 62bit
                        CPU TYPE Intel Xeon X5680 / NUMBER: 6 core
                        HOST TYPE: ESXi 5.0 patch3 Dell R710 and Supermicro + 4xNIC braodcom 5709 offloading without JumboFrame
                        STORAGE TYPE HP P2000 G3 4x10Gb/s / DISK NUMBER: 6x600GB SAS2 15k / RAID LEVEL: RAID10 from 6x600GB SAS2 15k iSCSI

                        |*TEST NAME*|*Avg Resp. Time ms*|*Avg IOs/sec*|*Avg MB/sec*|*% cpu load*|
                        |*Max Throughput-100%Read*|6.57|8844|276|0%|
                        |*RealLife-60%Rand-65%Read*|8.41|4296|33|1%|
                        |*Max Throughput-50%Read*|7.20|8125|253|0%|
                        |*Random-8k-70%Read*|8.04|4353|34|1%|

                        • 400. Re: New  !! Open unofficial storage performance thread
                          abirhasan Hot Shot

                          IOpsMBpsLatency% CPU Utilization
                          Max Throughput-100%Read6774211.78.810.8
                          RealLife-60%Rand-65%Read391730.611.721.3
                          Max Throughput-50%Read6896215.58.710.6
                          Random-8k-70%Read349627.312.523.1

                           

                          Latency  numbers cut in half basically. I read somewhere (but maybe not  in this  forum), to use as many workers in IOMETER as cpus & cores  in the VM,  but I'm guessing based in other numbers posted, that people  are only  using 1 worker

                           

                           

                           

                          abirhasan    http://imagicon.info/cat/5-59/text-smiley.png
                          • 401. Re: New  !! Open unofficial storage performance thread
                            alexxdavid Novice

                            Hi there

                             

                            Needs some advice regarding my setup

                             

                            Starting with the Servers :

                            2 x Dell R710 with quad port each reserved for iSCSI

                            48GB Ram Each

                             

                            San:

                            Dell MD 3220i setup as :

                            2 x Controllers with 2GB Cache each

                            8 x 300GB 10 Krpm in Raid 10 in slot 1-8 owned by controller 0

                            8 x 146GB 15 Krpm in Raid 10 in slot 17-24 owned by controller 1

                            No Hot Spare

                             

                            2 LUNS :

                            1 x 1.1TB ( the 8 x 300GB )

                            1 X 550GB ( the 8 x 146GB )

                             

                            Now as you can see i have space in the san to put another 8 SAS Drive in slot 9-16

                             

                            What would be the better solution from those :

                             

                            1 : Buy 8 X 3000GB and create another lun with 1.1TB and owned it by another controller = 3 Luns total, one controller will own 2 luns

                            2 : Buy 4 X 300GB and 4 X 146GB and expand both the raid 10 already in place  = Same 2 Luns, 1 luns per each controller

                            3 : Buy 4 X 300GB and 4 X 146GB and expand both raid 10 but create 4 Luns, 2 luns per each raid 10, 2 luns per controller

                             

                            Space is not an issue as total data for the business is under 1TB and right now, we still got massive space left, we have around 10 virtual machines.

                             

                            Right now, both luns can achieve ~ 12000 - 13000 iops from iometer 100% read from 4 network card and 250Mbps from 50%read

                             

                            Tell me your thoughts.

                             

                            Regards

                            • 402. Re: New  !! Open unofficial storage performance thread - Nimble Storage
                              dbrinkmann99 Novice

                              OpenPerformanceTest32 against a Nimble CS240, not bad for a bunch of 7.2K drives with some SSD for read cache

                               

                               

                              SERVER TYPE:Windows 2008 R2 VM
                              CPU TYPE / NUMBER: 5620
                              HOST TYPE: Cisco C200 M1
                              STORAGE TYPE / DISK NUMBER / RAID LEVEL: 
                              Nimble CS240 / Hybrid
                              Test name Latency Avg iops Avg MBps cpu load
                              Max Throughput-100%Read15.8937791180%
                              RealLife-60%Rand-65%Read4.44128981001%
                              Max Throughput-50%Read11.5050271570%
                              Random-8k-70%Read3.78153431190%

                               

                               

                              Old style VMTN communities table:

                              SERVER TYPE:Windows 2008 R2 VM
                              CPU TYPE / NUMBER: 5620
                              HOST TYPE: Cisco C200 M1
                              STORAGE TYPE / DISK NUMBER / RAID LEVEL: 
                              Nimble CS240 / Hybrid
                              |*TEST NAME*|*Avg Resp. Time ms*|*Avg IOs/sec*|*Avg MB/sec*|*% cpu load*|
                              |*Max Throughput-100%Read*|15.89|3779|118|0%|
                              |*RealLife-60%Rand-65%Read*|4.44|12898|100|1%|
                              |*Max Throughput-50%Read*|11.50|5027|157|0%|
                              |*Random-8k-70%Read*|3.78|15343|119|0%|
                              
                              • 403. Re: New  !! Open unofficial storage performance thread
                                zgz87 Novice

                                Hi everyone!

                                 

                                I would like to have your opinion about my results. I feel that I am doing something incorrectly since the MBps in the RealLife case and in the Random case are dramatically lower. Can this be possible? Also, is it normal to get the CPU utilization in all the tests equal to 0?

                                 

                                CPU type: Intel Xeon L5638 HC 2GHz (x2)

                                Host type: ESXi 5 / 96 Gb RAM / OpenSUSE 12.1 Kernel 3.1.0

                                Storage type: EMC2 VNX

                                Disk type: Pool of 30 x 15K550GB / RAID 5

                                LUN: 500 GB

                                Interface: FCoE SW driver (from ESXi vmware): ixgbe

                                 


                                              IOps          MBpsLatency% CPU Utilization
                                Max Throughput-100%Read3434107170
                                RealLife-60%Rand-65%Read51641150
                                Max Throughput-50%Read279087210
                                Random-8k-70%Read41631440

                                 

                                To end, I posted some questions about IOmeter in another thread that maybe someone could answer : http://communities.vmware.com/thread/397984

                                 

                                Thanks

                                • 404. Re: New  !! Open unofficial storage performance thread
                                  Ingens Novice

                                  Hi at all,

                                   

                                  You can compare these reports between both storages below;

                                   

                                  EMC VNXe 3100:

                                   

                                  'Test Type,Test Description
                                  0,IO-Test
                                  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
                                  TABLE oF RESULTS
                                  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


                                  SERVER TYPE: VM
                                  CPU TYPE / NUMBER: VCPU / 1
                                  HOST TYPE: CISCO UCS 200 M, 24GB RAM; 6 CPUs x Intel Xeon CPU E5645 2.40 Ghz
                                  STORAGE TYPE / DISK NUMBER / RAID LEVEL: EMC VNXe 3100 / 6 Disks (7.200 RPM) x 868.961 GB / RAID 6


                                  ##################################################################################
                                  TEST NAME--
                                     Av. Resp. Time ms Av. IOs/sek    Av. MB/sek-
                                  ##################################################################################


                                  Max Throughput-100%Read......___21.40____......._2825.00__........._88.28___


                                  RealLife-60%Rand-65%Read..___93.48____.........__562.42__...........__4.39___


                                  Max Throughput-50%Read........___15.76____.......__3766.27__........._117.69___


                                  Random-8k-70%Read...............___118.96____.......__440.12__.........__3.43___


                                  EXCEPTIONS: CPU Util. 32% - 15% - 18% - 15%;

                                  SUN SERIES 7110:


                                  ##################################################################################

                                   

                                  'Test Type,Test Description
                                  0,IO-Test
                                  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
                                  TABLE oF RESULTS
                                  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


                                  SERVER TYPE: VM
                                  CPU TYPE / NUMBER: VCPU / 1
                                  HOST TYPE: HP DL320 G5p, 8GB RAM; 4 CPUs x Intel Xeon X3320 2.50 Ghz
                                  STORAGE TYPE / DISK NUMBER / RAID LEVEL: SUN STORAGE 7110 / 16 Disks (10.000 RPM) x 137 GB / Double Parity


                                  ##################################################################################
                                  TEST NAME--
                                     Av. Resp. Time ms Av. IOs/sek    Av. MB/sek-
                                  ##################################################################################


                                  Max Throughput-100%Read......___21.71____......._2770.693__........._86.58___


                                  RealLife-60%Rand-65%Read..___84.02____.........__699.9__...........__5.47___


                                  Max Throughput-50%Read........___15.56____.......__3735.425__........._116.73___


                                  Random-8k-70%Read...............___77.47____.......__741.82__.........__5.79___


                                  EXCEPTIONS: CPU Util. 32% - 15% - 18% - 15%;


                                  ##################################################################################

                                   

                                  Please, could you provide feedback about the results achived?

                                   

                                  Ingens,

                                  Best regards

                                  1 25 26 27 28 29 Previous Next