1 2 3 4 Previous Next 56 Replies Latest reply on Mar 24, 2014 9:25 AM by dtwilley Go to original post
      • 45. Re: Equallogic Performance Issues
        alex555550 Enthusiast

        Hi all,

         

        with the new Equallogic firmware 6.0.1 it seems better. Also Cisco release a IOS Update 15.0(2)SE, that solves the Flow Control issue.

        • 46. Re: Equallogic Performance Issues
          alex.wu Novice

          hello,

           

          have you resolved the issue about flowcontrol?

           

          I met the same issue.. my cisco ISO is 12.2(58)se2  cisco 2960s switch

           

          Switch# show flowcontrol interface gigabitEthernet g1/0/21
          

          #

          Port       Send FlowControl  Receive FlowControl  RxPause TxPause
          

          #

                     admin    oper     admin    oper
          

          #

          ---------  -------- -------- -------- --------    ------- -------
          

          #

          Gi1/0/21   Unsupp.  Unsupp.  desired  off         0       0
          

           

          the port 21 is connected with EQL, but the "oper" under receive flowcontrol is always "OFF".

          • 47. Re: Equallogic Performance Issues
            chimera Novice

            Re: Storage Heartbeat VMK port.  (Mentioned in EQL iSCSI config guides for ESX)

             

            This will never affect performance. 

             

            Saying it will "never" affect performance is incorrect.  Without storage heartbeat, performance still can be affected.  If storage heartbeat is not setup and a NIC (with the lowest vmk mapping) fails, then "the Equallogic will not be able to accurately determine connectivity during the login process, and therefore suboptimal placement of iSCSI sessions will occur". If a storage heartbeat vmk is setup, then it gets tied to both pNIC's and so if a NIC fails, it uses the 2nd NIC to determine optimal placement of iSCSI sessions. So the heartbeat vmk in itself won't direct affect performance, but to say it will never affect performance is incorrect

             

             

            • 48. Re: Equallogic Performance Issues
              dwilliam62 Enthusiast

              Sorry that I wasn't clear.

               

              Under normal conditions the SHB VMK port has no impact. Since no iSCSI traffic passes through it.  However, a network outage isn't what I would consider a "performance" problem.  Since w/o SHB you can end up with a total outage, not just degraded performance.  I've seen it happen.   That's what I was referring to. 

               

              FYI: This issue is resolved in ESXi v5.1.  So you no longer need the SHB VMK port.

              • 49. Re: Equallogic Performance Issues
                Josh Enthusiast

                Sorry to as the dumb questions but I just want to clarify what the problem is.

                 

                You  have an array with two 1GB links connected on the active controller,  your hosts have multiple 1GB ( how many?) links connected to the  switching.

                 

                You are seeing a 100MB/s limit on IOMeter running within a single VM.

                 

                You say for the array "The combined throughput of both ethernet adapters can never exceed 1Gbit/s"

                 

                So for a single 1 gigabit link the maximum theoritical throughput is 108 megabytes per second.

                 

                If you have two separate VM's running on two separate hosts running I/O meter you are seeing 50MB/s throughput on each?

                VCIX6 - NV | VCAP5 - DCA / DCD / CID | vExpert 2014,2015,2016 | http://www.vcrumbs.com - My Virtualization Blog!
                • 50. Re: Equallogic Performance Issues
                  _VR_ Novice
                  >> If you have two separate VM's running on two separate hosts running I/O meter you are seeing 50MB/s throughput on each?
                  Correct.
                  Here is what it looks like on the EQL side:
                  eth0 enabled, eth1 disabled = 100 MB/s
                  eth0 disabled, eth1 enabled = 100 MB/s
                  eth0 enabled, eth1 enabled = 100MB/s (50MB/s on eth0, 50MB/s on eth1)
                  It doesnt matter how many hosts are running concurrent tests or how many nics each host has or how many paths are setup. The results are always the same.
                  • 51. Re: Equallogic Performance Issues
                    dwilliam62 Enthusiast

                    Do you still have a support case open with Dell?

                     

                    Don

                    • 52. Re: Equallogic Performance Issues
                      Kaeon Lurker

                      Hi KetchAdmin,

                       

                      I would very much like to know what settings you used to get your Equallogic speed up. Also, what tool do you use to get your performance numbers. We are using Equallogic storage with vsphere 5.1 and MEM configured.

                       

                      BR

                      • 53. Re: Equallogic Performance Issues
                        JNixERG Lurker

                        I am out of the office until Monday 4/22 .  If your request is urgent, please resend your message with URGENT in the subject and I will get back to you as quickly as possible.

                         

                        Steve Pearce

                        • 54. Re: Equallogic Performance Issues
                          dtwilley Lurker

                          I know its been a while but I found this while trying to diagnose a fault of our own.

                           

                          Environment

                          PS4100X (24x600Gb SAS), 2 x Cisco 3560X (12.2 (55), 3 x HP DL360p (2 x QP Broadcom 5719 NICS), VMWare 5.1 U2 (HP custom image).

                           

                          Problem

                          We see 50MB/s in total from the Equallogic regardless of how many hosts are used and how many NICs, if we test from two hosts the speed halves, if we test from two NICS we see 25MB/s on each NIC or 50MB/s if one NIC is tested.,

                           

                          I followed advice from this post as well as following all the usual best practice guides.

                          • Delayed ACK Disabled
                          • IOPS threshold changed from 1000 to 3
                          • Login Timeout changed to 60
                          • Equallogic Firmware updated from 6.0.2 to 6.0.4 then eventually to 7.0.2
                          • Ran the Hard drive firmware upgrade utility
                          • Broadcom TG3 drivers upgraded from 3.12.x to 3.13.x (both 50.1 and 50.2)
                          • Firmware upgraded on the hosts using the built in HP Intelligent Provisioning Utility
                          • Cisco config has flow control set to desired and confirmed as working on all iSCSI ports, portfast, confirmed disabled storm control, VLAN separation, again, as per guides

                           

                          I logged a call with Dell who wanted Diags and SANHQ output, they said the SAN was fine.  Logged a ticket with VMWare and they looked at DAVG in ESXTOP and found that this was showing values of 100-1000 during IO operations so VMWare pointed me at the switches as they said it wasn't VMWare and Dell had already said it wasn't the SAN.

                           

                          I also found millions of oversize packets on iSCSI ports, its almost as if the Equallogic isn't correctly discovering path MTU and the Cisco was having to fragment packets before transmitting them to the hosts, maybe inducing the lag.

                           

                          Checked all the switch config, checked flow control was operational on the ports set to desired (as per this post), thanks

                           

                          Eventually I went to the Datacentre and disconnected all the cables to the SAN, plugged in a laptop and ran IOMeter and did some basic file copies between SAN volumes and again could only get 50MB/s.

                           

                          I got Dell on a remote session via the wireless on the laptop so they could see it directly and they've agreed to send out a specialist engineer Monday morning so we'll see what happens next.  They're also sending some spares but as yet I don't know what they are.

                           

                          This whole process started on the Monday and it took me until Friday to convince Dell of the problem.

                           

                          I'll update this post when we know more in case anyone else has to go through the same hell I have.

                           

                          Dave Twilley VCP5

                          Datek Solutions Ltd.

                          • 55. Re: Equallogic Performance Issues
                            RamzyMasarweh Lurker


                            Thanks A lot Dave .. please keep us posted

                            regards,

                            Ramzy

                            • 56. Re: Equallogic Performance Issues
                              dtwilley Lurker

                              So I met the Dell Skytech guy at the datacentre this morning, his laptop gets 100MB/s either direct to the SAN or via the switches so looks like my laptop test was just an anomaly!  He spoke to some guys at base and they said try an earlier TG3 driver, that made no difference but its interesting that the one on record as testing/working with EQL is the 3.12xx driver which is known to corrupt datastores!

                               

                              Anyway, we're now escalating it with Dell as a Broadcom NIC issue as we have nowhere left to go.  I have an Intel QP NIC on standby.

                               

                              In the meantime I'm sticking VSphere 5.5 on to the host and will see what happens.  What's annoying is we see 100-200MB/s on the NICs when doing vMotion, so this is either limited to just iSCSI or its not a Broadcom issue.

                              1 2 3 4 Previous Next