1 2 Previous Next 15 Replies Latest reply on Dec 2, 2015 9:52 AM by HariRajan

    Slow Hard Disk I/O while coping large file in Esxi Host

    proSandip Lurker

      Dear All,

       

      I have Cisco UCS C-220 M3L server. I install VMware esxi 5.1 on it. I have VMware 5.1. I have configure RAID 1 and RAID 0 (configure 2 VD) in Raid 0 (VD 0) I install Esxi 5.1 OS and my Virtual machine are store in Raid 1 (VD 1). But now I am feeling serious problem in Coping Files and Deploy OVF files. There is very slow Coping process (takes more than 200 minutes for coping 30 GB VMDK files).

      What could be the problems and how to solved this ? Please kindly help me.

       

      UCS C-220 M3L Specification

      UCS-CPU-E5-2609

      2.4 GHz E5-2609/80W 4C/10MB Cache/DDR3 1066MHz

      2

      UCS-MR-1X082RY-A

      8GB DDR3-1600-MHz RDIMM/PC3-12800/dual rank/1.35v

      4

      UCS-HDD2TI2F213

      2TB SAS 7.2K RPM 3.5 inch HDD/hot plug/drive sled   mounted    

      3

      UCSC-RAID-11-C220

      Cisco UCS RAID SAS 2008M-8i Mezz Card for C220 (0/1/10/5/50)

      1

        • 1. Re: Slow Hard Disk I/O while coping large file in Esxi Host
          RaZaKKaZaR Enthusiast

          Hello proSandip,

           

          Are these copies all to local disk or SAN?

           

          Also, you mentioned that you installed vSphere 5.1 on the C220 M3.

           

          Per the UCS Interoperability Matrix (http://www.cisco.com/web/techdoc/ucs/interoperability/matrix/matrix.html), vSphere 5.1 is not supported on the C-Series servers yet.

           

          It is possible that the drivers included with the 5.1 media are not optimal for your server.  Is it a requirement for you to run 5.1?  Is it possible that you can try installing the Cisco-customized 5.0 update 1 ISO to see if you get better performance?  Of course, if you are already in production, don't proceed with such a step.

           

          -Trevor

           

          Sent from my iPad

          • 2. Re: Slow Hard Disk I/O while coping large file in Esxi Host
            DLA_Mike Novice

            I believe there are issues with the battery on the raid Controller. Without the battery there or functioning properly, you won't get any caching.

            I just saw this issue somewhere in ciscos documentation. Open up a tac case and see what support says.

            1 person found this helpful
            • 3. Re: Slow Hard Disk I/O while coping large file in Esxi Host
              proSandip Lurker

              Dear RaZaKKaZaR

               

              When I install Esxi 5.1 in Cisco UCS 220M3L. There is no any issue related to drivers and others. I make 5 virtual machin in that Esxi Host. now when I try to copy file from my Storage to Esxi host copying process is very slow it show more than 200minutes for 30 GB data copy. then I try to copy file from One virtual machine to another Virtual Machine problem Is same. why this I/O it too slow ?? and How can i solve this problem ?

              It is related to Raid card or HDD

              I have 2TB SAS 7.2K RPM * 2 HDD

              Cisco UCS RAID SAS 2008M-8i Mezz Card

              • 4. Re: Slow Hard Disk I/O while coping large file in Esxi Host
                proSandip Lurker

                Dear fren

                 

                When I install Esxi 5.1 in Cisco UCS 220M3L. There is no any issue related to drivers and others. I make 5 virtual machin in that Esxi Host. now when I try to copy file from my Qnap Storage to  Cisco Esxi host copying process is very slow it show more than 200minutes for 30 GB data copy. then I try to copy file from One virtual machine to another Virtual Machine within Cisco Esxi host (Same Datastore SAS HDD Raid 1) problem Is same. why this I/O it too slow ?? and How can i solve this problem ?

                It is related to Raid card or HDD

                I have 2TB SAS 7.2K RPM * 2 HDD

                Cisco UCS RAID SAS 2008M-8i Mezz Card

                 

                UCS.png

                • 5. Re: Slow Hard Disk I/O while coping large file in Esxi Host
                  RaZaKKaZaR Enthusiast

                  Hello proSandip,

                   

                  ESXi 5.1 will install fine on a C-Series.  However, if the optimal driver versions are not installed, that can cause performance problems.

                   

                  I've experienced this personally for B-Series when implementing vSphere on UCS for customers.

                   

                  Are all of your VMs running locally?  Or are some\all on SAN?  Do you FC\iSCSI\NAS?

                   

                  Any other machines in your environment experiencing slow I/O rates when accessing this same storage?

                  • 6. Re: Slow Hard Disk I/O while coping large file in Esxi Host
                    proSandip Lurker

                    Dear RaZaKKaZaR

                     

                    I make some Virtual machin in Esxi host 5.1 and some are in Qnap Storage (ISCSI) Please provide me Esxi ISO file link that perform best in Cisco UCS 220M3L servers. thank you

                    • 7. Re: Slow Hard Disk I/O while coping large file in Esxi Host
                      RaZaKKaZaR Enthusiast

                      Per the Support Matrix, there isn't a C-Series firmware version that supports vSphere 5.1 yet.

                       

                      Firmware 1.4.7 was just released, but its not on the matrix yet. Perhaps it supports vSphere 5.1?  You will need to verify with Cisco TAC.

                       

                      If firmware 1.4.7 doesn't support vSphere 5.1 either, see if they can give you an approximate timeframe for the release of a firmware that does.

                       

                      Other items to check in your environment:

                      -what are the speeds of your switch and array ports?

                      -Network switch and Qnap firmware? Any recommendations from manufacturers to update?

                      -Regarding copies between VMs, are your VMs using the VMXNET 3 network adapter type instead of the default e1000?  If not, replace your VMs' NICs with VMXNET 3 network cards, and make sure VMware Tools are installed in the OS.  VMXNET3 can improve network performance for VMs.

                      • 8. Re: Slow Hard Disk I/O while coping large file in Esxi Host
                        cmanucy Hot Shot

                        I would just like to add some information to this thread in the hopes that it helps someone else...

                         

                        I have two of these servers that I've been bringing up to speed to get ready for production.  I also noticed very poor I/O when it came to iSCSI connections compared to other servers using the same infrastructure. 

                         

                        I currently have open tickets with both Cisco and VMWare on the issue.  I believe it to be some kind of low-level driver issue - Cisco does now support ESX 5.1 with the new 1.4(7) firmware, at least according to the tech I've spoken with.

                         

                        Just as a point of reference, I'm unable to get much more than about 40MB/sec using a 100% sequential read test with IOMeter.  Using the same storage and network gear, I'm able to get 110MB/sec from an older UCS 200 M2 server.  Random IO tests are equally dismal - about 1/10th the IO I expect.

                         

                        I also have the 10G VIC's in these servers, and it doesn't matter.  Still only get the same dismal IO over a 10G connection as I do from a single iSCSI using the built-in Intel 350's.

                         

                        I have also verified this same performance issue with another farm's setup, using a completely different SAN and network gear.  I'll try and keep this thread updated as I go since I believe this to be a fixable bug.

                        • 9. Re: Slow Hard Disk I/O while coping large file in Esxi Host
                          yelbaglff Novice

                          Out of curiosity, did either of you install using the Cisco custom ESXi 5.1 ISO?

                           

                          https://communities.cisco.com/message/110145

                           

                           

                          Also, verify your adapter firmware is at a compatible version.  SSH to your host and run these commands.

                           

                          esxcfg-nics -l

                          ethtool -i 'vmnic from previous output'

                           

                           

                          Verify compatibility here:

                           

                          http://www.cisco.com/web/techdoc/ucs/interoperability/matrix/matrix.html

                          • 10. Re: Slow Hard Disk I/O while coping large file in Esxi Host
                            cmanucy Hot Shot

                            No I did not try the Cisco custom ISO, but I could.  I have a feeling I have some time before this is fixed.

                             

                            Just for everyone's info, here's my setup:

                             

                             

                            ~ # esxcfg-nics -l
                            Name    PCI           Driver      Link Speed     Duplex MAC Address       MTU    Description
                            vmnic0  0000:01:00.00 igb         Up   1000Mbps  Full   a4:93:4c:aa:47:4a 1500   Intel Corporation I350 Gigabit Network Connection
                            vmnic1  0000:01:00.01 igb         Up   1000Mbps  Full   a4:93:4c:aa:47:4b 1500   Intel Corporation I350 Gigabit Network Connection
                            vmnic2  0000:0b:00.00 enic        Up   10000Mbps Full   a4:4c:11:13:b0:a8 1500   Cisco Systems Inc Cisco VIC Ethernet NIC
                            vmnic3  0000:0c:00.00 enic        Up   10000Mbps Full   a4:4c:11:13:b0:a9 1500   Cisco Systems Inc Cisco VIC Ethernet NIC
                            ~ # ethtool -i vmnic0
                            driver: igb
                            version: 2.1.11.1
                            firmware-version: 1.6-1
                            bus-info: 0000:01:00.0
                            ~ # ethtool -i vmnic1
                            driver: igb
                            version: 2.1.11.1
                            firmware-version: 1.6-1
                            bus-info: 0000:01:00.1
                            ~ # ethtool -i vmnic2
                            driver: enic
                            version: 2.1.2.38
                            firmware-version: 2.1(1a)
                            bus-info: 0000:0b:00.0
                            ~ # ethtool -i vmnic3
                            driver: enic
                            version: 2.1.2.38
                            firmware-version: 2.1(1a)
                            bus-info: 0000:0c:00.0

                             

                             

                             

                             

                            I do note that Cisco has a higher driver requirement for the Intel 350's....

                             

                            Manufacturer     = Intel

                             

                            Adapter Driver   = 3.2.10

                            Adapter Firmware = 1.6-1

                            Boot Code / BIOS = v1.3.98

                            I might try that custom ISO and see if it makes any difference on one of the hosts.  I'll report back on my findings.

                            • 11. Re: Slow Hard Disk I/O while coping large file in Esxi Host
                              cditmo Novice

                              I am working on-site ih a customer data center today and have limited access to the phone and email. I will respond to emails when I'm back in the office.

                               

                              If you need immediate assistance please call the CDI office at 704-716-0010.

                               

                              Jonathan

                              • 12. Re: Slow Hard Disk I/O while coping large file in Esxi Host
                                cmanucy Hot Shot

                                So I took the VMWare / Cisco image and installed that on another UCS 220 M3 host I had.  Drivers ARE different vs. what I'd installed with the help of the Cisco engineer:

                                 

                                 

                                ~ # esxcfg-nics -l
                                Name    PCI           Driver      Link Speed     Duplex MAC Address       MTU    Description
                                vmnic0  0000:01:00.00 igb         Up   1000Mbps  Full   a4:4c:11:2a:2d:4e 1500   Intel Corporation I350 Gigabit Network Connection
                                vmnic1  0000:01:00.01 igb         Down 0Mbps     Half   a4:4c:11:2a:2d:4f 1500   Intel Corporation I350 Gigabit Network Connection
                                vmnic2  0000:0b:00.00 enic        Up   10000Mbps Full   a4:4c:11:13:c8:60 1500   Cisco Systems Inc Cisco VIC Ethernet NIC
                                vmnic3  0000:0c:00.00 enic        Up   10000Mbps Full   a4:4c:11:13:c8:61 1500   Cisco Systems Inc Cisco VIC Ethernet NIC
                                ~ # ethtool -i vmnic0
                                driver: igb
                                version: 3.2.10
                                firmware-version: 1.6-1
                                bus-info: 0000:01:00.0
                                ~ # ethtool -i vmnic1
                                driver: igb
                                version: 3.2.10
                                firmware-version: 1.6-1
                                bus-info: 0000:01:00.1
                                ~ # ethtool -i vmnic2
                                driver: enic
                                version: 2.1.2.22
                                firmware-version: 2.1(1a)
                                bus-info: 0000:0b:00.0
                                ~ # ethtool -i vmnic3
                                driver: enic
                                version: 2.1.2.22
                                firmware-version: 2.1(1a)
                                bus-info: 0000:0c:00.0
                                However, even with the updated drivers on the gig nics, it made no difference in the IO on the box... so the problem still exists.  I'm going to go update my tickets with VMWare and Cisco...
                                • 13. Re: Slow Hard Disk I/O while coping large file in Esxi Host
                                  cmanucy Hot Shot

                                  Just as an update to anyone reading this thread.... this problem has been CONFIRMED by Cisco.  They were able to replicate the poor performance when attaching to a NetApp iSCSI datastore.

                                   

                                  They have also opened a ticket with VMWare.

                                   

                                  Curiously, they found when they install CentOS 6.x on the UCS (bare metal), they did not experience the IO issues with the SAN.  I have not had a chance to try and duplicate this test scenario with my own environment...

                                  • 14. Re: Slow Hard Disk I/O while coping large file in Esxi Host
                                    gleunis Lurker

                                    Hi,

                                     

                                    Was there ever found a solution? We have a similar problem and can't find any reason why it'ts that slow.

                                     

                                    Kind regards

                                    1 2 Previous Next