1 2 Previous Next 23 Replies Latest reply on Nov 16, 2008 4:11 PM by Jae Ellers Go to original post
      • 15. Re: ESX over NFS or iSCSI
        mldmld Hot Shot

        Thanks Dalepa,

         

        6509 is a Cisco switch ?

         

         

         

        Which FAS do you use ?

         

         

         

        So you have 2 vif of 4 ports. Do you mean your filer has a 2 levels VIF ?

         

         

         

        I mean you created 2 multi VIF of 4 ethernet port, 4 ethernet ports connected to each Cisco switches

         

         

         

        Then you agregate it in one Vif in single mode ?

         

         

         

        But you saytoo you configure  4 active / 3 passive / head ?

         

         

        On the other hand, what is the network configuration on the ESX side ?

         

         

         

         

         

         

        I'm very surprised by the number of disks you put online for an agregate. With such many disk, the IO throughput is the higher one can get. But is this agreate reliable ? I mean, with so many disks, did you experience volume crash, even with RAID DP feature ? How long theses agregate exists ?

         

         

         

        In my company, we create agregate of 13 disks. Do you think it could be cause a performance issue for ESX/NFS ?

         

         

         

        Sorry to ask so many questions, but I would like to avoid pitfalls !

         

         

         

        Best regards

         

         

         

        ML

        • 16. Re: ESX over NFS or iSCSI
          dalepa Hot Shot
          mldmld wrote:

          Thanks Dalepa,

           

          6509 is a Cisco switch ?   yep

           

           

          Which FAS do you use ?  fas3070 R200 and fas3050

           

           

           

          So you have 2 vif of 4 ports. Do you mean your filer has a 2 levels VIF ?   yep a trunk and two vifs

           

           

           

          I mean you created 2 multi VIF of 4 ethernet port, 4 ethernet ports connected to each Cisco switches  yep

           

           

           

          Then you agregate it in one Vif in single mode ?  yep

           

           

           

          But you saytoo you configure 4 active / 3 passive / head ?   one vif is standby, the other is prefered

           

           

          On the other hand, what is the network configuration on the ESX side ?   see photo above

           

           

           

           

           

           

           

           

          I'm very surprised by the number of disks you put online for an agregate. With such many disk, the IO throughput is the higher one can get. But is this agreate reliable ? I mean, with so many disks, did you experience volume crash, even with RAID DP feature ? How long theses agregate exists ?

           

           

          we use a raid size of 28.   we have 24 clustered filers across north america   0 data loss over 10 years.

           

           

           

           

           

           

          In my company, we create agregate of 13 disks. Do you think it could be cause a performance issue for ESX/NFS ?

           

           

           

           

           

          yes maybe.  the more disk you have the better performance.

           

           

           

          Sorry to ask so many questions, but I would like to avoid pitfalls !

           

           

           

           

           

          We have done several conference calls hosted by Netapp with several fortune 500 companies on this topic...

           

           

          Send a private email here and I'll put you in contact with our Netapp Rep if you are interested.

           

           

           

          Best regards

           

           

           

          ML

           

          1 person found this helpful
          • 17. Re: ESX over NFS or iSCSI
            mldmld Hot Shot

             

            Hi,

             

             

            I've just read http://www.netapp.com/go/techontap/matl/downloads/NAS-presentation.pdf . I joined to my entry.

             

             

            They state to take care about NFS and single link usage on slide 13.

             

             

            It is said : "NFS traffic uses singleVMKernel IP address per vSwitch / set of team NICsThis means that a single uplinkwill be selected for ALL traffic with:

             

             

            •Load balancing based only on source IP/MAC/Port

             

             

            •Load balancing with source IP and a SINGLE DESTINATION IP

             

             

            •Single datastore"

             

             

             

             

             

            Does it means that NFS traffic will only uses one uplink whatever the configuration selected on the NIC teaming on the ESX server Storage Vmkernel; if only one IP adress is used on the storage side, even with a VIF ?

             

             

            To avoid this : does it means that I should

             

             

            - on the ESX side, select a Load Balancing policy and the same on the Cisco switches

             

             

            - on the NetApp side, declare the same number of IP adresses as the number of active-active trunking NICs. So 2 IP adresses for 2 x 2 NICs in a two level VIF like in the document 's example.

             

             

            - Create a DNS entry for each IP adresses.(like nfs1.toaster.company.com, nfs2.toaster.company.com)

             

             

            - Then when connecting the datastores on an ESX, I should use the nfs1.toaster.company.com name and nfs2.toaster.company.com alternatively ?

             

             

            -  Spread VM on all volume

             

             

            Thanks

             

             

            ML

             

             

            • 18. Re: ESX over NFS or iSCSI
              WillemB Hot Shot

               

              Urgrue, I've noticed you have extreme problems on iSCSI which is strange since all is well at my company (>100 VM's and growing without problems). Sounds to me like a network communications problem. I can think of a few possibilities (probably won't be any of them but we're here to 'try' and help ).

               

               

              1. It sounds like a flopping duplex setting. You've probably checked it out but if not set it all to fixed 1000Mbps speed settings. Cisco switches are known to do flip-flops on auto-negotiation.

               

               

              2. I've heard people having different MTU settings between iSCSI NAS and the host.

               

               

              3. Latency could be too big for iSCSI to handle. Can result in unexplainable slowdown.

               

               

              4. I've heard of people experiencing unexplainable slowdowns with iSCSI. Somebody noted that for them the cause was due to a slow part of network. Both sides we're connected at 1000Mbps but somewhere in the middle there was a piece of network machinery running at 100Mbps.

               

               

              The limiting 100Mbps piece of network wasn't a bandwidth bottleneck but the speed transaction was very problematic for the iSCSI connection.

               

               

               

               

               

              To continue on the discussion here's someone who has tested FC,iSCSI and NFS. There are positive comments towards both iSCSI and NFS. The ending argument is very interesting.

               

               

              http://blog.scottlowe.org/2007/09/21/nfs-for-vmware-storage/

               

               

              Quote "Hi,I made last year a test to compare FC, iSCSI and NFS using iozone on a physical Linux RH 4.

              The storage was a NetApp FAS3020, and the server a HP BL20p.

              FC & iSCSI where very close

              NFS was really slower."

               

               

              And another quote "FC is always faster with 1 or 2 ESX hosts...however the more ESX host you add the faster NFS performs. This is because of FC SCSI reservations in ESX. Ideally only one host can read or write to a LUN at a time with FCP. With NFS this is not a limitation. Hence the more hosts, the better performace on NFS then presenting a number of LUNs to several hosts on FC...."

               

               

              • 19. Re: ESX over NFS or iSCSI
                Hot Shot

                 

                Dalepa,

                 

                 

                 

                 

                 

                I am looking for reference companies using VMware over NFS to NetApp storage. Is it OK if I contact you offline ?

                 

                 

                • 20. Re: ESX over NFS or iSCSI
                  kcollo Enthusiast

                  I have just started using ASIS on a VMware ISCSI volume.  We do thin provisioning as well on the NAS. Here is a link to a blog that I put up and will be updating

                   

                  http://blog.colovirt.com/2008/10/23/netapp-deduplication-a-sis-and-vmware/

                   

                   

                  Asis has been running on the volume for a few days and there are only 9 virtual machines there.  I will be moving over a lot more in the next few weeks.  The current compression is as follows:

                   

                   

                  Filesystem        used        saved        %saved

                  /vol/vmware/     96GB      99GB       51%

                   

                   

                  kevin@colovirt.com

                   

                   

                  • 21. Re: ESX over NFS or iSCSI
                    kcollo Enthusiast

                     

                    Got an update.  Getting better deduplication.  Still moving over development and non critical VMs to the sis volume.  This is an ISCSI LUN formatted VMFS.  

                     

                     

                    NAS> df -sh /vol/testVol

                     

                    Filesystem

                    used

                    saved

                    %saved

                    /vol/testVol/

                    519GB

                    754GB

                    59%

                     

                     

                     

                     

                    Saving 749 gig so far!  ASIS (NetApp dedipe) setup information can be found here:

                     

                    http://blog.colovirt.com/2008/11/14/netapp-enabling-deduplication-asis-on-a-volume/

                    • 22. Re: ESX over NFS or iSCSI
                      mike.laspina Virtuoso

                       

                      Hi,

                       

                       

                      I'm getting exceptional performance with iSCSI.

                       

                       

                      http://blog.laspina.ca/roller/Ubiquitous/entry/x4500_zfs_and_iscsi_performance

                       

                       

                       

                       

                       

                       

                       

                      http://blog.laspina.ca/

                       

                       

                      • 23. Re: ESX over NFS or iSCSI
                        Jae Ellers Master

                         

                        If you use iSCSI with hardware initiators you can manually load balance to use 2 links at once on the ESX side.

                         

                         

                        ESX runs great on blades.   You just need the right blades.  I'd love to hear arguments to the contrary.

                         

                         

                        1 2 Previous Next