1 2 3 Previous Next 31 Replies Latest reply on Jul 9, 2009 10:23 AM by chethank

    I/O performance of vSphere

    Novice

       

      I am a senior performance engineer at VMware. I have spent quite a bit of time on performance analysis of ESX storage stack starting from ESX 3.0. We at VMware are constantly working on improving I/O performance of ESX. I have done few experiments to drive extremely high I/O load on single instance of ESX. With 3.5, I obtained 100,000 I/OPs with an I/O load that is most representative of real applications (100,000 I/Ops) until I ran out of hardware. The performance envelope was pushed furthe with vSphere when we achieved 350,000 I/Ops in an experiment done at EMC labs. I wrote a blog highlighting the results with some details on the experiments in VMware's performance blog VROOM.

       

       

      Having a well configured I/O system is critical for good application performance. Very often I hear questions from customers on storage performance, choice of virtual disk format - VMFS vs RDM, best practices etc., The answer is simple - follow the best practices that you would normally follow in native world when designing an I/O infrastructure for your application. ESX provides excellent I/O performance and can support even extreme I/O demands from applications as the results discussed in the blogs indicate. VMFS or RDM - you can expect similar performance though RDM can help during certain scenarios which are purely non-performance related.

       

       

      We can discuss more in this thread. Feel free to post your questions, comments on the blogs or any I/O related issues on this thread. I will try my best to respond. May be some one who has already faced a similar situation will jump in with a solution even we at VMware wouldn't have thought of!

       

       

      Chethan

       

       

        • 1. Re: I/O performance of vSphere
          rDale Enthusiast

           

          I have a question in order to get tose kinds of IOPS did you do any disk alignment with the vmfs vol or the vmdk files ?

           

           

          I often hear alot of blah about disk alignment being needed for SQL servers and Exchnage servers.

           

           

          what block size did you work with on the ntfs partions ?

           

           

          • 2. Re: I/O performance of vSphere
            Gabrie Master
            vExpert

             

            Q1:   A while ago I saw a discussion on VMFS blocksize. I learned that VMFS blocksize doesnot matter when Windows Guest is reading for example a 64kb block from vmdk on vmfs. So my question is, when does the blocksize matter if it has no influence on performance and why should I choose between 1 or 8MB blocksize?

             

             

             

            Q2:  What is the best way to track performance from Guest disk through Guest SCSI controller through Host HBA Queue depth to LUN Queue depth? Only vScsiStats and esxtop? Or other tools aswell?

             

             

             

             

             

            http://www.GabesVirtualWorld.com

             

             

            • 3. Re: I/O performance of vSphere
              Novice

               

              The virtual disks were created using the VI client, which automatically aligns the vmdk files. As far as guest OS is concerned, no file system was created on these virtual disks. For the guest OS (and iometer) they appeared as raw disks.

               

               

              But, if you were to create a file system for a particular application, my suggestion would be to go with the best practices suggested by Microsoft for particular applications. For e.g., MS recommends 64K NTFS block size for SQL server files.

               

               

              Hope that helps.

               

               

              • 4. Re: I/O performance of vSphere
                Novice

                 

                You choose different block sizes depending on the size of the largest file (vmdks) on the VMFS datastore. For e.g., with 1MB block size, maximum size for a single vmdk is 256GB. The limit increases as you increase the block size.

                 

                 

                vScsiStats and esxtop are very popular. You can also use tools provided by storage vendors (such as Navisphere Analyzer from EMC) to complement esxtop data. 

                 

                 

                • 5. Re: I/O performance of vSphere
                  Gabrie Master
                  vExpert

                   

                  chethank wrote:

                  You choose different block sizes depending on the size of the largest file (vmdks) on the VMFS datastore. For e.g., with 1MB block size, maximum size for a single vmdk is 256GB. The limit increases as you increase the block size.

                   

                  vScsiStats and esxtop are very popular. You can also use tools provided by storage vendors (such as Navisphere Analyzer from EMC) to complement esxtop data. 

                   

                   

                  I understand about the blocksize and the 256GB limit for a VMDK on a 1MB blocksize VMFS. But why shouldn't I just create only 8MB blocked VMFS in the future? Why is it better to create 1MB block size VMFS when you don't need VMDK bigger then 256GB?

                   

                   

                   

                   

                   

                   

                   

                  http://www.GabesVirtualWorld.com

                   

                   

                  • 6. Re: I/O performance of vSphere
                    Novice

                     

                    There is no reason why you shouldn't use 8MB block size instead of 1MB. In case of 'thin' vmdk format unit of space allocation is the minimum block size used for VMFS. As the file grows space is allocated in chunks of the chosen block size. If the application has periods of bursty writes (which causes file growth) followed by period of inactivity, this may lead to some wastage in space. This may not ber very significant compared to the acual file size, but just something to keep in mind.

                     

                     

                    • 8. Re: I/O performance of vSphere
                      jasonboche Champion
                      vExpert
                      chethank wrote:

                       

                      The virtual disks were created using the VI client, which automatically aligns the vmdk files.

                       

                      Not in my experience.

                      The VI Client will align VMFS volumes at creation time, but not the guest OS in the .vmdk.

                      I've been aligning a lot of .vmdk's lately that were created with VI3.

                      Would like clarification here.

                       

                      As far as guest OS is concerned, no file system was created on these virtual disks. For the guest OS (and iometer) they appeared as raw disks.

                       

                       

                      But, if you were to create a file system for a particular application, my suggestion would be to go with the best practices suggested by Microsoft for particular applications. For e.g., MS recommends 64K NTFS block size for SQL server files.

                       

                       

                      Hope that helps.

                       

                       






                      [i]Jason Boche, vExpert[/i]

                      [boche.net - VMware Virtualization Evangelist|http://www.boche.net/blog/][/i]

                      [VMware Communities User Moderator|http://www.vmware.com/communities/content/community_terms/][/i]

                      [Minneapolis Area VMware User Group Leader|http://communities.vmware.com/community/vmug/us-central/minneapolis][/i]

                      • 9. Re: I/O performance of vSphere
                        depping Champion
                        VMware EmployeesUser Moderators

                         

                        I wrote an article about it a while ago. Especially with the new VMFS Grow feature it makes sense to always use 8MB Blocks. If you do you will always have the option to create the largest file possible. And indeed for Thin Provisioning you will grow in increments of 8MB vs 1MB which should lead to less locking and less growing. The overhead should be minumum. (max = 7MB x total amount of VMs)

                         

                         

                         

                        http://www.yellow-bricks.com/2009/05/14/block-sizes-and-growing-your-vmfs/

                         

                         

                         

                        Duncan

                        VMware Communities User Moderator | VCP | VCDX

                        -


                        Blogging: http://www.yellow-bricks.com

                        Twitter: http://www.twitter.com/depping

                         

                        If you find this information useful, please award points for "correct" or "helpful".

                         

                         

                        • 10. Re: I/O performance of vSphere
                          dconvery Virtuoso
                          vExpert

                           

                          Chethank -

                           

                           

                          I would like a clarification on alignment as well. What I am hearing you say is  to create VMFS in the VIC and it would be automatically aligned (I already knew that part). I am not sure I understand if the guest needs anything special to aling from default setting. Are you saying that the guest will be aligned as well? or is there something special I need to do within the guest?

                           

                           

                           

                           

                          Dave Convery

                          VMware vExpert 2009

                          http://www.dailyhypervisor.com

                          http://communities.vmware.com/servlet/JiveServlet/download/38-20623/vExpert_logo_100x57.jpg

                          Careful. We don't want to learn from this.

                          Bill Watterson, "Calvin and Hobbes"

                           

                           

                          • 11. Re: I/O performance of vSphere
                            depping Champion
                            VMware EmployeesUser Moderators

                            I think he only stated that he did not need to allign the additional disk cause IOMeter would access the raw disk without it being formatted with a file system?

                             

                            When a it's formatted with a filesystem it will need to be alligned depending on the filesystem used.

                             

                             

                            Duncan

                            VMware Communities User Moderator | VCP | VCDX

                            -


                            Blogging: http://www.yellow-bricks.com

                            Twitter: http://www.twitter.com/depping

                             

                            If you find this information useful, please award points for "correct" or "helpful".

                            • 12. Re: I/O performance of vSphere
                              IRivera Novice

                               

                              I think you'll find this article very helpful on aligning Windows .vmdk

                               

                               

                              http://www.vmware.com/pdf/esx3_partition_align.pdf

                               

                               

                              • 13. Re: I/O performance of vSphere
                                dconvery Virtuoso
                                vExpert

                                 

                                Yeah, I think I was reading into Jason's questions too much. I use the steps outlined in the partition alignment whitepaper for the guests right now.

                                 

                                 

                                Let me ask this: Is there an easy way to align the C: drive on a windows system? I have gone through the process using a helper VM to partition and format the VMDK and then install the OS on the newly-formatted disk, but there must be a better way. Is it really necessary?

                                 

                                 

                                 

                                 

                                 

                                Dave Convery

                                VMware vExpert 2009

                                http://www.dailyhypervisor.com

                                http://communities.vmware.com/servlet/JiveServlet/download/38-20623/vExpert_logo_100x57.jpg

                                Careful. We don't want to learn from this.

                                Bill Watterson, "Calvin and Hobbes"

                                 

                                 

                                • 14. Re: I/O performance of vSphere
                                  Novice

                                   

                                  CLARIFICATION: As some of the readers have pointed out, creation of a VMFS datastore from VI client automatically aligns VMFS on a LUN. And yes, this is what I did too. But this doesn't align a file system created on a virtual disk (.vmdk) in the guest OS. That has to be done manually. This document has some helpful information on alignment: http://www.vmware.com/pdf/esx3_partition_align.pdf . Sorry for the confusion in my earlier response.

                                   

                                   

                                  >> As far as guest OS is concerned, no file system was created on these virtual disks. For the guest OS (and iometer) they appeared as raw disks.

                                   

                                   

                                  In our case, since no file system was created on the virtual disk in the guest OS, the virtual disks didn't require any alignment.

                                   

                                   

                                  Hope the clarification helps.

                                   

                                   

                                  1 2 3 Previous Next