12 Replies Latest reply on Sep 10, 2006 3:09 PM by DougBaer

    Proper vmfs3 sizing and best practice for ESX 3.0

    dwchan Expert

      With some of the recent changes to ESX filesystem (vmfs3), can anyone comment on the following.

       

      1.  in the past, all virtual machines share the same swap file on the esx host.  But with VI3, from what I understand, the virtual machine swap file reside on the same folder/area as the virtual machine vmdk.  So if i create a dedicate LUN for VM1 (let say 2 CPU, 4 G of RAM, 80G of disk), how much space I make the LUN?  Basically, how much extra space should I add for the SWAP?

       

      2.  For best practice, is there a guideline on how many padded/free space i should leave empty on each VMFS3 LUN.  You know how in Windows world, it is ideal to leave 18% free space on the drive for best performance.  For wmfs3, is it a bad idea if i use up all the space on the LUN for VM, or is it a good idea to leave some percent free to increase performance?

       

      3.  Lastly, does defrag program like diskeeper have any performance benefit inside a virtual machine?

       

      dwc

        • 1. Re: Proper vmfs3 sizing and best practice for ESX 3.0
          Quotient Expert

          Thought I best get in before this gets swamped...

           

          To answer in turn, IMO:

           

          1. For the best flexibility, allow enough space for all your vRAM + enough space for a snapshot allowance if you are not using independent vDisks. 

           

          The snapshot allowance could be different for various VMs.  It’s best to classify them ASAP according to workload and snapshot lifetime and then present VMFS LUNs of a suitable size whilst avoiding snapshot workload on LUNs with high performance VM requirements.  If you are using VMware Consolidated Backup or are intending to, you need to consider your backup strategy.  Are you going to backup VMs individually or as a group?  The latter will have concurrent snapshot space requirements (i.e. you’ll need more free space) whereas the former will be a consecutive requirement.

           

          If you are able to determine the following metrics per VM you’ll be well placed to calculate the correct amount of disk space to use:

           

          vRAM Size

          vDisk Writes (MBps)

          vDisk Reads (MBps)

          vDisk Size

          Operational Snapshot Allowance (%)

          vcbMounter Snapshot Allowance (%)

           

          Then use Disk Read / Write to split up your I/O workload into various categories based on how you want to manage disk workload via manual load balancing and you will have basic groupings of VMs to calculate size for.  I'm now using the following formulae on a per VM basis in most cases.  Remember to get all units in the same order of magnitude, i.e. all MB or all GB…

           

          For a concurrent VCB strategy:

          Total vDisk Allowance = ([/b]

           

          For a consecutive VCB strategy:

          Total vDisk Allowance = ([/b]

           

          Then add all these up and you have your total requirement per VMFS datastore, as follows:

           

          For a concurrent VCB strategy:

          Add up all the VM Totals

           

          For a consecutive VCB strategy:

          Find your VM with the largest disk allocation (all disks added together) and calulate this:

           

          Largest vcbSnapAllow = [/b]

           

          Add up all the VM Totals and add the Largest vcbSnapAllow value from above...

           

          Works for me anyway…

           

          2.  Provided you use the proper methods of disk interaction, i.e. vmkfstools / VC, I wouldn’t expect the need for any VMFS3 maintenance.  Performance gains for any other physical disk subsystem would apply for VMFS LUNs.  I suspect that leaving a large amount of space free on a VMFS LUN would have little impact on performance, good or bad.  However, free space should be allocated for operational needs.  Doing so also means that you don't have to worry about the many small files that go along with each VM, which take up much more room than they let on, due to the large cluster size…

           

          3. File de-fragmentation utilities will always be beneficial on NTFS (or until MS fix it).  Always leave at least 19% free space for proper file-system maintenance.  Use DiskKeeper, Raxco, O&O if you can.  I personally like url=http://www.dirms.comDIRMS (Do It Right MicroSoft) and Buzzsaw[/url].  They’re definitely the most efficient..! 

           

          Cheers,

          Ben

          • 2. Re: Proper vmfs3 sizing and best practice for ESX 3.0
            mstahl75 Master

            You should also take into account your block size and the fact that all configuration files are also created on the VMFS. I know those file won't be very large but depending on your block size you might run into some issues.

             

            When we created our LUNs were were unaware that the swap file was being stored on the LUN with all the other files. The main thing is to be sure that there is enough space left to account for all the VM memory minus VM memory reservations plus a fudge factor to take into account all the other config and log files.

             

            If you don't have enough space you might have VMs that will not power on.

             

            If you use other features you might even need more space.

            • 3. Re: Proper vmfs3 sizing and best practice for ESX 3.0
              BUGCHK Master

              I know those file won't be very large but depending on your block size you might run into some issues.

               

              I've read somewhere that VMFS3 supports suballocations - a file system block of ,e.g. 1 MB, can be used to store several smaller files.

              • 4. Re: Proper vmfs3 sizing and best practice for ESX 3.0
                dwchan Expert

                it is the allocation of VM Swap files is what i am worry about.  If the VM is allocated with 3.6G of physical memory, and I have enough memory on the host to minimize swap (i.e. 16G physical memory, with 4 - 3.6G VM)  Would the VM swap go beyond the phyical allocated memory?  Basically what I am asking is the worst case scensorio

                 

                dwc

                • 5. Re: Proper vmfs3 sizing and best practice for ESX 3.0
                  Expert

                  Yes that is correct. BTW, the ESX Server SAN Configuration Guide does have a small discussion on what kind of LUN sizes one would prefer for ones VMFS3 volumes. The discussion is definitely not as detailed as you want, but FWIW.

                  • 6. Re: Proper vmfs3 sizing and best practice for ESX 3.0
                    Quotient Expert

                    VM SWAP is allocated on a per VM basis.

                     

                    In the example you have given, you would be required to allocate 3.6GB of disk space per VM - just for the VM's swap file.

                     

                    Although having 16GB will reduce the likelihood of the file being used, ESX will want to create it regardless.

                     

                    The file is used as per ESXv2, just the default location has changed and it is now on a per VM basis rather than a per host basis (where the file is a pool for all VMs)...

                    • 7. Re: Proper vmfs3 sizing and best practice for ESX 3.0
                      mstahl75 Master

                      The swap file size is equal to the amount or RAM allocated minus the amount of RAM reserved for each VM. In VI2 the reserved amount defaulted to half of the total. In VI3 the reservation defaults to 0.

                       

                      So, it would depend on what kind of reservations are on the individual VMs and total RAM.

                      • 8. Re: Proper vmfs3 sizing and best practice for ESX 3.0
                        mstahl75 Master

                        Interesting, I did not know that. I will look into it.

                         

                        Thanks for the information.

                        • 9. Re: Proper vmfs3 sizing and best practice for ESX 3.0
                          dwchan Expert

                          So if I allocate 3.6G of memory to a VM, the VM Swap file would be 3.6G right from the start?  or does it start from 0 to what ever it need?  In addition, what do you mean by this "amount of RAM reserved for each VM"

                           

                          dwc

                          • 10. Re: Proper vmfs3 sizing and best practice for ESX 3.0
                            mstahl75 Master

                            On a VM, under Edit Settings > Resources > Memory[/b] the is a value called Reservation[/b]. That is the amount of memory that ESX will reserve[/i] for that VM. That memory isn't available to other VMs.

                             

                            So, if that value is set to 0 on your VM with 3.6 GB of memory then the swap file will be 3.6 GB when you power on the VM. ESX 3.0 allocates that much space on the VMFS when the VM is first powered on. Now, if you have a reservation of 1 GB on the VM then the swap file that gets allocated is only supposed to be 2.6 GB.

                            • 11. Re: Proper vmfs3 sizing and best practice for ESX 3.0
                              nwallaceml Enthusiast

                              If you are able to determine the following metrics

                              per VM you’ll be well placed to calculate the correct

                              amount of disk space to use:

                               

                              vRAM Size

                              vDisk Writes (MBps)

                              vDisk Reads (MBps)

                              vDisk Size

                              Operational Snapshot Allowance (%)

                              vcbMounter Snapshot Allowance (%)

                               

                              Cheers,

                              Ben

                               

                              I understand that everybody's environment is different, and as such snapshot allowances will be different for everybody. However, for us folks who do not have enough resources to test and/or are just too lazy... would you mind sharing what you use for snapshot allowances in your equations? For instance, what % snapshot allowance do you use for:

                                -  a VM that is pretty static, like a web server?

                                -  a VM that sees a lot of change, like a database server?

                               

                              Thanks

                              • 12. Re: Proper vmfs3 sizing and best practice for ESX 3.0
                                DougBaer Master
                                VMware Employees

                                As far as Operational Snapshot allowance, I'd think about whether you will want to use the snapshot feature at all with the VM in question. Maybe I just come from the older ESX 2.x world and don't really see the value in regularly using the snapshot feature for my server machines, but I don't plan on utilizing the feature much.

                                 

                                I use the feture constantly with the Workstation product, and may use the feature occasionally while testing a patch/upgrade/whatever. However, I SERIOUSLY doubt I will maintain snapshotted versons of my production servers' VMs unless my ESX servers are hosting test/dev machines.

                                 

                                As for the vcbMounter Snapshotallowance, I would look at the largest VMDK on the VMFS and use that as the allowance, if I were going to use VCB.

                                 

                                Doug