5 Replies Latest reply on Feb 8, 2006 2:41 AM by Chemosh

    SAN disk carving strategy with ESX hosts and IBM BladeCenter

    dchase Novice

      Hello everyone,

       

      I have been reading through the forums looking for best practices on carving up a brand new SAN for use with a brand new IBM BladeCenter.  We are coming from multiple standalone ESX hosts and we purchased an IBM BladeCenter with 18 blades (13 HS20 two-way and 5 HS40 four-way), and a Hitachi AMS500 SAN with close to 4TB of space available to us.  This is for a QA/Dev lab and the SAN primarily contains SATA disks for the majority of the VMs we will be running.  We have about 1TB of FC we're reserving for performance testing.  My question relates to the SATA disk space we have available to us.

       

      We will be doing 100% boot from SAN.  None of the blades will have locally attached storage.

       

      So far here are my assumptions which have been confirmed by the forums.  If anything seems wrong please let me know:

            

           1) We will want to carve out one LUN for each of the ESX boot volumes.

       

           2) We will want to carve out one large data storage LUN for every 10-15 VMs that will be running off it.  This LUN can be shared among blades (we only plan to have a max of 6-8 VMs running per blade, hence the LUNs will need to be shared).  The LUNs available to ESX will need to be set to Public access mode.

       

           3) Each LUN will be available to all blade servers.  However only two or three blade servers may be running VMs from a given LUN.

       

      My questions are as follows.

       

           1) Is it better to create fewer large raid groups and many small LUNs, or should the raid-group to LUN ratio be more 1:1 for performance reasons?  I assume a LUN spanned across a RAID group will be striped against all the disks in the group.  So two LUNs created on the same raid group will span all of the spindles, causing a decrease in performance.

       

           2) Is it better to carve many smaller LUNs or a few larger LUNs for VMDK storage?  Larger LUNs would be easier to manage, but in terms of performance, what have people experienced?

       

           3) What size boot LUN would you carve out for each ESX boot volume?

       

           4) If you had 18 blades and 4TB of disk space over maybe 25-30 disks, how would you setup your raid groups and LUNs?

       

      I mainly ask these questions because, while possible, we would rather not have to re-carve the disks at a later time because we goofed on the initial setup.

       

      Any and all suggestions are greatly welcomed and appreciated.  We are new to booting from SAN and best practices with regards to VMWare (we have been a growing ESX shop for the past year and a half, and plan to move this practice into a production environment later this year) and would like to hear some input/feedback.

       

      Thank you,

      Dave C

        • 1. Re: SAN disk carving strategy with ESX hosts and IBM BladeCenter
          bchancellor Hot Shot

          ;    1) We will want to carve out

          one LUN for each of the ESX boot volumes.

           

           

               2) We will want to

          carve out one large data storage LUN for every 10-15

          VMs that will be running off it.  This LUN can be

          shared among blades (we only plan to have a max of

          6-8 VMs running per blade, hence the LUNs will need

          to be shared).  The LUNs available to ESX will need

          to be set to Public access mode.

           

          That looks okay, you will want to consider spanning the running VMs over the luns however, and setting a prefered path for the HBA.  So you dont want esxbox01 to have all 6 of it's VM's on lun04 for instance.... Your ideal setup is to spread the lun love.

           

           

           

               3) Each LUN will be

          available to all blade servers.  However only two or

          three blade servers may be running VMs from a given

          LUN.

           

            That's fine, with the obvious exception of boot luns

           

           

          My questions are as follows.

           

               1) Is it better to

          create fewer large raid groups and many small LUNs,

          or should the raid-group to LUN ratio be more 1:1 for

          performance reasons?  I assume a LUN spanned across a

          RAID group will be striped against all the disks in

          the group.  So two LUNs created on the same raid

          group will span all of the spindles, causing a

          decrease in performance.

           

          It's better to keep the 1:1 if you are going to have multiple luns.  Otherwise you begin to have random contention between devices.  Keep in mind striping doesn't spread I/O across all disks.

           

          For instance: Your average I/O is 4k, you setup your RAID group to have 5 disks with a stripe width of 128k.   Durring writes or sequential reads your I/O is only moving from one disk to another every 32 I/Os.  When you have 6 systems sharing the RAID group you have the potential of 192 I/Os waiting for one drive while the other 4 sit idle (Not to say this happens every time, but it does happen sparatically).

           

               2) Is it better to

          carve many smaller LUNs or a few larger LUNs for VMDK

          storage?  Larger LUNs would be easier to manage, but

          in terms of performance, what have people

          experienced?

           

          This is up to you and largely determined by the I/O activity of your VMs.  If  you keep to your above plan of 10-15 VMs per lun you should be fine, for all but the most I/O intensive VMs

           

               3) What size boot LUN

          would you carve out for each ESX boot volume?

           

          A decent rule of thumb is 2-3 physical servers per disk.  Set your boot RAID group(s) up seperatly from your VMFS raid groups.  So 18 servers divided by 2 or 3 servers per disk =  6 to 9 physical drives needed for boot.  Now take the size of those drives and devide by the number of servers and you have your boot size.

           

           

           

               4) If you had 18 blades

          and 4TB of disk space over maybe 25-30 disks, how

          would you setup your raid groups and LUNs?

           

           

          Assuming:

          - You have 30 drives roughly 136GB each

          - You have 18 physical servers

          - You plan to host up to 144VMs (8 per server)

          - At 15 VMs per lun we need 10 luns

           

          RG1 : 8 disks RAID-10 (ESX Boot) ~= 544GB usable

            (Each of your 18 servers gets a 30GB boot lun)

           

          RG2: 5 disks Raid-5  (I hate raid 5, but it's for QA). ~= 544GB usable

                   - 2 vmfs luns in this group

           

          RG3: 5 disks Raid-5

                   - 2 vmfs luns in this group

           

          RG4: 5 disks Raid-5

                   - 2 vmfs luns in this group

           

          RG5: 5 disks Raid-5

                   - 2 vmfs luns in this group

           

          Two leftover disks for hot spare or some such.

           

          Disclaimer: There are probably typos in this post, but it's too much work to click on spell check and wade through it all.  Especially since I didn't start out planning to put this much effort into it.

           

          -Brett

          • 2. Re: SAN disk carving strategy with ESX hosts and IBM BladeCenter
            AMcCreath Master

            Brett, you should have added a PayPal donation link for that quality of post mate. Nice!

             

            • 3. Re: SAN disk carving strategy with ESX hosts and IBM BladeCenter
              Jwoods Expert

              No kidding!  dchase, you have to give Brett some "points" love for this one!  Good stuff!

              • 4. Re: SAN disk carving strategy with ESX hosts and IBM BladeCenter
                Chemosh Novice

                No kidding! dchase, you have to give Brett some "points" love for this one! Good stuff! [/i]

                 

                Definitely!  Kudos for the great response bcancellor.  I've actually been looking into ESX of blades for a few weeks now, and am leaning more towards HP personally.  You've definitely given me a few more little tidbitsto help me with my research.  Thanks!

                • 5. Re: SAN disk carving strategy with ESX hosts and IBM BladeCenter
                  Chemosh Novice

                  ....And I totally agree with you on RAID 5; it's good for webservers, and that's about all I care to use it for.  I guess a few years messing with write intensive databases will teach anyone to get away from RAID 5.