4 Replies Latest reply on Oct 12, 2005 1:04 PM by yess

    Wanted: SAN Best Practices

    yess Lurker

      We recently defined an additional LUN to be shared between two ESX hosts.  Things didn't go as planned.  Some existing SAN volumes changed LUN numbers and had different LUN numbers on each path (I forgot to mention that each host is connected to the storage via two fabrics).  The end result was that the number of volumes increased because ESX did not realise it was dealing with the same LUN with a different number on each path.   We ended up with corruption on the volume and had to scratch the files.

       

      Without getting into excessive finger pointing there is some debate about the cause of the issue.  The SAN team suggest that as ESX claims to support path fail-over it should be able to cope, and that anyway Windows (running Veritas Volume Manager) never has a problem because it writes a signature to the disk.

       

      You won't be surprised if I tell you that the ESX team do not entirely agree.  Their view is that if the LUN numbers of the volume change then all bets are off.

       

      From discussions with our experts, and those of the storage vendor, here is a set of guidelines that I am working on:

       

      Guidelines for a happy marriage[/u]

      1. Never change the LUN number of a volume.  Use LUN number mapping on the SAN storage to prevent LUNs numbers changing or being reordered when a volume is added or deleted.

      2. Add space to ESX by adding a SAN volume and extending the vmfs file system across the new volume using ESX functionality.

       

      Steps to add storage /u

      1. Check that dual-paths are healthy on all ESX nodes.

      2. Shutdown one path between the storage and the ESX hosts.

      3. Make the change to the storage. (See point 1).

      4. Verify all volumes are available on all hosts.

      5. Double-check the SAN configuration and bring up the second path.

      6. Verify all volumes are available.

       

      The business about going to a single path comes from the storage vendor.

       

      Does this sound sensible? Anyone got any better ideas?

       

      Cheers!

        • 1. Re: Wanted: SAN Best Practices
          sbeaver Guru
          vExpertUser Moderators

          I am going to step in with one part

           

          >2. Add space to ESX by adding a SAN volume and extending the vmfs file system across the new volume using ESX functionality.

           

          The recommended practice is to NOT extend VMFS volumes but yet have 2 VMFS partition presented to ESX.  The extending that ESX uses does not spread the load over the space but just dumps data until its filled.  Use more smaller luns

           

          My 2 cents

          • 2. Re: Wanted: SAN Best Practices
            Jwoods Expert

            Not using dual pathing yet, so can't help ya too much there.  However, you don't want to go down the volume expansion route...bad idea.  Risk losing data this way.  Best practice, just create another volume.

             

            Message was edited by:

                    Jwoods  just a bit late...

            • 3. Re: Wanted: SAN Best Practices
              Dave Suraci Enthusiast

              My 4 cents...

               

              Do not disable one of the paths out of the ESX host.  Instead, implement preferred pathing. 

               

              When adding storage to the SAN, create new LUNS and format them accordingly.  Do not span LUNS, and NEVER reorder or renumber them.  If you can, have a second SAN storage device(s) and mirror the LUNs between them, setting the mirror up as Read-Only.  Then Mask out the read-only LUNs in the MUI for those servers in the farm so that no VM writes to them, other than the data being mirrored.  Then, should you lose your primary SAN storage, you can modify the LUN masking info in the MUI to see the mirrored LUN, and bring up the appropriate VM, whose VMDK files now live on the secondary SAN storage.  Also, make sure that you have 2 dual path SAN to the SAN storage, so that in case you lose one port, you don't have controller flip-flopping on the SAN end, the ESX hosts will just communicate to the SAN storage via the secondary port on the same controller.

              • 4. Re: Wanted: SAN Best Practices
                yess Lurker

                The idea behind extending the file system over multiple LUNs was to avoid wasting \[expensive] SAN storage.  We already have more than a few LUNs with a around 10GB free space waiting for a new owner. 

                 

                But I take your point.  If we use multiple volumes we reduce the impact of a corruption of any single LUN.

                 

                Thanks for the feedback so far.  Good stuff.  Keep it coming!