1 2 3 Previous Next 31 Replies Latest reply on Aug 12, 2015 12:09 AM by jwhitehv Go to original post
      • 15. Re: High-Capacity VSAN 6 Nodes
        jwhitehv Enthusiast
        vExpert

        Let's say it's a minimum 3 node cluster, that's 216 TB before overhead.  It work load can consume that much space, but still perform well with only 6 sockets in the cluster?  It seems like with disk that dense, you would run out of compute power before you could fill it up.  Is it a few VMs that are massive?  A lot of little VMs that don't consume much CPU?  Something else I'm missing?

         

        Not sure what you mean. Environments don't have a uniform compute to storage ratio and adjusting the core count per socket from 4-12 would be my first choice of scaling up compute requirements. 6 sockets can represent 72 cores. This client has a ton of data to store, but no analytics to run against it. One needs to weigh the cost of high-end CPUs against the cost of additional VSAN socket costs.    

        • 16. Re: High-Capacity VSAN 6 Nodes
          jwhitehv Enthusiast
          vExpert

          Does NexentaConnect, SoftNAS, etc answer that question?

          Answer what question? Definitely not, "How are other users scaling VSAN?"

          • 17. Re: High-Capacity VSAN 6 Nodes
            elerium Hot Shot
            vExpert

            The VSAN HCL is pretty slow to update. Since release of 6.0 i don't think any NVME cards or SATA disks have made it into HCL yet. Of all the parts on the HCL, raid controller is most important followed by SSD. I'm not sure how important magnetic disks are to the HCL so far. The generalized thing I keep reading is that if you don't use enterprise/quality parts or try to cheap out with consumer level parts, things will suck and data will be lost. In particular, SSD drives will need power loss protection capability which isn't found on many consumer level drives.

             

            I'm willing to take some risks since my cluster is used for development purposes, so I've built my VSAN6 cluster with magnetic drives and SSD not on HCL. I'm using WD RE4 4TB SAS, and Intel P3700 1.6TB for SSD on Dell R730xd which has 16 3.5" bays. I end up with 56TB raw or 28TB usable (ftt=1) storage for 2U of rackspace. Seeing that other users have reported great success with the P3700 and both Intel/VMWare have released a performance whitepaper with the P3700 in use, I decided to go with this SSD. The only grief I've had is actually with the Dell H730 raid controller and that was when it was on the HCL firmware version and on HBA mode (both recommended on HCL). I told the VMWare support engineer that I flashed to a newer non-HCL firmware from Dell to fix PSOD/controller resets and change from HBA to RAID0 and it's been golden since. Support didn't give me any kind of grief at all for not following HCL.

             

            In regards to VMWare not supporting you when you aren't using HCL parts, I have not found this to be the case. I had 2 separate tickets open, support checked if everything was on HCL and warned me that data loss could occur from not following HCL recommendations and asked if I understood. Then proceeded to assist with my tickets.

             

            In regards to licensing, I don't like how nodes that don't contribute to storage but are in the same cluster still require VSAN licensing, otherwise even with socket license pricing, it's an amazing performance for value product.

            • 18. Re: High-Capacity VSAN 6 Nodes
              jwhitehv Enthusiast
              vExpert

              There are some 6TB and 4TB SAS drives on the HCL:

              VSAN6-6TB.png

              HGST drives and only HP as an OEM. So if building on anything other than HP, users can't get OEM support on their drives.

               

              VSAN6-4TB.png

              Better, as users can get 4TB drives from four OEMs (I'm not counting IBM since the sale of System X to Lenovo). Cisco is noticeably absent from both lists.

              • 19. Re: High-Capacity VSAN 6 Nodes
                jonretting Enthusiast

                This is kinda nit picky. You would have the only listed options of HGST or OEM HP. That isn't to say your cluster wouldn't be supported by vmware, if you went with the Dell edition. I think the issue comes down to a frequent mantra of limited HCL testing capabilities, or something to that tune. Granted the discussion would no longer be technical, but might it be wiser a subject. As previous users mention the lack of NVME PCIe and many other things. Is the HCL a lofty goal, but under serviced? Thinking aloud. -Jon

                • 20. Re: High-Capacity VSAN 6 Nodes
                  zdickinson Expert

                  I think he was answering my question about small compute power with a large amount of storage.  And yes, I can see that if you used something like Nexenta to serve up large amounts of storage you would need a small amount of compute.

                   

                  72 cores in a 3 node cluster seems small to me.  I guess you are correct, it depends on the environment.

                   

                  Back to your initial question of whether or not anyone had used vSAN at high density...  I don't have a direct answer, but Duncan has a had a small series of large scale deployments of vSAN.  Not sure of the density.  Thank you, Zach.

                   

                  http://www.yellow-bricks.com/2015/07/06/virtual-san-is-breaking-down-silos-for-united-utilities/

                  Virtual SAN enabling PeaSoup to simplify cloud

                  • 21. Re: High-Capacity VSAN 6 Nodes
                    jwhitehv Enthusiast
                    vExpert

                    This is kinda nit picky. You would have the only listed options of HGST or OEM HP. That isn't to say your cluster wouldn't be supported by vmware, if you went with the Dell edition. I think the issue comes down to a frequent mantra of limited HCL testing capabilities, or something to that tune.

                    Wait, what? You're suggesting that VMware support is obligated to fully support systems not on the HCL?

                    • 22. Re: High-Capacity VSAN 6 Nodes
                      jwhitehv Enthusiast
                      vExpert

                      In regards to VMWare not supporting you when you aren't using HCL parts, I have not found this to be the case. I had 2 separate tickets open, support checked if everything was on HCL and warned me that data loss could occur from not following HCL recommendations and asked if I understood. Then proceeded to assist with my tickets.

                      That's helpful to know from a customer point of view. I'm approaching from a consultant's point of view, following the various design whitepapers from VMware which all hammer the HCL. On a conceptual basis, I can understand that the underlying point might be to use enterprise grade SSDs, controllers with stable drivers and a certain minimum queue depth, etc. But that's not what's happening. Effectively, the VSAN HCL is a list of parts that partners and consultants are limited to recommending from. A customer might make a decision to allow a deviation from it, but I can't imagine making a recommendation of parts not on the HCL, knowing that VMware is only going to give "best effort" support.

                      • 23. Re: High-Capacity VSAN 6 Nodes
                        elerium Hot Shot
                        vExpert

                        I absolutely agree, if I were consulting for customers, I'd stick to VSAN HCL only as that is stressed very much by VMWare as best practice there is risk from deviating from it. The current HCL does greatly limit in terms of price/capacity (no large SATA) and performance (no NVME drives).

                         

                        It does look like they are slowly updating HCL, the WD RE4 4TB SAS was originally not in HCL when I implemented, I checked today and it's been qualified!

                        vsan-hcl-re4.png

                        • 24. Re: High-Capacity VSAN 6 Nodes
                          dlboone Lurker
                          VMware Employees

                          Not sure what the discrepancy is.  I'm seeing both high capacity drives and PCI-e SSDs for 6.0.  The Fusion-IOs show up under SanDisk.

                          • 25. Re: High-Capacity VSAN 6 Nodes
                            Bleeder Hot Shot

                            Not sure what the discrepancy is.  I'm seeing both high capacity drives and PCI-e SSDs for 6.0.  The Fusion-IOs show up under SanDisk.

                             

                            Hmm, did those get HCL'd for 6.0 recently?  I remember checking the SX300-1600 earlier in the month, and it was only HCL'd for 5.5.

                            • 26. Re: High-Capacity VSAN 6 Nodes
                              jonretting Enthusiast

                              No -- I'm suggesting they are not necessarily mutually inclusive. In my opinion. Moreover that the HCL is under serviced. Thanks, -Jon

                              • 27. Re: High-Capacity VSAN 6 Nodes
                                jwhitehv Enthusiast
                                vExpert

                                No -- I'm suggesting they are not necessarily mutually inclusive. In my opinion. Moreover that the HCL is under serviced.

                                Well, I agree that product seems to be slow getting to the HCL. But again, from a consultant or partner point of view, it's difficult to recommend product not on the HCL hoping to get support instead of with product on it, knowing the customer will get support.

                                • 28. Re: High-Capacity VSAN 6 Nodes
                                  jwhitehv Enthusiast
                                  vExpert

                                  Not sure what the discrepancy is.  I'm seeing both high capacity drives and PCI-e SSDs for 6.0.  The Fusion-IOs show up under SanDisk.

                                  Yes, there are now PCIe drives certified for VSAN 6! These have literally appeared today! The Intel 3700 family on the scaling guide is still not there though.

                                   

                                      

                                  ModelDevice TypePart NumberCapacity
                                  Fusion ioMemory PX600-1000PCI-ESDFACAMOP-1T00-SF11000
                                  Fusion ioMemory PX600-1300PCI-ESDFACAMOP-1T30-SF11300
                                  Fusion ioMemory PX600-2600PCI-ESDFACAMOP-2T60-SF12600
                                  Fusion ioMemory PX600-5200PCI-ESDFACCMOP-5T20-SF15200
                                  Fusion ioMemory SX300-1300PCI-ESDFACAMOS-1T30-SF11250
                                  Fusion ioMemory SX300-1600PCI-ESDFACAMOS-1T60-SF11600
                                  Fusion ioMemory SX300-3200PCI-ESDFACAMOS-3T20-SF13200
                                  Fusion ioMemory SX300-6400PCI-ESDFACAMOS-6T40-SF16400
                                  • 29. Re: High-Capacity VSAN 6 Nodes
                                    Bleeder Hot Shot

                                    Well, better late than never?

                                     

                                    I'm a bit disappointed that there isn't a better way to be notified of updates to the VSAN HCL.  It looks like I'll have to check for updates manually.  I had been relying on RSS updates to the Virtual Blocks blog.