1 2 Previous Next 15 Replies Latest reply on Jul 10, 2020 11:32 AM by TheBobkin

    vSAN Stretched Cluster and HA isolation addresses

    MtheG92 Novice

      Hi folks

       

      There is always this recommendation about the 2 HA isolation addresses which you should define in a vSAN stretched cluster environment.

       

      If you assume the following example vSAN stretched cluster architecture, what would be the effectively benefit from these two addresses (mentioned as vSphere HA Isolation Address A/B)?

      design.jpg

       

      Example-1:

      Assume that I configure a virtual interface address on Switch-01a as the das.isolationaddress0 and another virtual interface address on Switch-01b as das.isolationaddress1. No Datastore heartbeat is configured. Since my ESXi hosts are redundant connected to all uplink switches, there would be no benefit in defining there das.isolationaddresses even if I would configure on virtual interface address per switch. If Switch-01a fails than the das.isolationaddress0 would be unavailable but the HA heartbeat on all affected hosts would not fail since all hosts are reachable over Switch-02a. So what is the benefit of the HA isolation addresses in this case? 

       

      Example-2:

      Assume that I configure a virtual interface address on Switch-01a as the das.isolationaddress0 and another virtual interface address on Switch-01b as das.isolationaddress1. No Datastore heartbeat is configured. Now the ISL breaks for whatever reason and the vSAN (HA) networks get partitioned into 2 segments. It gets network partitioned because all hosts within a site can communicate with each other but not with hosts from the other site since the network heartbeat goes via the vSAN network. But now the Witness kicks in and forms the "surviving" site with the Preferred Site. But again, what would be the benefit of the HA isolation addresses in this case?

       

      There is no use case reagarding to the picture above, where I really benefit from configureing 2 or 4 das.isolationaddress on the vSAN realted network switches. Do I miss sth. about the vSphere HA mechanism and vSAN?

       

      Thanks in advance,

      MtheG

        • 1. Re: vSAN Stretched Cluster and HA isolation addresses
          ZibiM Enthusiast

          Hello

           

          I'm not a big expert, but I'll give a shot :-)

          I'd say this is related to few things:

          1. This is stretched cluster and unavailability of one site must not impact 2nd site - having single isolation address located on the site that ceased to be available can upset things

          2. Especially that the recommended response for HA isolation in VSAN scenarios is power off VMs and restart

           

          In order to address that you create 2 isolation addresses - 1 local for each site

          In case of site disconnect each half of the cluster can reach the local isolation address and only the witness connectivity decides who will stay alive

           

          So looking back at your question - recommended HA isolation response

          PS. And I just realized I need to make sure that SVI I requested will survive single switch failure :-)

          • 2. Re: vSAN Stretched Cluster and HA isolation addresses
            MtheG92 Novice

            Hi ZibiM

             

            Thank you for your feedback.

             

            To your answers:

             

            1. This is stretched cluster and unavailability of one site must not impact 2nd site - having single isolation address located on the site that ceased to be available can upset things

            In case of a site isolation there would be a benefit for vSphere HA to validate a host isolation inside the surviving site. BUT if you have a fully redundant network infrastructure like the most of that I  have seen than there is not a "real" chance for a isolation (except the host or FDM service crashes) or partition within this site.

             

            So when does a host declares itself isolated?

             

            • It is not receiving any communication from the master
            • It cannot ping the isolation address
            • It is not receiving any election traffic of any other hosts in the cluster

             

            In case of the site isolation there will be a new master election (if your initial master was on the other partitioned site) and therefore you have a master within your site. In case of a switch outage there would be always an uplink from the ESXi to the remaining switch due the fully redundant network stack. So, if this very unlikely event happens where all uplinks of your ESXi hosts are down, I will face a total cluster shutdown and once more there is no benefit of this isolation address.

             

            2. Especially that the recommended response for HA isolation in VSAN scenarios is power off VMs and restart

            I know, you are right. Also, I need at least one isolation address to fulfill the vSphere HA requirements.

             

            PS. And I just realized I need to make sure that SVI I requested will survive single switch failure :-)

            Yes, that is a good point. In most cases we have two isolation addresses per site (total of four addresses), one for each ESXi uplink switch. Another situation is, where you can point the isolation address to e.g. a HSRP IP address of the two ESXi uplink switches.

             

            But beside all of that, I am asking for the technical explanation of this requirement, because I don’t see a benefit in this technique regarding to an infrastructure I described initially. If you are planning a stretched cluster and you are facing a conceptual discussion with the network guys you only have the statement regarding to the mandatory requirement of at least one proper isolation address.

             

            Kind regards,

            MtheG

            • 3. Re: vSAN Stretched Cluster and HA isolation addresses
              ZibiM Enthusiast

              But beside all of that, I am asking for the technical explanation of this requirement, because I don’t see a benefit in this technique regarding to an infrastructure I described initially. If you are planning a stretched cluster and you are facing a conceptual discussion with the network guys you only have the statement regarding to the mandatory requirement of at least one proper isolation address.

              Yes, I can wholeheartedly agree to that.

              It would be great to get some technical explanation what's behind this req, and not just "because"

               

              Unfortunately I'm finding the whole networking part of the VSAN really lacking, which is bit depressing really considering that this is network based storage.

              • 4. Re: vSAN Stretched Cluster and HA isolation addresses
                depping Guru
                User ModeratorsVMware Employees

                you need two isolation addresses just in case you have two failures at the same time. Just imagine the link between the locations fails, or the preferred location fail. If next an isolation occurs in the Secondary site and your isolation address is located in the preferred site and unreachable how would an isolation be determined, or better said a false positive be determined? You unfortunately would not be able to do that. hence we tell people to add 2 isolation addresses, one for each location, simply to allow "isolation detection" to work even in the case of a situation where a failure has already occurred.

                1 person found this helpful
                • 5. Re: vSAN Stretched Cluster and HA isolation addresses
                  HassanAlKak88 Expert
                  vExpert

                  Hello,

                   

                  the following more explain the isolation address's use cases and recommendations: Advanced Options | vSAN Stretched Cluster Guide | VMware

                  • 6. Re: vSAN Stretched Cluster and HA isolation addresses
                    ZibiM Enthusiast

                    Thank you Duncan for coming by.

                     

                    I have additional question regarding this topic.

                    You wrote a while ago that in the VSAN cluster HA traffic is automatically moved to the VSAN network:

                    I want vSphere HA to use a specific Management VMkernel interface | Yellow Bricks

                     

                    I have a question why we need to then mess with HA advanced settings in order to provide isolation address ?

                    Why we cannot just set up gateway address on the VSAN vmkernel and be done with this ?

                    • 7. Re: vSAN Stretched Cluster and HA isolation addresses
                      depping Guru
                      VMware EmployeesUser Moderators

                      Because HA by default always uses the default gateway of the management interface, and vSAN doesn't allow you to specify the gateway in the current release. This is something that is being worked on, and I will ask the HA team, again, to improve this process. As I think it can be done much easier indeed.

                      • 8. Re: vSAN Stretched Cluster and HA isolation addresses
                        MtheG92 Novice

                        Hi

                         

                        Thank you for your feedback but if you read through the conversation you would recognize that we are already aware of this recommendations and explenations:

                        In case of a site isolation there would be a benefit for vSphere HA to validate a host isolation inside the surviving site. BUT if you have a fully redundant network infrastructure like the most of that I  have seen than there is not a "real" chance for a isolation (except the host or FDM service crashes) or partition within this site.

                        This implies the following statement from your storagehub reference: This would enable vSphere HA to validate host isolation even in the case of a partitioned scenario (network failure between sites).

                         

                        The discussion is more about "defending" this recommendation on technical basis regarding to a given infrastructure like the one I initially described.

                         

                        Kind regards,

                        MtheG

                        • 9. Re: vSAN Stretched Cluster and HA isolation addresses
                          MtheG92 Novice

                          Hi Duncan

                           

                          In this case as far as I know, the ping of the isolation address would fail but due the fully redundant/cross-cabled network stack on both sites, all the hosts in the surviving site would be able to communicate to each other. Also, an FDM master should be automatically elected in this surviving site/partition if the original FDM master was in the failed site. Hence only one of three criteria for a host isolation event would be met – right?

                           

                          An isolation event would occur if the hosts could not communicate with each other, could not ping the isolation address and could not ping the FDM master. But if this happens in the given infrastructure initially described, then I would face a total cluster shutdown because my hosts are not able to communicate with the isolation address and therefore it would also be very likely that they cannot communicate with each other because the uplink switches must have failed as well. You could configure two isolation address per site (e.g. one per uplink switch) or HSRP to avoid the situation of an unpingable isolation address in case of single switch failure. But in such an event you would be able to communicate with the other hosts anyway and therefore one isolation criteria would not be met.

                           

                          In my opinion, this isolation address would only be useful if you could not provide a fully redundant network infrastructure. Or do I miss something?

                           

                          Kind regards,

                          MtheG

                          • 10. Re: vSAN Stretched Cluster and HA isolation addresses
                            depping Guru
                            User ModeratorsVMware Employees

                            I am not sure what you want me to say? If you don't feel there's a chance this situation can occur then you don't follow the recommendation. I don't see what the problem is, to be honest. An isolation can also occur as a result of a NIC failure, driver failures, VLAN configuration issues etc. But again, it is a recommendation. This is an "insurance, a fail-safe mechanism. But again, if you don't see any value or don't believe this can ever happen in your environment then you don't configure it.

                            • 11. Re: vSAN Stretched Cluster and HA isolation addresses
                              MtheG92 Novice

                              Hi Duncan

                               

                              From my point of view there is no real benefit of using this isolation address in my initially described infrastructure. The only reason to use it is to fulfill the minimum vSphere HA requirements of providing at least one isolation address.

                               

                              The reason why I came up with this question are situations like this:

                               

                              Fictive Scenario: Conceptual design meeting

                               

                              VMware guy: I need at least two isolation addresses on the vSAN subnet for HA purpose, one in every site.

                              Network guy: Why? We do not use “direct” layer-3 functionality on this segment, but you have full layer-2 connectivity.

                              VMware guy: We need these two addresses to allow a granular level of potential isolation and segmentation events determination.

                              Network guy: Why? The whole network stack is redundant/cross-cabled on both sites. There would be no such situation where you would face a network partition within one of these two sites where the network infrastructure itself would not have dramatically failed.

                              For example, if the ISL breaks and in your surviving site one of the two ESXi uplink switches fails, then you would be able to communicate via the second uplink switch. Hence a potential isolation situation is not likely except the host itself fails. In a scenario where both uplink switches will fail, there would be a more problematic situation because your whole ESXi infrastructure is down/isolated.

                              Long story short, it does not look like there is a real benefit or even a more granular detection of isolation or partition events with such isolation addresses.

                              VMware guy: Yeah… well… it is a "mandatory" requirement to fulfill the minimum vSphere HA requirements and it’s also a common best practice for vSAN Stretched Clusters.

                               

                              I was more interested in the discussion to validate the arguments because maybe I missed something…

                               

                              Kind regards,

                              MtheG

                              • 12. Re: vSAN Stretched Cluster and HA isolation addresses
                                depping Guru
                                User ModeratorsVMware Employees

                                Like I said, if you don't see a value then you don't configure it. You don't even need to configure the Isolation Response to begin with. If you are certain the network provides a 100% guarantee that an isolation can never occur then you don't configure it. You won't need an isolation address and you won't need an isolation response if isolations can't happen.

                                 

                                Now having said that, it would be the very first time in a lifetime that any network administrator would guarantee a 100% uptime and can guarantee that there's no way for an ESXi host to ever reach a state where it thinks it is isolated. Like I said, it is a fail safe mechanism. You don't need to use it.

                                 

                                We recommend to use it, simply because there's no such a thing as a 100% guarantee. How big is the risk? Small. What is the cost of configuration? Little. Considering you are building a stretched cluster, we recommend to implement every single failsafe mechanism. Whether that is for a physical network fabric issue, or for a driver issue / host side network issue etc.

                                 

                                So yes, I do think you are missing something and are looking at the recommendation purely from a network admin point of view, but there are many components that can fail, and personally, I'd rather be safe than sorry, especially when it comes at little to no cost.

                                • 13. Re: vSAN Stretched Cluster and HA isolation addresses
                                  MtheG92 Novice

                                  Hi Duncan

                                  So yes, I do think you are missing something and are looking at the recommendation purely from a network admin point of view, but there are many components that can fail, and personally, I'd rather be safe than sorry, especially when it comes at little to no cost.

                                  You are right, my discussion was a way to “isolation address” centric for sure, there are way more components to be considered when planning a high available vSAN stretched cluster and sure, I am aware of this. But my intention was to focus this discussion around this isolation address component.

                                  We recommend to use it, simply because there's no such a thing as a 100% guarantee. How big is the risk? Small. What is the cost of configuration? Little. Considering you are building a stretched cluster, we recommend to implement every single failsafe mechanism. Whether that is for a physical network fabric issue, or for a driver issue / host side network issue etc.

                                  That is a good statement and the usual outcome in such situation like described in my last post.

                                   

                                  Thank you very much at this point for your time and feedback and have a nice weekend.

                                   

                                  Cheers,

                                  MtheG

                                  • 14. Re: vSAN Stretched Cluster and HA isolation addresses
                                    ZibiM Enthusiast

                                    Just one last comment :-)

                                     

                                    Please take into the consideration that recommendations are for all environments - big and small

                                    In your case your network team can argue that you have just a pair of switches at each site - so the extra isolation address is not needed.

                                    Even in the worst possible scenario you still will have local L2 connectivity and your ESXi servers will be able to talk to each other.

                                     

                                    Think about environments bigger than that - something like 64 node cluster, 32 nodes at each site, 8 servers per rack, 4 pairs of switches (2-rack network pod)

                                    In such scenario network partition is much bigger worry - you could get it even within the site.

                                    In case like that independent isolation address helps to determine which half of the site is alive and should work, and which is isolated and should power off and autokill in order to avoid messing up the data.

                                     

                                    Good luck with your talks with network team :-)

                                    1 2 Previous Next