12 Replies Latest reply on Oct 15, 2019 8:03 AM by jaqen

    UAG Load Balancing using HAProxy

    KFM Enthusiast

      Hi community, this problem has been bugging me for a while now so I figured I'd reach out to the community and hopefully get this thing working! Bear with me, this is a long one....

       

      I work for a service provider and want to building out a scalable UAG-pair-per-tenant design behind a pair of HAProxy load-balancers for external (i.e. over the Internet) access. Note that we are using the Horizon DaaS product not the Horizon View product. Apart from the difference in name, I believe the UAG functions identically in both environments.

       

      My final solution would be two pfSense (community edition) firewalls with the HAProxy package installed on both to provide HA and load-balancing functionality to the tenant UAGs behind them. As we onboard more tenants, I would add another pair of UAGs for each. HAProxy will selectively pick which UAG is required for the incoming connection based on SNI. For example:

      • TenantA  - external URL: daas.tenanta.com
      • TenantB - external URL: daas.tenantb.com
      • and so on....

      Connections arriving at daas.tenanta.com will be directed to a UAG pair for tenantA. Connections arriving at daas.tenantb.com will be directed to a UAG pair for tenantB. And so on and so forth.

       

      As this is my first attempt at building out a HAProxy solution I've decided to keep it simple and use just one pfSense/HAproxy load-balancer, one UAG and one tenant. This is what I have now:

      • Single pfSense firewall with HAProxy package installed
      • Single UAG (10.0.0.5) for tenantA
      • external URL daas.tenanta.com with public  IP address (1.1.1.1)
      • One internal-facing VIP (10.0.0.1) which acts as the gateway for the UAG
      • Firewall does 1:1 NAT from daas.tenanta.com to VIP (i.e. 1.1.1.1 -> 10.0.0.1)
      • Firewall rules allowing any -> VIP for ports 443, 8443 and 4172.
      • UAG is configured using apsetup.sh script. It configures the following settings:
        • proxyDestinationURL: https://tenant_appliance_ip
        • pcoipExternalUrl: 1.1.1.1:4172
        • blastExternalUrl: daas.tenanta.com:8443
        • tunnelExternalUrl: daas.tenanta.com:443
      • UAG gateway is the VIP (10.0.0.1)
      • HAProxy configuration:
        • One frontend for ports 443, 8443, 4172
        • Three backends for ports 443, 8443 and 4172 all with tenantA UAG as backend server
          • I've also tried one backend for just port 443 with tenantA UAG as backend server as this seems to work for Blast connections via browser
        • Frontend acl uses SNI for daas.tenanta.com to send to tenantA UAG backend

       

      In this given configuration, I've observed the following:

      • Browser access to both the user portal (daas.tenanta.com) and admin portal (daas.tenanta.com/admin) work fine
      • Initiating a Blast connection to the desktop via the browser, works fine.
      • Using the Horizon Client, I can authenticate successfully but then get the "could not establish tunnel connection" error message. This is what I ultimately need to get working!

       

      Some things I'm not sure about:

      • Should I be using layer 4 (tcp) or layer 7 (ssl/https) load balancing?
      • Should I have multiple backends, one for each port
      • I really want to use source IP affinity as per Load Balancing across VMware Unified Access Gateway Appliances as I think HAProxy can see the client IP address. (i.e. in the HAProxy logs I can see the client IP address.) I don't really want to go with the other two methods if I can help it.
      • Why does Blast via a browser work when I just have one backend listening on 443? When I do a netstat on my client I can see an active connection to 1.1.1.1:8443.

       

      Some things I've tried:

      • Note that when I revert the solution to just a plain firewall bypassing the load-balancer, everything (browser and Horizon client) works fine. i.e. traditional port-forwarding/NAPT to the UAG with FW ACLs allowing any -> UAG:443,8443,4172.
      • I've also used a second tenant to test the SNI ACL and that seems to work fine too. i.e. I can use blast via browser to both daas.tenanta.com and daas.tenantb.com.
      • Collected debug logs on Horizon View client
      • Analysed debug logs on tenant appliance
      • Analysed UAG logs
      • In the above three log collections, I saw nothing obvious to my untrained eyes

       

      I'm really at my wits end here so any help would be much appreciated!

        • 1. Re: UAG Load Balancing using HAProxy
          LarryBlanco2 Expert

          I don't know specifically which protocol you are using because you list all 3 in your rule, but in the case of PCoIP, HAproxy does not support UDP.  You will have to use the 3 external IP setup.   One for the HAProxy and one for each UAG per tenant.

           

          Take a look at this thread: HAproxy loadbalancer and View 6.1.1

           

          Hope that helps you get past or at least answer your question.

           

          Larry

          1 person found this helpful
          • 2. Re: UAG Load Balancing using HAProxy
            KFM Enthusiast

            Hi Larry,

             

            Thanks for your response. I did read that thread prior to posting - it contains a wealth of knowledge. I guess it's good to know that it is possible but using method 2 where the LB is used for the initial connection then all subsequent traffic goes direct to the UAGs.

             

            If that is really the case then that's a shame with HAProxy. It's interesting that the LB product developed by loadbalancer.org seems to support UDP even though it's purportedly built on HAProxy (http://pdfs.loadbalancer.org/Vmware_Horizon_Deployment_Guide.pdf ) Perhaps they've incorporated bits of IPVS which does layer 4 load-balancing into their product.

             

            It also seems a little strange that I can't build a connection using the Horizon Client after authenticating. i.e. I don't get a list of entitled desktops. I wouldn't have thought this would be using UDP.

            As for which protocols I am using, I was hoping to use both Blast and PCoIP. This means a largely seamless experience for both internal and external users with no need to dictate which protocol gets used depending on where they're connecting from. They just fire up the Horizon View client and connect (obviously assuming the FQDN is the same both internally and externally).

             

            I'll try method 2 and let you know how I get on.

             

             

            Thanks,

            Kam

            • 3. Re: UAG Load Balancing using HAProxy
              techguy129 Expert
              vExpert

              The browser works because you are essentially reverse proxying through the UAG to the connection servers web portal. When you establish a connection to the backend resource (desktop/RDSH) through the portal you are connecting from the connection server.

               

               

              The error you mention has me suspecting that you may not have the appropriate ports open from the UAG to the resources (desktops / RDSH) on the backend. If you review the logs on the UAG should indicate the problem.

               

               

              Since you can authenticate using the client and connect successfully to the HTML portal it sounds like your HAProxy is setup correctly.

              • 4. Re: UAG Load Balancing using HAProxy
                KFM Enthusiast

                Thanks for your reply. I don't believe it's a connection problem between the UAG and the desktop resources as they're all on the same network - i.e. for now, there is no inner-firewall.

                 

                After reading Network Ports in VMware Horizon 7: VMware Horizon 7 version 7.2  it's clear why the browser works - all communication happens over TCP - there is no UDP. And as was already mentioned, HAProxy, for better or for worse, does not do UDP. As mentioned specifically in the HAProxy version 1.7.11 - Starter Guide, it says HAProxy "will not see IP packets nor UDP datagrams". Major bummer

                 

                I do find it interesting that when I do a packet capture of a NAPT, non-loadbalancing connection, I do not see any UDP packets (PCoIP 4172) until the actual display starts up. This corresponds to what is mentioned in the Network Ports article in the link above - that all login traffic happens over TCP 443. So if there are no UDP packets during the login stage, then why can I not even bring up the list of available desktops when using the Horizon View client and through a load-balanced connection? Perhaps markbenson can help as he authored this Load Balancing across VMware Unified Access Gateway Appliances and helped immensely in Can BEAT run over a different port than UDP 8443?


                Assuming I somehow resolve the "could not establish tunnel" issue, I could have two connection server entries - one for external (uses Blast only) and one for internal which has no protocol restrictions.

                 

                I guess for now in order to retain the consistent "one connection server" user experience, it looks like I will need to go down the multiple VIP method instead.

                • 5. Re: UAG Load Balancing using HAProxy
                  markbenson Master
                  VMware Employees

                  KFM - you certainly need to fix your Horizon tunnel issue first. HTML Access (browser) doesn't use the tunnel, but native Horizon clients do. After authentication, Horizon clients establish the tunnel connection using the value of tunnelExternalUrl. e.g. https://daas.tenanta.com:443. This is one of the secondary connections and you MUST ensure that gets routed to the same UAG as used for the initial primary connection. If the tunnelExternalUrl is not usable by the client, the tunnel connection is blocked or misrouted to the wrong UAG appliance it will fail. You then won't get the list of entitled desktops. All secondary protocols (Tunnel, Blast, PCoIP) must be routed to the same UAG as the primary connection.

                  1 person found this helpful
                  • 6. Re: UAG Load Balancing using HAProxy
                    Bertman1980 Novice

                    Hello KFM,

                    Do you have a sollution for you HAProxy issues ? We have 2 pod and one UAG and no LB installed jet, but sometimes we have dubble logon issues, We think it because of round robin without LB. So we are looking for a good/free load balancer that does the job.

                    • 7. Re: UAG Load Balancing using HAProxy
                      KFM Enthusiast

                      So yes, I did get it working with HAProxy, albeit only one as I didn't get time to deploy pfSense/HAProxy in a HA setup. I also did get it to load-balance between two UAGs. I can send you the HAProxy configuration if you need it for reference? I never got any double login issues.

                       

                      In a nutshell, HAProxy will not work with UDP which means that we cannot use Method 1 - Source IP Affinity. I did get it working with Method 2 - Multiple Port Number Groups and also Method 3 - Multiple VIPs. The configuration for those three methods are here: Load Balancing across VMware Unified Access Gateway Appliances.

                      • 8. Re: UAG Load Balancing using HAProxy
                        Bertman1980 Novice

                        Hello,

                        If you can send me the config i will appreciate that.

                        I saw that you put the load balancer in front of the UAG. I was thinking to put the load balancer between UAG and the connection server.

                        Bert

                        • 9. Re: UAG Load Balancing using HAProxy
                          KFM Enthusiast

                          The intention was to have two UAGs in an active/active setup. This requires a load-balancer in front to distribute the incoming sessions to the UAG with the least connections.

                           

                          As I was deploying the UAG for a Horizon DaaS deployment, I didn't need to put another LB in front of the tenant appliances since these are natively HA out of the box - no further configuration or LB required. I haven't worked with Horizon View for many years so I can't say if putting a LB in front of the connection server is a supported topology.

                           

                          As for the HAProxy config, here it is below. Note that the design I came up with was to have one public IP address upon which all tenant portal URLs would resolve to. HAProxy would then use SNI to forward the request to that particular tenant's UAG pair. This allowed me to scale out the number of tenant appliances whilst LB the connections through a pair of pfSense/HAProxy appliances. There are obviously a number of different ways you could design this - each with their respective pros and cons.

                           

                          Hope that helps!

                           

                          # Automaticaly generated, dont edit manually.

                          # Generated on: 2019-04-19 08:26

                          global

                              maxconn            1000

                              log            /var/run/log    local0    info

                              stats socket /tmp/haproxy.socket level admin

                              uid            80

                              gid            80

                              nbproc            1

                              hard-stop-after        15m

                              chroot                /tmp/haproxy_chroot

                              daemon

                              server-state-file /tmp/haproxy_server_state

                           

                          listen HAProxyLocalStats

                              bind 127.0.0.1:2200 name localstats

                              mode http

                              stats enable

                              stats refresh 2

                              stats admin if TRUE

                              stats show-legends

                              stats uri /haproxy/haproxy_stats.php?haproxystats=1

                              timeout client 5000

                              timeout connect 5000

                              timeout server 5000

                           

                          frontend Universal_UAG_Frontend

                              bind            publicIP:443 name publicIP:443  

                              mode            tcp

                              log            global

                              option            tcplog

                              timeout client        30000

                              tcp-request inspect-delay    5s

                              acl            TenantA    req.ssl_sni -i daas.TenantA.com

                              acl            TenantB    req.ssl_sni -i daas.TenantB.com

                              tcp-request content accept if { req.ssl_hello_type 1 }

                              use_backend TenantA_UAG_Pool_ipvANY  if  TenantA

                              use_backend TenantB_UAG_Pool_ipvANY  if  TenantB

                           

                          backend TenantA_UAG_Pool_ipvANY

                              mode            tcp

                              id            100

                              log            global

                              stick-table type ip size 50k expire 1h

                              stick on src

                              balance            leastconn

                              timeout connect        30000

                              timeout server        30000

                              retries            3

                              option            httpchk GET /favicon.ico

                              server            TenantA-UAG1 TenantA_UAG1_PrivateIP:443 id 106 check-ssl check inter 1000  verify none

                              server            TenantA-UAG2 TenantA_UAG2_PrivateIP:443 id 102 check-ssl check inter 1000  verify none

                           

                          backend TenantB_UAG_Pool_ipvANY

                              mode            tcp

                              id            103

                              log            global

                              stick-table type ip size 50k expire 1h

                              stick on src

                              balance            leastconn

                              timeout connect        30000

                              timeout server        30000

                              retries            3

                              option            httpchk GET /favicon.ico

                              server            TenantB-UAG1 TenantB_UAG1_PrivateIP:443 id 101 check-ssl check inter 1000  verify none

                              server            TenantB-UAG2 TenantB_UAG2_PrivateIP:443 id 102 check-ssl check inter 1000  verify none

                          • 10. Re: UAG Load Balancing using HAProxy
                            Bertman1980 Novice

                            Thank you for sharing, i will take a look at it.

                            • 11. Re: UAG Load Balancing using HAProxy
                              heloma Lurker

                              hi,

                              Five things to consider when you deploy loadbalancing in front of UAG.

                              1- make usre the secure tunnel option is checked in the UAG settings.

                              2- In Layer 7 mode (full ssl proxy), the certificate deployed on UAG MUST be the same as the one deployed on the haproxy.

                              3- In the UAG, you have to indicate the public IP address used by the clients.

                              4- You have to consider 3 ports (4 if you want to deploy blast). tcp/443, tcp/4172 and udp/4172. this latter is for pcoip.

                              5-  Use the session persistency based on source IP, so the client connection is stuck to the same backend UAG.

                               

                              keep in mind that all communications between external clients and connection servers or virtual desktops are ALL going through UAG. therefore, you have to make sure that your pfsense haproxy has the required performance to handle all the traffic.

                               

                              the F5 deployment may provide more insight.... https://www.f5.com/pdf/deployment-guides/vmware-horizon-view-dg.pdf

                               

                               

                               

                               

                              cheers.

                               

                               

                              heloma.

                              • 12. Re: UAG Load Balancing using HAProxy
                                jaqen Lurker

                                I know this is an old thread but wanted to point out, the Horizon Client does not provide SNI information. So when your ACL is looking for an SNI, and you have no default backend definied - it's going to ultimately fail.

                                 

                                That very first connection from the client does - in order to vet the certificate. When you go to use a resource however, it does not. Ultimately the "trick" is just to have the default backed blindly point to your pool of UAGs.

                                 

                                You can verify by simply disabling the UDP tunnel and configuring UAG to force everything through TCP/443 - client still won't work properly. I had an article save somewhere, but vmware's response was "tough cookies".