KFM Tracker

Re: vCAV 3.0 & vCD 9.7 Initial Setup Cert issue

KFM — Fri, 13 Dec 2019 06:04:42 GMT

To be clear, the hostname on the actual management appliance does not need to be the same as the public FQDN. In our environment we use our internal naming scheme to assign hostnames to the appliances and everything works fine from the Internet.

The trickiest thing I've found with vCAv is DNS resolution (depending if you're using split DNS) and the firewall rules depending if you deploy the components in the VMware recommended zones (trusted for all components except for the Tunnel which lives in the DMZ).

To the OP - are you still experiencing issues with your deployment?

Re: vCD 9.7 Installation Issues

KFM — Fri, 13 Dec 2019 05:34:23 GMT

With all due respect, your answer is (a) not correct; and (b) not helpful.

Nowhere is it mentioned that deploying vCD through ESXi is not supported. When you apply some logic to it, why would it not be supported? What would the resultant VM run on? Why the ESXi host itself! Whether I deploy through ESXi or vCenter, it should not make a difference.

Anyway, I finally got around to resolving this myself. So for those who may be stuck deploying any OVA/OVF directly to an ESXi host, this official VMware ovftool document says:

"If you are deploying with the ovftool command targeting an ESXi host, you must “inject” the parameters into the resulting VM when it is powered on. This is because the ESXi host lacks a cache to store the OVF parameters, as with vCenter Server. Therefore, you must use the --X:injectOvfEnv debug option with the --poweron flag in the command line"

The direct link is here: Online Documentation - OVF Tool User's Guide (U2) - VMware {code}

So once I added the two additional parameters, everything worked a treat.

Re: How to Install Obsoleted Package?

KFM — Sun, 15 Sep 2019 13:01:34 GMT

Hi, thanks for your prompt reply! I'm aware of the potential issues that may arise from installing software into vendor blackbox appliances, however I am simply trying to troubleshoot an underlying network problem. Ultimately I can easily redeploy the appliances without the 3rd party software from a scripted install once the testing is complete.

How to Install Obsoleted Package?

KFM — Sun, 15 Sep 2019 12:47:25 GMT

So I've deployed some vCloud Availability appliances which I believe are based on Photon OS. I want to install tcpdump and netcat to do some troubleshooting but because this is a locked down appliance I had to manually edit the photon.repo file to enable the the photon repository in /etc/yum.repos.d.

Now when i try to install either package I get the following message:

root@photonvm [ /etc/yum.repos.d ]# tdnf install tcpdump

Found 1 problem(s) while resolving

1. installed package photon_vasecurity-11.3.0-7315477.noarch obsoletes tcpdump provided by tcpdump-4.9.2-1.ph2.x86_64

Error(1301) : Hawkey general runtime error

root@photonvm [ /etc/yum.repos.d ]# tdnf install netcat

Found 1 problem(s) while resolving

1. installed package photon_vasecurity-11.3.0-7315477.noarch obsoletes netcat provided by netcat-0.7.1-5.ph2.x86_64

Error(1301) : Hawkey general runtime error

Some further info on the photon_vasecurity package...

root@photonvm [ /etc/yum.repos.d ]# tdnf info photon_vasecurity

Name : photon_vasecurity

Arch : noarch

Epoch : 0

Version : 11.3.0

Release : 7315477

Install Size : 108.83k 111439 (111439)

Repo : @System

Summary : VA Security Hardening scripts for VMware

URL : (null)

License : commercial

Description : Virtual Appliance Security Hardening for Photon VMware.

I have no idea how to proceed. I get that photon_vasecurity obsoletes tcpdump but the former is a totally different package to the latter! Is the photon_vasecurity package deliberately obsoleting tcpdump and netcat (and probably a whole lot more other packages) from a hardening perspective? If this is the case, how can I override or manually install the two packages?

Any pointers would be much appreciated - thanks!

Re: vCloud Director and Cisco ACI integration

KFM — Fri, 13 Sep 2019 00:51:36 GMT

Regarding ACI and vCD, I have just deployed this and there is no integration. We just need to import the networks pushed down by ACI as external networks into vCD.

I did have to deploy and configure a NSX Manager instance but didn't prep any clusters for NSX. It just needs to be associated with a vCenter before you can add a vCenter instance as a resource provider for a pVDC.

Apart from that, I'm not sure what you really mean by the following comment.

have a use case for vCD, but NSX is a dagger

Re: vCD 9.7 & Datastore cluster : Compatibility

KFM — Wed, 28 Aug 2019 00:51:41 GMT

Nope, no fix for this, just the workaround which you mentioned.

Unfortunately VMware weren't particularly helpful in qualifying what operations are "safe" to do in vCenter that won't conflict with anything in vCD. It would be helpful to have an official KB article of what is safe and what is not safe to do. They wanted me to list all the things I wanted to do so they could qualify it but I don't really have time to test things for VMware so I pointed them to this post.

Re: vCD 9.7 & Datastore cluster : Compatibility

KFM — Mon, 05 Aug 2019 05:30:41 GMT

I'm going to open a SR to ask for an official stance on this but in the meantime...

Did we get to the bottom of this? In the past (i.e. vCD 8.x) we tagged both datastores and datastore clusters and we could see both in vCD - happy days.

Now, I'm using vCD 9.7 and when creating a new Provider VDC under the Add Storage window, I can only see storage policies that correspond to datastores that are NOT in any datastore cluster. It doesn't see any datastores that are part of a datastore cluster - exactly what the OP mentions.

The other question is around SDRS settings and whether they are compatible with vCD. For example, assuming we have a number of datastores in a datastore cluster:

Can we enable anti-affinity for VMDKs and thus when vCD creates VMs, the system will automatically place VMDKs on separate backend datastores?
What mechanism chooses which datastore to provision VM disks on? Is it vCD or does vCD just call vCenter API and let SDRS choose?
If SDRS migrates a VMDK from one datastore to another (or is done manually to balance space), will this be picked up by vCD when viewing VM properties as a vCD admin as per the following screenshot (apologies since most of it is blurred out).

Re: How to add "Provider VDC" via HTML5 UI of vCloud Director 9.7 ?

KFM — Thu, 01 Aug 2019 12:07:20 GMT

To the OP, did adding it via API work even if you couldn't see the vCenter server in the inventory? i.e. if you query the API for vCenter server do you see it and thus were you able to add a Provider VDC?

Re: vCD 9.7 Installation Issues

KFM — Thu, 01 Aug 2019 12:01:07 GMT

I can confirm I can mount the NFS point to any other server.

I wasn't able to solve this one yet when I tried the same deployment in production it worked. Go figure.

Re: vCloud Director and Cisco ACI integration

KFM — Thu, 01 Aug 2019 08:35:21 GMT

I guess that leads me to my next question which is whether we can operate vCD without NSX present at all (because we're looking at using ACI for all the SDN stuff). My guess is "no". Here's my scenario:

vCenter is installed
vCD 9.7 cell appliance is installed and configured
NO NSX Manager deployed

So in order to add a vCenter into vCD using the https://vcdcell/cloud UI, part of the setup necessarily includes entering NSX manager details. Without this (because we're not going to use it, ergo it's not installed), I cannot proceed. Onto the next way then....I can use the https://vcdcell/provider UI to attach a vCenter alone but then is it mandatory that I register an NSX(-T) manager or can I just leave the vCenter as a standalone instance? I assume yes because the documentation (Attach a vCenter Server Instance Alone or Together with an NSX Manager Instance ) says:

Assuming I just leave the vCenter as a standalone, no-NSX instance, then in the https://vcdcell/provider UI under vSphere Resources I can see the vCenter itself and also objects in the Distributed Switches and Port Groups but no Hosts. If I use the https://vcdcell/cloud UI then under vSphere Resources I can also see objects in Switches & Port Groups, nothing in Hosts but this time nothing in vCenters. This means I cannot add any Provider VDCs as they need to be backed by a vCenter server resource. See the below screenshots as an example:

Maybe my assumption is correct and the VMware documentation is spot on, but is the reason I cannot see the vCenter server in the https://vcdcell/cloud UI because I haven't attached any NSX manager to the vCenter server and not just some bug or display/sync issue? Meaning that I really cannot operate vCD without a NSX instance?

I guess if the answer is we need an NSX manager then I suppose we can deploy one but not actually use any of the NSX components (DLR, DFW, VXLAN, etc) and hence we won't be billed for any of the components.

vCD 9.7 Installation Issues

KFM — Thu, 25 Jul 2019 13:20:05 GMT

I'm reaching out to the community because I'm at my wits end at deploying the vCD 9.7 OVA using all the methods available. Some info:

I have deployed a NFS server and can confirm the permissions, etc are all correct (more on this later)
I have deployed a RabbitMQ server
Am using VMware_vCloud_Director-9.7.0.4343-14046945_OVF10.ova
Am testing all this in a nested VMware ESXi installation on VMware Workstation
Am attempting to deploy the first primary small appliance with embedded DB
I have forward and reverse DNS resolution entries for the vCD cell

So I've tried the following:

Deploying the OVA to my vESXi host simply by following the bouncing ball and supplying all required information when deploying a VM from OVF/OVA file. All seems fine and I can login to the VAMI on port 5480 however I get the dreaded "no nodes found in the cluster" error in the vCD Database Availability tab.

When I login to the console, and view the setupvcd.log file I get the following:

And the contents of the update-appliance-type.log file

Both log files imply that I have not selected a valid cell type yet I am 100% certain that I chose "primary small" during the OVA deployment.

I can confirm that the vCD cell can reach the NFS transfer server, has mounted the NFS share and that it has successfully written to the directory.

I can also confirm that the user and group ownership are correct.

Getting sick and tired of typing the same thing over and over again during the UI OVA wizard, I tried deploying the OVA to my vESXi host using the ovftool using the same parameters. Now, after powering on I get the following start job which never succeeds and I am left with a non-functioning vCD instance.

This times out and eventually I get to the VAMI screen saying the networking is not configured. Basically the vCD appliance is useless at this point. I cannot even login using the credentials I used during the OVA deployment. Nothing much can be done besides deleting the appliance and starting again.

Here's a copy of the ovftool syntax I'm using. It reflects the same properties I'm using during the UI deployment.

ovftool \

--noSSLVerify \

--acceptAllEulas \

--datastore='Datastore1' \

--allowAllExtraConfig \

--net:"eth0 Network"="VM Network" \

--net:"eth1 Network"="VM Network 2" \

--name=vCD \

--diskMode=thin \

--prop:"vami.ip0.VMware_vCloud_Director"="192.168.32.51" \

--prop:"vami.ip1.VMware_vCloud_Director"="10.0.0.51" \

--prop:"vami.DNS.VMware_vCloud_Director"="192.168.32.30" \

--prop:"vami.domain.VMware_vCloud_Director"="kfmlab.local" \

--prop:"vami.gateway.VMware_vCloud_Director"="192.168.32.2" \

--prop:"vami.netmask0.VMware_vCloud_Director"="255.255.255.0" \

--prop:"vami.netmask1.VMware_vCloud_Director"="255.255.255.0" \

--prop:"vami.searchpath.VMware_vCloud_Director"="kfmlab.local" \

--prop:"vcloudapp.enable_ssh.VMware_vCloud_Director"="True" \

--prop:"vcloudapp.expire_root_password.VMware_vCloud_Director"="False" \

--prop:"vcloudapp.nfs_mount.VMware_vCloud_Director"="192.168.32.61:/storage" \

--prop:"vcloudapp.ntp-server.VMware_vCloud_Director"="0.au.pool.ntp.org" \

--prop:"vcloudapp.varoot-password.VMware_vCloud_Director"="Password01!" \

--prop:"vcloudconf.db_pwd.VMware_vCloud_Director"="Password01!" \

--prop:"vcloudconf.admin_email.VMware_vCloud_Director"="admin@kfmlab.local" \

--prop:"vcloudconf.admin_fname.VMware_vCloud_Director"="vcdadmin" \

--prop:"vcloudconf.admin_pwd.VMware_vCloud_Director"="Password01!" \

--prop:"vcloudconf.admin_uname.VMware_vCloud_Director"="administrator" \

--prop:"vcloudconf.inst_id.VMware_vCloud_Director"="1" \

--prop:"vcloudconf.sys_name.VMware_vCloud_Director"="vcd01" \

--deploymentOption="primary-small"

Any help/assistance/ideas would be greatly appreciated!

Re: vCD Architecture Scale-Out

KFM — Wed, 15 May 2019 01:46:13 GMT

Hi Daniel,

It certainly does help, thank you! Sounds like we may have to deploy a LB in front of the cell. I guess it would have to be deployed anyway when we eventually deploy additional cells so no harm done doing it from the start.

vCD Architecture Scale-Out

KFM — Mon, 13 May 2019 05:54:30 GMT

Hi all,

I'm looking at deploying a vCD environment, initially starting with just a single cell and single RabbitMQ server to gauge actual demand/use of vCD functionality. I will be using the 9.7 appliance with an embedded PSQL database. An external NFS transfer server will also be deployed. No load-balancer required as yet since we only have one cell anyway.

My question is can we build out an environment at a later date when we actually need to expand? For instance, deploy a single cell now and expand to 2, 3, n-number cells with a load-balancer in front. Can we do the same with RabbitMQ too? I can't seem to find any information regarding cell/environment expansion. It appears that the vCD architecture is designed to support scaling-out but just wanted to confirm.

Thanks for your time!

Kam

Re: vCloud Director and Cisco ACI integration

KFM — Thu, 09 May 2019 12:08:01 GMT

Yep, that's what I suspect will be required. I think so long as all the moving parts have an API that can be consumed, then it makes it potentially easier for some orchestration engine to do some of the heavy lifting that would normally be available out of the box.

vCloud Director and Cisco ACI integration

KFM — Thu, 09 May 2019 05:10:56 GMT

Hi all,

I am setting up a new greenfield DC and was just wondering if anyone has any experiencing or knowledge of vCD and Cisco ACI integration? For better or for worse we're not looking at NSX in doing any of the SDN stuff. I know you can use vCD to spin up Edge gateways and DLRs and attach logical switches to them but I was wondering is there the same/similar level of integration with Cisco ACI and the APIC cluster that controls it. Or even if there isn't, how do we expose ACI created dvPGs/networks to vCD?

Worse case scenario is that we simply present tenant's dvPG in vCenter to vCD then manually stitch them to the L3 gateway. This is what we've done in the past and unless someone can shed some light on it, will probably continue to do this with ACI.

Thanks in advanced!

Kam

Re: UAG Load Balancing using HAProxy

KFM — Thu, 18 Apr 2019 14:04:11 GMT

The intention was to have two UAGs in an active/active setup. This requires a load-balancer in front to distribute the incoming sessions to the UAG with the least connections.

As I was deploying the UAG for a Horizon DaaS deployment, I didn't need to put another LB in front of the tenant appliances since these are natively HA out of the box - no further configuration or LB required. I haven't worked with Horizon View for many years so I can't say if putting a LB in front of the connection server is a supported topology.

As for the HAProxy config, here it is below. Note that the design I came up with was to have one public IP address upon which all tenant portal URLs would resolve to. HAProxy would then use SNI to forward the request to that particular tenant's UAG pair. This allowed me to scale out the number of tenant appliances whilst LB the connections through a pair of pfSense/HAProxy appliances. There are obviously a number of different ways you could design this - each with their respective pros and cons.

Hope that helps!

# Automaticaly generated, dont edit manually.

# Generated on: 2019-04-19 08:26

global

maxconn 1000

log /var/run/log local0 info

stats socket /tmp/haproxy.socket level admin

uid 80

gid 80

nbproc 1

hard-stop-after 15m

chroot /tmp/haproxy_chroot

daemon

server-state-file /tmp/haproxy_server_state

listen HAProxyLocalStats

bind 127.0.0.1:2200 name localstats

mode http

stats enable

stats refresh 2

stats admin if TRUE

stats show-legends

stats uri /haproxy/haproxy_stats.php?haproxystats=1

timeout client 5000

timeout connect 5000

timeout server 5000

frontend Universal_UAG_Frontend

bind publicIP:443 name publicIP:443

mode tcp

log global

option tcplog

timeout client 30000

tcp-request inspect-delay 5s

acl TenantA req.ssl_sni -i daas.TenantA.com

acl TenantB req.ssl_sni -i daas.TenantB.com

tcp-request content accept if { req.ssl_hello_type 1 }

use_backend TenantA_UAG_Pool_ipvANY if TenantA

use_backend TenantB_UAG_Pool_ipvANY if TenantB

backend TenantA_UAG_Pool_ipvANY

mode tcp

id 100

log global

stick-table type ip size 50k expire 1h

stick on src

balance leastconn

timeout connect 30000

timeout server 30000

retries 3

option httpchk GET /favicon.ico

server TenantA-UAG1 TenantA_UAG1_PrivateIP:443 id 106 check-ssl check inter 1000 verify none

server TenantA-UAG2 TenantA_UAG2_PrivateIP:443 id 102 check-ssl check inter 1000 verify none

backend TenantB_UAG_Pool_ipvANY

mode tcp

id 103

log global

stick-table type ip size 50k expire 1h

stick on src

balance leastconn

timeout connect 30000

timeout server 30000

retries 3

option httpchk GET /favicon.ico

server TenantB-UAG1 TenantB_UAG1_PrivateIP:443 id 101 check-ssl check inter 1000 verify none

server TenantB-UAG2 TenantB_UAG2_PrivateIP:443 id 102 check-ssl check inter 1000 verify none

Re: UAG Load Balancing using HAProxy

KFM — Wed, 17 Apr 2019 03:52:05 GMT

So yes, I did get it working with HAProxy, albeit only one as I didn't get time to deploy pfSense/HAProxy in a HA setup. I also did get it to load-balance between two UAGs. I can send you the HAProxy configuration if you need it for reference? I never got any double login issues.

In a nutshell, HAProxy will not work with UDP which means that we cannot use Method 1 - Source IP Affinity. I did get it working with Method 2 - Multiple Port Number Groups and also Method 3 - Multiple VIPs. The configuration for those three methods are here: Load Balancing across VMware Unified Access Gateway Appliances.

Re: UAG Load Balancing using HAProxy

KFM — Wed, 02 Jan 2019 14:29:24 GMT

Thanks for your reply. I don't believe it's a connection problem between the UAG and the desktop resources as they're all on the same network - i.e. for now, there is no inner-firewall.

After reading Network Ports in VMware Horizon 7: VMware Horizon 7 version 7.2 it's clear why the browser works - all communication happens over TCP - there is no UDP. And as was already mentioned, HAProxy, for better or for worse, does not do UDP. As mentioned specifically in the HAProxy version 1.7.11 - Starter Guide, it says HAProxy "will not see IP packets nor UDP datagrams". Major bummer

I do find it interesting that when I do a packet capture of a NAPT, non-loadbalancing connection, I do not see any UDP packets (PCoIP 4172) until the actual display starts up. This corresponds to what is mentioned in the Network Ports article in the link above - that all login traffic happens over TCP 443. So if there are no UDP packets during the login stage, then why can I not even bring up the list of available desktops when using the Horizon View client and through a load-balanced connection? Perhaps markbenson can help as he authored this Load Balancing across VMware Unified Access Gateway Appliances and helped immensely in Can BEAT run over a different port than UDP 8443?

Assuming I somehow resolve the "could not establish tunnel" issue, I could have two connection server entries - one for external (uses Blast only) and one for internal which has no protocol restrictions.

I guess for now in order to retain the consistent "one connection server" user experience, it looks like I will need to go down the multiple VIP method instead.

Re: UAG Load Balancing using HAProxy

KFM — Wed, 02 Jan 2019 07:13:00 GMT

Hi Larry,

Thanks for your response. I did read that thread prior to posting - it contains a wealth of knowledge. I guess it's good to know that it is possible but using method 2 where the LB is used for the initial connection then all subsequent traffic goes direct to the UAGs.

If that is really the case then that's a shame with HAProxy. It's interesting that the LB product developed by loadbalancer.org seems to support UDP even though it's purportedly built on HAProxy (http://pdfs.loadbalancer.org/Vmware_Horizon_Deployment_Guide.pdf ) Perhaps they've incorporated bits of IPVS which does layer 4 load-balancing into their product.

It also seems a little strange that I can't build a connection using the Horizon Client after authenticating. i.e. I don't get a list of entitled desktops. I wouldn't have thought this would be using UDP.

As for which protocols I am using, I was hoping to use both Blast and PCoIP. This means a largely seamless experience for both internal and external users with no need to dictate which protocol gets used depending on where they're connecting from. They just fire up the Horizon View client and connect (obviously assuming the FQDN is the same both internally and externally).

I'll try method 2 and let you know how I get on.

Thanks,

Kam

UAG Load Balancing using HAProxy

KFM — Wed, 02 Jan 2019 01:28:37 GMT

Hi community, this problem has been bugging me for a while now so I figured I'd reach out to the community and hopefully get this thing working! Bear with me, this is a long one....

I work for a service provider and want to building out a scalable UAG-pair-per-tenant design behind a pair of HAProxy load-balancers for external (i.e. over the Internet) access. Note that we are using the Horizon DaaS product not the Horizon View product. Apart from the difference in name, I believe the UAG functions identically in both environments.

My final solution would be two pfSense (community edition) firewalls with the HAProxy package installed on both to provide HA and load-balancing functionality to the tenant UAGs behind them. As we onboard more tenants, I would add another pair of UAGs for each. HAProxy will selectively pick which UAG is required for the incoming connection based on SNI. For example:

TenantA - external URL: daas.tenanta.com
TenantB - external URL: daas.tenantb.com
and so on....

Connections arriving at daas.tenanta.com will be directed to a UAG pair for tenantA. Connections arriving at daas.tenantb.com will be directed to a UAG pair for tenantB. And so on and so forth.

As this is my first attempt at building out a HAProxy solution I've decided to keep it simple and use just one pfSense/HAproxy load-balancer, one UAG and one tenant. This is what I have now:

Single pfSense firewall with HAProxy package installed
Single UAG (10.0.0.5) for tenantA
external URL daas.tenanta.com with public IP address (1.1.1.1)
One internal-facing VIP (10.0.0.1) which acts as the gateway for the UAG
Firewall does 1:1 NAT from daas.tenanta.com to VIP (i.e. 1.1.1.1 -> 10.0.0.1)
Firewall rules allowing any -> VIP for ports 443, 8443 and 4172.
UAG is configured using apsetup.sh script. It configures the following settings:
- proxyDestinationURL: https://tenant_appliance_ip
- pcoipExternalUrl: 1.1.1.1:4172
- blastExternalUrl: daas.tenanta.com:8443
- tunnelExternalUrl: daas.tenanta.com:443
UAG gateway is the VIP (10.0.0.1)
HAProxy configuration:
- One frontend for ports 443, 8443, 4172
- Three backends for ports 443, 8443 and 4172 all with tenantA UAG as backend server
  - I've also tried one backend for just port 443 with tenantA UAG as backend server as this seems to work for Blast connections via browser

Frontend acl uses SNI for daas.tenanta.com to send to tenantA UAG backend

In this given configuration, I've observed the following:

Browser access to both the user portal (daas.tenanta.com) and admin portal (daas.tenanta.com/admin) work fine
Initiating a Blast connection to the desktop via the browser, works fine.
Using the Horizon Client, I can authenticate successfully but then get the "could not establish tunnel connection" error message. This is what I ultimately need to get working!

Some things I'm not sure about:

Should I be using layer 4 (tcp) or layer 7 (ssl/https) load balancing?
Should I have multiple backends, one for each port
I really want to use source IP affinity as per Load Balancing across VMware Unified Access Gateway Appliances as I think HAProxy can see the client IP address. (i.e. in the HAProxy logs I can see the client IP address.) I don't really want to go with the other two methods if I can help it.
Why does Blast via a browser work when I just have one backend listening on 443? When I do a netstat on my client I can see an active connection to 1.1.1.1:8443.

Some things I've tried:

Note that when I revert the solution to just a plain firewall bypassing the load-balancer, everything (browser and Horizon client) works fine. i.e. traditional port-forwarding/NAPT to the UAG with FW ACLs allowing any -> UAG:443,8443,4172.
I've also used a second tenant to test the SNI ACL and that seems to work fine too. i.e. I can use blast via browser to both daas.tenanta.com and daas.tenantb.com.
Collected debug logs on Horizon View client
Analysed debug logs on tenant appliance
Analysed UAG logs
In the above three log collections, I saw nothing obvious to my untrained eyes

I'm really at my wits end here so any help would be much appreciated!