vRA's 7.5, vSphere initial Setup fails on "Setup tenant administrator"

[_MikeW_]

Background: We're looking to migrate from Foreman+vSphere to vRA+vSphere. I've taken the vRA install and config and the vRO course.

Test lab is a clean install of vSphere Client version 6.7.0.30000, VMware ESXi, 6.7.0, 8169922 and vRA 7.5.

After initial install I've attempted to run the vSphere Initial setup within vRA catalog and ran into some problems. I found that vSphere 6.5 systems needs to have the vCenter plugin updated (vRO vCenter plug-in  for vSphere 6.5 service releases ) which I did. After that it did progress further but now I am seeing the following error. I'm hoping someone with more experience might have some insight as to what might be the cause because my google-fu isn't turning anything up.

[2019-05-29 15:30:45.052] [I] ********************************************************

[2019-05-29 15:30:45.054] [I] Validate endpoint connection settings

[2019-05-29 15:30:45.059] [I] ********************************************************

[2019-05-29 15:30:45.098] [I] Validation of connection to vCenter server

[2019-05-29 15:30:45.253] [I] Connection to endpoint [vcenter.blackbox.local] completed successfully.

[2019-05-29 15:30:45.262] [I] Validation endpoint compute resource Blackbox

[2019-05-29 15:30:45.294] [I] Compute resource with name [Blackbox] was found in vCenter instance [vcenter.blackbox.local]

[2019-05-29 15:30:45.318] [I] ********************************************************

[2019-05-29 15:30:45.322] [I] Locate Cafe / Infrastrucure Hosts

[2019-05-29 15:30:45.325] [I] ********************************************************

[2019-05-29 15:30:45.581] [I] Cafe host for tenant vsphere.local for user administrator found.

[2019-05-29 15:30:45.585] [I] Cafe host for tenant vsphere.local for user administrator found.

[2019-05-29 15:30:45.588] [I] Cafe host for system tenant created successfully

[2019-05-29 15:30:45.649] [I] Infrastructure host already configured.

[2019-05-29 15:30:45.723] [I] ********************************************************

[2019-05-29 15:30:45.727] [I] Setup tenant

[2019-05-29 15:30:45.733] [I] ********************************************************

[2019-05-29 15:30:45.792] [I] Target tenant vsphere.local

[2019-05-29 15:30:45.811] [I] ********************************************************

[2019-05-29 15:30:45.813] [I] Setup tenant administrator

[2019-05-29 15:30:45.816] [I] ********************************************************

[2019-05-29 15:30:45.868] [E] Error in (Workflow:vSphere Initial Setup / Get Local User (item39)#10) TypeError: Cannot call method "indexOf" of null

[2019-05-29 15:30:45.890] [I] --------------------------------------------------

[2019-05-29 15:30:45.893] [E] Unable to execute step: Setup tenant administrator

[2019-05-29 15:30:45.895] [I] --------------------------------------------------

[2019-05-29 15:30:45.914] [I] Rollback creates entities

[2019-05-29 15:30:45.967] [E] Workflow execution stack:

***

item: 'vSphere Initial Setup/item1', state: 'failed', business state: 'Setup / Locate tenant administrator', exception: 'TypeError: Cannot call method "indexOf" of null (Workflow:vSphere Initial Setup / Get Local User (item39)#10)'

workflow: 'vSphere Initial Setup' (daf78a25-baf7-4e8e-a12a-b7b1b0576795)

|  'attribute': name=errorCode type=string value=TypeError: Cannot call method "indexOf" of null (Workflow:vSphere Initial Setup / Get Local User (item39)#10)

|  'attribute': name=vmTemplateNames type=Array/string value=__NULL__

|  'attribute': name=selectedTemplateNames type=Array/string value=__NULL__

|  'attribute': name=stepName type=string value=Setup tenant administrator

|  'attribute': name=InterfaceTypeId type=string value=vSphere

|  'attribute': name=enterpriseGroupName type=string value=Fabric Group

|  'attribute': name=machinePrefixName type=string value=vm

|  'attribute': name=businessGroupName type=string value=Business Group

|  'attribute': name=reservationName type=string value=Reservation

|  'attribute': name=entitlementName type=string value=Endpoint Provisionig

|  'attribute': name=adminServiceName type=string value=Deployment Infrastructure

|  'attribute': name=machineNumberLength type=number value=2.0

|  'attribute': name=vCenterServerPort type=number value=443.0

|  'attribute': name=vCenterServerPath type=string value=/sdk

|  'attribute': name=entities type=Array/Any value=#{#vCACCAFE:VCACHost#dunes://service.dunes.ch/CustomSDKObject?id='5f975cbd-1cee-4e48-956b-835bfee3d17b'&dunesName='vCACCAFE:VCACHost'#;#vCAC:VCACHost#dunes://service.dunes.ch/CustomSDKObject?id='91000668-5d28-4dd1-84b4-43a7620a5bdd'&dunesName='vCAC:VCACHost'#}#

|  'attribute': name=tenantCafeHost type=vCACCAFE:VCACHost value=__NULL__

|  'attribute': name=iaasHost type=vCAC:VCACHost value=dunes://service.dunes.ch/CustomSDKObject?id='91000668-5d28-4dd1-84b4-43a7620a5bdd'&dunesName='vCAC:VCACHost'

|  'attribute': name=systemTenantCafeHost type=vCACCAFE:VCACHost value=dunes://service.dunes.ch/CustomSDKObject?id='5f975cbd-1cee-4e48-956b-835bfee3d17b'&dunesName='vCACCAFE:VCACHost'

|  'attribute': name=reservationNetworkName type=string value=

|  'attribute': name=reservationStorage type=string value=

|  'attribute': name=resourcePoolName type=string value=

|  'attribute': name=managementEndpointEntity type=vCACCAFE:Endpoint value=__NULL__

|  'attribute': name=connectionTimeout type=number value=60.0

|  'attribute': name=operationTimeout type=number value=60.0

|  'attribute': name=availableStoreNames type=Array/string value=__NULL__

|  'attribute': name=availableResourcePoolNames type=Array/string value=__NULL__

|  'attribute': name=availableNetworkNames type=Array/string value=__NULL__

|  'attribute': name=cafeHosts type=Array/vCACCAFE:VCACHost value=#{#vCACCAFE:VCACHost#dunes://service.dunes.ch/CustomSDKObject?id='5f975cbd-1cee-4e48-956b-835bfee3d17b'&dunesName='vCACCAFE:VCACHost'#;#vCACCAFE:VCACHost#dunes://service.dunes.ch/CustomSDKObject?id='789a623f-65df-4272-8896-886b1f66c9ac'&dunesName='vCACCAFE:VCACHost'#;#vCACCAFE:VCACHost#dunes://service.dunes.ch/CustomSDKObject?id='951dd9f8-cf9a-4b5a-8f26-f2463fedbba6'&dunesName='vCACCAFE:VCACHost'#}#

|  'attribute': name=systemTenantDomain type=string value=vsphere.local

|  'attribute': name=cafeHostHostName type=string value=vra.blackbox.local

|  'attribute': name=systemTenantUserName type=string value=Administrator@vsphere.local

|  'attribute': name=systemTenantName type=string value=vsphere.local

|  'attribute': name=tenantAdminEntity type=Any value=Any#__NULL__

|  'attribute': name=targetTenantName type=string value=vsphere.local

|  'attribute': name=managementEndpointHosts type=Array/vCAC:Entity value=__NULL__

|  'input': name=endPointName type=string value=vCenter

|  'input': name=endPointHost type=string value=vcenter.blackbox.local

|  'input': name=endPointComputeResourceName type=string value=Blackbox

|  'input': name=endPointAdminUserName type=string value=administrator@drwho.org

|  'input': name=endPointAdminPassword type=SecureString value=__NULL__

|  'input': name=newTenantAdminFirstName type=string value=

|  'input': name=newTenantAdminLastName type=string value=

|  'input': name=newTenantAdminUsername type=string value=

|  'input': name=newTenantAdminPassword type=SecureString value=__NULL__

|  'input': name=newTenantAdminEmailAddress type=string value=

|  'input': name=newTenantName type=string value=

|  'input': name=createNewTenant type=boolean value=false

|  'input': name=useCurrentTenant type=boolean value=true

|  'input': name=systemTenantUserPassword type=SecureString value=__NULL__

|  'input': name=dataCollectionTimeOut type=number value=240.0

|  'input': name=existingTenant type=string value=

|  'input': name=existingTenantAdminUsername type=string value=

|  'input': name=existingTenantAdminPassword type=SecureString value=__NULL__

|  'no outputs'

*** End of execution stack.

[daphnissov]

To be quite honest, I never have recommended using this feature because of how wonky (due to complexity) it is. And because you'll almost certainly have to clean up after it, I usually recommend going through the configuration steps manually so you understand what's necessary to get vRA working and so you'll have more control over the process.

[_MikeW_]

I've actually been walking through the vRA 7.5 config PDF (700 pages so not nearly done) and when I got to endpoints to add vSphere I couldn't get that to work properly so figured perhaps the workflow would be better. When I run the "test connection" for adding the vCenter endpoint in vRA it fails with "System exception". Which, is clear as mud to me there as well. I'm guess there is some fiddly bit somewhere that I need to set for either of these to work I'm just not clear where/what.

[daphnissov]

Show a screenshot of your log at Infrastructure => Monitoring => Log, please.

[_MikeW_]

So it looks like I had filled in the request form incorrectly. Once I corrected that it stopped having issues with the "Tenant" creation (I didn't want a new one anyway).

Now it seems to be stalling at trying to find compute. Loops then fails. While trying to see what could cause that I noticed connectivity between IaaS->vCenter was "odd". I can ping from IaaS to vCenter server but I can't get to the vCenter http port from the IaaS host (just trying to get to the url via chrome on the IaaS host). Completely disabled the firewall on IaaS host to verify it's not that (this is all on private lab) and didn't seem to change anything. I double checked my vSphere firewall allows private ip space (it does).

I'm walking through various things now to try and figure out what's blocking the connection because I suspect this is the cause for the timeout during the Workflow.

[_MikeW_]

Once I resolved the connectivity between iaas and DNS issues I was able to run the initial vsphere setup workflow.