jrhaakenson's Posts

My issue is fixed and resolved now.  It seems that a VM i had migrated from one environment to the vSAN datastore on this environment was corrupted somehow.  It may have been corrupted by some Veeam backup software running backup tasks overnight and messing up the VM file structure on the vSAN datastore but I'm not completely sure on that.  The migrated VM on the datastore was in a directory of VM_Name->VM_Name->VM files rather than just VM_Name->VM files.  There was an extra VM_Name directory on the vSAN datastore (possibly created by Veeam).  So I completely deleted the troublesome VM directory on the vSAN datastore, through vCenter (luckily this VM was unimportant), rebooted the ESXi host, and upon reboot the host services all started and the host reconnected to vCenter and the vSAN network again.  I don't really understand how a single powered off VM on the vSAN datastore could cause the ESXi host management services to fail starting up (even after multiple restarts) but that seems to be what was causing the issue.  All resolved now.

Good news.  After 2.5 months, I have a working solution for this.  The binary download repository for vRLCM is https://download2.vmware.com.  There are a large number of other VMware download repositories that constantly change IP addresses, but we won't get into that here.  First make sure your DNS server can resolve VMware's plethora of download repositories.  But my vRLCM DNS could so this was not the issue. What pointed me in the right direction was running the command: curl -v https://download2.vmware.com.  This output a refused TLS 1.2 connection.  I fixed a similar issue on my vIDM appliance by modifying the /etc/ssh/sshd_config file line 117 to remove some troublesome ciphers.  So I accessed my vRLCM sshd_config file located in /etc/ssh/sshd_config and scrolled down to line 117.  The first two ciphers listed are aes256-gcm@openssh.com and aes128-gcm@openssh.com.  By removing these two ciphers and saving the sshd_config file I was able to finally open a TLS session with https://download2.vmware.com and download the binary files I needed to upgrade my managed vRealize appliance from the vRLCM.  My cipher list on line 117 of sshd_config only contains aes256-ctr,aes192-ctr,aes128-ctr now and this seems to work.  I'm not sure what the issue with the first two ciphers mentioned above was or why they were allowed to certain VMware update repositories (such as vrealize-update.vmware.com) but not download2.vmware.com.  Furthermore I'm not sure why these two ciphers by default do not affect other users but affected my appliance.  I requested answers to these questions from my open VMware support ticket but they have not been able to provide answers at this time.  Still I hope this information is useful to any users who experience update issues with vRealize appliances.  I have used it to fix update issues on both vRLCM and vIDM in my environment.

I found the solution.  In vSphere under Configure->Advanced Settings, the Advanced vCenter Server Setting vpxd.certmgmt.mode was configured as custom.  I changed it to thumbprint and it let me add the ESXi hosts.  I believe our intent is to manage our own certificates on the ESXi hosts, but I'll need check with my certificate admin to see how we are doing it.   If this value is set to custom does that mean that a custom certificate must be installed on the ESXi host for it to be managed by vSphere?  Likewise if it is set to Thumbprint, will vSphere add the SSL thumbprint and manage the host that way?