Good Afternoon,
1. For a couple of weeks now I have been working on completing what should be a fairly straight forward task of upgrading a fully integrated 6.0 VCSA to 6.7 VCSA.
2. Stage 1: Deploy vCenter Server Appliance with an Embedded Platform Services Controller completes without issue.
Image 1.1. Upgrade Stage 2: Data transfer and appliance setup is in progress
3. Stage 2 completes two of the three tasks and then complains that the source vCenter has been powered off. This seems like a problem with the workflow. I've tried powering the old vCenter back on and retrying the operation but it simply fails again in error.
Image 1.2 VCSA 6.7 Upgrade Failure Error:
4. The new appliance has an error which indicates the IMPORT failed. It still has the old IP address on the console. When I reboot the target appliance it switches to the new Appliance FQDN and IP address but the appliance is clearly broken. The console still shows Upgrade Import error:
Image 1.3 VCSA 6.7 VCSA appliance
5. When I logon to the VCSA 6.7 at https://FQDN.TLD/:5480 I see the following error message:
Image 1.4 VCSA 6.7 Appliance Broken message.
I think I am doing everything correctly. I don't know if there is another way to go forward. I want to say that several years back there was talk of being able to stand up a new appliance. (Standalone) and them manually import the configuration? If anyone has seen this problem and have a solution I am all ears.
PS. I have a ticket open with VMware support and I have provided them with the logs but I have not received any actionable responses from them yet. Other than try powering on the previous appliance to complete the final step.
Thank you.
Hello Lee,
It was a pleasure working with you on this case.
To ensure clarity on the resolution of your issue and as a record for yourself below is a summary of what we worked on:
Summary:
6.0 to 6.7 appliance upgrade is failing with unable to lookup hostname
Cause and Resolution:
Found the following from the source 6.0 vCenter:
IP/Hostname has been masked for security reasons.
---------------------------------------------
# /opt/vmware/share/vami/vami_get_network
interface: eth0
config_present: true
config_flags: STATICV4+STATICV6
config_ipv4addr: 192.168.1.10
config_netmask: 255.255.255.0
config_broadcast: 192.168.1.255
config_gatewayv4:
config_ipv6addr: fe80::251:45ff:fef9:441d
config_prefix: 64
config_gatewayv6:
autoipv6:
active_ipv4addr: 192.168.1.10
active_netmask: 255.255.0.0
active_broadcast: 192.168.1.255
active_ipv6addr:
active_prefix:
active_gatewayv4: 192.168.1.1
active_gatewayv6:
---------------------------------------------
From the above output, note that "config_netmask" and "active_netmask" has two different values.
The "active_netmask: 255.255.0.0" is the correct subnet for this environment
After upgrade:
Ran the following command on the new upgraded 6.7 vCenter appliance:
---------------------------------------------
/opt/vmware/share/vami/vami_config_net
Network Configuration for eth0
IPv4 Address: 192.168.1.10
Netmask: 255.255.255.0
IPv6 Address:
Prefix:
Global Configuration
IPv4 Gateway:
IPv6 Gateway:
Hostname: abc.xyz.com
DNS Servers: 127.0.0.1, 192.168.1.10
Domain Name:
Search Path:
Proxy Server:
---------------------------------------------
Manually edited the configuration using the options available in "/opt/vmware/share/vami/vami_config_net"
After changes:
---------------------------------------------
Network Configuration for eth0
IPv4 Address: 192.168.1.10
Netmask: 255.255.0.0
IPv6 Address:
Prefix:
Global Configuration
IPv4 Gateway: 192.168.1.1
IPv6 Gateway:
Hostname: abc.xyz.com
DNS Servers: 192.168.1.10
Domain Name:
Search Path:
Proxy Server:
---------------------------------------------
Once the changes were made, rebooted the vCenter and all services came back online and functional.
HA alarms were cleared and vCenter was stable.
Regards,
Jonathan
vSphere Install Upgrade Team
_______________________________________________________________________________________________________
"Did you find this helpful? Let us know by completing this survey (takes 1 minute!)"
Hello levans01
Thanks for posting in VMware communities.
A quick note: After stage 1 is completed take a snapshot of the appliance so that next time if it fails, you can resume from where stage 1 is complete.
By default, during stage 2, the source appliance will be powered off and then first boot happens on the destination upgraded vCenter appliance.
Can I have the SR number so that I can take a quick look at the logs if you have uploaded it.
Regards,
Jonathan
vSphere Install Upgrade Team
Johnathan,
Thanks for the quick answer and the TIP. Here's the SR 20099508402
Thank you.
Hello levans01
I have reviewed the logs.
Here are my findings:
===================================
bootstrap.log
2020-02-07T18:07:50.881Z ERROR transport Command ['/usr/bin/python3', '/usr/lib/vmware/cis_upgrade_runner/UpgradeOrchestrator.py', '-m', 'import', '-f', 'upgrade-import-config.json', '-o', '/var/log/vmware/upgrade/import.json', '--logDir', '/var/log/vmware/upgrade', '--logFileName', 'import-upgrade-runner.log', '--cancelFile', '/var/tmp/upgrade_cancel.op', '-l', 'en', '--logLevel', 'INFO', '--disableScreenLog'] exit-code=1, stdout=, stderr=
2020-02-07T18:07:50.881Z ERROR __main__ ERROR: Fatal error during upgrade IMPORT. For more details take a look at: /var/log/vmware/upgrade/import-upgrade-runner.log
2020-02-07T18:07:50.882Z INFO root Exiting with exit-code 1
import-upgrade-runner.log
2020-02-07T18:07:50.795Z INFO UpgradeRunner Loading upgrade workflow context from /storage/seat/cis-export-folder/system-data/UpgradeRunner.ctx..
2020-02-07T18:07:50.795Z INFO config.config_loader Source com.vmware.vpxd endpoint is not specified and its components could not be found automatically
2020-02-07T18:07:50.796Z WARNING networking_utils Could not find address info for 127.0.0.1
2020-02-07T18:07:50.796Z INFO config.credentials Credentials are not defined for component -- com.vmware.vpxd
2020-02-07T18:07:50.797Z ERROR networking_utils Could not validate host xyz.abc.local: [Errno -3] Temporary failure in name resolution
2020-02-07T18:07:50.798Z ERROR UpgradeRunner Upgrade Runner has encountered an exception
Traceback (most recent call last):
File "/usr/lib/vmware/cis_upgrade_runner/UpgradeRunner.py", line 1771, in main
credentials.loadCredentials(configData)
File "/usr/lib/vmware/cis_upgrade_runner/py/config/credentials.py", line 74, in loadCredentials
===================================
I have scrubbed your FQDN for privacy and security. Changed it to xyz.abc.local
From the above logs, it looks like name resolution is failing.
Plan:
Make sure to keep the source appliance, the jumpbox from where you run the installer and the destination appliance all three machines on the same port group, same esxi host. This will eliminate most of the network problems as the VMs will be communicating within the virtual switch portgroup.
Make sure to keep vCenter DRS in manual mode
From the jump box, using command prompt verify both forward and reverse lookup of the vCenter FQDN.
Regards,
Jonathan
vSphere Install Upgrade Team
Johnathon,
I will try that to see if it solves my problem. I will post and update on my steps and the outcomes.
Thanks!
1. Forward and Reverse lookup names are working from the Jump Box where the installer is running for the VCSA FQDN.
2. All machines are in the same port group on the VDS. They are all pointing to the same DNS servers.
3. I move all machines into the host same host. (Source Appliance, Destination Appliance, Windows VUM machine and Jumpbox.)
4. Started Stage 2 of the installation - Failed at the same point. (It failed) #2 made it to 100% before generating an error.
Image 1.1 Final error.
Hello,
The engineer working on the SR has been notified.
We both will work internally with you to get on a remote session and get this sorted tomorrow morning hours.
Regards,
Jonathan
VMware Install Upgrade Team
Hello Lee,
It was a pleasure working with you on this case.
To ensure clarity on the resolution of your issue and as a record for yourself below is a summary of what we worked on:
Summary:
6.0 to 6.7 appliance upgrade is failing with unable to lookup hostname
Cause and Resolution:
Found the following from the source 6.0 vCenter:
IP/Hostname has been masked for security reasons.
---------------------------------------------
# /opt/vmware/share/vami/vami_get_network
interface: eth0
config_present: true
config_flags: STATICV4+STATICV6
config_ipv4addr: 192.168.1.10
config_netmask: 255.255.255.0
config_broadcast: 192.168.1.255
config_gatewayv4:
config_ipv6addr: fe80::251:45ff:fef9:441d
config_prefix: 64
config_gatewayv6:
autoipv6:
active_ipv4addr: 192.168.1.10
active_netmask: 255.255.0.0
active_broadcast: 192.168.1.255
active_ipv6addr:
active_prefix:
active_gatewayv4: 192.168.1.1
active_gatewayv6:
---------------------------------------------
From the above output, note that "config_netmask" and "active_netmask" has two different values.
The "active_netmask: 255.255.0.0" is the correct subnet for this environment
After upgrade:
Ran the following command on the new upgraded 6.7 vCenter appliance:
---------------------------------------------
/opt/vmware/share/vami/vami_config_net
Network Configuration for eth0
IPv4 Address: 192.168.1.10
Netmask: 255.255.255.0
IPv6 Address:
Prefix:
Global Configuration
IPv4 Gateway:
IPv6 Gateway:
Hostname: abc.xyz.com
DNS Servers: 127.0.0.1, 192.168.1.10
Domain Name:
Search Path:
Proxy Server:
---------------------------------------------
Manually edited the configuration using the options available in "/opt/vmware/share/vami/vami_config_net"
After changes:
---------------------------------------------
Network Configuration for eth0
IPv4 Address: 192.168.1.10
Netmask: 255.255.0.0
IPv6 Address:
Prefix:
Global Configuration
IPv4 Gateway: 192.168.1.1
IPv6 Gateway:
Hostname: abc.xyz.com
DNS Servers: 192.168.1.10
Domain Name:
Search Path:
Proxy Server:
---------------------------------------------
Once the changes were made, rebooted the vCenter and all services came back online and functional.
HA alarms were cleared and vCenter was stable.
Regards,
Jonathan
vSphere Install Upgrade Team
_______________________________________________________________________________________________________
"Did you find this helpful? Let us know by completing this survey (takes 1 minute!)"