Old vCenter IP in /opt/vmware/etc/vami/ovfEnv.xml

[racom]

I was readdressing one part of vSphere 6.0 datacenter including VCSA and VR appliances. After changing VR IP I was able to connect it to VCSA. But since I've changed VCSA IP I can't change VCSA IP in VAMI because of "Bad exit code: 1" problem and VCSA can't connect VR. Replication jobs to DR run until reboot. Powering VR off and on didn't resolve problem and VRM service doesn't run. I can see errors in /var/log/boot.msg:

haveged already running

Waiting for network to come up (attempt 1 of 10)...

Configuration for eth0 found

No route to host

Exception in thread "main" java.lang.RuntimeException: java.util.concurrent.ExecutionException:

com.vmware.vim.vmomi.client.exception.ConnectionException: java.net.NoRouteToHostException: No

route to host

        at com.vmware.jvsl.run.CheckedRunnable.withoutChecked(CheckedRunnable.java:27)

        at com.vmware.jvsl.run.VlsiRunnable$1.run(VlsiRunnable.java:104)

        at com.vmware.jvsl.run.ExecutorRunnable.withExecutor(ExecutorRunnable.java:17)

... etc ...

        at com.vmware.jvsl.evs.EVServiceRegistrar.register(EVServiceRegistrar.java:63)

        ... 10 more

Initialization error: No route to host

Traceback (most recent call last):

  File "/opt/vmware/share/htdocs/service/hms/cgi/boot.py", line 153, in main

    __ROUTINES__[name]()

  File "/opt/vmware/share/htdocs/service/hms/cgi/boot.py", line 76, in func

    return fn(*args)

  File "/opt/vmware/share/htdocs/service/hms/cgi/boot.py", line 520, in update_extension_login

    ovf.hms_register_extension()

  File "/opt/vmware/share/htdocs/service/hms/cgi/ovf.py", line 438, in hms_register_extension

    jks, passwd, alias

  File "/opt/vmware/share/htdocs/service/hms/cgi/ovf.py", line 360, in register_extension

    commands.execute(cmd, 'stdout', __HMS_HOME__)

  File "/opt/vmware/share/htdocs/service/hms/cgi/commands.py", line 326, in execute

    raise Exception(last_line)

Exception: No route to host

Unable to obtain password from OVF Environment.

<notice -- Nov 10 12:27:05.529875000> 'vaos start' exits with status 0

I've found old vCenter IP in /opt/vmware/etc/vami/ovfEnv.xml and try to change it but it change back after reboot. vSphere Replication is configured to use the embedded vPostgreSQL database and it looks like old VCSA IP is somewhere there.

I can see this errors in /opt/vmware/hms/logs/hms.log:

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'primary

GroupResumedEventHandler' defined in class path resource [META-INF/eventlog.xml]: Cannot resolv

e reference to bean 'primaryGroupPauseResumeEventHandler' while setting bean property 'targetOb

ject'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creat

ing bean with name 'primaryGroupPauseResumeEventHandler': Injection of autowired dependencies f

ailed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not a

utowire field: private com.vmware.hms.server.ManagedObjectFactory com.vmware.hms.eventhandlers.

PrimaryGroupPauseResumeEventHandler.managedObjectFactory; nested exception is org.springframewo

rk.beans.factory.BeanCurrentlyInCreationException: Error creating bean with name 'siteManager':

Requested bean is currently in creation: Is there an unresolvable circular reference?

Any idea to solve it, please?

[mvalkanov]

Hi,

One setting that must be up to date is VirtualCenter.FQDN (see https://vcenter_address/mob/?moid=VpxSettings&doPath=setting%5b%22VirtualCenter%2eFQDN%22%5d - this should also be visible at the Web Client, but I don't remember the exact location).

The VR appliance on every boot retrieves vCenter address from the OVF environment.

This OVF environment is kept at the ovfEnv.xml you mention below.

If the ovfEnv.xml is not empty, "Save and Restart" from the VR appliance VAMI UI should get you working with the new vCenter address (the Bad exit code: 1 error happens when ovfEnv.xml is empty - usually if the VR appliance is powered on directly from host, or if somehow the data in vCenter DB about the VR appliance OVF environment is missing - can happen if the host of the appliance is removed from vCenter and then re-added back - the VR appliance VM managed object id changes and the old environment is gone). Given that you have a non-empty ovfEnv.xml, it is strange that you hit the "Bad exit code: 1" error in the VAMI UI.

The OVF environment is pushed to the VR appliance at each power on of the VR appliance through vCenter. Reboot is not enough, but a power off and power on (and the power on must not be through ESXi or command line, but through vCenter).

The OVF environment source of truth is kept at vCenter DB at table VPX_EXT_DATA with key something like com.vmware.vim.vsm and VirtualMachine:vm-190 where vm-190 is the managed object id of the VR appliance VM in vCenter.

Perhaps you can try to check if the OVF environment data at vCenter DB is correctly updated with the new vCenter address?

If the above info does not help resolve the issue, please consider opening a support ticket.

Regards,

Martin

[racom]

Hi Martin,

thanks a lot for your answer. It looks like VirtualCenter.FQDN key is bad. I can see old IP of VCSA value there.

After IP change I was moving VCSA to other VLAN and I switched VM Network via vSphere Client to host 'cause management IP of VCSA became unaccessible. I couldn't login using vSphere Web Client nor VAMI then with "Unable to authenticate user" message. I logged to SSH console, run shell and found that network mask and brd stayed the old ones. I've set correct settings in /etc/sysconfig/network/ifcfg-eth0 and restarted network service. I've found old VCSA IP in /etc/hosts and correct it using vim. Everything starts to work then except vSphere Replication. This manual changes probably lead to disproportion with database keys. Is any way to correct it and change the VirtualCenter.FQDN key? VCSA was replicated so I have replica files in DR area if it'd be usable somehow.

I can open a support ticket if solution is out of scope of this forum.

Regards

Tomas

[mvalkanov · Accepted solution]

Hi,

You can update the incorrect value using the vSphere Web Client:

vCenter -> Manage -> Settings -> Advanced vCenter Server Settings -> Edit -> VirtualCenter.FQDN

Afterwards, power off the VR appliance and power it back on.

Once the correct VirtualCenter.FQDN value appears in the ovfEnv.xml, open the VAMI UI, verify the vCenter address is correct, hit "Save and Restart" and everything should be fine.

Regards,

Martin

[racom]

Hi Martin,

it works. Thanks a lot for your support.

Regards

Tomas

[racom]

Hi,

can I have one more question?

Almost everything works well now. But I found target site not authenticated in vCenter vSphere Replication tab and if I try to reconnect it I got "javax.net.ssl.sslhandshakeexception remote host closed connection during handshake" error message. If I log to DR site vCenter and try to reconnect I got "Cannot connect to the specified site, site might not be available on the network or the network configuration may not be correct replication" message. Both sites were connectable before I changed VCSA and VR IPs. Is it related with IP changes?

Regards

Tomas