12 Replies Latest reply on Nov 15, 2019 3:31 PM by dirch201110141

    HMS services failed to start in vsphere replication appliance 8.2

    Akide Novice

      recently i made a upgrade for vsphere replication appliance from 8.1 to 8.2, after upgrade i went to web console ip:5480 saws that

      VRM service is

      stopped

       

      As the same configuration like before the VRM services still not start. And when i went to web interface ip/dr/#/home, there shows that HMS services can not access.

      After that i check the HMS services by shell in appliance got these information attached below:

       

       

       

      * hms.service - vSphere Replication Management Server

         Loaded: loaded (/lib/systemd/system/hms.service; enabled; vendor preset: enabled)

         Active: failed (Result: exit-code) since Sun 2019-06-09 23:06:35 UTC; 19s ago

        Process: 3873 ExecStart=/usr/java/default/bin/java $JAVA_FILE_ENCODING_OPTION $JAVA_METASPACE_OPTIONS $JAVA_GC_OPTIONS $JAV

        Process: 3869 ExecStartPre=/opt/vmware/hms/bin/checkHmsConfigured.sh (code=exited, status=0/SUCCESS)

        Process: 3787 ExecStartPre=/opt/vmware/hms/bin/checkHmsAppliance.sh (code=exited, status=0/SUCCESS)

      Main PID: 3873 (code=exited, status=1/FAILURE)

       

       

      Jun 09 23:06:34 vsr hms[3873]:         at com.vmware.hms.monitor.host.HostInventoryManagerImpl.lambda$0(HostInventoryManagerI

      Jun 09 23:06:34 vsr hms[3873]:         at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)

      Jun 09 23:06:34 vsr hms[3873]:         at com.vmware.hms.monitor.host.HostInventoryManagerImpl.getHostVersion(HostInventoryMa

      Jun 09 23:06:34 vsr hms[3873]:         at com.vmware.hms.vmcrypt.hks.impl.HostKeyServiceImpl.getHostDatastoreMounts(HostKeySe

      Jun 09 23:06:34 vsr hms[3873]:         at com.vmware.hms.vmcrypt.hks.impl.HostKeyServiceImpl.init(HostKeyServiceImpl.java:95)

      Jun 09 23:06:34 vsr hms[3873]:         at com.vmware.hms.HmsService.startupProcedureStage1(HmsService.java:232)

      Jun 09 23:06:34 vsr hms[3873]:         ... 9 more

      Jun 09 23:06:35 vsr systemd[1]: hms.service: Main process exited, code=exited, status=1/FAILURE

      Jun 09 23:06:35 vsr systemd[1]: hms.service: Unit entered failed state.

      Jun 09 23:06:35 vsr systemd[1]: hms.service: Failed with result 'exit-code'.

       

       

      Do anyone have idea to fix this problem?

      Thanks

        • 2. Re: HMS services failed to start in vsphere replication appliance 8.2
          Akide Novice

          Could i know which log file is related to this services.

          I am not sure the location of log file.

          Thanks

          • 3. Re: HMS services failed to start in vsphere replication appliance 8.2
            daphnissov Guru
            Community WarriorsvExpert

            I bet Google knows where those log files are.

            • 4. Re: HMS services failed to start in vsphere replication appliance 8.2
              Akide Novice

              Log details from /opt/vmware/hms/logs/hms.log

               

              root@vsr [ ~ ]# cat /opt/vmware/hms/logs/hms.log | grep -v -e 'INFO' -e 'DEBUG' -e 'TRACE'

               

              2019-06-10 08:50:31.574 WARN  org.hibernate.orm.deprecation [main] (..hibernate.id.SequenceGenerator)  | HHH90000014: Found use of deprecated [org.hibernate.id.SequenceGenerator] sequence-based id generator; use org.hibernate.id.enhanced.SequenceStyleGenerator instead.  See Hibernate Domain Model Mapping Guide for details.

              2019-06-10 08:50:31.574 WARN  org.hibernate.orm.deprecation [main] (..hibernate.id.SequenceGenerator)  | HHH90000014: Found use of deprecated [org.hibernate.id.SequenceGenerator] sequence-based id generator; use org.hibernate.id.enhanced.SequenceStyleGenerator instead.  See Hibernate Domain Model Mapping Guide for details.

              2019-06-10 08:50:31.575 WARN  org.hibernate.orm.deprecation [main] (..hibernate.id.SequenceGenerator)  | HHH90000014: Found use of deprecated [org.hibernate.id.SequenceGenerator] sequence-based id generator; use org.hibernate.id.enhanced.SequenceStyleGenerator instead.  See Hibernate Domain Model Mapping Guide for details.

              2019-06-10 08:50:31.575 WARN  org.hibernate.orm.deprecation [main] (..hibernate.id.SequenceGenerator)  | HHH90000014: Found use of deprecated [org.hibernate.id.SequenceGenerator] sequence-based id generator; use org.hibernate.id.enhanced.SequenceStyleGenerator instead.  See Hibernate Domain Model Mapping Guide for details.

              2019-06-10 08:50:40.952 WARN  com.vmware.hcs.gateway.vcd.VcdGatewayProperties [main] (..gateway.vcd.VcdGatewayProperties)  | Couldn't find a property 'vcd-vapp-monitor-polling-interval'. Using the default.

              2019-06-10 08:50:40.953 WARN  com.vmware.hcs.gateway.vcd.VcdGatewayProperties [main] (..gateway.vcd.VcdGatewayProperties)  | Couldn't find a property 'vcd-vapp-status-timeout'. Using the default.

              2019-06-10 08:50:40.954 WARN  com.vmware.hcs.gateway.vcd.VcdGatewayProperties [main] (..gateway.vcd.VcdGatewayProperties)  | Couldn't find a property 'vcd-task-polling-admin-query'. Using the default.

              2019-06-10 08:50:40.954 WARN  com.vmware.hcs.gateway.vcd.VcdGatewayProperties [main] (..gateway.vcd.VcdGatewayProperties)  | Couldn't find a property 'service.name'. Using the default.

              2019-06-10 08:50:40.966 WARN  com.vmware.hcs.gateway.vcd.VcdGatewayProperties [main] (..gateway.vcd.VcdGatewayProperties)  | Couldn't find a property 'service.namespace'. Using the default.

              2019-06-10 08:50:40.966 WARN  com.vmware.hcs.gateway.vcd.VcdGatewayProperties [main] (..gateway.vcd.VcdGatewayProperties)  | Couldn't find a property 'service.vcloud-namespace'. Using the default.

              2019-06-10 08:50:40.966 WARN  com.vmware.hcs.gateway.vcd.VcdGatewayProperties [main] (..gateway.vcd.VcdGatewayProperties)  | Couldn't find a property 'service.view-right-name'. Using the default.

              2019-06-10 08:50:40.967 WARN  com.vmware.hcs.gateway.vcd.VcdGatewayProperties [main] (..gateway.vcd.VcdGatewayProperties)  | Couldn't find a property 'service.manage-right-name'. Using the default.

              2019-06-10 08:50:40.967 WARN  com.vmware.hcs.gateway.vcd.VcdGatewayProperties [main] (..gateway.vcd.VcdGatewayProperties)  | Couldn't find a property 'service.admin-right-name'. Using the default.

              2019-06-10 08:50:40.968 WARN  com.vmware.hcs.gateway.vcd.VcdGatewayProperties [main] (..gateway.vcd.VcdGatewayProperties)  | Couldn't find a property 'service.view-right-vcd-form'. Using the default.

              2019-06-10 08:50:40.968 WARN  com.vmware.hcs.gateway.vcd.VcdGatewayProperties [main] (..gateway.vcd.VcdGatewayProperties)  | Couldn't find a property 'service.manage-right-vcd-form'. Using the default.

              2019-06-10 08:50:40.970 WARN  com.vmware.hcs.gateway.vcd.VcdGatewayProperties [main] (..gateway.vcd.VcdGatewayProperties)  | Couldn't find a property 'vcd-client-connect-timeout'. Using the default.

              2019-06-10 08:50:40.970 WARN  com.vmware.hcs.gateway.vcd.VcdGatewayProperties [main] (..gateway.vcd.VcdGatewayProperties)  | Couldn't find a property 'vcd-client-read-timeout'. Using the default.

              []

              2019-06-10 08:50:52.496 WARN  com.vmware.jvsl.sessions.net.VcVimServiceVersions [main] (..sessions.net.VcVimServiceVersions)  | Latest VC VMODL version '6.7.2' is not supported.

              2019-06-10 08:50:52.750 ERROR com.vmware.hms.HmsService [main] (..vmware.hms.HmsService)  | stage 1 starting...FAILED

              2019-06-10 08:50:52.754 FATAL hms [main] (..vmware.hms.App)  |

               

               

              HMS SERVER ERROR

               

               

              java.lang.RuntimeException: HMS Server failed to start successfully:

              at com.vmware.hms.HmsService.startupProcedureStage1(HmsService.java:249)

              at com.vmware.hms.HmsService.onRefresh(HmsService.java:447)

              at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:880)

              at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:545)

              at com.vmware.jvsl.util.SpringService.init(SpringService.java:152)

              at com.vmware.jvsl.util.SpringService.init(SpringService.java:125)

              at com.vmware.jvsl.util.SpringService.<init>(SpringService.java:61)

              at com.vmware.hms.HmsSingletonSpringService.<init>(HmsSingletonSpringService.java:124)

              at com.vmware.hms.HmsSingletonSpringService.init(HmsSingletonSpringService.java:99)

              at com.vmware.hms.App.main(App.java:118)

              Caused by: java.lang.NullPointerException

              at com.vmware.hms.monitor.host.HostInventoryManagerImpl.loadHostVersion(HostInventoryManagerImpl.java:650)

              at com.vmware.hms.monitor.host.HostInventoryManagerImpl.lambda$0(HostInventoryManagerImpl.java:630)

              at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)

              at com.vmware.hms.monitor.host.HostInventoryManagerImpl.getHostVersion(HostInventoryManagerImpl.java:630)

              at com.vmware.hms.vmcrypt.hks.impl.HostKeyServiceImpl.getHostDatastoreMounts(HostKeyServiceImpl.java:141)

              at com.vmware.hms.vmcrypt.hks.impl.HostKeyServiceImpl.init(HostKeyServiceImpl.java:95)

              at com.vmware.hms.HmsService.startupProcedureStage1(HmsService.java:232)

              ... 9 more

              • 5. Re: HMS services failed to start in vsphere replication appliance 8.2
                daphnissov Guru
                vExpertCommunity Warriors

                It looks like a generic message. I'd try to first unregister and re-register the extension. Failing that, attach a new log file, don't paste a snippet.

                • 6. Re: HMS services failed to start in vsphere replication appliance 8.2
                  Akide Novice

                  Thanks for follow up, there is the log file of HMS service.

                  And i already tired to unregister VRMS and delete vSphere replication appliance and reinstall it.

                  This log took from a brand new installed vSphere replication appliance but problem is still occurred.

                  • 7. Re: HMS services failed to start in vsphere replication appliance 8.2
                    sajithru Novice

                    I'm having the same issue. Have you found a solution?

                    • 8. Re: HMS services failed to start in vsphere replication appliance 8.2
                      Akide Novice

                      No,i didn't find any useful information or solution. Through the test on my end, 8.1 version works properly but can not support vshpere 6.7 u2. In vmware docs, only 8.2 version support 6.7 u2, but it can not connect to vcenter properly. I quickly scanned logs, it's may a vmware software issue related to java give bad tracks back. 

                      • 9. Re: HMS services failed to start in vsphere replication appliance 8.2
                        sajithru Novice

                        Yes, me too think it may be due to a bug from VMware's end. I have a successfully configured VR 8.2 instance on one site but when I try to deploy it on the 2nd site it fails. Tried several fresh installments with no luck.

                        • 10. Re: HMS services failed to start in vsphere replication appliance 8.2
                          Akide Novice

                          Currently i got a solution may works for you, i change the VRM version to 8.1.2 or later which it's officially support VMware vSphere 6.7 Update 2.

                           

                          Through the test on my end VRM 8.1.2 works properly.

                           

                          release note attached below hope it helps:

                          VMware vSphere Replication 8.1.2 Release Notes

                          • 11. Re: HMS services failed to start in vsphere replication appliance 8.2
                            vmwaresbrf Lurker

                            Hi sajithru

                             

                            I'm having the same issue.

                             

                            I have successfully upgraded VR 8.1.2 to 8.2 instance on one site, but when I try to start it after upgrade on the 2nd site it fails to start with "Warning: Bad service status: Service not running"

                             

                            Have you found a solution?

                            • 12. Re: HMS services failed to start in vsphere replication appliance 8.2
                              dirch201110141 Novice

                              5 cents from me.
                              I also experienced problems with getting Replication server up and running, VRM service was stopped and was in no way shape to get running.

                              I got a "Bad exit code: 1". If i tried a new certificate i was told the certificate was wrong with nice red letters.

                              Log files told

                              - the hms service was not running. "hms not configured"

                              - route not found (or something like that)

                               

                              My setup: homelab with 6.7u3 in domain. I did the official rename feature after initial install as I got stuck getting the vCenter servername to work correct in the first place.

                               

                              After reinstalling 8.2 several times and trying 8.1.2 also and looking through 10 miles of logs, the following KB led me to check yet another file

                              VMware Knowledge Base (KB 2106709)

                              This points to checking the following:

                              /opt/vmware/etc/vami/ovfEnv.xml file. The file should contain the following properties:

                               

                              evs:GuestApi/evs:URL
                              evs:GuestApi/evs:Token
                              evs:GuestApi/evs:X509Thumbprint

                              evs:VCenterApi/evs:IP
                              evs:VCenterApi/evs:X509Thumbprint
                              evs:VCenterApi/evs:HttpPort
                              evs:VCenterApi/evs:HttpsPort
                              evs:VCenterApi/evs:SelfMoRef
                              evs:VCenterApi/evs:Address

                               

                              Looking into the xml-file there is valuable background info on the Replication setup and at least I found that VR was expecting to connect using the ancient vCenter name "photon-machine". The lazy fix was to create a new DNS-record, the right fix ... well, new install some other day.

                              My issue was probably not the same as OP, but the KB may be useful anyways.