Skip navigation

Last option to try when you don't have any option to try from the UI or the admin UI

 

If you have tried multiple options and couldn't resolve vROps related issues and decided to go with new deployment. Before you go for new deployment try one last option below

 

  • Run the Cluster Offline command on the MASTER node in the cluster.
    • $VMWARE_PYTHON_BIN /usr/lib/vmware-vcopssuite/utilities/sliceConfiguration/bin/vcopsClusterManager.py offline-cluster Maintenance
  • Run the below slice offline command on the analytics nodes in the cluster:
    • $VMWARE_PYTHON_BIN $ALIVE_BASE/../vmware-vcopssuite/utilities/sliceConfiguration/bin/vcopsConfigureRoles.py --action bringSliceOffline --offlineReason "recovery"
    • Run the above command on all the nodes DATA, REPLICA and last MASTER in the same order.
  • Reboot all the Nodes
  • Power-on all the Nodes
  • Run the below command to bring the slice online
    • $VMWARE_PYTHON_BIN $VCOPS_BASE/../vmware-vcopssuite/utilities/sliceConfiguration/bin/vcopsConfigureRoles.py --action bringSliceOnline
    • Run the above command on all the nodes in MASTER , REPLICA and DATA node in the same order.
  • Bring cluster online, run the below command on the MASTER Node.
    • $VMWARE_PYTHON_BIN /usr/lib/vmware-vcopssuite/utilities/sliceConfiguration/bin/vcopsClusterManager.py init-cluster

    

For unknown reason the filesystem on NSX-Manager will go bad and to recovering from this is different when compared with other filesystem recovery methods.

 

Problem: NSX Manager VM unable to boot. you will a screen something like below:

 

https://confluence.eng.vmware.com/download/attachments/268560128/image2017-12-21%2011%3A33%3A42.png?version=1&modificationDate=1513836222000&api=v2

 

 

Recovery Steps:

  •      Download and Connect the Ubuntu ISO to CD drive of the NSX-Manager VM.
  • Boot through CD drive and choose "try now" option.
  • Run the recovery command "fsck /dev/sda2"
  • NSX-Manager VM will recover back from the filesystem and will boot normally.
  • vROps nodes status will wait indefinitely @ "Waiting for Analytics" because of the time sync issue between the nodes

 

    com.integrien.analytics.AnalyticsMain.run - AnalyticsMain.run failed with error: Time difference between servers is:30134 ms. It is greater than 30000 ms. Unable to operate, terminating...

    java.lang.IllegalStateException: Time difference between servers is:30134 ms. It is greater than 30000 ms. Unable to operate, terminating...

      at com.integrien.alive.controller.Controller.verifyTimeDifferenceBetweenServers(Controller.java:1306)

      at com.integrien.alive.controller.Controller.doRun(Controller.java:626)

      at com.integrien.analytics.AnalyticsMain.doRun(AnalyticsMain.java:352)

      at com.integrien.analytics.AnalyticsMain.run(AnalyticsMain.java:1387)

      INFO  [Analytics Main Thread ] com.vmware.vcops.platform.common.PlatformEnvironment.exitSystem - Terminating process with exit code -1 ...

      INFO  [SystemExitThread ] com.vmware.vcops.platform.common.PlatformEnvironment.run - exitSystem has been called by:

      java.lang.Throwable at com.vmware.vcops.platform.common.PlatformEnvironment.exitSystem(PlatformEnvironment.java:200)

 

Resolution:

    If normal ntp sync doesn't work.

    You can try using this command "rcntp ntptimeset <ntp-server>"

 

 

  • Restoring the default dashboards manually ( this is an example of restoring the vRA dashboard ):

     

       Problem: After installing the vRA Mangement Pack, the default dashboards were not shown up and for this we figured out that post_install of Management Pack didn't run. To resolve this we manually imported the dashboards

 

Resolution:

The below is an example for restoring the vRA default dashboards on vROPs

 

    1. Locate the dashboard file location: /usr/lib/vmware-vcops/user/plugins/inbound/vCACAdapter3/conf/dashboards/Dashboards.json
    2. Execute the import command by passing the dashboard json file on vROps node:

 

vrops-mstrn-01:/usr/lib/vmware-vcops/tools/opscli # ./ops-cli.sh dashboard import all /usr/lib/vmware-vcops/user/plugins/inbound/vCACAdapter3/conf/dashboards/Dashboards.json

Starting vRealize Operations CLI

2016-10-20 10:15:30,116 Starting command Command: dashboard:import

2016-10-20 10:15:30,404 Connecting to server

2016-10-20 10:15:37,491 Try to login with maintenanceAdmin

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".

SLF4J: Defaulting to no-operation (NOP) logger implementation

SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

2016-10-20 10:15:38,561 Successfully logged in.

2016-10-20 10:15:38,561 Successfully connected to server

2016-10-20 10:15:43,810 Successfully imported dashboard /usr/lib/vmware-vcops/user/plugins/inbound/vCACAdapter3/conf/dashboards/Dashboards.json for user [all]. The configuration transformation will be scheduled in background.

2016-10-20 10:15:43,811 Triggered Command: dashboard:import vrops-mstrn-01:/usr/lib/vmware-vcops/tools/opscli #

 

After this log-in back into the vROps UI and you can notice that all default dashboards are back.