VirtualVen
Contributor
Contributor

VRA 7.0 failed login!

Jump to solution

Hi! vRealize experts,

Need your urgent help. I have a minimal (1 VRA7.0 and 1 IaaS) test environment - configured everything without any issue and able to publish a blue print and deploy a VM.

I am not sure what happened, one day when I tried login (url:https://vra1.mydomain.com/vcac ) to vRA portal it took a very long time (Signing-in screenshot attached).

Waited like 5 minute and hit refresh and it popped up with an error message "Identity Manager encountered an error Contact your admin... " screenshot attached.

I did a couple of reboots of both the servers (shutdown: IaaS server first and then vRA7 server; startup: vRA7 server first and then 5 minutes later IaaS server)  and don't see any improvement. Cleared browser cache as well. Sometimes it give me this error message "HTTP Status 404 - /SAAS/t/VSPHERE.LOCAL/apps/"

https://vra1.mydomain.com:5480 works fine and I don't see any error with certificates and all services are registered (at lease the ones that show up there).

I checked the catalina.out file and don't see any error.

Not sure where else to look, any help is deeply appreciated.

Many thanks

Regards

Ven

Thank you Ven
0 Kudos
1 Solution

Accepted Solutions
VirtualVen
Contributor
Contributor

Well looks like its not working, so rebuilding whole thing. Smiley Happy

Thank you Ven

View solution in original post

0 Kudos
5 Replies
Kedar_D
Enthusiast
Enthusiast

Hi Van,

Check following things.

1. Login to https://vra1.mydomain.com:5480​  make sure all services are registered.

2. Check time synchronization between vRA ,authentication source and IaaS.

3. When you restart the vRA service check for the catalina log files. If any servers are failing to register it should show up in the log while services are getting started.

Hope this helps!

------------------------------------------------------------ If you find this or any other answer useful please mark the answer as correct or helpful. Kedar http://virtualtechknow.wordpress.com/ ------------------------------------------------------------
0 Kudos
VirtualVen
Contributor
Contributor

Hi! Kedar,

Many thanks for the reply.

Yes I've done all that as part of my troubleshooting and they seem fine. All listed services are showing registered and time is in sync with host.

Don't see any errors in Catalina.out file.

I also noticed it works intermittently - like it worked for a while yesterday - I was able to login and make configure Branding but after like 20 minutes when I was saving my configuration it suddenly throws the mentioned errors and back to square one.

Anybody any help!!!!

Thank you

Regards

Ven

Thank you Ven
0 Kudos
VirtualVen
Contributor
Contributor

HI! Kedar,

I just rebooted the vra7 server and a part of catalina.out file is below:

2017-03-20 19:32:35,748 vcac: [component="cafe:notification" priority="INFO" thread="localhost-startStop-2" tenant="" context="" token=""] com.vmware.vcac.platform.service.rest.init.RestWebApplicationInitializer.createSpringApplicationContext:118 - Loading of Spring application context for service /notification-service ...

2017-03-20 19:32:35,792 vcac: [component="cafe:notification" priority="INFO" thread="localhost-startStop-2" tenant="" context="" token=""] com.vmware.vcac.platform.service.rest.init.RestWebApplicationInitializer.createSpringApplicationContext:145 - Spring application context for service /notification-service is loaded.

2017-03-20 19:32:35,803 vcac: [component="cafe:work-item" priority="INFO" thread="localhost-startStop-4" tenant="" context="" token=""] com.vmware.vcac.platform.service.rest.init.RestWebApplicationInitializer.mapSecurityFilter:171 - Initializing security filter for service /workitem-service ...

2017-03-20 19:32:35,804 vcac: [component="cafe:work-item" priority="INFO" thread="localhost-startStop-4" tenant="" context="" token=""] com.vmware.vcac.platform.service.rest.init.RestWebApplicationInitializer.mapSecurityFilter:176 - Security filter for service /workitem-service is mapped.

2017-03-20 19:32:35,810 vcac: [component="cafe:component-registry" priority="INFO" ed dump Java HotSpot(TM) 64-Bit Server VM (25.72-b15 mixed mode):

"serviceSubscribeAmqpTaskExecutor-89308" #616889 prio=5 os_prio=0 tid=0x00007f0f09e93000 nid=0x67c4 waiting on condition [0x00007f0ecb649000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000007853c1248> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

"serviceSubscribeAmqpTaskExecutor-89348" #616888 prio=5 os_prio=0 tid=0x00007f0eff140000 nid=0x67c3 waiting on condition [0x00007f0ecb504000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x0000000762fd5950> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

22-Mar-2017 21:32:17.802 INFO [Thread-5] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-nio-127.0.0.1-8082"]

22-Mar-2017 21:32:18.900 INFO [Thread-5] org.apache.catalina.core.StandardService.stopInternal Stopping service Catalina

22-Mar-2017 21:32:21.625 WARNING [localhost-startStop-8] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [software-service] appears to have started a thread named [metrics-logger-reporter-custom-1-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:

sun.misc.Unsafe.park(Native Method)

java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)

java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

java.lang.Thread.run(Thread.java:745)

22-Mar-2017 21:32:21.728 WARNING [localhost-startStop-9] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [console-proxy-service] appears to have started a thread named [metrics-logger-reporter-custom-1-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:

sun.misc.Unsafe.park(Native Method)

java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)

java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

java.lang.Thread.run(Thread.java:745)

22-Mar-2017 21:32:22.069 WARNING [localhost-startStop-9] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [console-proxy-service] appears to have started a thread named [ThreadPoolTaskScheduler-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:

sun.misc.Unsafe.park(Native Method)

java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)

java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

java.lang.Thread.run(Thread.java:745)

22-Mar-2017 21:32:22.090 WARNING [localhost-startStop-9] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [console-proxy-service] appears to have started a thread named [ThreadPoolTaskScheduler-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:

sun.misc.Unsafe.park(Native Method)

java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)

java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

Not sure if its because I didn't follow a proper restart order. But see lot of Java errors - I don't have a clue what these are....

Any help?

Thank you

Ven

Thank you Ven
0 Kudos
VirtualVen
Contributor
Contributor

Ok I did some digging and see  REGISTERED service are 31 from vRA console.    

S.NoServices
1advanced-designer-service
2approval-service
3authentication
4authorization
5branding-service
6catalog-service
7component-registry
8composition-service
9console-proxy-service
10content-management
11event-broker-service
12eventlog-service
13fabric-service
14files-service
15forms-service
16iaas-proxy-provider
17iaas-service
18identity
19licensing-service
20management-service
21network-service
22notification-service
23plugin-service
24portal-service
25properties-service
26reservation-service
27shell-ui-app
28software-service
29sts-service
30vco
31workitem-service

Not sure if this is the complete list of am I missing some.

Secondly, browsed the "catalina.out" file and here is the list of services that are listed as starting; initializing and registering:

Clearly not all services are showing up.

Services:

Starting up service /notification-service ...

Starting up service /component-registry ...

Starting up service /workitem-service ...

Starting up service /event-broker-service ...

Starting up service /identity ...

Starting up service /catalog-service ...

Starting up service /branding-service ...

Starting up service /iaas-proxy-provider ...

Starting up service /forms-service ...

Starting up service /software-service ...

Starting up service /console-proxy-service ...

Starting up service /content-management-service ...

Starting up service /properties-service ...

Starting up service /management-service ...

Starting up service /portal-service ...

Starting up service /reservation-service ...

Starting up service /composition-service ...

Starting up service /files-service ...

Starting up service /network-service ...

Starting up service /approval-service ...

Initializing:

Initialization completed successfully for [ licensing-service ]

Initialization completed successfully for [ plugin-service ]

Initialization completed successfully for [ component-registry ]

Initialization completed successfully for [ identity ]

Initialization completed successfully for [ authorization ]

Initialization completed successfully for [ authentication ]

Initialization completed successfully for [ notification-service ]

Initialization completed successfully for [ workitem-service ]

Initialization completed successfully for [ event-broker-service ]

Initialization completed successfully for [ eventlog-service ]

Initialization completed successfully for [ branding-service ]

Initialization completed successfully for [ catalog-service ]

Initialization completed successfully for [ advanced-designer-service ]

Initialization completed successfully for [ forms-service ]

Initialization completed successfully for [ iaas-proxy-provider ]

Initialization completed successfully for [ fabric-service ]

Initialization completed successfully for [ console-proxy-service ]

Initialization completed successfully for [ console-proxy-service ]

Initialization completed successfully for [ content-management ]

Initialization completed successfully for [ software-service ]

Initialization completed successfully for [ properties-service

Initialization completed successfully for [ portal-service ]

Initialization completed successfully for [ reservation-service ]

Initialization completed successfully for [ composition-service ]

Initialization completed successfully for [ management-service ]

Initialization completed successfully for [ network-service ]

Initialization completed successfully for [ approval-service ]

Initialization completed successfully for [ shell-ui-app ]

Initialization completed successfully for [ files-service ]

Registering:

Registering service licensing-service ...

Registering service plugin-service ...

Registering service component-registry ...

Registering service authorization ...

Registering service authentication ...

Registering service eventlog-service ...

Registering service fabric-service ...

Registering service console-proxy-service ...

Can anyone confirm the total that are needed for everything to work successfully and why some from above list are not matching, like the services are not being initialized and registered.

Thank you

Ven

Thank you Ven
0 Kudos
VirtualVen
Contributor
Contributor

Well looks like its not working, so rebuilding whole thing. Smiley Happy

Thank you Ven

View solution in original post

0 Kudos