VMware Cloud Community
aseaz404
Enthusiast
Enthusiast
Jump to solution

None of the cells have a vCenter proxy service running

Ran into this issue last night.  I've been reading several sources to see if I can pin point the cause but no luck so far other than communication between the cell and the db was impacted.  I've followed the instructions in KB 1035506 but no change.  I've shutdown the cell via the service command and then also tried with the cell management tool, no difference.  The QRTZ tables get cleared of all their entries but once I start up the cell again they seem to populate with the same set of records.

I'm on vCloud 5.1.2.1068441 and using MS-SQL 2012.  If anyone has any suggestions of where to look next it would be greatly appreciated, thanks in advance.

Reply
0 Kudos
1 Solution

Accepted Solutions
aseaz404
Enthusiast
Enthusiast
Jump to solution

Ok, just in case other folks run into this it appears that the KB (I referenced in my first post) is incomplete as of 10/24/2013.  The proper sequence is as follows, assuming you can afford the downtime:

1.  Shut down the cell VM or run the service vmware-vcd stop command on the cell.

2.  THIS STEP IS CURRENTLY NOT INCLUDED IN THE KB.  Shutdown your vCenter VM, that's what I had to do.  I suppose you can test just shutting down the service, I wasn't so inclined.

3.  Run the script to clean your QRTZ tables.

4.  NOT INCLUDED (LIKE STEP 2).  Power your vCenter VM back up, if you had shut down the service you can try starting it at this stage.

5.  Power up your cell or run the service vmware-vcd start command.

My translation of the explanation I received from the support folks was that both vcenter and the vcloud cell needed to stay off the db during cleanup and that the KB was missing the vcenter piece.  I was told the KB would be changed soon, but in the meantime this might help.

View solution in original post

Reply
0 Kudos
3 Replies
aseaz404
Enthusiast
Enthusiast
Jump to solution

I have a brief update, though not much progress toward a solution.  I have noticed that when the cell is back up the QRTZ table repopulate rather quickly, and it's always the same rows which leads me to think that there's a roadblock somewhere.  I'm attaching a worksheet with the contents of the QRTZ_JOB_DETAILS table.  Going through it I saw that there’s an LDAP sync that’s queued.  I tried to run an LDAP sync manually and it failed too with the error message:

com.vmware.vcloud.common.model.User cannot be cast to com.vmware.vcloud.common.model.IntegratedUserModel

I then put a trace on the SLQ server to watch what was happening on the DB and found:

exec sp_execute 43,N'java.lang.ClassCastException: com.vmware.vcloud.common.model.User cannot be cast to com.vmware.vcloud.common.model.IntegratedUserModel
                at com.vmware.ssdc.backend.dao.impl.HibernateIntegratedUserDao.deleteUnreferencedSystemLdapUsers(HibernateIntegratedUserDao.java:65)
                at com.vmware.ssdc.backend.ldap.LdapManagerImpl.garbageCollectSharedMembers(LdapManagerImpl.java:269)
                at com.vmware.ssdc.backend.ldap.LdapManagerImpl.syncLdap(LdapManagerImpl.java:317)
                at com.vmware.ssdc.backend.ldap.LdapManagerImpl.access$100(LdapManagerImpl.java:79)
                at com.vmware.ssdc.backend.ldap.LdapManagerImpl$3.invoke(LdapManagerImpl.java:359)
                at com.vmware.ssdc.backend.CAkimbiTask._invokeChildUnsafe(CAkimbiTask.java:120)
                at com.vmware.ssdc.backend.CAkimbiTask.access$500(CAkimbiTask.java:39)
                at com.vmware.ssdc.backend.CAkimbiTask$InvokeChildThread.innerRun(CAkimbiTask.java:237)
                at com.vmware.ssdc.backend.CAkimbiTask$InvokeChildThread.access$300(CAkimbiTask.java:156)
                at com.vmware.ssdc.backend.CAkimbiTask$InvokeChildThread$1.run(CAkimbiTask.java:175)
                at com.vmware.ssdc.backend.CAkimbiTask$InvokeChildThread$1.run(CAkimbiTask.java:164)
                at com.vmware.vcloud.common.threadpool.ThreadContextExecutor.execute(ThreadContextExecutor.java:48)
                at com.vmware.ssdc.backend.CAkimbiTask$InvokeChildThread.run(CAkimbiTask.java:189)
',0x31725CF2366946E29D77BEFAD6797064

Reply
0 Kudos
aseaz404
Enthusiast
Enthusiast
Jump to solution

Ok, just in case other folks run into this it appears that the KB (I referenced in my first post) is incomplete as of 10/24/2013.  The proper sequence is as follows, assuming you can afford the downtime:

1.  Shut down the cell VM or run the service vmware-vcd stop command on the cell.

2.  THIS STEP IS CURRENTLY NOT INCLUDED IN THE KB.  Shutdown your vCenter VM, that's what I had to do.  I suppose you can test just shutting down the service, I wasn't so inclined.

3.  Run the script to clean your QRTZ tables.

4.  NOT INCLUDED (LIKE STEP 2).  Power your vCenter VM back up, if you had shut down the service you can try starting it at this stage.

5.  Power up your cell or run the service vmware-vcd start command.

My translation of the explanation I received from the support folks was that both vcenter and the vcloud cell needed to stay off the db during cleanup and that the KB was missing the vcenter piece.  I was told the KB would be changed soon, but in the meantime this might help.

Reply
0 Kudos
MillardJK
Enthusiast
Enthusiast
Jump to solution

The KB has been changed. Now it simply says: To resolve this issue please contact VMware technical support.

Thanks, VMware: NOT HELPFUL.

——
Jim Millard
Kansas City, MO USA
Reply
0 Kudos