vCenter dead after BSOD. SSO Problem

[tman24]

Had a BSOD on our vCenter 5.1 server yesterday (first time!), and now none of the vCenter services will start. SQL (which runs locally) seems fine, but after looking at the logs, it appears that the SSO service is failing to start, causing everything else to break. If I look at the SSO server logs, I can see this repeated over and over;

12-Oct-2012 10:09:17.535 SEVERE [WrapperSimpleAppMain] org.apache.catalina.startup.Bootstrap.initClassLoaders Class loader creation threw exception java.lang.NoSuchFieldError: threadAllocatedMemorySupport

                at sun.management.VMManagementImpl.initOptionalSupportFields(Native Method)

                at sun.management.VMManagementImpl.<clinit>(VMManagementImpl.java:47)

                at sun.management.ManagementFactory.<clinit>(ManagementFactory.java:468)

                at java.lang.management.ManagementFactory.getPlatformMBeanServer(ManagementFactory.java:504)

                at org.apache.catalina.startup.Bootstrap.createClassLoader(Bootstrap.java:158)

                at org.apache.catalina.startup.Bootstrap.initClassLoaders(Bootstrap.java:91)

                at org.apache.catalina.startup.Bootstrap.init(Bootstrap.java:227)

                at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:425)

                at net.covalent.tomcat.wrapper.ThreadDumpWrapper.main(ThreadDumpWrapper.java:74)

                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                at java.lang.reflect.Method.invoke(Method.java:597)

                at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:240)

                at java.lang.Thread.run(Thread.java:662)

vCenter then just throws an error when starting saying SSO isn't running, and dies.

Everything was working fine before the BSOD, and no errors have been reported since (you know what Windows is like!), but If I have to re-install the whole thing, it's going to be messy. Any ideas?

[Chaz999]

few question, Does yor vCenter running under Windows if yes then

- what kind if OS you are using?

- Any Anti Virus if yes then who is the vendor?

thanks

[tman24]

Thanks. Host O/S is 2008R2, A/V is Trend ServerProtect. I don't think A/V has anything to do with the problem though. It's certainly never caused us any problems before even with large installs like VC5.1.

If I can't recover it soon, I think I'm going to have to re-install. Hopefully backing up the SQL DB and taking a copy of the SSL certs might make things easier, but it's quite a risk.

[Chaz999]

Just wondering if you have STOP error

thanks

[tman24]

Where would you expect to see a STOP error? Nothing in the VPXD logs, and the Windows event errors are useless.

[Chaz999]

usually When you referring to BSOD it means that you have STOP error during the Windows Bootup process or may be am I missing some thing ?

thanks

[tman24]

Ahhh, sorry, I was thinking VMware stop error, not Windows! The BSOD was totally unrelated to VC. I was re-installing an app that installed its own driver. Had already done this once before with no problem, but the app wasn't working properly. Came to the point where it was actuall installing the driver, and bam, BSOD. Error was not something I'd seen often, but I don't think it will happen again. The server is stable in all other respects.

[GreatWhiteTec]

I think the logical thing to do here will be to remove the app that caused the BSOD and perhaps move it to another server. What is the app you are referring to?

[Chaz999]

got it,

based on yor above coments about "Re-building" it seems to me your SQL and vCenter Application running on same VM,

keep in mind your SQL service must start first before your vCenter, there is delay startup option available in W2K8, you can chnage vCenter service set as delay start.

second thing run SQL Managment Studio and makesure that there is no password expire and also check under SQL Server Confgiuration Manager , Protocol of MSSQLSERVER under SQL Network Configuration TCP/IP should be enable

also check SYSTEM DSN entry under ODBC (64bit) there is no conenctivity issue between vCenter and SQL DB

Thanks

[tman24]

The app in question just happened to be VirtualCloneDrive (fake ISO  mounter), which I use on all servers that need it. I think the BSOD was a  'one-off' that tripped windows for some reason. Never seen it before,  will probably never see it again.

[tman24]

I appreciate your time.

I can confirm that the VM SQL DB us running fine. I've checked the settings is server config manager, and they are all correct (TCPIP enabled, and server is listening on 1433 as normal). I quickly checked the ODBC DSN link to SQL, and that passes all checks, so no problem there.

I firmly believe the problem isn't SQL, but is SSO Server which is not starting because of a Java problem or corruption related to the BSOD (that's what the error implies), but because SSO is so new to VC5.1, there's very little info out there about it at the moment.

Running out of time..... looking like a re-install now...

[GreatWhiteTec]

Get the minidump file from windows, and use debugging tools. This should tell you what caused the BSOD.

[GreatWhiteTec]

This should get you going.

http://support.dell.com/support/topics/global.aspx/support/kcs/document?docid=543301