Perf Charts service experienced an internal error.

[Khue]

So, I am now experiencing this issue. After looking through the forums, I didn't seem to find anything that helped. I eventually contacted VMware support and had the first BAD experience in 5 years with them. I upgraded my virtual center from version 4 to version 4 update 1. The performance charts overview section now gives me this error message. When I upgraded I downloaded the newest media available from the website and ran the installer. I backed up my databases. Everything seemed to install well. Then I started noticing 2 issues.

1) The hardware status tab in the vCenter UI was unavailable. I looked at the plugins and they were having an issue. I tried starting the management web service but it would not start. I quickly came to realize, that somewhere during the process, java was uninstalled. I reinstalled java and configured tom cat to point at the new java install. The hardware status tab now works perfectly. Issue resolved.

2) The overview of the perfmon logs "experiences an internal error." This issue started happening after the install/upgrade of vCenter UI. I am using SQL express and I have 10 esx hosts. While I understand that this is an unsupported configuration, everything worked fine prior to the update and something definitely happened with the tomcat/java setup when the installer did it's thing. VMware's suggestion for fixing the issue was to buy a SQL Server Full license. Their troubleshooting techniques seemed to focus on everything but tomcat and java.

After looking through some tomcat log files I found this line which seems to correspond to me clicking on the Performance Overview button:

org.apache.commons.digester.Digester : Parse Error at line 3 column 31: cvc-elt.1: Cannot find the declaration of element 'reportsDescriptionRepository'.

org.xml.sax.SAXParseException: cvc-elt.1: Cannot find the declaration of element 'reportsDescriptionRepository'.

at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)

at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.error(Unknown Source)

at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(Unknown Source)

at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(Unknown Source)

at com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator.handleStartElement(Unknown Source)

at com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator.startElement(Unknown Source)

at com.sun.org.apache.xerces.internal.impl.dtd.XMLDTDValidator.startElement(Unknown Source)

at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown Source)

at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$ContentDriver.scanRootElementHook(Unknown Source)

at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown Source)

at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(Unknown Source)

at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source)

at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)

at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)

at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)

at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source)

at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source)

at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)

at org.apache.commons.digester.Digester.parse(Digester.java:1785)

at com.vmware.vim.stats.report.repository.xml.bean.XMLBeanStorage.parseFile(Unknown Source)

at com.vmware.vim.stats.report.repository.xml.bean.XMLBeanStorage.registerFileChangeListeners(Unknown Source)

at com.vmware.vim.stats.report.repository.xml.bean.XMLBeanStorage.(Unknown Source)

at com.vmware.vim.stats.webui.startup.ReportDescriptorsRepositoryInitializer.createRepository(Unknown Source)

at com.vmware.vim.stats.webui.startup.ReportDescriptorsRepositoryInitializer.init(Unknown Source)

at com.vmware.vim.stats.webui.startup.StatsReportInitializer.createInitializers(Unknown Source)

at com.vmware.vim.stats.webui.startup.StatsReportInitializer.init(Unknown Source)

at com.vmware.vim.stats.webui.startup.StatsApplicationLauncher$1$1.run(Unknown Source)

at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)

at java.util.concurrent.FutureTask.run(Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

com.vmware.vim.stats.webui.startup.StatsApplicationLauncher : Task execution produced an error. Re-initialization attempt #2 will startup after 60 seconds...

Does this mean anything to anyone? I pulled this from stats.log out of the c:\program files\vmware\infrastructure\tomcat\logs folder.

[AndreTheGiant]

Have you enabled the SQL Browser service?

Do you have make some upgrade also on the SQL Express part?

Andre

[Khue]

Thanks for the reply Andre. The SQL Browser Service is enabled. All other aspects that require SQL Browser are working. I believe the Hardware Status tab and the Performance tab (advanced) use the Browser Service. I have not run any patches against SQL Server Express. I had a member of my Dev team look at the error I posted, he seems to think theres a problem with whatever is parsing the xml files. The element tag is present in all of the files in webapps\statsreport\report\.

[Khue]

I wanted to update this post. I sat down with this over the weekend and I figured out a fix for the problem. The issue seemed to be related to file permissions on the xml files located in C:\Program Files\VMware\Infrastructure\tomcat\webapps\statsreport\report. For some reason inheritance had been broken on the folder itself. The cause of the folder permission change at this point is unknown. I noticed that while all other subfolders in the "webapps" folder had inherited permissions, the report folder did not and system and the local administrators group had some permissions changed. They only had read and write access not full control. I restored permission inheritance, restarted the vCenter Server, and now everything seems to be working fine for the time being. I hope this helps someone in the future.

I would also like to point out that the SQL Server Express installation that I am using is supporting 10 hosts and over 50 virtual machines. Clearly, while this configuration is "not supported," it does work if you maintain everything properly.