VMware Cloud Community
Vodder
Enthusiast
Enthusiast

vCOPS Manager 5 no longer showing data

Hi,

We installed 2 instances of vCOPS Manager 5 across 2 datacentres. Each DC has 2 vCenter instances. They are not currently in linked mode.

All 4 started fine and were collecting data. However one of them changed after a couple of hours to showing everything as grey symbols with question marks. It was working and some data of the VMs is showing as green on the initial dashboard for health but the health rating is ?

From the custom UI login I found an error stating that it had stopped communicating and to check vcenter web services and storage service. These were OK.

I have restarted the vCOPS manger service and even restarted the whole vApp hwoever the problem persists.

Any ideas? Or which logs should be first port of call?

I checked things mentioned in other posts about timezones and perms and these are all OK. If anything I would expect it not to have started collecting anything at all if these were wrong.

The vApp is set to Medium, licensed for Enterprise. The number of VM's in the DC effected is no more than 1500

Any help appreciated


Cheers

Tim

@timgleed | VCAP5-DCA/DCD | VCAP4-DCA/DCD | VCP5 | VCP4 | VCP3 | VCP4-DT | VCA4-DT | VTSP4 | MCITP | PRINCE2 | ITIL | BSc Hons
Reply
0 Kudos
8 Replies
pgoel
Enthusiast
Enthusiast

Try 'repair' command

repair --ipaddress <ipaddress of Analytics VM>

Reply
0 Kudos
kmurthy
VMware Employee
VMware Employee

I wouldn't recommend a repair before first ascertaining what the actual problem is. Can you post a screenshot of the error and also the support bundle ? You can generate it from the admin UI (https://<UI VM>/admin; Status tab) or the custom UI (Support link at the top -> Logs).

Reply
0 Kudos
Vodder
Enthusiast
Enthusiast

Thanks guys. Looks like it started receiving data about 30 mins after I restarted the vApp. Will still like to investigate why it died though. Which logs are first port of call?

Regarding the repair command, what exactly does that do? Are there other commands or they listed somewhere?

@timgleed | VCAP5-DCA/DCD | VCAP4-DCA/DCD | VCP5 | VCP4 | VCP3 | VCP4-DT | VCA4-DT | VTSP4 | MCITP | PRINCE2 | ITIL | BSc Hons
Reply
0 Kudos
Vodder
Enthusiast
Enthusiast

Looks like that was a bit premature as it has failed again. This is really strange! Everything looked OK but then when I went back into a couple of hours later it stated data had stopped again around 6am, even though before it was showing health on the server as 100. You know on the green "impact" time bar for the VM and then a number on the end? Well this one for the VC VM was green 100 with a grey section where it was down. This is now showing it still grey.

THis is part of the collector log. This covers both VC's. Though the data below is also the same in vmware_adapter log 15 which I believe (15) refers to the VC that is not communicating.

{http://xml.apache.org/axis/}hostname:secondvm-external

java.net.SocketTimeoutException: Read timed out
at org.apache.axis.AxisFault.makeFault(AxisFault.java:101)
at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:154)
at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
at org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)
at org.apache.axis.client.Call.invokeEngine(Call.java:2784)
at org.apache.axis.client.Call.invoke(Call.java:2767)
at org.apache.axis.client.Call.invoke(Call.java:2443)
at org.apache.axis.client.Call.invoke(Call.java:2366)
at org.apache.axis.client.Call.invoke(Call.java:1812)
at com.vmware.vim.sms.SmsBindingStub.queryList(SmsBindingStub.java:19249)
at com.vmware.apputils.SmServiceConnection.queryList(SmServiceConnection.java:188)
at com.integrien.adapter.vmware.sms.SmsPerformanceCollector.querySMS(SmsPerformanceCollector.java:427)
at com.integrien.adapter.vmware.sms.SmsPerformanceCollector.collect(SmsPerformanceCollector.java:151)
at com.integrien.adapter.vmware.sms.SmsPerformanceCollector.access$100(SmsPerformanceCollector.java:35)
at com.integrien.adapter.vmware.sms.SmsPerformanceCollector$1.run(SmsPerformanceCollector.java:514)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:830)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:787)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at org.apache.axis.transport.http.HTTPSender.readHeadersFromSocket(HTTPSender.java:583)
at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:143)
... 17 more
2012-02-22 07:39:19,616 ERROR [Collector worker thread 25] (15) com.integrien.alive.common.adapter3.AdapterBase.collect - Collect threw an exception: null
2012-02-22 07:44:14,582 ERROR [Timer-1] (974) com.integrien.adapter.vmware.sms.SmsPerformanceCollector.collect - Unable to obtain SMS data after retries
AxisFault
faultCode: ServerFaultCode
faultSubcode:
faultString:
faultActor:
faultNode:
faultDetail:
{urn:sms}ProviderSyncFailedFault:null
at sun.reflect.GeneratedConstructorAccessor177.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:355)
at java.lang.Class.newInstance(Class.java:308)
at org.apache.axis.encoding.ser.BeanDeserializer.<init>(BeanDeserializer.java:104)
at org.apache.axis.encoding.ser.BeanDeserializer.<init>(BeanDeserializer.java:90)
at com.vmware.vim.sms.SyncInProgress.getDeserializer(SyncInProgress.java:87)
at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.axis.encoding.ser.BaseDeserializerFactory.getSpecialized(BaseDeserializerFactory.java:154)
at org.apache.axis.encoding.ser.BaseDeserializerFactory.getDeserializerAs(BaseDeserializerFactory.java:84)
at org.apache.axis.encoding.DeserializationContext.getDeserializer(DeserializationContext.java:464)
at org.apache.axis.encoding.DeserializationContext.getDeserializerForType(DeserializationContext.java:547)
at org.apache.axis.message.SOAPFaultDetailsBuilder.onStartChild(SOAPFaultDetailsBuilder.java:157)
at org.apache.axis.encoding.DeserializationContext.startElement(DeserializationContext.java:1035)
at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown Source)
at org.apache.xerces.parsers.AbstractXMLDocumentParser.emptyElement(Unknown Source)
at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
at org.apache.axis.encoding.DeserializationContext.parse(DeserializationContext.java:227)
at org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:696)
at org.apache.axis.Message.getSOAPEnvelope(Message.java:435)
at org.apache.axis.handlers.soap.MustUnderstandChecker.invoke(MustUnderstandChecker.java:62)
at org.apache.axis.client.AxisClient.invoke(AxisClient.java:206)
at org.apache.axis.client.Call.invokeEngine(Call.java:2784)
at org.apache.axis.client.Call.invoke(Call.java:2767)
at org.apache.axis.client.Call.invoke(Call.java:2443)
at org.apache.axis.client.Call.invoke(Call.java:2366)
at org.apache.axis.client.Call.invoke(Call.java:1812)
at com.vmware.vim.sms.SmsBindingStub.sync(SmsBindingStub.java:19345)
at com.vmware.apputils.SmServiceConnection.sync(SmServiceConnection.java:233)
at com.integrien.adapter.vmware.sms.SmsConnection.sync(SmsConnection.java:116)
at com.integrien.adapter.vmware.sms.SmsPerformanceCollector.querySMS(SmsPerformanceCollector.java:425)
at com.integrien.adapter.vmware.sms.SmsPerformanceCollector.collect(SmsPerformanceCollector.java:166)
at com.integrien.adapter.vmware.sms.SmsPerformanceCollector.access$100(SmsPerformanceCollector.java:35)
at com.integrien.adapter.vmware.sms.SmsPerformanceCollector$1.run(SmsPerformanceCollector.java:514)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
2012-02-22 07:44:19,995 ERROR [Collector worker thread 9] (15) com.integrien.alive.common.adapter3.AdapterBase.collect - Collect threw an exception: null
2012-02-22 07:49:18,813 ERROR [Collector worker thread 16] (15) com.integrien.alive.common.adapter3.AdapterBase.collect - Collect threw an exception: null
2012-02-22 07:54:19,043 ERROR [Collector worker thread 17] (15) com.integrien.alive.common.adapter3.AdapterBase.collect - Collect threw an exception: null
2012-02-22 07:59:18,546 ERROR [Collector worker thread 2] (15) com.integrien.alive.common.adapter3.AdapterBase.collect - Collect threw an exception: null
2012-02-22 08:04:19,576 ERROR [Collector worker thread 10] (15) com.integrien.alive.common.adapter3.AdapterBase.collect - Collect threw an exception: null
2012-02-22 08:09:20,016 ERROR [Collector worker thread 3] (15) com.integrien.alive.common.adapter3.AdapterBase.collect - Collect threw an exception: null
2012-02-22 08:14:19,789 ERROR [Collector worker thread 14] (15) com.integrien.alive.common.adapter3.AdapterBase.collect - Collect threw an exception: null
2012-02-22 08:19:21,304 ERROR [Collector worker thread 20] (15) com.integrien.alive.common.adapter3.AdapterBase.collect - Collect threw an exception: null
2012-02-22 08:24:20,457 ERROR [Collector worker thread 19] (15) com.integrien.alive.common.adapter3.AdapterBase.collect - Collect threw an exception: null
2012-02-22 08:29:20,501 ERROR [Collector worker thread 15] (15) com.integrien.alive.common.adapter3.AdapterBase.collect - Collect threw an exception: null
2012-02-22 08:34:19,171 ERROR [Collector worker thread 12] (15) com.integrien.alive.common.adapter3.AdapterBase.collect - Collect threw an exception: null
2012-02-22 08:34:51,578 ERROR [Timer-2] (15) com.integrien.adapter.vmware.sms.SmsPerformanceCollector.collect - Unable to obtain SMS data after retries
AxisFault
faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server.userException
faultSubcode:
faultString: java.net.SocketTimeoutException: Read timed out
faultActor:
faultNode:
faultDetail:
{http://xml.apache.org/axis/}stackTrace:java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:830)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:787)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at org.apache.axis.transport.http.HTTPSender.readHeadersFromSocket(HTTPSender.java:583)
at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:143)
at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
at org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)
at org.apache.axis.client.Call.invokeEngine(Call.java:2784)
at org.apache.axis.client.Call.invoke(Call.java:2767)
at org.apache.axis.client.Call.invoke(Call.java:2443)
at org.apache.axis.client.Call.invoke(Call.java:2366)
at org.apache.axis.client.Call.invoke(Call.java:1812)
at com.vmware.vim.sms.SmsBindingStub.queryList(SmsBindingStub.java:19249)
at com.vmware.apputils.SmServiceConnection.queryList(SmServiceConnection.java:188)
at com.integrien.adapter.vmware.sms.SmsPerformanceCollector.querySMS(SmsPerformanceCollector.java:427)
at com.integrien.adapter.vmware.sms.SmsPerformanceCollector.collect(SmsPerformanceCollector.java:151)
at com.integrien.adapter.vmware.sms.SmsPerformanceCollector.access$100(SmsPerformanceCollector.java:35)
at com.integrien.adapter.vmware.sms.SmsPerformanceCollector$1.run(SmsPerformanceCollector.java:514)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
@timgleed | VCAP5-DCA/DCD | VCAP4-DCA/DCD | VCP5 | VCP4 | VCP3 | VCP4-DT | VCA4-DT | VTSP4 | MCITP | PRINCE2 | ITIL | BSc Hons
Reply
0 Kudos
kmurthy
VMware Employee
VMware Employee

Please contact vmware support to get the 5.0 vApp hotfix. It should fix the issues that you are seeing.

Vodder
Enthusiast
Enthusiast

OK thanks, will do.

Is there any public info on this hotfix? Someone else refers to a hotfix on another thread. But I've been unable to find anything

@timgleed | VCAP5-DCA/DCD | VCAP4-DCA/DCD | VCP5 | VCP4 | VCP3 | VCP4-DT | VCA4-DT | VTSP4 | MCITP | PRINCE2 | ITIL | BSc Hons
Reply
0 Kudos
gradinka
VMware Employee
VMware Employee

I think there isn't...

Anyway, there is only 1 hotfix available for 5.0 at the moment, so it's the same thing.

You'll get it tru support.

Reply
0 Kudos
Vodder
Enthusiast
Enthusiast

Just to confirm the hotfix to build 598005 fixed the issue

@timgleed | VCAP5-DCA/DCD | VCAP4-DCA/DCD | VCP5 | VCP4 | VCP3 | VCP4-DT | VCA4-DT | VTSP4 | MCITP | PRINCE2 | ITIL | BSc Hons
Reply
0 Kudos