VMware Cloud Community
mattdevo1
Contributor
Contributor
‎08-04-2011 08:09 PM

vCloud Usage Meter v2.1 randomly failing to collect

We installed the vCloud Usage Meter v2.1 yesterday, the setup was fairly straight-forward except for the issue where you have to refresh the web page the first time it loads...

Some collections are randomly failing though, for example I can see 6 "OK" and 7 "Fail" collections under the monitor section for today (the failures are at random times, not in a single time period).  Nothing is or has been changed in our VMWare environment, and v2.0 has run for a few months without any failures.

When it fails to collect, we recieve an email with the following error:

A collection begun at 2011-08-05 10:55:00 has failed.

com.vmware.core.UnknownRemoteException: java.net.SocketException: Connection timed out

com.vmware.vim.managedobject.LicenseAssignmentManager.queryAssignedLicenses(LicenseAssignmentManager.java:70)

com.vmware.cloud.usagemetering.collector.LicenseFinder.find(LicenseFinder.scala:44)

com.vmware.cloud.usagemetering.collector.LicenseFinder.getLicense(LicenseFinder.scala:23)

com.vmware.cloud.usagemetering.collector.VcDataHandler.com$vmware$cloud$usagemetering$collector$VcDataHandler$$createSample(VcDataHandler.scala:145)

com.vmware.cloud.usagemetering.collector.VcDataHandler$$anonfun$collectSamples$1$$anonfun$apply$1.apply(VcDataHandler.scala:56)

com.vmware.cloud.usagemetering.collector.VcDataHandler$$anonfun$collectSamples$1$$anonfun$apply$1.apply(VcDataHandler.scala:49)

scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)

scala.collection.immutable.List.foreach(List.scala:45)

com.vmware.cloud.usagemetering.collector.VcDataHandler$$anonfun$collectSamples$1.apply(VcDataHandler.scala:49)

com.vmware.cloud.usagemetering.collector.VcDataHandler$$anonfun$collectSamples$1.apply(VcDataHandler.scala:42)

scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)

scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:35)

com.vmware.cloud.usagemetering.collector.VcDataHandler.collectSamples(VcDataHandler.scala:42)

com.vmware.cloud.usagemetering.collector.Collector$.collectAndSaveSamples(Collector.scala:73)

com.vmware.cloud.usagemetering.collector.Collector$$anonfun$scheduleCollecting$1$$anonfun$apply$1$$anon$1.run(Collector.scala:48)

java.util.TimerThread.mainLoop(Timer.java:512)

java.util.TimerThread.run(Timer.java:462)

Any help is much appreciated.


Regards

Matt

0 Kudos
20 Replies
dbriccetti
Hot Shot
Hot Shot
‎08-04-2011 09:36 PM

Hi Matt. Thanks for reporting the problem. Do all the emails report failing at LicenseAssignmentManager.java:70?

0 Kudos
mattdevo1
Contributor
Contributor
‎08-04-2011 09:48 PM

Yes they all fail at LicenseAssignmentManager.java:70

The meter has discovered 2 licenses, Enterprise plus and Standard.  I have both selected at the moment.

I've also noticed that when I run an aggregate report with both licenses selected, I only see Enterprise Plus in the report.  When I select the Standard license only, I see the report for Standard (I hope that makes sense?).


I’m not sure if this is related or just another bug….

Matt

0 Kudos
dbriccetti
Hot Shot
Hot Shot
‎08-04-2011 10:42 PM

Randy, I’m pulling in some other VMware resources to look into the timeout.

As to the second issue, I am able to reproduce it.

Thanks.

0 Kudos
mattdevo1
Contributor
Contributor
‎08-05-2011 05:34 AM

Twice tonight we have recieved a different error.  Once again, nothing has changed in our VMWare environment.  One hour it failed with the below error, the next hour it worked, the next hour it failed with the below error again...

This is ridiculous, but probably worth working out what the issue might be, or there will probably be some very small usage reports this month!

A collection begun at 2011-08-05 20:55:00 has failed.

com.vmware.core.UnknownRemoteException: java.net.SocketTimeoutException: Read timed out

com.vmware.vim.service.PropertyCollectorAdapterImpl.retrieveProperties(PropertyCollectorAdapterImpl.java:397)
com.vmware.sdk.toolkit.view.ViewBaseImpl.retrieveView(ViewBaseImpl.java:210)
com.vmware.vim.service.EntityFinderImpl.findEntities(EntityFinderImpl.java:338)
com.vmware.vim.service.EntityFinderImpl.findEntities(EntityFinderImpl.java:82)
com.vmware.cloud.usagemetering.collector.VcDataHandler.com$vmware$cloud$usagemetering$collector$VcDataHandler$$findPoweredOnVMs(VcDataHandler.scala:109)
com.vmware.cloud.usagemetering.collector.VcDataHandler$$anon$1.<init>(VcDataHandler.scala:96)
com.vmware.cloud.usagemetering.collector.VcDataHandler.createResourcePoolVMsMap(VcDataHandler.scala:95)
com.vmware.cloud.usagemetering.collector.VcDataHandler.collectSamples(VcDataHandler.scala:32)
com.vmware.cloud.usagemetering.collector.Collector$.collectAndSaveSamples(Collector.scala:73)
com.vmware.cloud.usagemetering.collector.Collector$$anonfun$scheduleCollecting$1$$anonfun$apply$1$$anon$1.run(Collector.scala:48)
java.util.TimerThread.mainLoop(Timer.java:512)
java.util.TimerThread.run(Timer.java:462)
0 Kudos
dbriccetti
Hot Shot
Hot Shot
‎08-05-2011 11:23 AM

Are any of you other service providers having this error?

0 Kudos
Eug74
VMware Employee
VMware Employee
‎08-05-2011 08:58 PM

Hey there,

please make sure that you have v2.1 GA installed. The GA has only been released few days ago. The v2.1 Beta had few bugs in it, but the GA is working just fine.

So, if you have dowloaded Beta only then please go back to:

http://www.vmware.com/download/download.do?downloadGroup=UMSV2

download 2.1 GA and it should work.

0 Kudos
mattdevo1
Contributor
Contributor
‎08-07-2011 12:09 AM

The ova we used is this one: vCloudUsageMeter-2.1.01.0-456909_OVF10.ova

This was downloaded on 4/7/2011 - I can only assume that this is the GA release?

We are still having random failures.  Yesterday there were 11 "OK" and 13 "Failure" collections.

0 Kudos
Eug74
VMware Employee
VMware Employee
‎08-07-2011 12:18 AM

The GA release was on the 29th of July 2011, so it is very unlikley that you are running a GA version of the metering tool.

Try re-downloading.

cheers

0 Kudos
mattdevo1
Contributor
Contributor
‎08-07-2011 12:33 AM

Sorry, typo....

This was downloaded on 4/8/2011

The filename of the version we are using is: vCloudUsageMeter-2.1.01.0-456909_OVF10.ova

0 Kudos
dbriccetti
Hot Shot
Hot Shot
‎08-07-2011 01:11 AM

Yes, that’s 2.1 GA. You are the only one to report this problem, so that could be helpful. Maybe there is something special about your environment. Very busy VC servers, perhaps? So busy that they time out. We could experiment with setting longer timeout values.

0 Kudos
dbriccetti
Hot Shot
Hot Shot
‎08-10-2011 10:21 PM

Let’s collect some wire-level details of what’s going on. Can you help us, please, Matt?

Please back up the current file, then place the attached file here:

/usr/local/apache-tomcat-7.0.14/webapps/um/WEB-INF/classes/log4j.properties

Restart the VM, or just Tomcat, and when the error occurs, please compress and send us:

/var/log/usgmtr/um.log (2.1.1), or

/usr/local/apache-tomcat-7.0.14/logs/catalina.out (2.1)

Thanks!

Preview file
1 KB
0 Kudos
mattdevo1
Contributor
Contributor
‎08-11-2011 02:10 AM

OK I have put the log4j.properties file on the server and rebooted.

When it throws the error again, I can send you through the log file.

I have a stupid question, but how will I get the log files off the appliance?  There seems to be very limited programs installed, what is your reccomendation?

0 Kudos
dbriccetti
Hot Shot
Hot Shot
‎08-11-2011 10:12 AM

You can use scp if you first start the sshd service (I am assuming you have changed the root password, and this won’t create any security problems for you).

service sshd start

Later:

service sshd stop

0 Kudos
mattdevo1
Contributor
Contributor
‎08-23-2011 07:31 PM

We have found the root cause of the issue....


This is extremely embarrassing, however the vCloud Usage Meter appliance was using an IP address that was already in use on our network, ie. a duplicate IP address.

Once we changed the IP address to something that wasn't being used, everything magically started to work....

Thanks for your assistance guys and I’m really sorry about the time spent on this one.

Note to self/everyone: Make sure you’re using a unique IP address!! Doh!

0 Kudos
dbriccetti
Hot Shot
Hot Shot
‎08-23-2011 07:40 PM

That's great news, Matt.

0 Kudos
SysDevHub
Contributor
Contributor
‎11-01-2011 02:30 AM

Hello, we are having similar issues.We have eliminated firewall and IP issue possibilities. Here is our error stack:

A collection from 209.203.0.197, begun at 2011-10-31 16:59:00, failed at 2011-10-31 16:59:02.

Usage Meter will wait 300 seconds and retry up to 2 more times.

java.util.NoSuchElementException: key not found: hardware

scala.collection.MapLike$class.default(MapLike.scala:223)

scala.collection.immutable.Map$Map1.default(Map.scala:93)

scala.collection.MapLike$class.apply(MapLike.scala:134)

scala.collection.immutable.Map$Map1.apply(Map.scala:93)

com.vmware.cloud.usgmtr.collect.mngobj.HostSystem$.apply(HostSystem.scala:11)

com.vmware.cloud.usgmtr.collect.vc.VCenterCollector$$anonfun$9.apply(VCenterCollector.scala:83)

com.vmware.cloud.usgmtr.collect.vc.VCenterCollector$$anonfun$9.apply(VCenterCollector.scala:82)

scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)

scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)

scala.collection.immutable.Map$Map3.foreach(Map.scala:152)

scala.collection.TraversableLike$class.map(TraversableLike.scala:206)

scala.collection.immutable.Map$Map3.map(Map.scala:132)

com.vmware.cloud.usgmtr.collect.vc.VCenterCollector.fetch(VCenterCollector.scala:82)

com.vmware.cloud.usgmtr.collect.VcDataHandler.getFromVc(VcDataHandler.scala:76)

com.vmware.cloud.usgmtr.collect.VcDataHandler.collectSamples(VcDataHandler.scala:30)

com.vmware.cloud.usgmtr.collect.Collector$.collectAndSaveSamples(Collector.scala:91)

com.vmware.cloud.usgmtr.collect.Collector$$anonfun$com$vmware$cloud$usgmtr$collect$Collector$$collectAll$1.apply(Collector.scala:70)

com.vmware.cloud.usgmtr.collect.Collector$$anonfun$com$vmware$cloud$usgmtr$collect$Collector$$collectAll$1.apply(Collector.scala:70)

com.vmware.cloud.usgmtr.commons.Parallelizer$$anonfun$2$$anon$1.call(Parallelizer.scala:18)

java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

java.util.concurrent.FutureTask.run(FutureTask.java:138)

java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

java.lang.Thread.run(Thread.java:662)

and the relevant catalina out file

cantina.jpg

Any help would be much appreaciated.

Kind Regards

0 Kudos
Eug74
VMware Employee
VMware Employee
‎11-01-2011 04:36 PM

Hi there,

The Usage Meter v2.2 has now been available for some time and it addressed a lot of bug issues. Please try and run it.

The link to v2.2 is the same as for the earlier versions

Cheers

0 Kudos
SysDevHub
Contributor
Contributor
‎11-01-2011 11:24 PM

Hello.

I should have included one last detail in that post. We are tryuing to use 2.2. The only reason I posted in this thread is because it appeared to be the most recently active and I hoped that would mean I was more likely to get a response from it.

0 Kudos
Eug74
VMware Employee
VMware Employee
‎11-02-2011 06:20 PM

I would sagest to email metering tool support team vcloudusagemeter@vmware.com for the best response.

0 Kudos