3.3.0 Monthly Reports not Running?

[phildw]

We upgraded to 3.3.0 early june and all data collection appeared fine.  However, at the start of July when we tried to run the monthly report for June, the appliance just sits there at 100% CPU and report never returns.

I've just upgraded to 3.3.1 but outcome appears to be the same:

top - 14:37:07 up 11 min,  1 user,  load average: 1.00, 1.07, 0.68

Tasks:  89 total,   2 running,  87 sleeping,   0 stopped,   0 zombie

Cpu(s): 99.7%us,  0.3%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

Mem:   3619928k total,  1676960k used,  1942968k free,    18876k buffers

Swap:  2105340k total,        0k used,  2105340k free,  1175632k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                               

4065 postgres  20   0  101m  47m  35m R 99.2  1.3   9:14.02 postgres     

Any thoughts before I raise a support case?

Looks like it the valid query running in postgres.

usgmtr=# select query from pg_stat_activity ;

                                                                                query                                                                                 

------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Select                                                                                                                                                                +

   sum(capped("Sample2"."vmBillingMemory",$1)) as "c0"                                                                                                                 +

From                                                                                                                                                                  +

   "SampleToLicense" "SampleToLicense1",                                                                                                                               +

   "Sample" "Sample2",                                                                                                                                                 +

   "Collection" "Collection3",                                                                                                                                         +

   "Vm" "Vm4",                                                                                                                                                         +

   "HostLatest" "HostLatest5",                                                                                                                                         +

   "License" "License6"                                                                                                                                                +

Where                                                                                                                                                                 +

   ((("SampleToLicense1"."vmId" in ((Select                                                                                                                            +

      "CustomerVm9"."vmId" as "CustomerVm9_vmId"                                                                                                                       +

    From                                                                                                                                                               +

      "CustomerVm" "CustomerVm9",                                                                                                                                      +

      "CustomerVmRule" "CustomerVmRule10"                                                                                                                              +

    Where                                                                                                                                                              +

      (("CustomerVmRule10"."createdBy" = $2) and ("CustomerVm9"."customerVmRuleId" = "CustomerVmRule10"."id"))                                                         +

   ) )) and (betweenco("SampleToLicense1"."start",$3,$4) and "SampleToLicense1"."reportExclusionReason" is null)) and ("SampleToLicense1"."sampleId" = "Sample2"."id"))

[dbriccetti]

Hi. We have some changes coming soon that should speed up report generation. In the meantime, how about deleting some of your older data from the database? You can do that on the Support page. It should make a huge difference.

[phildw]

I did see that, but I am trying to confirm what our VSPP obligations are for retaining data.  I thought we had to keep 3 years of usage data.

Phil

[dbriccetti]

Yes, you do. Choosing that delete option gives you this:

“This feature deletes data older than the specified number of days. If you want to delete data still within the required VSPP program retention period, you must back up the appliance first.”

[phildw]

I've archived the appliance and deleted everything down to 90 days.  Still cannot run reports, but now I get a Null pointer exception.  I'll raise an SR

[dbriccetti]

If you like, paste the stack trace here.

[phildw]

Where do I find the full trace?  I've grep'd through the support bundle and /opt and /var/log

[dbriccetti]

Can we see a screen shot of where you see the NullPointerException? I see the stack trace you sent Global Support Services. Does that error correspond in time with when you’re seeing the NPE?

[dbriccetti · Accepted solution]

The problem may be that Usage Meter is expecting each license it finds on each vCenter Server to have an “entityDisplayName”.

vSphere 5.5 Documentation Center

But I see from the documentation that this can be null. Let me consider, with GSS, what our options are.

[phildw]

Update to close this off.

We did indeed have some older licenses loaded in vSphere that ended up with a null in the entityDisplayName field.  We removed these licenses from vSphere which appeared to do the trick.

The gotcha here is that the vCloud appliance has 2 copies of license information.  1 that you can see though the web interface and is stored in the database, and a second in memory cache which is the license information read from the vSphere server(s) themselves.  It's this cached entry that contains the null, not the db itself.

This will be addressed in a subsequent release of the appliance.  Currently, if I reboot the appliace I can run monthly reports without issue, but after a collection then I can still get the error.  However all the data is correctly loaded in the database.  It's just the cache.

[dbriccetti]

Thanks, Phil. It might be helpful if you were to edit the tile and add something like “when license has empty entityDisplayName field”.