vCloud Director 5.5.2 - Provisioning vApp Fails : Data Truncation

[Justin1911]

Recently a vCloud environment I was working in was unable to provision any VMs.  The vcloud-container-debug.log had the following relevant entries:

2015-03-26 14:05:20,424 | ERROR    | backend-activity-pool-16  | JDBCExceptionReporter          | Data truncation | vcd=4c14b291-7ca6-4697-8127-838e6768c1e3,task=6ce4e6a8-33d5-4eaa-ae26-ced099fbcd34 activity=urn:uuid:6ce4e6a8-33d5-4eaa-ae26-ced099fbcd34

2015-03-26 14:05:20,424 | ERROR    | backend-activity-pool-16  | JDBCExceptionReporter          | Arithmetic overflow error converting expression to data type int. | vcd=4c14b291-7ca6-4697-8127-838e6768c1e3,task=6ce4e6a8-33d5-4eaa-ae26-ced099fbcd34 activity=urn:uuid:6ce4e6a8-33d5-4eaa-ae26-ced099fbcd34

After working with VMware support, it was determined that the cause of this was a disconnected datastore that was reporting an available size "in the exabytes".  VMware support analyzed the VCD database to determine this.  The environment was also using the *Any* storage policy.  But even when that policy was no longer being presented, provisioning still failed with the same error.

The resolution was to remove the disconnected datastore and provisioning completed successfully.

This involved a lengthy resolution process, so I wanted to take the time to share the findings should someone else ever run in to this one.

[IamTHEvilONE]

To explain a bit further as to what I saw when working on a case (not sure if it was yours or someone else Justin1911).

vCD Inventories all Datastores individually as data type BIG INT, and then sum all Datastores in a given policy to find the total capacity (Also BIG INT).

BIG INT in SQL is a signed value between -2^63 (-9,223,372,036,854,775,808) & 2^63-1 (9,223,372,036,854,775,807) from MS SQL and there is no larger INT type at this time (that I know of).  the "truncation" is when the sum of datastores size in bytes exceeds the maximum value allowed as a Big Int.

9 EB is an absolutely huge amount of data.  The only time I have seen this data truncation issue happen, is when the Storage Device is incorrectly representing the capacity of a Datastore/Volume to ESXi/vCenter.

In a sense, we are seeing the problem in vCloud Director, but I can't think of any single storage device that can offer this.  I have seen storage profiles in PetaBytes ... but that's a full order of magnitude away from ExaBytes.

So the solution, as mentioned, would be to remove the Datastore from vCenter Inventory and try to find out how or why the Storage Device is representing 9 EB when it probably shouldn't be.

[Justin1911]

If it was resolved last Friday it almost certainly was mine.

Thanks to the guys in support who put in the hard work and did the deep dive to resolve this.

[IamTHEvilONE]

Then probably not.  I saw one around beginning of February.