Just in case someone runs into the same issue.
Today after an Oracle DB crash, vCloud Director started complaining that it cannot execute queries anymore with the following message:
Internal Server Error
- could not execute query
- ORA-01555: snapshot too old: rollback segment number with name "" too small
ORA-22924: snapshot too old
Backup (expdp) also lead to the same issue with PROPERTY_MAP table.
ORA-31693: Table data object "VCLOUD"."PROPERTY_MAP" failed to load/unload and is being skipped due to error:
ORA-02354: error in exporting/importing data
ORA-01555: snapshot too old: rollback segment number with name "" too small
ORA-22924: snapshot too old
I engaged VMware support team and their suggestion was to clean up all inventory tables and force vCloud to re-sync back from vCenter.
The procedure goes like this: [ON YOUR OWN RESPONSIBILITY]
1. Make sure you keep latest valid backup of vCloud Director [just in case]
2. Graceful shutdown of all vCloud Director cells
3. Using Oracle SQL Developer connect to Oracle vCloud Director scheme.
4. Run the following queries:
delete from task;
update jobs set status = 3 where status = 1;
update last_jobs set status = 3 where status = 1;
delete from busy_object;
delete from QRTZ_SCHEDULER_STATE;
delete from QRTZ_FIRED_TRIGGERS;
delete from QRTZ_PAUSED_TRIGGER_GRPS;
delete from QRTZ_CALENDARS;
delete from QRTZ_TRIGGER_LISTENERS;
delete from QRTZ_BLOB_TRIGGERS;
delete from QRTZ_CRON_TRIGGERS;
delete from QRTZ_SIMPLE_TRIGGERS;
delete from QRTZ_TRIGGERS;
delete from QRTZ_JOB_LISTENERS;
delete from QRTZ_JOB_DETAILS;
delete from compute_resource_inv;
delete from custom_field_manager_inv;
delete from cluster_compute_resource_inv;
delete from datacenter_inv;
delete from datacenter_network_inv;
delete from datastore_inv;
delete from dv_portgroup_inv;
delete from dv_switch_inv;
delete from folder_inv;
delete from managed_server_inv;
delete from managed_server_datastore_inv;
delete from managed_server_network_inv;
delete from network_inv;
delete from resource_pool_inv;
delete from storage_pod_inv;
delete from task_inv;
delete from vm_inv;
delete from property_map;
commit; --most important one
5. Start up vCloud Director cells and check the cell.log to verify that cells are booting up properly.
6. Login to the web UI and check vCenters status - you should see that the Inventory is being synced.
7. Re-run DB export (expdp)
