VMware Cloud Community
pscheri
Enthusiast
Enthusiast

vCloud database upgrade fail

Hi! I am trying to upgrade vCloud Director to 5.1 from 1.5.1.

My installation consists in 1 vCloud Director Cell and an SQL 2008 Server.

I´ve already upgrade vCloud Director Server, but when I run the "upgrade" script, I get the following error according to the logs:

"Unable to upgrade the database: java.sql.SQLException: Cannot insert the value NULL into column 'memory_configured_mb', table 'vcloud01.dbo.computevm'; column does not allow nulls. INSERT fails."

The only additional modification to the DB, is the "GuestOS" type for ESXi, but I had already tried to remove that customization with an old backup.

Thanks in advance!

Pablo.-

2012-09-17 11:22:56,552 | DEBUG    | pool-1-thread-1           | LoggingProgressListener        | Upgrade Task Received progress report: com.vmware.vcloud.progress.ImmutableProgressReport@25f45576 {state: RUNNING, progress: 13%} |
2012-09-17 11:22:56,552 | DEBUG    | pool-1-thread-1           | SerialAggregateTask            | Steps to upgrade to 2.0.34: Beginning execution of task Record version 2.0.33.transition |
2012-09-17 11:22:56,557 | DEBUG    | pool-1-thread-1           | SerialAggregateTask            | Steps to upgrade to 2.0.34: Beginning execution of task Compute.groovy:22 |
2012-09-17 11:22:56,557 | DEBUG    | pool-1-thread-1           | RawSQLTask                     | Executing sql 'INSERT INTO computevm (id, computehub_id, vrp_id, creation_status, memory_min_mb,
        memory_configured_mb, memory_overhead_mb, cpu_min_mhz, num_vcpu, vmmoref)
        SELECT distinct vm.id, vrprp.computehub_id, vrp.id,
        0,
        vminv.mem_reservation, vminv.mem, vminv.mem_overhead,
        vminv.cpu_reservation, vminv.vcpu_count,
        vm.moref
        FROM vm vm, vm_inv vminv, vapp_vm vappvm,
        vrp vrp, vrp_rp vrprp,
        vm_container vmcont,
        org_prov_vdc o, org_prov_vdc_rp orp
        WHERE vm.moref = vminv.moref
        AND vm.vc_id = vminv.vc_id
        AND vm.id NOT IN (SELECT vm_id FROM deployed_vm)
        AND vm.id = vappvm.svm_id
        AND vappvm.vapp_id = vmcont.sg_id
        AND vmcont.org_vdc_id = o.id
        AND o.id = orp.org_prov_vdc_id
        AND orp.rp_moref = vrprp.sub_rp_moref
        AND orp.vc_id=vrprp.sub_rp_vc_id
        AND vrp.id = vrprp.vrp_id
        AND vminv.resource_pool_moref = vrprp.sub_rp_moref' |
2012-09-17 11:22:56,673 | WARN     | pool-1-thread-1           | SerialAggregateTask            | Steps to upgrade to 2.0.34: Task failed due to uncaught exception |
java.sql.SQLException: Cannot insert the value NULL into column 'memory_configured_mb', table 'vcloud01.dbo.computevm'; column does not allow nulls. INSERT fails.
        at net.sourceforge.jtds.jdbc.SQLDiagnostic.addDiagnostic(SQLDiagnostic.java:368)
        at net.sourceforge.jtds.jdbc.TdsCore.tdsErrorToken(TdsCore.java:2816)
        at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2254)
        at net.sourceforge.jtds.jdbc.TdsCore.getMoreResults(TdsCore.java:631)
        at net.sourceforge.jtds.jdbc.JtdsStatement.processResults(JtdsStatement.java:584)
        at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQL(JtdsStatement.java:546)
        at net.sourceforge.jtds.jdbc.JtdsStatement.executeImpl(JtdsStatement.java:723)
        at net.sourceforge.jtds.jdbc.JtdsStatement.execute(JtdsStatement.java:1157)
        at com.vmware.vcloud.upgrade.tasks.RawSQLTask.call(RawSQLTask.java:106)
        at com.vmware.vcloud.upgrade.tasks.RawSQLTask.call(RawSQLTask.java:1)
        at com.vmware.vcloud.upgrade.SerialAggregateTask.call(SerialAggregateTask.java:43)
        at com.vmware.vcloud.upgrade.SerialAggregateTask.call(SerialAggregateTask.java:1)
        at com.vmware.vcloud.upgrade.AbstractDelegatingTask.doCall(AbstractDelegatingTask.java:124)
        at com.vmware.vcloud.upgrade.TransactionTask.doCall(TransactionTask.java:74)
        at com.vmware.vcloud.upgrade.AbstractDelegatingTask.call(AbstractDelegatingTask.java:103)
        at com.vmware.vcloud.upgrade.AbstractDelegatingTask.call(AbstractDelegatingTask.java:1)
        at com.vmware.vcloud.upgrade.AbstractDelegatingTask.doCall(AbstractDelegatingTask.java:124)
        at com.vmware.vcloud.upgrade.UpgradeTask.doCall(UpgradeTask.java:79)
        at com.vmware.vcloud.upgrade.AbstractDelegatingTask.call(AbstractDelegatingTask.java:103)
        at com.vmware.vcloud.upgrade.AbstractDelegatingTask.call(AbstractDelegatingTask.java:1)
        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
2012-09-17 11:22:56,674 | DEBUG    | pool-1-thread-1           | RawSQLTask                     | Transaction for task 'com.vmware.vcloud.upgrade.SerialAggregateTask@67a9b034' will rollback |
java.sql.SQLException: Cannot insert the value NULL into column 'memory_configured_mb', table 'vcloud01.dbo.computevm'; column does not allow nulls. INSERT fails.
        at net.sourceforge.jtds.jdbc.SQLDiagnostic.addDiagnostic(SQLDiagnostic.java:368)
        at net.sourceforge.jtds.jdbc.TdsCore.tdsErrorToken(TdsCore.java:2816)
        at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2254)
        at net.sourceforge.jtds.jdbc.TdsCore.getMoreResults(TdsCore.java:631)
        at net.sourceforge.jtds.jdbc.JtdsStatement.processResults(JtdsStatement.java:584)
        at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQL(JtdsStatement.java:546)
        at net.sourceforge.jtds.jdbc.JtdsStatement.executeImpl(JtdsStatement.java:723)
        at net.sourceforge.jtds.jdbc.JtdsStatement.execute(JtdsStatement.java:1157)
        at com.vmware.vcloud.upgrade.tasks.RawSQLTask.call(RawSQLTask.java:106)
        at com.vmware.vcloud.upgrade.tasks.RawSQLTask.call(RawSQLTask.java:1)
        at com.vmware.vcloud.upgrade.SerialAggregateTask.call(SerialAggregateTask.java:43)
        at com.vmware.vcloud.upgrade.SerialAggregateTask.call(SerialAggregateTask.java:1)
        at com.vmware.vcloud.upgrade.AbstractDelegatingTask.doCall(AbstractDelegatingTask.java:124)
        at com.vmware.vcloud.upgrade.TransactionTask.doCall(TransactionTask.java:74)
        at com.vmware.vcloud.upgrade.AbstractDelegatingTask.call(AbstractDelegatingTask.java:103)
        at com.vmware.vcloud.upgrade.AbstractDelegatingTask.call(AbstractDelegatingTask.java:1)
        at com.vmware.vcloud.upgrade.AbstractDelegatingTask.doCall(AbstractDelegatingTask.java:124)
        at com.vmware.vcloud.upgrade.UpgradeTask.doCall(UpgradeTask.java:79)
        at com.vmware.vcloud.upgrade.AbstractDelegatingTask.call(AbstractDelegatingTask.java:103)
        at com.vmware.vcloud.upgrade.AbstractDelegatingTask.call(AbstractDelegatingTask.java:1)
        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
2012-09-17 11:22:56,675 | DEBUG    | pool-1-thread-1           | LoggingProgressListener        | Upgrade Task Received progress report: com.vmware.vcloud.progress.ImmutableProgressReport@260fa8d9 {state: FAILED, progress: 13%} |
2012-09-17 11:22:56,677 | ERROR    | main                      | UpgradeAgent                   | Unable to upgrade the database: java.sql.SQLException: Cannot insert the value NULL into column memory_configured_mb, table vcloud01.dbo.computevm; column does not allow nulls. INSERT fails. |
com.vmware.vcloud.upgrade.UpgradeAgentException: java.sql.SQLException: Cannot insert the value NULL into column 'memory_configured_mb', table 'vcloud01.dbo.computevm'; column does not allow nulls. INSERT fails.
        at com.vmware.vcloud.upgrade.UpgradeAgent.runUpgradeTasks(UpgradeAgent.java:1219)
        at com.vmware.vcloud.upgrade.UpgradeAgent.executeUpgrade(UpgradeAgent.java:745)
        at com.vmware.vcloud.upgrade.UpgradeAgent.start(UpgradeAgent.java:979)
        at com.vmware.vcloud.upgrade.UpgradeAgent.main(UpgradeAgent.java:881)
Caused by: java.sql.SQLException: Cannot insert the value NULL into column 'memory_configured_mb', table 'vcloud01.dbo.computevm'; column does not allow nulls. INSERT fails.
        at net.sourceforge.jtds.jdbc.SQLDiagnostic.addDiagnostic(SQLDiagnostic.java:368)
        at net.sourceforge.jtds.jdbc.TdsCore.tdsErrorToken(TdsCore.java:2816)
        at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2254)
        at net.sourceforge.jtds.jdbc.TdsCore.getMoreResults(TdsCore.java:631)
        at net.sourceforge.jtds.jdbc.JtdsStatement.processResults(JtdsStatement.java:584)
        at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQL(JtdsStatement.java:546)
        at net.sourceforge.jtds.jdbc.JtdsStatement.executeImpl(JtdsStatement.java:723)
        at net.sourceforge.jtds.jdbc.JtdsStatement.execute(JtdsStatement.java:1157)
        at com.vmware.vcloud.upgrade.tasks.RawSQLTask.call(RawSQLTask.java:106)
        at com.vmware.vcloud.upgrade.tasks.RawSQLTask.call(RawSQLTask.java:1)
        at com.vmware.vcloud.upgrade.SerialAggregateTask.call(SerialAggregateTask.java:43)
        at com.vmware.vcloud.upgrade.SerialAggregateTask.call(SerialAggregateTask.java:1)
        at com.vmware.vcloud.upgrade.AbstractDelegatingTask.doCall(AbstractDelegatingTask.java:124)
        at com.vmware.vcloud.upgrade.TransactionTask.doCall(TransactionTask.java:74)
        at com.vmware.vcloud.upgrade.AbstractDelegatingTask.call(AbstractDelegatingTask.java:103)
        at com.vmware.vcloud.upgrade.AbstractDelegatingTask.call(AbstractDelegatingTask.java:1)
        at com.vmware.vcloud.upgrade.AbstractDelegatingTask.doCall(AbstractDelegatingTask.java:124)
        at com.vmware.vcloud.upgrade.UpgradeTask.doCall(UpgradeTask.java:79)
        at com.vmware.vcloud.upgrade.AbstractDelegatingTask.call(AbstractDelegatingTask.java:103)
        at com.vmware.vcloud.upgrade.AbstractDelegatingTask.call(AbstractDelegatingTask.java:1)
        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

Reply
0 Kudos
5 Replies
pscheri
Enthusiast
Enthusiast

Solved!

The problem was that I had some inconsistency between vCloud Director and vCenter VMs. Actually there were some vApps in vCloud that its VMs were in Grayed state in vCenter server.

When the Upgrader tried to read the vmx file (through vCenter) to get the memory information, it gets a NULL value, and the installer failed.

Pablo.-

Reply
0 Kudos
shepherdz
VMware Employee
VMware Employee

pscheri wrote:

Solved!

The problem was that I had some inconsistency between vCloud Director and vCenter VMs. Actually there were some vApps in vCloud that its VMs were in Grayed state in vCenter server.

I'd like to make sure I understand what happened here to see if we can avoid this sort of problem in the future. To help with that, I'd be interested in more details on the resolution to your issue.

It sounds like your solution might have been to roll back the upgrade, fix the inconsistency, and re-run upgrade. Is that an accurate description? If so, what steps did you need to take to fix the VMs in your case?

When the Upgrader tried to read the vmx file (through vCenter) to get the memory information, it gets a NULL value, and the installer failed.

Pablo.-

As an aside, the upgrade process doesn't actually talk to vCenter. What the log messages you provided show is the upgrade process referencing cached information from vCenter stored in the vCD database. In this case, it basically boils down to the same behavior you described, but I just want to make sure no one stumbling across this thread in the future gets confused.

Zach Shepherd, Member of Technical Staff, VMware vCloud Director Engineering
Reply
0 Kudos
pscheri
Enthusiast
Enthusiast

Hi Zach,

First of all, sorry for my "not so clear" english, let me explain myself a little further.

Zach Shepherd escribió:

I'd like to make sure I understand what happened here to see if we can avoid this sort of problem in the future. To help with that, I'd be interested in more details on the resolution to your issue.

It sounds like your solution might have been to roll back the upgrade, fix the inconsistency, and re-run upgrade. Is that an accurate description? If so, what steps did you need to take to fix the VMs in your case?

The solution was:

  1. Restore the vCloud MSSQL DB from a previous backup.
  2. Start vCloud Director services (was not necesary to revert to vCloud VM Snapshot, with DB ver. 1.5 and vCloud ver. 5.1 it worked just fine)
  3. Search in vCenter Server console, the VMs that were grayed out and depends of vCloud. This was the only way to identify the conflicted VMs. It was not possible to do it directly from vCloud.
  4. Identify those vCenter´s VMs in the vCloud console, and completely remove them. Those VMs were not really there anymore, not even in the Datastores... I suppose they might have been deleted from storage, vCenter, o maybe they were just inconsistence VMs.
  5. Quiesce the cell.
  6. Stop the services.
  7. Upgrade the DB.

Actually, when the upgrade process failed, I could see that the Table in the DB had been generated, but it was fill with only 1 ROW with all NULL values.

As an aside, the upgrade process doesn't actually talk to vCenter. What the log messages you provided show is the upgrade process referencing cached information from vCenter stored in the vCD database. In this case, it basically boils down to the same behavior you described, but I just want to make sure no one stumbling across this thread in the future gets confused.

Thanks for the clarification!!

Pablo.-

Reply
0 Kudos
shepherdz
VMware Employee
VMware Employee

Thanks for the explanation.

Zach Shepherd, Member of Technical Staff, VMware vCloud Director Engineering
Reply
0 Kudos
JasonBurrell
Enthusiast
Enthusiast

I had the same problem, here is how I found out what VM was causing it:

Ran this query off of the 1/2 upgraded database and found the VM with mem that was NULL

SELECT distinct vm.id, vrprp.computehub_id, vrp.id,
        0,
        vminv.mem_reservation, vminv.mem, vminv.mem_overhead,
        vminv.cpu_reservation, vminv.vcpu_count,
        vm.moref
        FROM vm vm, vm_inv vminv, vapp_vm vappvm,
        vrp vrp, vrp_rp vrprp,
        vm_container vmcont,
        org_prov_vdc o, org_prov_vdc_rp orp
        WHERE vm.moref = vminv.moref
        AND vm.vc_id = vminv.vc_id
        AND vm.id NOT IN (SELECT vm_id FROM deployed_vm)
        AND vm.id = vappvm.svm_id
        AND vappvm.vapp_id = vmcont.sg_id
        AND vmcont.org_vdc_id = o.id
        AND o.id = orp.org_prov_vdc_id
        AND orp.rp_moref = vrprp.sub_rp_moref
        AND orp.vc_id=vrprp.sub_rp_vc_id
        AND vrp.id = vrprp.vrp_id
        AND vminv.resource_pool_moref = vrprp.sub_rp_moref
        order by mem

Once I had that I tried to find the name by moref ID.

SELECT TOP 1000 [id]

      ,[vc_id]

      ,[computer_name]

      ,[storage_allocation_mb]

      ,[validity_status]

      ,[guest_os]

      ,[is_vrouter]

      ,[is_published]

      ,[date_created]

      ,[template_id]

      ,[is_template]

      ,[needs_customization]

      ,[primary_nic_id]

      ,[guest_cust_vm_id]

      ,[guestos_id]

      ,[moref]

      ,[location_path]

      ,[cloud_uuid]

      ,[instance_uuid]

      ,[creation_status]

      ,[version_number]

      ,[suspend_proctype_id]

      ,[snapshot_moref]

      ,[dstore_moref]

      ,[ovf_env]

      ,[ovf_env_transports]

  FROM [VCLOUDDB].[dbo].[vm]

where moref = 'vm-8971'

It gave me the name of the VM and it's ID, the problem was the ID was not in vCenter so I had to dig through the vCloud UI until I found a machine that matched it, deleted it and after that the upgrade ran smoothly.

Hope this helps someone!

Reply
0 Kudos