VMware Cloud Community
Itzikr
Enthusiast
Enthusiast

Upgrade to 5.1 Failed and resulted in a corrupted DB

Hi,

i tried to upgrade vCD 1.5 to 5.1, the VCD upgrade itself went ok but it failed on the DB upgrade:

Unable to upgrade the database: java.sql.SQLException: Violation of UNIQUE KEY constraint
'uq_os_name_version_update'. Cannot insert duplicate key in object 'dbo.os'. The duplicate key value is
(VMware ESXi, 5.1.0, 0).

when i tried to run the same DB command again, it came with an error that i need to restore my DB !!

any idea how to fix this??

thanks

Itzik Reich
Reply
0 Kudos
15 Replies
shepherdz
VMware Employee
VMware Employee

Please file a support request for this to ensure it gets investigated properly. (Improper recovery from failed upgrades can lead to data loss, which I imagine you'd want to avoid.)

In the mean time, I can try to help.

Did you upgrade any ESX hosts before upgrading vCD?

Could you PM me your upgrade-*.log file(s) (located in /opt/vmware/vcloud-director/logs)? (I suggest this instead of attaching them here because they could contain sensitive data. If this is just a test environment and you're comfortable attaching them here, that's fine too.)

If you need to get back up and running, you should be able to do so by restoring your database from the backups you made prior to upgrade and then restoring your cell from a backup (or, if you don't have a pre-upgrade backup of your cell, you can just deploy a new vCD 1.5 cell using your response.properties  file (like you were trying to add a new cell to a multi-cell server  group).

Zach Shepherd, Member of Technical Staff, VMware vCloud Director Engineering
Reply
0 Kudos
FABSAN
Contributor
Contributor

did you ever find out what the issue was? we are looking to upgrade our environment to 5.1 from 1.5 as well. Thanks!

Reply
0 Kudos
shepherdz
VMware Employee
VMware Employee

FABSAN wrote:

did you ever find out what the issue was? we are looking to upgrade our environment to 5.1 from 1.5 as well. Thanks!

I never heard back after my comment above, but strongly suspect that the enountered this issue because an ESX host in the environment was upgraded prior to the vCD upgrade. As long as you upgrade vCD prior to any ESX host, you shouldn't be able to run into this issue.

One way to check to see if you're likely to enounter this (or another similar issue) during the database upgrade step of the upgrade process is to try the upgrade in a backup. To do this, you can deploy a database backup in an isolated environment (one that's so completely isolated that you couldn't accidentally connect to the real database if you mis-typed something). In that environment, you can deploy a new 5.1 cell and configure it using the response.properties file from the production environment like you were adding another cell to your environment (again, you really want to do this in an isolated environment so that you don't accidentally configure it to use the real database). Then you can run ./bin/upgrade from that new cell which is configured to use the restored backup of the database to simulate a "dry run" of upgrading the vCD database. If that upgrade fails, you'll probably want to dig into that failure with support to determine whether it's a real problem or simply an false-positive as a result of the "simulated" environment, but it it succeeds, you can be pretty confident you won't see an issue like this when you go to upgrade "for real". (Note that even if the upgrade succeeds, you won't be able to start the system to verify anything; you've only duplicated the database and not the other dependencies like vCenter and vShield.)

Zach Shepherd, Member of Technical Staff, VMware vCloud Director Engineering
Reply
0 Kudos
adam5983
Contributor
Contributor

I'm having the exact same problem.

Unfortuantely i've upgraded my vcentre and some of the hosts to 5.1 already...

I've moved all vms onto the old 5.0 host and removed all 5.1 hosts from inventory, and restored vcloud drictor and database to prior the 5.1 upgrade.

I then did a update, still fails at the same spot with the same error message.

Help please!

Reply
0 Kudos
Itzikr
Enthusiast
Enthusiast

Hi,

sorry for not replying before, im still stuck at the same situation, i HAVE upgraded my vCenter / ESXi hosts to 5.1 prior the failed vcD 5.1 upgrade and have no backup! (insert your cynical comment here..)

can this situation be fixed, i really dont want to recreate all of the vCD enviornment again..

thanks!

Itzik Reich
Reply
0 Kudos
gaadmin
Contributor
Contributor

This is an interesting post, I have the exact same problem. Brought about by the same circumstance I believe.

We upgraded a couple of our ESXi hosts under VCD1.5 to ESXi 5.1 and the subsequent DB upgrade fails.

Luckily (I hope) we took your advice and tested the upgrade on a copy of our DB...


Can you advise the best way forward from here?

Thanks,
Alex

Reply
0 Kudos
shepherdz
VMware Employee
VMware Employee

adam5983 wrote:

I'm having the exact same problem.

Unfortuantely i've upgraded my vcentre and some of the hosts to 5.1 already...

I've moved all vms onto the old 5.0 host and removed all 5.1 hosts from inventory, and restored vcloud drictor and database to prior the 5.1 upgrade.

I then did a update, still fails at the same spot with the same error message.

Help please!

If the database backup was taken prior to the vCD 5.1 upgrade, but after a host was upgraded, you'd need to do some extra clean up. If the backup was taken prior to any of the host upgrades, I wouldn't expect the upgrade to fail with the same error message. Either way, I'd suggest opening a support request so that you can get support to take a look at your system and see what's going on with it.

Itzik wrote:

Hi,

sorry for not replying before, im still stuck at the same situation, i HAVE upgraded my vCenter / ESXi hosts to 5.1 prior the failed vcD 5.1 upgrade and have no backup! (insert your cynical comment here..)

can this situation be fixed, i really dont want to recreate all of the vCD enviornment again..

thanks!

It should be possible to fix this even without a backup, but I suspect it will involve direct manipulation of the vCD database. You're going to want to involve someone from support to make sure that the recovery process goes smoothly. (If you haven't already, I'd actually suggest starting to make a backup of your current broken database. That may save some time when you get in touch with support and will ensure that you have something to fall back on if the first attempt at recovery goes poorly for some reason.)

gaadmin wrote:

This is an interesting post, I have the exact same problem. Brought about by the same circumstance I believe.

We upgraded a couple of our ESXi hosts under VCD1.5 to ESXi 5.1 and the subsequent DB upgrade fails.

Luckily (I hope) we took your advice and tested the upgrade on a copy of our DB...


Can you advise the best way forward from here?

Thanks,
Alex

In this case, the best way forward would definitely be to reach out to support. [I know I sound like a broken record at this point, but I'd hate to give someone specific advice here without entirely understanding their deployment environment (or to give someone advice and have someone else try to follow it despite their specific environment or the process that led up to the error being a bit different) and have something go wrong; upgrade just isn't an area where you want to guess-and-check your way through things.]

Having a copy where you can test things should simplify the process; I suspect support will supply you with a short workaround to try on the copy and, when it works, use for the real system.

Zach Shepherd, Member of Technical Staff, VMware vCloud Director Engineering
Reply
0 Kudos
IamTHEvilONE
Immortal
Immortal

Hey everyone,

I saw this thread going and wanted to chime in, as I am on the VMware Support Team who reply to your requests.

I have seen this come through support, and we have done spot fixes for people. 

However, I am getting the proposed fix verified as okay by our development staff to ensure that it doesn't have any long lasting effects on the environment.

Best Regards,

Jon Hemming

Reply
0 Kudos
gaadmin
Contributor
Contributor

Thanks for the update Jon, do you have a rough timeframe for the hotfix?

Also, we find ourselves in limbo between vSphere 5.0 and 5.1. Given you're working on a fix for the vCloud issue, does it make sense for us to continue upgrading our ESX hosts to 5.1 or should we hold off?

Thanks,

Alex

Reply
0 Kudos
IamTHEvilONE
Immortal
Immortal

The specific issue here is that vCloud Director 1.5.1, or prior, has been connected to vCenter 5.1 with an ESXi 5.1 host attached to it.  Because it doesn't have the bits to support it, 5.1 gets flagged as unsupported in the database.

Officially, vCloud 1.5.1 and previous hasn't been tested with vCenter 5.1 or ESXi 5.1 ... so it was doing it's job.

If you have already hit the error state during the upgrade of vCloud 1.5.x or previous to 5.1, and don't have a backout plan, I don't think it will cause more problems by upgrading the rest of the hosts.

Problem is you will not have a functioning vCloud Director interfrace ... so you have to think how much of an impact this would be to the company.

If you can, please file a support request for vCloud Director if you want to try and get this resolved quickly. 

Once you have a support request number, send it to me via Private Message and I will attach notes to it for the tech to work with.  Just know that I'm based in EDT.

Reply
0 Kudos
Itzikr
Enthusiast
Enthusiast

Hi,

i can see that this issue has received a VMware KB:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=203662...

my question is can it be fixed without a restore of the vCD DB (which i dont have..)

Thanks

Itzik Reich
Reply
0 Kudos
IamTHEvilONE
Immortal
Immortal

I think, given your situation, is to take a backup of the DB as is, then perform steps 3 through 6.

The upgrade process should continue from last failure, but we wanted to try and be as clean as possible.

Reply
0 Kudos
Itzikr
Enthusiast
Enthusiast

Thanks but I do t think that will work because the vCD vm complains that the RPM's have already been installed

Itzik

Itzik Reich
Reply
0 Kudos
IamTHEvilONE
Immortal
Immortal

I think we just have to run the db upgrade component if we can't fully roll back.

However, I would really appreciate it if you can file a support ticket and PM me the ticket number.  I can tag the appropraite people to look at it in your time zone.

Reply
0 Kudos
Itzikr
Enthusiast
Enthusiast

Hi,

as you can see, apart from running the first sql command succesfully, the other 2 fail..

Itzik Reich
Reply
0 Kudos