vCenter 5.0 update 1 upgrade - database not upgraded?

[crazygibbon]

Hi,

Upgraded our vCenter 5.0 server to update 1 - had no error messages, and upgraded the components (update manager etc), rebooted and all seemed fine until I checked the vCenter service status. I have 6 alerts in there:-

Under license services, I have:-

Threshold Usage Tracking Service - Cannot obtain user-defined license thresholds

Asset Properties History Service - Cannot store hosts' MAC addresses in the vCenter Server Database

Assignments Feeding Service - Cannot obtain license assignments for VRAM usage

License Usage History Service - Cannot store license usage in the vCenter Server database

Also have:-

VMware vSphere Update Manager Extension - unable to retrieve health data from http://<vcenter>/downloads/health.xm;

VMware vCenter Storage Monitoring Service - Service initialization failed

This has all the hallmarks of the upgrade having upgraded vCenter, but not having done the database upgrade. Database is SQL 2005 on a separate server, and worked fine as vCenter 5.0 initially - my account has DBO permissions on the vCenter database and on MSDB.

Anyone else had this problem?  I have an SR logged with VMware, but thought I'd see if this has happened to anyone else...

JD

[som94]

Hi crazygibbon,

Please maske sure you have completed all the pre upgraded checker and allthe path are correct .

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=200428...

"Life would be so much easier if we only had the source code"

[crazygibbon]

Already done, everything seems ok but still no go. Any more ideas?

JD

[chriswahl]

Hi JD,

Just wanted to confirm with you that I upgraded my vCenter 5.0 to 5.0u1 over the weekend without a hitch. Not that it helps you that much, but does confirm that it is supposed to work properly. I logged in using the same AD service account that is connecting to the vCenter DB to run the update.

[robert_eckdale]

I am seeing the identical service failures / messages as you on two vCenter servers (linked), although my VMware vSphere Update Manager Extension is in the green.

[crazygibbon]

Yep, sorted the update manager - forgot to upgrade the plugin on the vSphere client   Still got the other errors though, had the guy from VMware Webex'd in yesterday and he seemed to think the problem lay with the FlexLM licence manager but can't help but think he's barking up the wrong tree; that's only there to provide licences for an old ESX 3.5 host we have running an old data warehouse system.

Sorely tempted to uninstall, restore the database and have another go but I'll lose a couple of days of performance stats...  Anyone got any other ideas?

JD

[crazygibbon]

Having investigated the stats.log file, I am getting errors connecting to the database following the upgrade:-

[2012-03-19 08:52:57,648 main  INFO  com.vmware.vim.common.lifecycle.InitializerExecutor] Initializing com.vmware.vim.stats.webui.startup.StatsReportInitializer...
[2012-03-19 08:52:57,695 pool-1-thread-1  INFO  com.vmware.vim.stats.webui.startup.StatsReportInitializer] Start STATs report initialization.
[2012-03-19 08:52:57,866 pool-1-thread-1  INFO  com.vmware.vim.common.vim.VcDataSourceInitializer] Start VC DataSource configuration.
[2012-03-19 08:52:59,364 Thread-2  ERROR com.vmware.vim.common.lifecycle.InitializerExecutor] Initialization error; attempt 2 will begin in 60 seconds...
java.util.concurrent.ExecutionException: java.lang.IllegalStateException: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Login failed for user '<DOMAIN\MACHINE>$'.)
    at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
    at java.util.concurrent.FutureTask.get(FutureTask.java:83)
    at com.vmware.vim.common.lifecycle.InitializerExecutor$MonitorCallback.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.IllegalStateException: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Login failed for user '<DOMAIN\MACHINE>$'.)
    at com.vmware.vim.common.lifecycle.InitializerExecutor$MonitorCallback$1.run(Unknown Source)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

I've removed the domain and machine names from this post for security, but for some reason it appears to be trying to use the local machine account against the remote database (which will obviously not work). Never did it until the upgrade!  I assume the license services uses Java to talk to the DB in the same way, hence the issue. Any ideas on how to proceed?

JD

[crazygibbon]

OK, seem to have fixed it - changed the account the VMware VirtualCenter Management Webservices service uses from local system to the same domain account as is used to connect to the SQL DB - lo and behold my perf stats, licence reporting etc now work.

Gonna reboot the server now and make sure the change stays consistent after a reboot.....

JD

[crazygibbon]

And it did

JD

[chriswahl]

For future reference, using the domain user account for the webservices service is a documented best practice.

http://kb.vmware.com/kb/2003790

For SQL Server DSNs configured with Windows authentication, use the same user account for the VMware VirtualCenter Management Webservices service and the DSN user.

[crazygibbon]

In that case, any idea why the update changed this?  Was working fine up until the moment update 1 was applied - in theory it would never have worked as it would not have had the permissions required to connect to the external database if I were using the local system account.

JD

[chriswahl]

Good question!

I'm fairly certain I set the account properly prior to upgrading in the home lab, but just noticed my service is also set to Local System.

I wrote up a quick blog article to get some eyes on this. Either Local System worked before, or the upgrade process changes out the log on account to Local System.

http://wahlnetwork.com/2012/03/20/vcenter-5-upgrade-1-webservices-service-error-with-local-system/

[robert_eckdale]

Configuring the VMware VirtualCenter Management Webservices to run as the same service account as used by  VMware VirtualCenter Server resolved the issue. I should also note that I am certain, that prior to the update both services were configured to use the same AD Domain user.

Thanks for the solution.

[crazygibbon]

Hey, that makes two bugs I've found in vSphere so far   Hope this post helps others who find themselves in the same position, I wonder if VMware are taking note of this and will fix it in the next update?

JD

[heykoo]

Thank you!

[syn3rgy]

I had the same issue, after performing the upgrade to 5.0 update 1 my service was running as local system and not my domain account. After changing the "VMware VirtualCenter Management Webservices" to use my domain account all the errors disappeared

Thanks again for the help guys

[sabzilla]

Also chiming in to say this fixed my problem as well. Did the 5.0U1 update last night, came into the office and saw red everywhere. Thanks!

[nsolimini]

What if your vcenter box is on a domain, but your SQL box is not apart of the domain. How would I correct this issue then?