VMware VirtualCenter Server service starting takes a long time (>1hour)

[MikelAanstoot]

We've updated our VMware vSphere cluster to 4.1 somewhere in august this year. Until last week it was running whithout a problem. Last week we restarted our SQL 2005 database servers (3 servers in a cluster) were the VMware VIM database also stands.

Since the last restart of the database and the vCenter server the VMware VirtualCenter Server service takes a long time to start. Whave investigated the issue and see the following:

• xvpd log: DBHelper::Init() takes a very long time?

• Callin: Xpxd::DbHelper::Init()

• SQL execution took too long: SELECT MO_REF, TYPE_NAME, ARRAY_NUM FROM VPX_TYPE_MAP

• execution elapsed time: 5273 ms

• Bind parameters:

• VpxdDbHelper:Init took 1254117 ms

• odbcad32.cpl: the previously used SQL account doesn't work anymore, Windows Integrated Authentication does?

• An odbc connection to another database also doesn't work

• connections from other machines or servers give no problem

• connections to a stand-alone SQL 2005 server also gives no problems

Our conclusion: it's something on the vCenter server what has changed which causes ODBC to fail. On the specific SQL 2005 database server are more databases, all with no issue so far.

Extra information: it seems that the SQL Client connection fails to the SQL 2005 Cluster where the VI databases is located. The database server and database can be connected from other servers/clients whithout a problem. I've moved the database to a standalone SQL 2005 servers, and changed the odbc settings (both 32 and 64 bit). Now vCenter Server is running smoothly, but why can't i connect to the SQL 2005 Cluster form the vCenter Server?

[vickey0rana]

have you upgraded you database as well? ousing the old one?

[MikelAanstoot]

Yes, also the database was updated in august this year. Since last wednesday nothing really has changed, only the database servers are restarted (and afterwards, off course, alse the vCenter Server was restarted).

I've added extra information to the first message.

[logiboy123]

Database connections are probably persistant. That means even though information has changed (maybe passwords, or the service account was locked out etc) it didn't effect the vCenter database connection because the server was still up.

The fact that moving your database to a new server fixed the issue seems to back up this line of reasoning. I don't have an answer for you, but I do have some hard won guidlines.

1) Use a different service accounts for each vSphere function; vCenter, Update Manager, VMware View etc. This means that if someone makes a change to the Update Manager account and UM fails then it won't affect the rest of your vSphere environment.

2) When installing a vSphere service, log on locally to the server using the service account that will be used for that service. Create the ODBC connection for that connection whilst logged in as the user for that service. Make sure to create the oppropriate 32bit or 64bit DSN. Update Manager for example is still a 32bit application and connection type even when installed on a 64bit platform.

3) Do not make all of these service accounts Domain Admins. Talk about sloppy implementation techniques. I make my service accounts Domain Admins for the installation only as well as making them dbowner on the MSDB database in SQL. After the install/update I remove this access and then leave them as remote desktop users of the server. Only svc_vCenter remains an admin on the local vCenter box and I'm not sure it even needs this.

4) Take screenshots of every single step you take when doing installs and upgrades. Then you can go back over what you did at a later stage to see if there is a problem. Sloppy and lazy consultants don't want to do this because it adds accountability to their process.

Regards,

logiboy123