Hi,
i wanted to update my whole environment to vSphere 4 Update 1. This included everything from ESX hosts to VCenter and Update Manager....
With this move i also wanted to get rid of the SQL Express setup i'm using for two years now. With 10 hosts, three Clusters and over 200 VMs it does not perform very well at all
So i decided to use a Oracle DB hosted by my favored DBA - with the thoughts of less problems and better performance.
I installed everything as usual - setup a VM with Win2k3, all Service Packs and security updates. Downloaded Oracle 10g Client, installed ODBC, configure tnsnames.ora (got all settings from dba). Created an ODBC connection - connection succesfull.
Installation of vCenter itself was succesfull. Started up vSphere Client, created a cluster, moved 4 of the 10 hosts to the new VC. Everything works fine. After a couple of hours my problems began... vCenter service shutted itself down. I figured some errors when viewing the Event log and google'd a lot - first impression: It does not look like as if Oracle would be a favored DB for VC.
Then i found some KB articles and instructed my DBA to change some settings (like increasing open cursors, expire_time=1 (we have firewalls between vc and oracle db) and a KB article that stated to set KeepAlive settings for tcpip on the W2k3 system.
After i did all of this i still get the same errors like:
Unable to recover from HY000:3135
SQL execution failed: INSERT INTO VPX_EVENT (EVENT_ID, CHAIN_ID, EVENT_TYPE, EXTENDED_CLASS, CREATE_TIME, USERNAME, CATEGORY, VM_ID, VM_NAME, HOST_ID, HOST_NAME, COMPUTERESOURCE_ID, COMPUTERESOURCE_TYPE, COMPUTERESOURCE_NAME, DATACENTER_ID, DATACENTER_NAME, DATASTORE_ID, DATASTORE_NAME, NETWORK_ID, NETWORK_NAME, NETWORK_TYPE, DVS_ID, DVS_NAME, CHANGE_TAG_ID) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
Execution elapsed time: 18922 ms
Diagnostic data from driver is HY000:1:3135:[Oracle][ODBC][Ora]ORA-03135: Verbindung verloren
datatype: 1, size: 4,arraySize: 1
................
Error inserting events: "ODBC error: () - " is returned when executing SQL statement "INSERT INTO VPX_EVENT (EVENT_ID, CHAIN_ID, EVENT_TYPE, EXTENDED_CLASS, CREATE_TIME, USERNAME, CATEGORY, VM_ID, VM_NAME, HOST_ID, HOST_NAME, COMPUTERESOURCE_ID, COMPUTERESOURCE_TYPE, COMPUTERESOURCE_NAME, DATACENTER_ID, DATACENTER_NAME, DATASTORE_ID, DATASTORE_NAME, NETWORK_ID, NETWORK_NAME, NETWORK_TYPE, DVS_ID, DVS_NAME, CHANGE_TAG_ID) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
Transaction took 18922 ms with 0 statements
An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC error: () - " is returned when executing SQL statement "INSERT INTO VPX_EVENT (EVENT_ID, CHAIN_ID, EVENT_TYPE, EXTENDED_CLASS, CREATE_TIME, USERNAME, CATEGORY, VM_ID, VM_NAME, HOST_ID, HOST_NAME, COMPUTERESOURCE_ID, COMPUTERESOURCE_TYPE, COMPUTERESOURCE_NAME, DATACENTER_ID, DATACENTER_NAME, DATASTORE_ID, DATASTORE_NAME, NETWORK_ID, NETWORK_NAME, NETWORK_TYPE, DVS_ID, DVS_NAME, CHANGE_TAG_ID) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
Forcing shutdown of VMware VirtualCenter now
I'm really wondering why the execution times out, maybe it means there are still dead connections and thats why the execution time grows... but when i go back in logs it seems as if almost every database transaction "fails" at the first try and there is a warning in vpxd.log
I hope someone has a idea how i could fix that problem.
Thanks for your help in advance!
Greetings Manuel
Just wanted everyone else know that the problem has been solved.
I do not know the specifics but apparently it was the ODBC Driver that caused some lags/connection interruption. While this happened the VC Service crashed. My DBA gave me the Oracle 10g 10.0.4.0 Client package and it seems to me that the included ODBC Driver does not work very well.
I was able to fix this problem with the Oracle Instant Client ODBC Driver, now everything runs really smooth and fast for months.
Just a conclusion: The original ODBC Driver problem was only on a Win2k3 box, VMware Update Manager worked fine with the same ODBC Driver on a WindowsXP box.
Greetz Manuel