Loosing connection to vCenter 5.0 Server appliance every hour.

[Webio]

Hello,

is there a way to debug why I'm loosing connection to one of my two vCenter 5.0 Server appliances? I'm connecting using Windows vSphere Client build 623373. I'm having problems only with one of my vCenter appliances and problems have started when I've removed vCenter Appliance from inventory on one server (ESXi 5.0U1 702118) and imported it to another ESXi (5.0U1 721882 - latest patch). During first startup on second server I've selected that I've moved VM. I'm using Essentials license so I had no other way to move vCenter appliance between servers other that removing and importing. I was trying to find anything in Events, Alarms etc. but I couldn't find anything (vCenter is not being restarted just disconnected and after few minutes I can connect again). Problems with vCenter are affecting my daily Veeam Backup & Replication backup jobs. Anyone have any idea where I should look for solution (maybe which log file inside /var/log?)?

Regards

P.S. I've also moved vCenter appliance server back to first ESXi server (ESXi 5.0U1 702118) but connectivity issues are still occuring.

[Webio]

Today I see that vCenter Server appliance web configuration panel (5480 port) is active (I had also opened SSH for about 12h) but I see there that vCenter service is not running. Where I can look for information what is causing service being stopped? Administration panel is not providing any logs.

[Webio]

Hello,

I've imported clean vCenter appliance hoping that it will not happen again but I was wrong. Once again I've found out that my vSphere client has lost connection to vCenter I've checked appliance web config panel (IP:5480) and I've noticed that vCenter service is not running. No one have any idea how to debug this problem and find out why vCenter service is being stopped?

Regards

[Webio]

When I'm beeing disconnected from vCenter I can see this type of messages in /var/log/messages:

Jul  2 18:50:01 localhost /usr/sbin/cron[1765]: (root) CMD ([ -x /usr/lib64/sa/sa1 ] && exec /usr/lib64/sa/sa1 -S ALL 1 1)
Jul  2 18:50:01 localhost /usr/sbin/cron[1767]: (root) CMD ( test -x /usr/sbin/vpxd_periodic && /usr/sbin/vpxd_periodic >/dev/null 2>&1)
Jul  2 19:00:01 localhost /usr/sbin/cron[3685]: (root) CMD ( test -x /usr/sbin/vpxd_periodic && /usr/sbin/vpxd_periodic >/dev/null 2>&1)
Jul  2 19:00:01 localhost /usr/sbin/cron[3688]: (root) CMD ([ -x /usr/lib64/sa/sa1 ] && exec /usr/lib64/sa/sa1 -S ALL 1 1)
Jul  2 19:00:03 localhost syslog-ng[2159]: Log statistics; dropped='pipe(/dev/xconsole)=3363', dropped='pipe(/dev/tty10)=0', processed='center(queued)=35716', processed='center(received)=32443', processed='destination(messages)=162', processed='destination(mailinfo)=8', processed='destination(mailwarn)=0', processed='destination(localmessages)=0', processed='destination(cron)=73', processed='destination(mailerr)=0', processed='destination(ldapmessages)=32273', processed='destination(netmgm)=0', processed='destination(warn)=1064', processed='destination(console)=1064', processed='destination(mail)=8', processed='destination(null)=0', processed='destination(xconsole)=1064', processed='destination(firewall)=0', processed='destination(acpid)=0', processed='source(src)=32443'
Jul  2 19:02:31 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 1778 seconds 134
Jul  2 19:02:31 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
Jul  2 19:02:35 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 3 seconds (quick failure 1) 1
Jul  2 19:02:35 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
Jul  2 19:02:38 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 3 seconds (quick failure 2) 1
Jul  2 19:02:38 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
Jul  2 19:03:41 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
Jul  2 19:03:41 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
Jul  2 19:04:14 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 33 seconds 1
Jul  2 19:04:14 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
Jul  2 19:05:16 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
Jul  2 19:05:16 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'

This time vCenter service is still running but vSphere client was disconnected. This problems are interrupting my Veeam backup jobs. I've didn't changed anythin in vCenter VM like decreasing RAM etc. vCenter Appliance settings are just like they where imported from OFV template.

[Webio]

I had some entries in /var/log/vmware/vpx/vpxd.log related to DB problems (I'm using embedded):

2012-07-03T00:33:57.909+02:00 [7FFFDF37C710 error 'Default'  opID=SWI-64f33f78] [VdbStatement] Diagnostic data from driver is  57011:1:-964:[IBM][CLI Driver][DB2/LINUXX8664] SQL0964C  The transaction  log for the database is full.  SQLSTATE=57011
-->
2012-07-03T00:33:57.909+02:00  [7FFFDF37C710 error 'Default' opID=SWI-64f33f78] [VdbStatement]  Diagnostic data from driver is 01504:2:513:[IBM][CLI  Driver][DB2/LINUXX8664] SQL0513W  The SQL statement will modify an  entire table or view.  SQLSTATE=01504
-->
2012-07-03T00:33:57.909+02:00 [7FFFDF37C710 error 'Default' opID=SWI-64f33f78] [VdbStatement] Bind parameters:
2012-07-03T00:33:57.909+02:00  [7FFFDF37C710 error 'Default' opID=SWI-64f33f78]  VpxdStatsCollector::ClearTempTable failed: "ODBC error: (57011) -  [IBM][CLI Driver][DB2/LINUXX8664] SQL0964C  The transaction log for the  database is full.  SQLSTATE=57011
--> " is returned when executing SQL statement "delete from VPX_TEMPTABLE0"
2012-07-03T00:33:57.909+02:00  [7FFFDF37C710 error 'Default' opID=SWI-64f33f78]  VpxdStatsCollector::ProcessTempTable Failed to process temp table. The  table can't be returned to insert further perf data ("ODBC error:  (57011) - [IBM][CLI Driver][DB2/LINUXX8664] SQL0964C  The transaction  log for the database is full.  SQLSTATE=57011
--> " is returned when executing SQL statement "call process_temptable0_proc")
2012-07-03T00:33:57.909+02:00  [7FFFDF37C710 error 'Default' opID=SWI-64f33f78] Alert:false@  /build/mts/release/bora-455964/bora/vpx/vpxd/stats/vpxdStatsCollector.cpp:193

which lead me to this blog entry:

http://www.virtualserverguy.com/blog/2012/2/23/vcenter-server-appliance-database.html

and after updating DB settings my problem looks solved.

Regards

[beckham007fifa]

Could you please sum up your issue in few sentences, what I have understood from your posts is the services for vcenter is getting stopped and the hosts connected to the vcenter is getting disconnected every hour( in the first post).

which of these problem is more predominent?

[Webio]

More times I've lost connection to vCenter than vCenter service stopped. I've noticed also bigger /dev/sdb1              20G  1.4G   18G   8% /storage/core partition usage (now I've cleaned it but during my problems partition usage was at about 75% level). Most of my vCenter problems was during Veeam VM backup jobs. When I've updated embedded DB settings according to mentioned blog entry I didn't have any problems during Veeam backup process and during two days I've only lost connection once. /dev/sdb1 /storage/core partition is also not being filled again.