We had a storage outage on our VIO Infrastructure.
If I restart the deployment I get the following error in the VC:
Task execution failed: Task failed on the following nodes: ['10.205.208.14', '10.205.208.13']. Refer logs for more details.
The IPs are from the DB server, so I login to these an tried to restart the mysql service:
sudo /etc/init.d/mysql restart
But this fails.
In the /var/log/syslog I found the following errors:
Mar 14 12:36:22 database01 ntpd[1572]: Listening on routing socket on fd #22 for interface updates
Mar 14 12:36:35 database01 mysqld: 170314 12:36:35 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out)
Mar 14 12:36:35 database01 mysqld: #011 at gcomm/src/pc.cpp:connect():162
Mar 14 12:36:35 database01 mysqld: 170314 12:36:35 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():208: Failed to open backend connection: -110 (Connection timed out)
Mar 14 12:36:35 database01 mysqld: 170314 12:36:35 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1379: Failed to open channel 'my_wsrep_cluster' at 'gcomm://10.205.208.12,10.205.208.13,10.205.208.14': -110 (Connection timed out)
Mar 14 12:36:35 database01 mysqld: 170314 12:36:35 [ERROR] WSREP: gcs connect failed: Connection timed out
Mar 14 12:36:35 database01 mysqld: 170314 12:36:35 [ERROR] WSREP: wsrep::connect(gcomm://10.205.208.12,10.205.208.13,10.205.208.14) failed: 7
Mar 14 12:36:35 database01 mysqld: 170314 12:36:35 [ERROR] Aborting
Mar 14 12:36:35 database01 mysqld:
Mar 14 12:36:35 database01 mysqld: 170314 12:36:35 [Note] WSREP: Service disconnected.
Mar 14 12:36:36 database01 mysqld: 170314 12:36:36 [Note] WSREP: Some threads may fail to exit.
Mar 14 12:36:36 database01 mysqld: 170314 12:36:36 [Note] /usr/sbin/mysqld: Shutdown complete
Mar 14 12:36:36 database01 mysqld:
Mar 14 12:36:36 database01 mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended
Mar 14 12:37:01 database01 CRON[2010]: (root) CMD (/usr/bin/viomon > /dev/null 2>&1)
Someone a hint how I can fix this?
