VMware Cloud Community
OsburnM
Hot Shot
Hot Shot
Jump to solution

VIO6 Recover from Lights Out - Deployment Status Stuck in 'OUTAGE'

Subject sums it up--  Have a 3-node w/HA deployment in VIO6.  Simulated a lights out scenario.  The VIO manager vAPP and the 3 VM controllers all powered back on; however, the deployment is stuck on 'OUTAGE' status and I cant seem to get it back to Normal/Ready/Running.  Any ideas?

root@manager [ ~ ]# viocli get deployment

LOG ANALYTICS IP                LOG ANALYTICS PORT   READY

loginsight.x.x   9000                  8/8

PUBLIC VIP     PRIVATE VIP    HIGH AVAILABILITY

x.x.x.x   1x.x.x.x   Enabled

NODE NAME                            ROLE         VALIDATION   STATUS    IP

controller-6xff8jsggl                Controller   Success      Running   x.x.x.x

controller-fd49gb96dl                Controller   Success      Running   x.x.x.x

controller-x8kc6fqgwn                Controller   Success      Running   x.x.x.x

manager.x.x   Manager      Success      Running   x.x.x.x

SERVICE        CONTROLLER                       READY   FAILURES

barbican       barbican-api                      0/2       -

               barbican-ks-listener              0/2       -

               barbican-worker                   0/2       -

cinder         cinder-api                        0/2       -

               cinder-scheduler                  0/2       -

               cinder-volume                     2/2       -

glance         glance-api                        0/2       -

               glance-vmw-replicator             0/1       -

heat           heat-api                          0/2       -

               heat-cfn                          0/2       -

               heat-engine                       0/2       -

horizon        horizon                           0/2       -

ingress        ingress                           2/2       -

               ingress-error-pages               1/1       -

keystone       keystone-api                      0/2       -

mariadb        mariadb-server                    3/3       -

               mariadb-ingress                   0/2       -

               mariadb-ingress-error-pages       2/2       -

               mariadb1-etcd                     3/3       -

memcached      memcached1-memcached              1/1       -

               memcached1-memcached-secondary    1/1       -

neutron        neutron-dhcp-agent-default        0/3       -

               neutron-metadata-agent-default    0/3       -

               neutron-server                    0/2       -

nova           nova-api-metadata                 0/2       -

               nova-api-osapi                    0/2       -

               nova-conductor                    0/2       -

               nova-consoleauth                  0/1       -

               nova-mksproxy                     0/1       -

               nova-placement-api                0/2       -

               nova-scheduler                    0/2       -

nova-compute   compute-f327078c-c8-compute       1/1       -

openvswitch    openvswitch-db                    3/3       -

               openvswitch-vswitchd              3/3       -

rabbitmq       rabbitmq1-rabbitmq                3/3       -

vioshim        vioadmin1-vioshim                 0/1       -

vioutils       node-config-manager               3/3       -

OpenStack Deployment State: OUTAGE

Reply
0 Kudos
1 Solution

Accepted Solutions
OsburnM
Hot Shot
Hot Shot
Jump to solution

Closing this thread-- all these issues with VIO 6 & DHCP are related to a busted deployment with the services passwords all expired in the base images.  VMware needs to fix before this product is even usable out of the box.

See the following thread:VIO 6 DHCP Agents Bug? with solution

View solution in original post

Reply
0 Kudos
3 Replies
xiangfeiz
VMware Employee
VMware Employee
Jump to solution

You can check from Pod level if any Pod failed.

kubectl get pods -A

Reply
0 Kudos
OsburnM
Hot Shot
Hot Shot
Jump to solution

openstack        neutron-dhcp-agent-default-8pjx8                              0/1 Init:0/2      0      8m44s
openstack        neutron-dhcp-agent-default-mmv5k                              0/1 Init:0/2      1      19h
openstack        neutron-dhcp-agent-default-wrswb                              0/1 Init:0/2      1      19h
openstack        neutron-metadata-agent-default-bhc94                          0/1 Init:0/2      2      20h
openstack        neutron-metadata-agent-default-ccdfh                          0/1 Init:0/2      1      20h
openstack        neutron-metadata-agent-default-sr8d4                          0/1 Init:0/2      1      20h

These two services seem stuck and I can't get them to stop or start.

Reply
0 Kudos
OsburnM
Hot Shot
Hot Shot
Jump to solution

Closing this thread-- all these issues with VIO 6 & DHCP are related to a busted deployment with the services passwords all expired in the base images.  VMware needs to fix before this product is even usable out of the box.

See the following thread:VIO 6 DHCP Agents Bug? with solution

Reply
0 Kudos