VMware Cloud Community
ZeMiracle
Enthusiast
Enthusiast

VIO Disaster Recovery, not possible ?

Hi,

I try to figure how we can restore VIO from scratch...

It seem it's not possible...

We have a VIO deployment A

  1. With the Viocli, we backup Management server and openstack Database.
  2. We Backup Nsx Manager.
  3. We delete the VIO deployment
    -it delete all the edges and the edge resource pool

We install a new VIO deployment

  1. We delete all the backup edge and meta data proxy of this new deployment
  2. We restore Nsx Manager
  3. We modify the Resource Pool ID of all the edges to match the new resource pool id
  4. With the viocli we restore the managment server and the database.
  5. We stop manually the rabbit Mq on all the database nodes
  6. We launch a viocli deployment configure

At this point we are able to stop vio deployment, but the start command failed to ssh the nodes.

All node appear to have 0.0.0.0 address.

Does VIO have a disaster recovery procedure ?

It seem it was not planned at all.

Reply
0 Kudos
9 Replies
admin
Immortal
Immortal

Hi ZeMiracle,

can you please let me know:

1) did you follow Back Up NSX Manager Data to backup NSX?

2) did you follow Restore an NSX Manager Backup to restore NSX?

3) considering your step "6", the exact procedure to restore VIO, you used? Did you follow Restore VMware Integrated OpenStack from a Backup ?

Cheers,

Domenico

Reply
0 Kudos
ZeMiracle
Enthusiast
Enthusiast

Hello Domenico,

Thank for trying to help Smiley Wink

1) did you follow Back Up NSX Manager Data to backup NSX? ->Yes

2) did you follow Restore an NSX Manager Backup to restore NSX? - Yes

3) considering your step "6", the exact procedure to restore VIO, you used? Did you follow Restore VMware Integrated OpenStack from a Backup ?

We don't want to restore VIO on the same deployment, we want to restore VIO after a complete rebuild of the system (on a brand new deployment).

The fact is, VIO 3.0 is not intend to be restored on a new deployment (after a crash recovery for exemple), and it's a big issue !

Exemple 1 : Because VIO store Edge into a Resource Pool... edge keep the resource pool ID into their configuration.

After restoring NSX on a new VIO deployment, because the old resource pool id don't exist anymore, you have to change the resource pool id into each Edge in order to see them redeploy.

We have pass every obstacle we have found to be able to restore a functionnal ... but at the end, we don't succeed to make it manageable from the Vcenter... seem we still have a problem into the OMS.

Cédric.

Reply
0 Kudos
admin
Immortal
Immortal

Hi Cedric,

I managed to restore VIO deployment from scratch multiple times (Compact mode) and I suppose it should be the same for HA mode if a previous NSX backup has been correctly performed.

Please, try the following, after restoring vCenter Server  and NSX from a previous backup:

1) Deploy a new OMS.

2) Use viocli restore mgmt_server -d <Deployment_name> <DIR_NAME> <NFS_VOLUME> to restore OpenStack Management Server

3) Use viocli recover -d <Deployment_name> -n <Nodes_to_recover> -dn <Folder_Name> -nfs <NFS_VOLUME> to recover one node or a group of nodes.    

(If you want to restore all the nodes, write after the n variable all the node to be restored)

In order to know all the restore-able nodes, you can run 

root@localhost:~# viocli recover -d VIO -n xxx

Invalid node name: xxx

Available node names: VIO-Compute-0, VIO-ControlPlane-0, ....

Please, let me know if you can try the above and if it works.

Cheers,

Domenico

Reply
0 Kudos
ZeMiracle
Enthusiast
Enthusiast

Hello,

Thanks a lot for this answer.

I'm surprised you have been able to get a functionnal VIO environment with no additional steps.

I'm certain that you can't recover your edges after a nsx restore on a new VIO 3.0 deployment if you don't change the resource pool id the edge are deployed in.

After doing a viocli restore, i don't proceed a "viocli recover" (no mention of this command in the restore procedure).

After restoring the Oms server and Openstack DB, i have multiples problem (from ssh key, to rabbit mq credential etc...).

Maybee the recover command is the step that is missing...

What is the action of this command ?

Ced

Reply
0 Kudos
admin
Immortal
Immortal

Hi Ced,

as told you in my last message, I managed to restore VIO deployment (from scratch) multiple time... but the Compact Version only, not the HA version. Smiley Happy

I suppose it is the same process if a correct NSX backup has been correctly performed.

The exact procedure to re-deploy a VIO deployment from a previous backup is the one I mentioned in my last post, considering that there are two sets of information required to re-buildthe deleted deployment:

1) credentials, metadata, VM info, network info, etc

2) OpenStack data in the OpenStack database.

so:

- running viocli backup mgmt_server -d <Deployment_name> <NFS_VOLUME> command, saves the data set 1

- running viocli backup backup openstack_db -d <Deployment_name> <NFS_VOLUME> command, saves the data set 2

and:

- running restore mgmt_server -d <Deployment_name> <DIR_NAME> <NFS_VOLUME> command, restores data set 1

- running viocli recover -d <Deployment_name> -n <Nodes_to_recover> -dn <Folder_Name> -nfs <NFS_VOLUME> command, recover all the nodes with the database.

Please, if you have time, try the steps and let me know if it works.

Cheers,

Domenico

Reply
0 Kudos
ZeMiracle
Enthusiast
Enthusiast

Hi Domenico,

As i say, i play the restore process multiple time in order to try to make it work (in HA).

The nsx resource pool id issue is valable for HA or compact mode (same problem).

I play theses command multiple times... and it's not enought to get your env functionnal (VIO 3.0).

- running restore mgmt_server -d <Deployment_name> <DIR_NAME> <NFS_VOLUME> command, restores data set 1

- running viocli recover -d <Deployment_name> -n <Nodes_to_recover> -dn <Folder_Name> -nfs <NFS_VOLUME> command, recover all the nodes with the database.

You can log into your VIO, but can't manage your VM since there is a issue with rabbit-mq credential.

Ced

Reply
0 Kudos
admin
Immortal
Immortal

Hi Cedric,

VIO deployed "Compact mode" does not use any NSX resource... only VDS... so there is no nsx resource pool.

Cheers,

Domenico

Reply
0 Kudos
ZeMiracle
Enthusiast
Enthusiast

In VIO 3.0, i confirm i can deploy in compact mode with NSX Smiley Happy

I do it several times... i never do a VDS deployment.

Reply
0 Kudos
admin
Immortal
Immortal

Hi Cedric,

probably there is a misunderstanding. Smiley Happy

Compact deployment mode means a VIO deployment  requiring few resources and memory and it uses vDS (virtual distributed switch), as per:

VMware Integrated OpenStack Deployment in Compact Mode and Architectural Overview of VDS Deployments

differently from HA mode that needs more resources and a NSX networking back-end, as per:

Architectural Overview of NSX Deployments

So, as per my previous messages I can confirm that backup and recovery processes works fine with Compact Mode and vDS.

Cheers,

Domenico

Reply
0 Kudos