How to remove a vROPs node manually?

[wpskern]

We are running vROPs 7.0.0.10098133

I have two nodes in a 8 node vROPs cluster (master/replica/4x data) that failed become a member of the vROPs cluster while adding one as a data node (7) and the other as a remote collector (8).

I tried KB VMware Knowledge Base because that helped me previously but now the curl command returns only that the node already has been remove but both failed nodes are still visible in the cluster management page.

I already did shutdown all nodes in the correct order and powered them on in the right order but without result.

I looked at the casa.db.script and the nodes are still in that file. (VMware Knowledge Base )

The current cluster status is FAILURE "Finish adding new node(s) and Waiting to finish cluster expansion

The status is greyed out

Any idea how to remove the two failed nodes if he curl command from VMware Knowledge Base does not work for the ip address, hostname or fqdn?

[RickVerstegen]

Did you already tried to take the cluster offline and turn it online again? Otherwise I suggest you to open a SR at GSS.

[GayathriS]

Hi

-->If you click on that respective node 7 and node 8 (which you want to remove) do you get option to remove that  ?

-->Those appliance on vcenter is powered off ?

-->You see those node entry in casa db script file .

Regards

Gayathri

[wpskern]

I can't take the cluster offline because it's still in a FAILED status and status of "finish adding new nodes".

I then modified the casa.db.script to get the cluster offline. That worked but the cluster is still in a status of "finish adding new nodes".

[sxnxr]

I would suggest letting GSS do this. I have had to do it a couple of times and i just logged a call

[wpskern · Accepted solution]

Answer from support which solved the problem:

- SSH to all 6 working nodes
- if SSH is disabled, on admin page you can enable SSH by clicking on the button
- when you are connected on all nodes, bring the nodes manually offline by running this command on all nodes:
$VMWARE_PYTHON_BIN $VCOPS_BASE/../vmware-vcopssuite/utilities/sliceConfiguration/bin/vcopsConfigureRoles.py --action bringSliceOffline --offlineReason "Maintenance"
- also run this command on all nodes to mark cluster like Offline
echo -e "\e[1;32mCasa.db.script edit initiated\e[0m";service vmware-casa stop;sleep 10;cp --backup=t /storage/db/casa/webapp/hsqldb/casa.db.script /storage/db/casa/webapp/hsqldb/casa.db.script.backup;sed -ri 's/"onlineState":"\w+"/"onlineState":"OFFLINE"/g;s/"initialization_state":"\w+"/"initialization_state":"NONE"/g;s/"online_state":"\w+"/"online_state":"OFFLINE"/g;s/"online_state_reason":"\w+"/"online_state_reason":""/g;s/"remove_node_state":"\w+"/"remove_node_state":"NONE"/g;s/"installation_state":"\w+"/"installation_state":"DONE"/g' /storage/db/casa/webapp/hsqldb/casa.db.script;sleep 2;service vmware-casa start;echo -e "\e[1;32mCluster Status has been modified\e[0m";grep "onlineState" /storage/db/casa/webapp/hsqldb/casa.db.script;
- when all 6 nodes are offline, and cluster is offline
- take the snapshots without memory and quescing of all the nodes in vROps cluster
- when you are done with the snapshots of all nodes
- connect to the master node and try to select 2 nodes, and delete them
 
If option for you till be grayed out after running step mentioned above, please continue with this steps:
- run this command to bring nodes online (you need to run this on all nodes)
$VMWARE_PYTHON_BIN $VCOPS_BASE/../vmware-vcopssuite/utilities/sliceConfiguration/bin/vcopsConfigureRoles.py --action bringSliceOnline
- run this command on all nodes to set status of the cluster online:
echo -e "\e[1;32mCasa.db.script edit initiated\e[0m";service vmware-casa stop;sleep 10;cp --backup=t /storage/db/casa/webapp/hsqldb/casa.db.script /storage/db/casa/webapp/hsqldb/casa.db.script.backup;sed -ri 's/"onlineState":"\w+"/"onlineState":"ONLINE"/g;s/"initialization_state":"\w+"/"initialization_state":"NONE"/g;s/"online_state":"\w+"/"online_state":"ONLINE"/g;s/"online_state_reason":"\w+"/"online_state_reason":""/g;s/"remove_node_state":"\w+"/"remove_node_state":"NONE"/g;s/"installation_state":"\w+"/"installation_state":"DONE"/g' /storage/db/casa/webapp/hsqldb/casa.db.script;sleep 2;service vmware-casa start;echo -e "\e[1;32mCluster Status has been modified\e[0m";grep "onlineState" /storage/db/casa/webapp/hsqldb/casa.db.script;

After running this commands on all the nodes, status of cluster should be Online, after which you should be able to select the nodes on admin page and delete them in the GUI (one by one).