rxjoseph
Enthusiast
Enthusiast

One host out of the 32 host in my vSAN is not utilizing disk

Jump to solution

hello Experts

Firstly i need an expert to answer my query as I have some good experience in vSAN but this issue is biting  me

I have a 32 Host ESXi 6 vSphere 6 vSAN cluster

Currently when i do a vsan.disks_stats from the RVC console I get the nice report with the disk size and used space

I can see all of the disk is used well but there is one exception, once of the servers is not using any vSAN disk

The Host has 98 VMs but see the disks stats

its only using 0.12 and rest of the Host are all using 70 %

Can someone guide me through the correct steps to troubleshoot please

PS C:\Users\RPARPJ016> get-vmhost sapmesxnlams030.crb.apmoller.net |get-vm |measure

Count    : 98

pastedImage_4.png

vAN components status its 0 on host 32 while all others have components

pastedImage_0.png

1 Solution

Accepted Solutions
TheBobkin
VMware Employee
VMware Employee

Hello rxjoseph​,

It is indeed in vSAN Decom State as I initially advised:

{

   "uuid": "5a744f74-02a6-aa68-d932-0cc47aa3c50c",

   "owner": "5a744f74-02a6-aa68-d932-0cc47aa3c50c",

   "health": "Healthy",

   "revision": "3",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "7c88fc7eea9891e32bc7b35532085ee5",

   "valueLen": "80",

   "content": {"decomState": 6, "decomJobType": 1, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 2},

   "errorStr": "(null)"

}

This is simply remediated by either running:

#localcli vsan maintenancemode cancel

(Specifically localcli not esxcli here as if it tries to talk through hostd/vpxa it will just refuse as it is not in MM at the ESXi level)

or

Put the host in MM via the GUI and take it out again.

Bob

View solution in original post

7 Replies
TheBobkin
VMware Employee
VMware Employee

Hello rxjoseph

Check if the host is in vSAN Decom-state via SSH:

#cmmds-tool find -t NODE_DECOM_STATE -f json

It will have decomState:6 if it is.

To confirm the UUID of the node you can use this run on that host:

#cmmds-tool whoami

Note that vSAN doesn't automatically balance data across disks unless any disk passes 80% utilised (default) so if you add a new host to a cluster or re-join cluster after a long maintenance window you will have to use proactive rebalance to move some data onto it.

This can be performed via the GUI or via RVC:

> vsan.proactive_rebalance <pathToCluster>

Bob

0 Kudos
rxjoseph
Enthusiast
Enthusiast

Hi Bob

This is the result from the host having the issue

[root@sapmesxnlams030:~] cmmds-tool find -t NODE_DECOM_STATE -f json

{

"entries":

[

{

   "uuid": "5a0c1a86-6cd4-d308-6fe9-0cc47aa35d58",

   "owner": "5a0c1a86-6cd4-d308-6fe9-0cc47aa35d58",

   "health": "Healthy",

   "revision": "3",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

[root@sapmesxnlams030:~] cmmds-tool whoami

5a744f74-02a6-aa68-d932-0cc47aa3c50c

Can you advise on this output please

0 Kudos
TheBobkin
VMware Employee
VMware Employee

Hello rxjoseph​,

"This is the result from the host having the issue

[root@sapmesxnlams030:~] cmmds-tool find -t NODE_DECOM_STATE -f json

{

"entries":

[

{

   "uuid": "5a0c1a86-6cd4-d308-6fe9-0cc47aa35d58",

   "owner": "5a0c1a86-6cd4-d308-6fe9-0cc47aa35d58",

   "health": "Healthy",

   "revision": "3",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}"

Did this command on this node only return a single NODE_DECOM_STATE entry? Running this on any host should return the entries (of this type) for all nodes in the cluster.

We can also see clearly that this is not the NODE_DECOM_STATE entry for this node:

   "uuid": "5a0c1a86-6cd4-d308-6fe9-0cc47aa35d58",

   "owner": "5a0c1a86-6cd4-d308-6fe9-0cc47aa35d58"

Where whoami (and also likely HOSTNAME entry) shows a different UUID (5a744f74-02a6-aa68-d932-0cc47aa3c50c).

There are a good few potential causes of your issue here but determining the node we are looking at is step 1.

Please confirm this is not a stretched-cluster, check there are no duplicate/stale entries for other types in CMMDS from nodes being removed/re-added improperly (as the fact you are hitting this on '32nd' node has multiple alarm bells triggered here that indicate this is not coincidence).

Bob

0 Kudos
rxjoseph
Enthusiast
Enthusiast

[root@sapmesxnlams030:~] cmmds-tool whoami

5a744f74-02a6-aa68-d932-0cc47aa3c50c

[root@sapmesxnlams030:~] cmmds-tool find -t NODE_DECOM_STATE -f json

{

"entries":

[

{

   "uuid": "5a0c1a86-6cd4-d308-6fe9-0cc47aa35d58",

   "owner": "5a0c1a86-6cd4-d308-6fe9-0cc47aa35d58",

   "health": "Healthy",

   "revision": "3",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "5bbca3f5-de1a-8d0a-37fe-0cc47aa39058",

   "owner": "5bbca3f5-de1a-8d0a-37fe-0cc47aa39058",

   "health": "Healthy",

   "revision": "1",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "59308d44-0f85-3610-af00-0cc47a39bdf4",

   "owner": "59308d44-0f85-3610-af00-0cc47a39bdf4",

   "health": "Healthy",

   "revision": "3",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "5a16c225-dd64-9a10-6ddd-0cc47aa43420",

   "owner": "5a16c225-dd64-9a10-6ddd-0cc47aa43420",

   "health": "Healthy",

   "revision": "3",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "5a5882a0-4efa-5b14-c76f-0cc47aa4e6fe",

   "owner": "5a5882a0-4efa-5b14-c76f-0cc47aa4e6fe",

   "health": "Healthy",

   "revision": "6",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "59265031-8888-a314-1578-0cc47aa3ff14",

   "owner": "59265031-8888-a314-1578-0cc47aa3ff14",

   "health": "Healthy",

   "revision": "3",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "5923a30d-88aa-d318-95e8-0cc47aa4341c",

   "owner": "5923a30d-88aa-d318-95e8-0cc47aa4341c",

   "health": "Healthy",

   "revision": "3",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "58e354cb-4330-1020-831e-0cc47aa38d48",

   "owner": "58e354cb-4330-1020-831e-0cc47aa38d48",

   "health": "Healthy",

   "revision": "3",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "596cadb4-3b02-8520-294a-0cc47ad375d8",

   "owner": "596cadb4-3b02-8520-294a-0cc47ad375d8",

   "health": "Healthy",

   "revision": "3",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "58e2e57f-d791-7a26-fd87-0cc47aa38f34",

   "owner": "58e2e57f-d791-7a26-fd87-0cc47aa38f34",

   "health": "Unhealthy",

   "revision": "14",

   "type": "NODE_DECOM_STATE",

   "flag": "0",

   "minHostVersion": "0",

   "md5sum": "851e1bbf87c931166b19910ad0e12a01",

   "valueLen": "80",

   "content": {"decomState": 6, "decomJobType": 1, "decomJobUuid": "68adbf24-7c18-e9c4-c395-6e91737ddb88", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 2},

   "errorStr": "(null)"

}

,{

   "uuid": "5933ecd5-53a6-8330-331e-0cc47aa39058",

   "owner": "5933ecd5-53a6-8330-331e-0cc47aa39058",

   "health": "Unhealthy",

   "revision": "7",

   "type": "NODE_DECOM_STATE",

   "flag": "0",

   "minHostVersion": "0",

   "md5sum": "a7d42aefc212f4555e9679b5feb75446",

   "valueLen": "80",

   "content": {"decomState": 6, "decomJobType": 1, "decomJobUuid": "93849f8b-c59f-9889-c7ce-6835609ab21f", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 2},

   "errorStr": "(null)"

}

,{

   "uuid": "591bc984-ef61-ec3c-df82-0cc47ad31d24",

   "owner": "591bc984-ef61-ec3c-df82-0cc47ad31d24",

   "health": "Healthy",

   "revision": "3",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "596cfb1f-72e7-9842-0ddc-0cc47ad3552e",

   "owner": "596cfb1f-72e7-9842-0ddc-0cc47ad3552e",

   "health": "Healthy",

   "revision": "3",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "596ca832-8df1-6c44-0f1b-0cc47ad36e78",

   "owner": "596ca832-8df1-6c44-0f1b-0cc47ad36e78",

   "health": "Healthy",

   "revision": "3",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "59347d01-05e8-2358-d3a1-0cc47aa39114",

   "owner": "59347d01-05e8-2358-d3a1-0cc47aa39114",

   "health": "Healthy",

   "revision": "2",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "596ca740-c5b4-e458-b98f-0cc47ad36046",

   "owner": "596ca740-c5b4-e458-b98f-0cc47ad36046",

   "health": "Unhealthy",

   "revision": "6",

   "type": "NODE_DECOM_STATE",

   "flag": "0",

   "minHostVersion": "0",

   "md5sum": "8f66969748bf2a364a89157140196f88",

   "valueLen": "80",

   "content": {"decomState": 6, "decomJobType": 0, "decomJobUuid": "acc847b2-2b02-4dcf-88cd-19809360c8f0", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 2},

   "errorStr": "(null)"

}

,{

   "uuid": "59100dc4-4fb2-1768-cded-0cc47aa3d068",

   "owner": "59100dc4-4fb2-1768-cded-0cc47aa3d068",

   "health": "Healthy",

   "revision": "2",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "596cabf1-94ec-9868-9742-0cc47ad3553a",

   "owner": "596cabf1-94ec-9868-9742-0cc47ad3553a",

   "health": "Healthy",

   "revision": "3",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "5a744f74-02a6-aa68-d932-0cc47aa3c50c",

   "owner": "5a744f74-02a6-aa68-d932-0cc47aa3c50c",

   "health": "Healthy",

   "revision": "3",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "7c88fc7eea9891e32bc7b35532085ee5",

   "valueLen": "80",

   "content": {"decomState": 6, "decomJobType": 1, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 2},

   "errorStr": "(null)"

}

,{

   "uuid": "591be406-6eb1-466c-6d14-0cc47aa3d074",

   "owner": "591be406-6eb1-466c-6d14-0cc47aa3d074",

   "health": "Healthy",

   "revision": "2",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "5a7456d8-b3a8-807c-d9cd-0cc47a665adc",

   "owner": "5a7456d8-b3a8-807c-d9cd-0cc47a665adc",

   "health": "Healthy",

   "revision": "2",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "591d231a-35d5-4494-8320-0cc47aa435c0",

   "owner": "591d231a-35d5-4494-8320-0cc47aa435c0",

   "health": "Healthy",

   "revision": "2",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "5bda19f2-75c0-5d94-933b-002590fb44e4",

   "owner": "5bda19f2-75c0-5d94-933b-002590fb44e4",

   "health": "Healthy",

   "revision": "3",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "596ca77d-d466-eb94-31ff-0cc47ad35f76",

   "owner": "596ca77d-d466-eb94-31ff-0cc47ad35f76",

   "health": "Healthy",

   "revision": "3",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "59153d37-ffb9-22a0-dbb5-0cc47aa3ce98",

   "owner": "59153d37-ffb9-22a0-dbb5-0cc47aa3ce98",

   "health": "Healthy",

   "revision": "3",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "58e34a33-0b1a-fba0-6dcc-0cc47ad31a90",

   "owner": "58e34a33-0b1a-fba0-6dcc-0cc47ad31a90",

   "health": "Healthy",

   "revision": "5",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "5933a42f-f3ad-b6a6-b1a7-0cc47aa38f34",

   "owner": "5933a42f-f3ad-b6a6-b1a7-0cc47aa38f34",

   "health": "Healthy",

   "revision": "3",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "5922669b-ca13-f2a6-37ce-0cc47aa4e646",

   "owner": "5922669b-ca13-f2a6-37ce-0cc47aa4e646",

   "health": "Healthy",

   "revision": "3",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "5a16b201-9cb1-a5a8-6115-0cc47aa3ff04",

   "owner": "5a16b201-9cb1-a5a8-6115-0cc47aa3ff04",

   "health": "Healthy",

   "revision": "3",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "5935cede-5bed-b5ac-6141-0cc47a665ae0",

   "owner": "5935cede-5bed-b5ac-6141-0cc47a665ae0",

   "health": "Healthy",

   "revision": "2",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "58e2f116-9e65-ffb8-775c-0cc47aa39058",

   "owner": "58e2f116-9e65-ffb8-775c-0cc47aa39058",

   "health": "Unhealthy",

   "revision": "56",

   "type": "NODE_DECOM_STATE",

   "flag": "0",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "590962bb-9b55-e4c8-a963-0cc47aa432b8",

   "owner": "590962bb-9b55-e4c8-a963-0cc47aa432b8",

   "health": "Healthy",

   "revision": "3",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "59251342-4fb6-1fe8-0d26-0cc47aa43398",

   "owner": "59251342-4fb6-1fe8-0d26-0cc47aa43398",

   "health": "Healthy",

   "revision": "3",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "58e30fc3-8242-24ea-9917-0cc47ad31d24",

   "owner": "58e30fc3-8242-24ea-9917-0cc47ad31d24",

   "health": "Unhealthy",

   "revision": "16",

   "type": "NODE_DECOM_STATE",

   "flag": "0",

   "minHostVersion": "0",

   "md5sum": "b288881e5ead776bb4cb39b8d3335280",

   "valueLen": "80",

   "content": {"decomState": 6, "decomJobType": 1, "decomJobUuid": "bb990c8b-f1af-f25a-1751-f4f6a7f994fb", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 2},

   "errorStr": "(null)"

}

,{

   "uuid": "5bd9bcf8-dba3-f8fa-5552-0cc47a63a188",

   "owner": "5bd9bcf8-dba3-f8fa-5552-0cc47a63a188",

   "health": "Unhealthy",

   "revision": "11",

   "type": "NODE_DECOM_STATE",

   "flag": "0",

   "minHostVersion": "0",

   "md5sum": "daadf5124d0e7499073d755f6f49758c",

   "valueLen": "80",

   "content": {"decomState": 6, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 2},

   "errorStr": "(null)"

}

,{

   "uuid": "5934d953-5534-eafc-19c5-0cc47aa39020",

   "owner": "5934d953-5534-eafc-19c5-0cc47aa39020",

   "health": "Healthy",

   "revision": "3",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "596ca6e5-91d9-effc-534c-0cc47ad3caa0",

   "owner": "596ca6e5-91d9-effc-534c-0cc47ad3caa0",

   "health": "Healthy",

   "revision": "3",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

]

}

[root@sapmesxnlams030:~]

0 Kudos
TheBobkin
VMware Employee
VMware Employee

Hello rxjoseph​,

It is indeed in vSAN Decom State as I initially advised:

{

   "uuid": "5a744f74-02a6-aa68-d932-0cc47aa3c50c",

   "owner": "5a744f74-02a6-aa68-d932-0cc47aa3c50c",

   "health": "Healthy",

   "revision": "3",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "7c88fc7eea9891e32bc7b35532085ee5",

   "valueLen": "80",

   "content": {"decomState": 6, "decomJobType": 1, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 2},

   "errorStr": "(null)"

}

This is simply remediated by either running:

#localcli vsan maintenancemode cancel

(Specifically localcli not esxcli here as if it tries to talk through hostd/vpxa it will just refuse as it is not in MM at the ESXi level)

or

Put the host in MM via the GUI and take it out again.

Bob

View solution in original post

rxjoseph
Enthusiast
Enthusiast

Thank you Bob you are absolutely a vSAN guru

issue is now sorted with the step you provide d and now i can see host is populating objects

One last question i see few more host with Decom state 6 which means those hosts are detached from the vSAN ?

pastedImage_0.png

0 Kudos
TheBobkin
VMware Employee
VMware Employee

Hello rxjoseph​,

Yes, I noted you have 37 entries which means there are extraneous ones from node references not being cleaned following decommissioning - this can happen if they are/were still referenced in some Object (generally unhealthy ones) when this was performed (or they weren't decommissioned correctly e.g.).

I will say that the only time I have seen stale CMMDS entries pose an issue is with Stretched-clusters e.g. failed Witnesses being added multiple times and duplicate Fault-Domain references as came up recently here on Communities. If they are 'Unhealthy' references then they can likely be cleaned up without issue but of course care has to be taken before doing this - at the very least 1) Ensure all cluster members are present and clustered, 2)Ensure that all data is healthy and 3) Ensure the entries being removed are NOT current cluster members - if you are uncomfortable or not 100% confident doing this then please contact GSS for assistance, we do not bite.

There are 6 'Unhealthy' node entries which doesn't add up for a 32-node cluster (32+6=38), are you positive this is a 32-node cluster and/or everyone is present?

Of these, 3 were decommissioned/last seen in MM with 'Ensure Accessibility', 3 with MM 'No Action' (I would never advise this unless their disks are already decommissioned/long gone) and 1 with no MM (going to assume the boot device failed and was re-installed or some other hardware failure).

   "uuid": "58e2e57f-d791-7a26-fd87-0cc47aa38f34",

   "owner": "58e2e57f-d791-7a26-fd87-0cc47aa38f34",

   "health": "Unhealthy",

   "revision": "14",

   "type": "NODE_DECOM_STATE",

   "flag": "0",

   "minHostVersion": "0",

   "md5sum": "851e1bbf87c931166b19910ad0e12a01",

   "valueLen": "80",

   "content": {"decomState": 6, "decomJobType": 1, "decomJobUuid": "68adbf24-7c18-e9c4-c395-6e91737ddb88", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 2},

   "errorStr": "(null)"

  

   "uuid": "5933ecd5-53a6-8330-331e-0cc47aa39058",

   "owner": "5933ecd5-53a6-8330-331e-0cc47aa39058",

   "health": "Unhealthy",

   "revision": "7",

   "type": "NODE_DECOM_STATE",

   "flag": "0",

   "minHostVersion": "0",

   "md5sum": "a7d42aefc212f4555e9679b5feb75446",

   "valueLen": "80",

   "content": {"decomState": 6, "decomJobType": 1, "decomJobUuid": "93849f8b-c59f-9889-c7ce-6835609ab21f", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 2},

   "errorStr": "(null)"

   "uuid": "596ca740-c5b4-e458-b98f-0cc47ad36046",

   "owner": "596ca740-c5b4-e458-b98f-0cc47ad36046",

   "health": "Unhealthy",

   "revision": "6",

   "type": "NODE_DECOM_STATE",

   "flag": "0",

   "minHostVersion": "0",

   "md5sum": "8f66969748bf2a364a89157140196f88",

   "valueLen": "80",

   "content": {"decomState": 6, "decomJobType": 0, "decomJobUuid": "acc847b2-2b02-4dcf-88cd-19809360c8f0", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 2},

   "errorStr": "(null)"

  

   "uuid": "58e2f116-9e65-ffb8-775c-0cc47aa39058",

   "owner": "58e2f116-9e65-ffb8-775c-0cc47aa39058",

   "health": "Unhealthy",

   "revision": "56",

   "type": "NODE_DECOM_STATE",

   "flag": "0",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

   "uuid": "58e30fc3-8242-24ea-9917-0cc47ad31d24",

   "owner": "58e30fc3-8242-24ea-9917-0cc47ad31d24",

   "health": "Unhealthy",

   "revision": "16",

   "type": "NODE_DECOM_STATE",

   "flag": "0",

   "minHostVersion": "0",

   "md5sum": "b288881e5ead776bb4cb39b8d3335280",

   "valueLen": "80",

   "content": {"decomState": 6, "decomJobType": 1, "decomJobUuid": "bb990c8b-f1af-f25a-1751-f4f6a7f994fb", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 2},

   "errorStr": "(null)"

  

   "uuid": "5bd9bcf8-dba3-f8fa-5552-0cc47a63a188",

   "owner": "5bd9bcf8-dba3-f8fa-5552-0cc47a63a188",

   "health": "Unhealthy",

   "revision": "11",

   "type": "NODE_DECOM_STATE",

   "flag": "0",

   "minHostVersion": "0",

   "md5sum": "daadf5124d0e7499073d755f6f49758c",

   "valueLen": "80",

   "content": {"decomState": 6, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 2},

   "errorStr": "(null)"

Bob

0 Kudos