7 Replies Latest reply on Jan 18, 2019 12:10 PM by TheBobkin

    One host out of the 32 host in my vSAN is not utilizing disk

    rxjoseph Novice

      hello Experts

       

      Firstly i need an expert to answer my query as I have some good experience in vSAN but this issue is biting  me

       

      I have a 32 Host ESXi 6 vSphere 6 vSAN cluster

      Currently when i do a vsan.disks_stats from the RVC console I get the nice report with the disk size and used space

      I can see all of the disk is used well but there is one exception, once of the servers is not using any vSAN disk

       

      The Host has 98 VMs but see the disks stats

      its only using 0.12 and rest of the Host are all using 70 %

      Can someone guide me through the correct steps to troubleshoot please

      PS C:\Users\RPARPJ016> get-vmhost sapmesxnlams030.crb.apmoller.net |get-vm |measure

      Count    : 98

       

       

      vAN components status its 0 on host 32 while all others have components

        • 1. Re: One host out of the 32 host in my vSAN is not utilizing disk
          TheBobkin Virtuoso
          VMware EmployeesvExpert

          Hello rxjoseph

           

          Check if the host is in vSAN Decom-state via SSH:

          #cmmds-tool find -t NODE_DECOM_STATE -f json

          It will have decomState:6 if it is.

          To confirm the UUID of the node you can use this run on that host:

          #cmmds-tool whoami

          Note that vSAN doesn't automatically balance data across disks unless any disk passes 80% utilised (default) so if you add a new host to a cluster or re-join cluster after a long maintenance window you will have to use proactive rebalance to move some data onto it.

          This can be performed via the GUI or via RVC:

          > vsan.proactive_rebalance <pathToCluster>

           

           

          Bob

          • 2. Re: One host out of the 32 host in my vSAN is not utilizing disk
            rxjoseph Novice

            Hi Bob

            This is the result from the host having the issue

             

            [root@sapmesxnlams030:~] cmmds-tool find -t NODE_DECOM_STATE -f json

            {

            "entries":

            [

            {

               "uuid": "5a0c1a86-6cd4-d308-6fe9-0cc47aa35d58",

               "owner": "5a0c1a86-6cd4-d308-6fe9-0cc47aa35d58",

               "health": "Healthy",

               "revision": "3",

               "type": "NODE_DECOM_STATE",

               "flag": "2",

               "minHostVersion": "0",

               "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

               "valueLen": "80",

               "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

               "errorStr": "(null)"

            }

             

            [root@sapmesxnlams030:~] cmmds-tool whoami

            5a744f74-02a6-aa68-d932-0cc47aa3c50c

             

            Can you advise on this output please

            • 3. Re: One host out of the 32 host in my vSAN is not utilizing disk
              TheBobkin Virtuoso
              VMware EmployeesvExpert

              Hello rxjoseph,

               

               

              "This is the result from the host having the issue

              [root@sapmesxnlams030:~] cmmds-tool find -t NODE_DECOM_STATE -f json

               

               

              {

              "entries":

              [

              {

                 "uuid": "5a0c1a86-6cd4-d308-6fe9-0cc47aa35d58",

                 "owner": "5a0c1a86-6cd4-d308-6fe9-0cc47aa35d58",

                 "health": "Healthy",

                 "revision": "3",

                 "type": "NODE_DECOM_STATE",

                 "flag": "2",

                 "minHostVersion": "0",

                 "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

                 "valueLen": "80",

                 "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

                 "errorStr": "(null)"

               

              }"

               

              Did this command on this node only return a single NODE_DECOM_STATE entry? Running this on any host should return the entries (of this type) for all nodes in the cluster.

              We can also see clearly that this is not the NODE_DECOM_STATE entry for this node:

                 "uuid": "5a0c1a86-6cd4-d308-6fe9-0cc47aa35d58",

                 "owner": "5a0c1a86-6cd4-d308-6fe9-0cc47aa35d58"

              Where whoami (and also likely HOSTNAME entry) shows a different UUID (5a744f74-02a6-aa68-d932-0cc47aa3c50c).

               

              There are a good few potential causes of your issue here but determining the node we are looking at is step 1.

              Please confirm this is not a stretched-cluster, check there are no duplicate/stale entries for other types in CMMDS from nodes being removed/re-added improperly (as the fact you are hitting this on '32nd' node has multiple alarm bells triggered here that indicate this is not coincidence).

               

               

              Bob

               

               

               

              • 4. Re: One host out of the 32 host in my vSAN is not utilizing disk
                rxjoseph Novice

                [root@sapmesxnlams030:~] cmmds-tool whoami

                5a744f74-02a6-aa68-d932-0cc47aa3c50c

                [root@sapmesxnlams030:~] cmmds-tool find -t NODE_DECOM_STATE -f json

                {

                "entries":

                [

                {

                   "uuid": "5a0c1a86-6cd4-d308-6fe9-0cc47aa35d58",

                   "owner": "5a0c1a86-6cd4-d308-6fe9-0cc47aa35d58",

                   "health": "Healthy",

                   "revision": "3",

                   "type": "NODE_DECOM_STATE",

                   "flag": "2",

                   "minHostVersion": "0",

                   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

                   "valueLen": "80",

                   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "5bbca3f5-de1a-8d0a-37fe-0cc47aa39058",

                   "owner": "5bbca3f5-de1a-8d0a-37fe-0cc47aa39058",

                   "health": "Healthy",

                   "revision": "1",

                   "type": "NODE_DECOM_STATE",

                   "flag": "2",

                   "minHostVersion": "0",

                   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

                   "valueLen": "80",

                   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "59308d44-0f85-3610-af00-0cc47a39bdf4",

                   "owner": "59308d44-0f85-3610-af00-0cc47a39bdf4",

                   "health": "Healthy",

                   "revision": "3",

                   "type": "NODE_DECOM_STATE",

                   "flag": "2",

                   "minHostVersion": "0",

                   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

                   "valueLen": "80",

                   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "5a16c225-dd64-9a10-6ddd-0cc47aa43420",

                   "owner": "5a16c225-dd64-9a10-6ddd-0cc47aa43420",

                   "health": "Healthy",

                   "revision": "3",

                   "type": "NODE_DECOM_STATE",

                   "flag": "2",

                   "minHostVersion": "0",

                   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

                   "valueLen": "80",

                   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "5a5882a0-4efa-5b14-c76f-0cc47aa4e6fe",

                   "owner": "5a5882a0-4efa-5b14-c76f-0cc47aa4e6fe",

                   "health": "Healthy",

                   "revision": "6",

                   "type": "NODE_DECOM_STATE",

                   "flag": "2",

                   "minHostVersion": "0",

                   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

                   "valueLen": "80",

                   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "59265031-8888-a314-1578-0cc47aa3ff14",

                   "owner": "59265031-8888-a314-1578-0cc47aa3ff14",

                   "health": "Healthy",

                   "revision": "3",

                   "type": "NODE_DECOM_STATE",

                   "flag": "2",

                   "minHostVersion": "0",

                   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

                   "valueLen": "80",

                   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "5923a30d-88aa-d318-95e8-0cc47aa4341c",

                   "owner": "5923a30d-88aa-d318-95e8-0cc47aa4341c",

                   "health": "Healthy",

                   "revision": "3",

                   "type": "NODE_DECOM_STATE",

                   "flag": "2",

                   "minHostVersion": "0",

                   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

                   "valueLen": "80",

                   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "58e354cb-4330-1020-831e-0cc47aa38d48",

                   "owner": "58e354cb-4330-1020-831e-0cc47aa38d48",

                   "health": "Healthy",

                   "revision": "3",

                   "type": "NODE_DECOM_STATE",

                   "flag": "2",

                   "minHostVersion": "0",

                   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

                   "valueLen": "80",

                   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "596cadb4-3b02-8520-294a-0cc47ad375d8",

                   "owner": "596cadb4-3b02-8520-294a-0cc47ad375d8",

                   "health": "Healthy",

                   "revision": "3",

                   "type": "NODE_DECOM_STATE",

                   "flag": "2",

                   "minHostVersion": "0",

                   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

                   "valueLen": "80",

                   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "58e2e57f-d791-7a26-fd87-0cc47aa38f34",

                   "owner": "58e2e57f-d791-7a26-fd87-0cc47aa38f34",

                   "health": "Unhealthy",

                   "revision": "14",

                   "type": "NODE_DECOM_STATE",

                   "flag": "0",

                   "minHostVersion": "0",

                   "md5sum": "851e1bbf87c931166b19910ad0e12a01",

                   "valueLen": "80",

                   "content": {"decomState": 6, "decomJobType": 1, "decomJobUuid": "68adbf24-7c18-e9c4-c395-6e91737ddb88", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 2},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "5933ecd5-53a6-8330-331e-0cc47aa39058",

                   "owner": "5933ecd5-53a6-8330-331e-0cc47aa39058",

                   "health": "Unhealthy",

                   "revision": "7",

                   "type": "NODE_DECOM_STATE",

                   "flag": "0",

                   "minHostVersion": "0",

                   "md5sum": "a7d42aefc212f4555e9679b5feb75446",

                   "valueLen": "80",

                   "content": {"decomState": 6, "decomJobType": 1, "decomJobUuid": "93849f8b-c59f-9889-c7ce-6835609ab21f", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 2},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "591bc984-ef61-ec3c-df82-0cc47ad31d24",

                   "owner": "591bc984-ef61-ec3c-df82-0cc47ad31d24",

                   "health": "Healthy",

                   "revision": "3",

                   "type": "NODE_DECOM_STATE",

                   "flag": "2",

                   "minHostVersion": "0",

                   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

                   "valueLen": "80",

                   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "596cfb1f-72e7-9842-0ddc-0cc47ad3552e",

                   "owner": "596cfb1f-72e7-9842-0ddc-0cc47ad3552e",

                   "health": "Healthy",

                   "revision": "3",

                   "type": "NODE_DECOM_STATE",

                   "flag": "2",

                   "minHostVersion": "0",

                   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

                   "valueLen": "80",

                   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "596ca832-8df1-6c44-0f1b-0cc47ad36e78",

                   "owner": "596ca832-8df1-6c44-0f1b-0cc47ad36e78",

                   "health": "Healthy",

                   "revision": "3",

                   "type": "NODE_DECOM_STATE",

                   "flag": "2",

                   "minHostVersion": "0",

                   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

                   "valueLen": "80",

                   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "59347d01-05e8-2358-d3a1-0cc47aa39114",

                   "owner": "59347d01-05e8-2358-d3a1-0cc47aa39114",

                   "health": "Healthy",

                   "revision": "2",

                   "type": "NODE_DECOM_STATE",

                   "flag": "2",

                   "minHostVersion": "0",

                   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

                   "valueLen": "80",

                   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "596ca740-c5b4-e458-b98f-0cc47ad36046",

                   "owner": "596ca740-c5b4-e458-b98f-0cc47ad36046",

                   "health": "Unhealthy",

                   "revision": "6",

                   "type": "NODE_DECOM_STATE",

                   "flag": "0",

                   "minHostVersion": "0",

                   "md5sum": "8f66969748bf2a364a89157140196f88",

                   "valueLen": "80",

                   "content": {"decomState": 6, "decomJobType": 0, "decomJobUuid": "acc847b2-2b02-4dcf-88cd-19809360c8f0", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 2},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "59100dc4-4fb2-1768-cded-0cc47aa3d068",

                   "owner": "59100dc4-4fb2-1768-cded-0cc47aa3d068",

                   "health": "Healthy",

                   "revision": "2",

                   "type": "NODE_DECOM_STATE",

                   "flag": "2",

                   "minHostVersion": "0",

                   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

                   "valueLen": "80",

                   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "596cabf1-94ec-9868-9742-0cc47ad3553a",

                   "owner": "596cabf1-94ec-9868-9742-0cc47ad3553a",

                   "health": "Healthy",

                   "revision": "3",

                   "type": "NODE_DECOM_STATE",

                   "flag": "2",

                   "minHostVersion": "0",

                   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

                   "valueLen": "80",

                   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "5a744f74-02a6-aa68-d932-0cc47aa3c50c",

                   "owner": "5a744f74-02a6-aa68-d932-0cc47aa3c50c",

                   "health": "Healthy",

                   "revision": "3",

                   "type": "NODE_DECOM_STATE",

                   "flag": "2",

                   "minHostVersion": "0",

                   "md5sum": "7c88fc7eea9891e32bc7b35532085ee5",

                   "valueLen": "80",

                   "content": {"decomState": 6, "decomJobType": 1, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 2},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "591be406-6eb1-466c-6d14-0cc47aa3d074",

                   "owner": "591be406-6eb1-466c-6d14-0cc47aa3d074",

                   "health": "Healthy",

                   "revision": "2",

                   "type": "NODE_DECOM_STATE",

                   "flag": "2",

                   "minHostVersion": "0",

                   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

                   "valueLen": "80",

                   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "5a7456d8-b3a8-807c-d9cd-0cc47a665adc",

                   "owner": "5a7456d8-b3a8-807c-d9cd-0cc47a665adc",

                   "health": "Healthy",

                   "revision": "2",

                   "type": "NODE_DECOM_STATE",

                   "flag": "2",

                   "minHostVersion": "0",

                   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

                   "valueLen": "80",

                   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "591d231a-35d5-4494-8320-0cc47aa435c0",

                   "owner": "591d231a-35d5-4494-8320-0cc47aa435c0",

                   "health": "Healthy",

                   "revision": "2",

                   "type": "NODE_DECOM_STATE",

                   "flag": "2",

                   "minHostVersion": "0",

                   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

                   "valueLen": "80",

                   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "5bda19f2-75c0-5d94-933b-002590fb44e4",

                   "owner": "5bda19f2-75c0-5d94-933b-002590fb44e4",

                   "health": "Healthy",

                   "revision": "3",

                   "type": "NODE_DECOM_STATE",

                   "flag": "2",

                   "minHostVersion": "0",

                   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

                   "valueLen": "80",

                   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "596ca77d-d466-eb94-31ff-0cc47ad35f76",

                   "owner": "596ca77d-d466-eb94-31ff-0cc47ad35f76",

                   "health": "Healthy",

                   "revision": "3",

                   "type": "NODE_DECOM_STATE",

                   "flag": "2",

                   "minHostVersion": "0",

                   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

                   "valueLen": "80",

                   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "59153d37-ffb9-22a0-dbb5-0cc47aa3ce98",

                   "owner": "59153d37-ffb9-22a0-dbb5-0cc47aa3ce98",

                   "health": "Healthy",

                   "revision": "3",

                   "type": "NODE_DECOM_STATE",

                   "flag": "2",

                   "minHostVersion": "0",

                   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

                   "valueLen": "80",

                   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "58e34a33-0b1a-fba0-6dcc-0cc47ad31a90",

                   "owner": "58e34a33-0b1a-fba0-6dcc-0cc47ad31a90",

                   "health": "Healthy",

                   "revision": "5",

                   "type": "NODE_DECOM_STATE",

                   "flag": "2",

                   "minHostVersion": "0",

                   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

                   "valueLen": "80",

                   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "5933a42f-f3ad-b6a6-b1a7-0cc47aa38f34",

                   "owner": "5933a42f-f3ad-b6a6-b1a7-0cc47aa38f34",

                   "health": "Healthy",

                   "revision": "3",

                   "type": "NODE_DECOM_STATE",

                   "flag": "2",

                   "minHostVersion": "0",

                   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

                   "valueLen": "80",

                   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "5922669b-ca13-f2a6-37ce-0cc47aa4e646",

                   "owner": "5922669b-ca13-f2a6-37ce-0cc47aa4e646",

                   "health": "Healthy",

                   "revision": "3",

                   "type": "NODE_DECOM_STATE",

                   "flag": "2",

                   "minHostVersion": "0",

                   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

                   "valueLen": "80",

                   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "5a16b201-9cb1-a5a8-6115-0cc47aa3ff04",

                   "owner": "5a16b201-9cb1-a5a8-6115-0cc47aa3ff04",

                   "health": "Healthy",

                   "revision": "3",

                   "type": "NODE_DECOM_STATE",

                   "flag": "2",

                   "minHostVersion": "0",

                   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

                   "valueLen": "80",

                   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "5935cede-5bed-b5ac-6141-0cc47a665ae0",

                   "owner": "5935cede-5bed-b5ac-6141-0cc47a665ae0",

                   "health": "Healthy",

                   "revision": "2",

                   "type": "NODE_DECOM_STATE",

                   "flag": "2",

                   "minHostVersion": "0",

                   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

                   "valueLen": "80",

                   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "58e2f116-9e65-ffb8-775c-0cc47aa39058",

                   "owner": "58e2f116-9e65-ffb8-775c-0cc47aa39058",

                   "health": "Unhealthy",

                   "revision": "56",

                   "type": "NODE_DECOM_STATE",

                   "flag": "0",

                   "minHostVersion": "0",

                   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

                   "valueLen": "80",

                   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "590962bb-9b55-e4c8-a963-0cc47aa432b8",

                   "owner": "590962bb-9b55-e4c8-a963-0cc47aa432b8",

                   "health": "Healthy",

                   "revision": "3",

                   "type": "NODE_DECOM_STATE",

                   "flag": "2",

                   "minHostVersion": "0",

                   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

                   "valueLen": "80",

                   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "59251342-4fb6-1fe8-0d26-0cc47aa43398",

                   "owner": "59251342-4fb6-1fe8-0d26-0cc47aa43398",

                   "health": "Healthy",

                   "revision": "3",

                   "type": "NODE_DECOM_STATE",

                   "flag": "2",

                   "minHostVersion": "0",

                   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

                   "valueLen": "80",

                   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "58e30fc3-8242-24ea-9917-0cc47ad31d24",

                   "owner": "58e30fc3-8242-24ea-9917-0cc47ad31d24",

                   "health": "Unhealthy",

                   "revision": "16",

                   "type": "NODE_DECOM_STATE",

                   "flag": "0",

                   "minHostVersion": "0",

                   "md5sum": "b288881e5ead776bb4cb39b8d3335280",

                   "valueLen": "80",

                   "content": {"decomState": 6, "decomJobType": 1, "decomJobUuid": "bb990c8b-f1af-f25a-1751-f4f6a7f994fb", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 2},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "5bd9bcf8-dba3-f8fa-5552-0cc47a63a188",

                   "owner": "5bd9bcf8-dba3-f8fa-5552-0cc47a63a188",

                   "health": "Unhealthy",

                   "revision": "11",

                   "type": "NODE_DECOM_STATE",

                   "flag": "0",

                   "minHostVersion": "0",

                   "md5sum": "daadf5124d0e7499073d755f6f49758c",

                   "valueLen": "80",

                   "content": {"decomState": 6, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 2},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "5934d953-5534-eafc-19c5-0cc47aa39020",

                   "owner": "5934d953-5534-eafc-19c5-0cc47aa39020",

                   "health": "Healthy",

                   "revision": "3",

                   "type": "NODE_DECOM_STATE",

                   "flag": "2",

                   "minHostVersion": "0",

                   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

                   "valueLen": "80",

                   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

                   "errorStr": "(null)"

                }

                ,{

                   "uuid": "596ca6e5-91d9-effc-534c-0cc47ad3caa0",

                   "owner": "596ca6e5-91d9-effc-534c-0cc47ad3caa0",

                   "health": "Healthy",

                   "revision": "3",

                   "type": "NODE_DECOM_STATE",

                   "flag": "2",

                   "minHostVersion": "0",

                   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

                   "valueLen": "80",

                   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

                   "errorStr": "(null)"

                }

                ]

                }

                [root@sapmesxnlams030:~]

                • 5. Re: One host out of the 32 host in my vSAN is not utilizing disk
                  TheBobkin Virtuoso
                  VMware EmployeesvExpert

                  Hello rxjoseph,

                   

                  It is indeed in vSAN Decom State as I initially advised:

                  {

                     "uuid": "5a744f74-02a6-aa68-d932-0cc47aa3c50c",

                     "owner": "5a744f74-02a6-aa68-d932-0cc47aa3c50c",

                     "health": "Healthy",

                     "revision": "3",

                     "type": "NODE_DECOM_STATE",

                     "flag": "2",

                     "minHostVersion": "0",

                     "md5sum": "7c88fc7eea9891e32bc7b35532085ee5",

                     "valueLen": "80",

                     "content": {"decomState": 6, "decomJobType": 1, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 2},

                     "errorStr": "(null)"

                  }

                   

                  This is simply remediated by either running:

                  #localcli vsan maintenancemode cancel

                  (Specifically localcli not esxcli here as if it tries to talk through hostd/vpxa it will just refuse as it is not in MM at the ESXi level)

                  or

                  Put the host in MM via the GUI and take it out again.

                   

                   

                  Bob

                  1 person found this helpful
                  • 6. Re: One host out of the 32 host in my vSAN is not utilizing disk
                    rxjoseph Novice

                    Thank you Bob you are absolutely a vSAN guru

                    issue is now sorted with the step you provide d and now i can see host is populating objects

                    One last question i see few more host with Decom state 6 which means those hosts are detached from the vSAN ?

                    • 7. Re: One host out of the 32 host in my vSAN is not utilizing disk
                      TheBobkin Virtuoso
                      vExpertVMware Employees

                      Hello rxjoseph,

                       

                       

                      Yes, I noted you have 37 entries which means there are extraneous ones from node references not being cleaned following decommissioning - this can happen if they are/were still referenced in some Object (generally unhealthy ones) when this was performed (or they weren't decommissioned correctly e.g.).

                      I will say that the only time I have seen stale CMMDS entries pose an issue is with Stretched-clusters e.g. failed Witnesses being added multiple times and duplicate Fault-Domain references as came up recently here on Communities. If they are 'Unhealthy' references then they can likely be cleaned up without issue but of course care has to be taken before doing this - at the very least 1) Ensure all cluster members are present and clustered, 2)Ensure that all data is healthy and 3) Ensure the entries being removed are NOT current cluster members - if you are uncomfortable or not 100% confident doing this then please contact GSS for assistance, we do not bite.

                       

                      There are 6 'Unhealthy' node entries which doesn't add up for a 32-node cluster (32+6=38), are you positive this is a 32-node cluster and/or everyone is present?

                      Of these, 3 were decommissioned/last seen in MM with 'Ensure Accessibility', 3 with MM 'No Action' (I would never advise this unless their disks are already decommissioned/long gone) and 1 with no MM (going to assume the boot device failed and was re-installed or some other hardware failure).

                       

                         "uuid": "58e2e57f-d791-7a26-fd87-0cc47aa38f34",

                         "owner": "58e2e57f-d791-7a26-fd87-0cc47aa38f34",

                         "health": "Unhealthy",

                         "revision": "14",

                         "type": "NODE_DECOM_STATE",

                         "flag": "0",

                         "minHostVersion": "0",

                         "md5sum": "851e1bbf87c931166b19910ad0e12a01",

                         "valueLen": "80",

                         "content": {"decomState": 6, "decomJobType": 1, "decomJobUuid": "68adbf24-7c18-e9c4-c395-6e91737ddb88", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 2},

                         "errorStr": "(null)"

                        

                         "uuid": "5933ecd5-53a6-8330-331e-0cc47aa39058",

                         "owner": "5933ecd5-53a6-8330-331e-0cc47aa39058",

                         "health": "Unhealthy",

                         "revision": "7",

                         "type": "NODE_DECOM_STATE",

                         "flag": "0",

                         "minHostVersion": "0",

                         "md5sum": "a7d42aefc212f4555e9679b5feb75446",

                         "valueLen": "80",

                         "content": {"decomState": 6, "decomJobType": 1, "decomJobUuid": "93849f8b-c59f-9889-c7ce-6835609ab21f", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 2},

                         "errorStr": "(null)"

                       

                         "uuid": "596ca740-c5b4-e458-b98f-0cc47ad36046",

                         "owner": "596ca740-c5b4-e458-b98f-0cc47ad36046",

                         "health": "Unhealthy",

                         "revision": "6",

                         "type": "NODE_DECOM_STATE",

                         "flag": "0",

                         "minHostVersion": "0",

                         "md5sum": "8f66969748bf2a364a89157140196f88",

                         "valueLen": "80",

                         "content": {"decomState": 6, "decomJobType": 0, "decomJobUuid": "acc847b2-2b02-4dcf-88cd-19809360c8f0", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 2},

                         "errorStr": "(null)"

                        

                         "uuid": "58e2f116-9e65-ffb8-775c-0cc47aa39058",

                         "owner": "58e2f116-9e65-ffb8-775c-0cc47aa39058",

                         "health": "Unhealthy",

                         "revision": "56",

                         "type": "NODE_DECOM_STATE",

                         "flag": "0",

                         "minHostVersion": "0",

                         "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

                         "valueLen": "80",

                         "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

                         "errorStr": "(null)"

                       

                         "uuid": "58e30fc3-8242-24ea-9917-0cc47ad31d24",

                         "owner": "58e30fc3-8242-24ea-9917-0cc47ad31d24",

                         "health": "Unhealthy",

                         "revision": "16",

                         "type": "NODE_DECOM_STATE",

                         "flag": "0",

                         "minHostVersion": "0",

                         "md5sum": "b288881e5ead776bb4cb39b8d3335280",

                         "valueLen": "80",

                         "content": {"decomState": 6, "decomJobType": 1, "decomJobUuid": "bb990c8b-f1af-f25a-1751-f4f6a7f994fb", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 2},

                         "errorStr": "(null)"

                        

                         "uuid": "5bd9bcf8-dba3-f8fa-5552-0cc47a63a188",

                         "owner": "5bd9bcf8-dba3-f8fa-5552-0cc47a63a188",

                         "health": "Unhealthy",

                         "revision": "11",

                         "type": "NODE_DECOM_STATE",

                         "flag": "0",

                         "minHostVersion": "0",

                         "md5sum": "daadf5124d0e7499073d755f6f49758c",

                         "valueLen": "80",

                         "content": {"decomState": 6, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 2},

                         "errorStr": "(null)"

                       

                       

                      Bob