VMware Cloud Community
marsherian
Enthusiast
Enthusiast

vCenter Disk components marked as absent - Parent disks are present.

I have a nested environment, which was cleanly shutdown in preparation for maintenance of the physical servers where they are hosted.

There were some communications issues at first, but I was eventually able to get all six nodes in the cluster to rejoin. All VMs in the nested environment came back healthy, except the vCenter server. I was able to use the parent vCenter's instance of RVC to connect to each of the hosts in the nested cluster. I then ran a vsan.vm_object_info for the vCenter VM.

Object info for vCenter VM from RVC

vsan.vm_object_info /10.25.0.21/ha-datacenter/vms/hzn-vc.mars.lab.adms.local/

VM hzn-vc.mars.lab.adms.local:

  Namespace directory

    Couldn't find info about DOM object ''

  Disk backing: [vsanDatastore] 6aa5265a-1ea1-5ca9-422e-005056b392af/hzn-vc.mars.lab.adms.local_11.vmdk

    DOM Object: 7ea5265a-6525-4da8-dd7b-005056b392af (v5, owner: unknown, proxy owner: None, policy: No POLICY entry found in CMMDS)

      RAID_0

        Component: 8eda275a-a5f8-d7fa-146f-005056b392af (state: ABSENT (6), host: 594d4721-e1d2-a466-5f38-005056b34157, md: 52ede984-77a1-85e5-99cf-721d8144fd57, ssd: 5210251c-5a07-06c6-0fba-2da17d51d9aa,

                                                         votes: 3, usage: 0.2 GB, proxy component: false)

        Component: 8eda275a-a64b-dcfa-3892-005056b392af (state: ABSENT (6), host: Unknown, md: 52b7da24-ee49-c806-2550-25b12584c0e1, ssd: Unknown, note: LSOM object not found,

                                                         votes: 1, proxy component: false)

        Component: 8eda275a-e762-defa-5ab3-005056b392af (state: ABSENT (6), host: Unknown, md: 52c56620-f893-9267-9731-bed3fb50767f, ssd: Unknown, note: LSOM object not found,

                                                         votes: 1, proxy component: false)

        Component: 8eda275a-93d2-dffa-1809-005056b392af (state: ABSENT (6), host: Unknown, md: 528ed5b2-e6f7-eee4-b687-6315c1d4731b, ssd: Unknown, note: LSOM object not found,

                                                         votes: 1, proxy component: false)

        Component: 8eda275a-5458-e1fa-b331-005056b392af (state: ABSENT (6), host: Unknown, md: 522cc47a-c25d-c60f-07ea-c387518831a4, ssd: Unknown, note: LSOM object not found,

                                                         votes: 1, proxy component: false)

        Component: 8eda275a-7d0c-e3fa-ab17-005056b392af (state: ABSENT (6), host: 594d4721-e1d2-a466-5f38-005056b34157, md: 528e315c-8ed7-99be-a528-a167efc3438d, ssd: 5210251c-5a07-06c6-0fba-2da17d51d9aa,

                                                         votes: 2, usage: 1.2 GB, proxy component: false)

  Disk backing: [vsanDatastore] 6aa5265a-1ea1-5ca9-422e-005056b392af/hzn-vc.mars.lab.adms.local_10.vmdk

    DOM Object: 7ea5265a-2d99-7f71-e7c3-005056b392af (v5, owner: unknown, proxy owner: None, policy: No POLICY entry found in CMMDS)

      RAID_0

        Component: 8fda275a-365f-171d-8c49-005056b392af (state: ABSENT (6), host: 594d4721-e1d2-a466-5f38-005056b34157, md: 5238f31e-8a6b-7231-ae8f-5c7ff481ffb8, ssd: 5210251c-5a07-06c6-0fba-2da17d51d9aa,

                                                         votes: 5, usage: 0.0 GB, proxy component: false)

        Component: 8fda275a-d7c1-1a1d-bd92-005056b392af (state: ABSENT (6), host: Unknown, md: 522cc47a-c25d-c60f-07ea-c387518831a4, ssd: Unknown, note: LSOM object not found,

                                                         votes: 1, proxy component: false)

        Component: 8fda275a-0e0c-1c1d-e1de-005056b392af (state: ABSENT (6), host: Unknown, md: 52c56620-f893-9267-9731-bed3fb50767f, ssd: Unknown, note: LSOM object not found,

                                                         votes: 1, proxy component: false)

        Component: a5369d5a-0ce2-a4b3-47f4-005056b392af (state: ABSENT (6), host: 594d4782-6f02-c98b-6e2a-005056b3fb13, md: 5294941b-1cc0-63fa-7db5-f5bed035a093, ssd: 523fce6a-8bad-c3a8-0e04-a2eab1dc1d48,

                                                         votes: 4, usage: 0.1 GB, proxy component: false)

        Component: 8fda275a-a864-1e1d-1e19-005056b392af (state: ABSENT (6), host: Unknown, md: 52b8d8af-bdeb-d4ca-e46d-45f8feffee99, ssd: Unknown, note: LSOM object not found,

                                                         votes: 1, proxy component: false)

        Component: 8fda275a-0d01-201d-7e12-005056b392af (state: ABSENT (6), host: Unknown, md: 52b7da24-ee49-c806-2550-25b12584c0e1, ssd: Unknown, note: LSOM object not found,

                                                         votes: 1, proxy component: false)

  Disk backing: [vsanDatastore] 6aa5265a-1ea1-5ca9-422e-005056b392af/hzn-vc.mars.lab.adms.local_5.vmdk

    DOM Object: 7ba5265a-13a3-d947-10fa-005056b392af (v5, owner: unknown, proxy owner: None, policy: No POLICY entry found in CMMDS)

      RAID_0

        Component: 8fda275a-8744-0d33-f5a5-005056b392af (state: ABSENT (6), host: Unknown, md: 528ed5b2-e6f7-eee4-b687-6315c1d4731b, ssd: Unknown, note: LSOM object not found,

                                                         votes: 2, proxy component: false)

        Component: 8fda275a-ed3c-1033-efb6-005056b392af (state: ABSENT (6), host: 594d4721-e1d2-a466-5f38-005056b34157, md: 52037a92-0d7a-a9e1-da61-76aad3d02271, ssd: 5210251c-5a07-06c6-0fba-2da17d51d9aa,

                                                         votes: 1, usage: 1.7 GB, proxy component: false)

        Component: 9f369d5a-9d83-401b-cd07-005056b392af (state: ABSENT (6), host: Unknown, md: 520dca0c-4084-1e4b-0d27-6fc3e58c69bc, ssd: Unknown, note: LSOM object not found,

                                                         votes: 1, proxy component: false)

        Component: 8fda275a-488a-1333-f495-005056b392af (state: ABSENT (6), host: 594d4721-e1d2-a466-5f38-005056b34157, md: 52ede984-77a1-85e5-99cf-721d8144fd57, ssd: 5210251c-5a07-06c6-0fba-2da17d51d9aa,

                                                         votes: 1, usage: 1.7 GB, proxy component: false)

        Component: 8fda275a-75da-1433-8f07-005056b392af (state: ABSENT (6), host: Unknown, md: 52b8d8af-bdeb-d4ca-e46d-45f8feffee99, ssd: Unknown, note: LSOM object not found,

                                                         votes: 1, proxy component: false)

        Component: 8fda275a-cd79-1633-7164-005056b392af (state: ABSENT (6), host: Unknown, md: 5248d763-e44e-223c-ccbc-83cac14a3ca3, ssd: Unknown, note: LSOM object not found,

                                                         votes: 1, proxy component: false)

  Disk backing: [vsanDatastore] 6aa5265a-1ea1-5ca9-422e-005056b392af/hzn-vc.mars.lab.adms.local_1.vmdk

    DOM Object: 7aa5265a-a876-986f-8495-005056b392af (v5, owner: unknown, proxy owner: None, policy: spbmProfileName = vSAN Default Storage Policy, SCSN = 5624, stripeWidth = 6, cacheReservation = 0, spbmProfileGenerationNumber = 1, hostFailuresToTolerate = 0, proportionalCapacity = 0, spbmProfileId = aa6d5a82-1c88-45da-85d3-3d74b91a5bad, forceProvisioning = 0, CSN = 205)

      RAID_0

        Component: 8fda275a-15b3-8c4c-015c-005056b392af (state: ACTIVE (5), host: 594d4721-e1d2-a466-5f38-005056b34157, md: 52037a92-0d7a-a9e1-da61-76aad3d02271, ssd: 5210251c-5a07-06c6-0fba-2da17d51d9aa,

                                                         votes: 2, usage: 0.3 GB, proxy component: false)

        Component: 8fda275a-312d-904c-c464-005056b392af (state: ACTIVE (5), host: 594d4721-e1d2-a466-5f38-005056b34157, md: 5238f31e-8a6b-7231-ae8f-5c7ff481ffb8, ssd: 5210251c-5a07-06c6-0fba-2da17d51d9aa,

                                                         votes: 1, usage: 0.3 GB, proxy component: false)

        Component: 9d369d5a-7ef4-abcb-59e3-005056b392af (state: ACTIVE (5), host: 594d4721-e1d2-a466-5f38-005056b34157, md: 52c8daec-b508-6b77-0ad2-370f42faf555, ssd: 5210251c-5a07-06c6-0fba-2da17d51d9aa,

                                                         votes: 1, usage: 0.3 GB, proxy component: false)

        Component: 8fda275a-667b-924c-e6b9-005056b392af (state: ACTIVE (5), host: 594d4721-e1d2-a466-5f38-005056b34157, md: 52fbaf49-b277-a89c-9050-4e4de201e9da, ssd: 5210251c-5a07-06c6-0fba-2da17d51d9aa,

                                                         votes: 1, usage: 0.3 GB, proxy component: false)

        Component: 8fda275a-5ba9-934c-1aaf-005056b392af (state: ACTIVE (5), host: 594d488f-4f6b-48e2-9fee-005056b30d27, md: 5261a25d-1909-c635-6432-3f22e4a49b86, ssd: 529f2efe-107c-c754-6f4e-bac24cad216c,

                                                         votes: 4, usage: 0.3 GB, proxy component: false)

        Component: 9d369d5a-632a-afcb-891f-005056b392af (state: ACTIVE (5), host: 594d4782-6f02-c98b-6e2a-005056b3fb13, md: 52e73dc8-40b2-61f2-53eb-6d0a8ce8944e, ssd: 523fce6a-8bad-c3a8-0e04-a2eab1dc1d48,

                                                         votes: 4, usage: 0.3 GB, proxy component: false)

  Disk backing: [vsanDatastore] 6aa5265a-1ea1-5ca9-422e-005056b392af/hzn-vc.mars.lab.adms.local_7.vmdk

    DOM Object: 7ca5265a-f991-ddba-7c6e-005056b392af (v5, owner: unknown, proxy owner: None, policy: No POLICY entry found in CMMDS)

      RAID_0

        Component: 8fda275a-23ea-9286-f29a-005056b392af (state: ABSENT (6), host: 594d4721-e1d2-a466-5f38-005056b34157, md: 5238f31e-8a6b-7231-ae8f-5c7ff481ffb8, ssd: 5210251c-5a07-06c6-0fba-2da17d51d9aa,

                                                         votes: 2, usage: 1.1 GB, proxy component: false)

        Component: 8fda275a-2d7d-9686-e6c2-005056b392af (state: ABSENT (6), host: Unknown, md: 522cc47a-c25d-c60f-07ea-c387518831a4, ssd: Unknown, note: LSOM object not found,

                                                         votes: 1, proxy component: false)

        Component: 8fda275a-8005-9886-a6ad-005056b392af (state: ABSENT (6), host: Unknown, md: 52b7da24-ee49-c806-2550-25b12584c0e1, ssd: Unknown, note: LSOM object not found,

                                                         votes: 1, proxy component: false)

        Component: 8fda275a-b1d0-9986-09a4-005056b392af (state: ABSENT (6), host: Unknown, md: 52ed7a36-867f-3bf8-d3a9-d6ce09cec33b, ssd: Unknown, note: LSOM object not found,

                                                         votes: 1, proxy component: false)

        Component: 95369d5a-74a8-dbec-e828-005056b392af (state: ABSENT (6), host: Unknown, md: 52f2f60d-af30-1359-97ae-88d0e7cab6f7, ssd: Unknown, note: LSOM object not found,

                                                         votes: 3, proxy component: false)

        Component: 8fda275a-6cce-9d86-4527-005056b392af (state: ABSENT (6), host: 594d4721-e1d2-a466-5f38-005056b34157, md: 52fbaf49-b277-a89c-9050-4e4de201e9da, ssd: 5210251c-5a07-06c6-0fba-2da17d51d9aa,

                                                         votes: 1, usage: 1.1 GB, proxy component: false)

  Disk backing: [vsanDatastore] 6aa5265a-1ea1-5ca9-422e-005056b392af/hzn-vc.mars.lab.adms.local_8.vmdk

    DOM Object: 7da5265a-3dc6-e9fa-93a9-005056b392af (v5, owner: unknown, proxy owner: None, policy: No POLICY entry found in CMMDS)

      RAID_0

        Component: 91da275a-b3cf-d95e-70d7-005056b392af (state: ABSENT (6), host: Unknown, md: 52ed7a36-867f-3bf8-d3a9-d6ce09cec33b, ssd: Unknown, note: LSOM object not found,

                                                         votes: 2, proxy component: false)

        Component: 98369d5a-efb0-9b3f-2127-005056b392af (state: ABSENT (6), host: Unknown, md: 52119754-66b0-c1cd-dd73-2b08c47bf163, ssd: Unknown, note: LSOM object not found,

                                                         votes: 1, proxy component: false)

        Component: 98369d5a-47c6-9e3f-1008-005056b392af (state: ABSENT (6), host: Unknown, md: 52f2f60d-af30-1359-97ae-88d0e7cab6f7, ssd: Unknown, note: LSOM object not found,

                                                         votes: 4, proxy component: false)

        Component: 91da275a-0af5-e15e-fc97-005056b392af (state: ABSENT (6), host: Unknown, md: 52b7da24-ee49-c806-2550-25b12584c0e1, ssd: Unknown, note: LSOM object not found,

                                                         votes: 1, proxy component: false)

        Component: 91da275a-0359-e45e-f646-005056b392af (state: ABSENT (6), host: Unknown, md: 52c56620-f893-9267-9731-bed3fb50767f, ssd: Unknown, note: LSOM object not found,

                                                         votes: 1, proxy component: false)

        Component: 98369d5a-7413-a03f-ac81-005056b392af (state: ABSENT (6), host: 594d4782-6f02-c98b-6e2a-005056b3fb13, md: 525fd058-73d7-2301-1675-5d770f45881e, ssd: 523fce6a-8bad-c3a8-0e04-a2eab1dc1d48,

                                                         votes: 4, usage: 0.0 GB, proxy component: false)

  Disk backing: [vsanDatastore] 6aa5265a-1ea1-5ca9-422e-005056b392af/hzn-vc.mars.lab.adms.local_9.vmdk

    DOM Object: 7da5265a-ca35-b837-5f35-005056b392af (v5, owner: unknown, proxy owner: None, policy: No POLICY entry found in CMMDS)

      RAID_0

        Component: 90369d5a-3c35-fe5f-b175-005056b392af (state: ABSENT (6), host: Unknown, md: 520dca0c-4084-1e4b-0d27-6fc3e58c69bc, ssd: Unknown, note: LSOM object not found,

                                                         votes: 4, proxy component: false)

        Component: 91da275a-3d3b-069c-f7c1-005056b392af (state: ABSENT (6), host: Unknown, md: 522cc47a-c25d-c60f-07ea-c387518831a4, ssd: Unknown, note: LSOM object not found,

                                                         votes: 1, proxy component: false)

        Component: 91da275a-fcf0-079c-cc6a-005056b392af (state: ABSENT (6), host: Unknown, md: 52b8d8af-bdeb-d4ca-e46d-45f8feffee99, ssd: Unknown, note: LSOM object not found,

                                                         votes: 1, proxy component: false)

        Component: 91da275a-8cea-0a9c-6fa8-005056b392af (state: ABSENT (6), host: Unknown, md: 52b7da24-ee49-c806-2550-25b12584c0e1, ssd: Unknown, note: LSOM object not found,

                                                         votes: 1, proxy component: false)

        Component: 91da275a-6640-0c9c-5f1d-005056b392af (state: ABSENT (6), host: 594d488f-4f6b-48e2-9fee-005056b30d27, md: 5261a25d-1909-c635-6432-3f22e4a49b86, ssd: 529f2efe-107c-c754-6f4e-bac24cad216c,

                                                         votes: 3, usage: 0.0 GB, proxy component: false)

        Component: 91da275a-b610-0e9c-7b7c-005056b392af (state: ABSENT (6), host: 594d4721-e1d2-a466-5f38-005056b34157, md: 52037a92-0d7a-a9e1-da61-76aad3d02271, ssd: 5210251c-5a07-06c6-0fba-2da17d51d9aa,

                                                         votes: 3, usage: 0.1 GB, proxy component: false)

  Disk backing: [vsanDatastore] 6aa5265a-1ea1-5ca9-422e-005056b392af/hzn-vc.mars.lab.adms.local.vmdk

    DOM Object: 79a5265a-a5b9-ec3f-8b4f-005056b392af (v5, owner: unknown, proxy owner: None, policy: No POLICY entry found in CMMDS)

      RAID_0

        Component: 95da275a-0330-162e-2f3a-005056b392af (state: ABSENT (6), host: Unknown, md: 522cc47a-c25d-c60f-07ea-c387518831a4, ssd: Unknown, note: LSOM object not found,

                                                         votes: 2, proxy component: false)

        Component: 95da275a-a359-1a2e-4e13-005056b392af (state: ABSENT (6), host: 594d4721-e1d2-a466-5f38-005056b34157, md: 52037a92-0d7a-a9e1-da61-76aad3d02271, ssd: 5210251c-5a07-06c6-0fba-2da17d51d9aa,

                                                         votes: 2, usage: 2.0 GB, proxy component: false)

        Component: a2369d5a-f915-4d5c-8b00-005056b392af (state: ABSENT (6), host: 594d4721-e1d2-a466-5f38-005056b34157, md: 52c8daec-b508-6b77-0ad2-370f42faf555, ssd: 5210251c-5a07-06c6-0fba-2da17d51d9aa,

                                                         votes: 2, usage: 2.0 GB, proxy component: false)

        Component: a2369d5a-58bf-505c-46b7-005056b392af (state: ABSENT (6), host: Unknown, md: 52119754-66b0-c1cd-dd73-2b08c47bf163, ssd: Unknown, note: LSOM object not found,

                                                         votes: 1, proxy component: false)

        Component: 95da275a-6ff8-1e2e-b500-005056b392af (state: ABSENT (6), host: Unknown, md: 52c56620-f893-9267-9731-bed3fb50767f, ssd: Unknown, note: LSOM object not found,

                                                         votes: 1, proxy component: false)

        Component: 95da275a-e714-202e-f898-005056b392af (state: ABSENT (6), host: Unknown, md: 52ed7a36-867f-3bf8-d3a9-d6ce09cec33b, ssd: Unknown, note: LSOM object not found,

                                                         votes: 1, proxy component: false)

  Disk backing: [vsanDatastore] 6aa5265a-1ea1-5ca9-422e-005056b392af/hzn-vc.mars.lab.adms.local_3.vmdk

    DOM Object: 7aa5265a-099e-04da-a3c3-005056b392af (v5, owner: unknown, proxy owner: None, policy: No POLICY entry found in CMMDS)

      RAID_0

        Component: 95da275a-e3a0-79a9-c2de-005056b392af (state: ABSENT (6), host: Unknown, md: 52b8d8af-bdeb-d4ca-e46d-45f8feffee99, ssd: Unknown, note: LSOM object not found,

                                                         votes: 2, proxy component: false)

        Component: 95da275a-e7fb-7da9-14c6-005056b392af (state: ABSENT (6), host: 594d4721-e1d2-a466-5f38-005056b34157, md: 52ede984-77a1-85e5-99cf-721d8144fd57, ssd: 5210251c-5a07-06c6-0fba-2da17d51d9aa,

                                                         votes: 3, usage: 0.5 GB, proxy component: false)

        Component: 8e369d5a-c19c-9413-de90-005056b392af (state: ABSENT (6), host: Unknown, md: 52db9dab-6d58-f114-8c5c-5588eeda8d74, ssd: Unknown, note: LSOM object not found,

                                                         votes: 3, proxy component: false)

        Component: 95da275a-21c7-80a9-41f8-005056b392af (state: ABSENT (6), host: 594d488f-4f6b-48e2-9fee-005056b30d27, md: 5274912e-0b9f-3852-0f33-2fbf15bea171, ssd: 529f2efe-107c-c754-6f4e-bac24cad216c,

                                                         votes: 3, usage: 0.5 GB, proxy component: false)

        Component: 95da275a-218d-82a9-93af-005056b392af (state: ABSENT (6), host: Unknown, md: 52ed7a36-867f-3bf8-d3a9-d6ce09cec33b, ssd: Unknown, note: LSOM object not found,

                                                         votes: 1, proxy component: false)

        Component: 95da275a-ee6c-85a9-9de7-005056b392af (state: ABSENT (6), host: Unknown, md: 52c56620-f893-9267-9731-bed3fb50767f, ssd: Unknown, note: LSOM object not found,

                                                         votes: 1, proxy component: false)

  Disk backing: [vsanDatastore] 6aa5265a-1ea1-5ca9-422e-005056b392af/hzn-vc.mars.lab.adms.local_4.vmdk

    DOM Object: 7ba5265a-9a3e-480d-453e-005056b392af (v5, owner: unknown, proxy owner: None, policy: No POLICY entry found in CMMDS)

      RAID_0

        Component: 9eda275a-78e6-a113-8c46-005056b392af (state: ABSENT (6), host: Unknown, md: 528ed5b2-e6f7-eee4-b687-6315c1d4731b, ssd: Unknown, note: LSOM object not found,

                                                         votes: 2, proxy component: false)

        Component: 9eda275a-c014-a613-00cb-005056b392af (state: ABSENT (6), host: Unknown, md: 52b7da24-ee49-c806-2550-25b12584c0e1, ssd: Unknown, note: LSOM object not found,

                                                         votes: 1, proxy component: false)

        Component: 9eda275a-0972-a713-c67b-005056b392af (state: ABSENT (6), host: Unknown, md: 52c56620-f893-9267-9731-bed3fb50767f, ssd: Unknown, note: LSOM object not found,

                                                         votes: 1, proxy component: false)

        Component: 9eda275a-19b9-a813-5f2b-005056b392af (state: ABSENT (6), host: 594d488f-4f6b-48e2-9fee-005056b30d27, md: 52048511-5131-3411-e6d2-986fb75c7452, ssd: 529f2efe-107c-c754-6f4e-bac24cad216c,

                                                         votes: 4, usage: 1.7 GB, proxy component: false)

        Component: 9eda275a-ee70-aa13-cd81-005056b392af (state: ABSENT (6), host: 594d4721-e1d2-a466-5f38-005056b34157, md: 528e315c-8ed7-99be-a528-a167efc3438d, ssd: 5210251c-5a07-06c6-0fba-2da17d51d9aa,

                                                         votes: 4, usage: 1.7 GB, proxy component: false)

        Component: 9eda275a-e2fc-ab13-23c1-005056b392af (state: ABSENT (6), host: Unknown, md: 52ed7a36-867f-3bf8-d3a9-d6ce09cec33b, ssd: Unknown, note: LSOM object not found,

                                                         votes: 1, proxy component: false)

  Disk backing: [vsanDatastore] 6aa5265a-1ea1-5ca9-422e-005056b392af/hzn-vc.mars.lab.adms.local_2.vmdk

    DOM Object: 7aa5265a-7413-35a6-a1df-005056b392af (v5, owner: unknown, proxy owner: None, policy: No POLICY entry found in CMMDS)

      RAID_0

        Component: 2d6d9d5a-59e4-d495-1e9a-005056b392af (state: ABSENT (6), host: hzn-01.mars.lab.adms.local, md: naa.6000c297e1ea4fde014717abe46ebe14, ssd: naa.6000c297a4864d24785e941576c4d388,

                                                         votes: 4, usage: 2.3 GB, proxy component: false)

        Component: a3da275a-d7b9-0a59-b16f-005056b392af (state: ABSENT (6), host: Unknown, md: 522cc47a-c25d-c60f-07ea-c387518831a4, ssd: Unknown, note: LSOM object not found,

                                                         votes: 1, proxy component: false)

        Component: a3da275a-9252-0c59-54f3-005056b392af (state: ABSENT (6), host: Unknown, md: 52b8d8af-bdeb-d4ca-e46d-45f8feffee99, ssd: Unknown, note: LSOM object not found,

                                                         votes: 1, proxy component: false)

        Component: a3da275a-e7ba-0d59-143d-005056b392af (state: ABSENT (6), host: Unknown, md: 528ed5b2-e6f7-eee4-b687-6315c1d4731b, ssd: Unknown, note: LSOM object not found,

                                                         votes: 1, proxy component: false)

        Component: 9a369d5a-22e9-f381-3cdc-005056b392af (state: ABSENT (6), host: Unknown, md: 52d6ff5e-e98b-ff67-25fd-815472c9fca1, ssd: Unknown, note: LSOM object not found,

                                                         votes: 3, proxy component: false)

        Component: a3da275a-5439-1259-a6d9-005056b392af (state: ABSENT (6), host: 594d4721-e1d2-a466-5f38-005056b34157, md: 52fbaf49-b277-a89c-9050-4e4de201e9da, ssd: 5210251c-5a07-06c6-0fba-2da17d51d9aa,

                                                         votes: 3, usage: 2.3 GB, proxy component: false)

  Disk backing: [vsanDatastore] 6aa5265a-1ea1-5ca9-422e-005056b392af/hzn-vc.mars.lab.adms.local_6.vmdk

    DOM Object: 7ca5265a-d9ef-5b75-c0d3-005056b392af (v5, owner: unknown, proxy owner: None, policy: No POLICY entry found in CMMDS)

      RAID_0

        Component: a5da275a-ebee-51ce-e6c0-005056b392af (state: ABSENT (6), host: Unknown, md: 528ed5b2-e6f7-eee4-b687-6315c1d4731b, ssd: Unknown, note: LSOM object not found,

                                                         votes: 2, proxy component: false)

        Component: a5da275a-4145-56ce-1e6a-005056b392af (state: ABSENT (6), host: 594d4721-e1d2-a466-5f38-005056b34157, md: 52037a92-0d7a-a9e1-da61-76aad3d02271, ssd: 5210251c-5a07-06c6-0fba-2da17d51d9aa,

                                                         votes: 1, usage: 0.2 GB, proxy component: false)

        Component: a5da275a-9493-59ce-b80a-005056b392af (state: ABSENT (6), host: 594d4721-e1d2-a466-5f38-005056b34157, md: 52fbaf49-b277-a89c-9050-4e4de201e9da, ssd: 5210251c-5a07-06c6-0fba-2da17d51d9aa,

                                                         votes: 1, usage: 0.1 GB, proxy component: false)

        Component: a5da275a-98de-5cce-6bc7-005056b392af (state: ABSENT (6), host: 594d4721-e1d2-a466-5f38-005056b34157, md: 52ede984-77a1-85e5-99cf-721d8144fd57, ssd: 5210251c-5a07-06c6-0fba-2da17d51d9aa,

                                                         votes: 1, usage: 0.2 GB, proxy component: false)

        Component: 8b369d5a-1cff-b0c8-e416-005056b392af (state: ABSENT (6), host: Unknown, md: 52008055-88dd-1a4a-d388-63d61263e375, ssd: Unknown, note: LSOM object not found,

                                                         votes: 3, proxy component: false)

        Component: a5da275a-9f24-61ce-8f91-005056b392af (state: ABSENT (6), host: Unknown, md: 52ed7a36-867f-3bf8-d3a9-d6ce09cec33b, ssd: Unknown, note: LSOM object not found,

                                                         votes: 1, proxy component: false)

>

In order to determine why the host locations are unknown I generated a report of the UUIDs of the disks in each ESXi VM:

Procedure to generate and audit disk UUIDs when cmmd-tool failed to find the UUIDs reported.

#following command was run on each host, and the output aggregated.

esxcli vsan storage list | awk '/^naa/{gsub(/\n$/,""); printf("%s ",$0)};/VSAN UUID/{print}'  | sed -r "s/^/$(hostname)\t/g"

#sample output:

hzn-01.mars.lab.adms.local naa.6000c297a4864d24785e941576c4d388 VSAN UUID: 521e3014-5694-91d1-d3ef-4df366c7db93

hzn-01.mars.lab.adms.local naa.6000c29fc40b471801b74fc0d466bb6b VSAN UUID: 523fded3-760f-5da4-0ff4-5e1c13c0145b

hzn-01.mars.lab.adms.local naa.6000c297e1ea4fde014717abe46ebe14 VSAN UUID: 527d1dde-ac68-2ed1-ba8c-c7067cd992e7

hzn-01.mars.lab.adms.local naa.6000c29c4be50d9ea143b6497ccbb2b4 VSAN UUID: 528a6313-83a3-7db2-f385-b98604f51b7a

hzn-01.mars.lab.adms.local naa.6000c29771dd641d19f0b1bf2ba5e3bf VSAN UUID: 528e9277-67c6-58ae-06ab-cd664681a9b7

hzn-01.mars.lab.adms.local naa.6000c293106b2d535c9863f485f4e80b VSAN UUID: 52beaf18-f1dc-57ed-c5d5-9fa180cd7c35

hzn-01.mars.lab.adms.local naa.6000c292a4c2587ed8fbe33b5976ac98 VSAN UUID: 52d99e7b-46e3-6c1f-6fee-510175616b7f

hzn-01.mars.lab.adms.local naa.6000c295efaff2c5333c2ff90c8dafad VSAN UUID: 52ffe1a3-3584-1b3b-a0ed-272ff9706fbc

#The concatenated results were searched for the md values from the output of vsan.vm_object_info

$ for i in $(awk  '/ABSENT/ && /LSOM object not found/{print $9}' ~/hzn-vc-disk_info.txt  | sort -u | sed -r 's/,//g' ) ; do grep $i ~/hzn-clu-md-uuids.txt ; done  | sort

hzn-03.mars.lab.adms.local      naa.6000c29013892f9f520df0f5313b993d    VSAN UUID: 520dca0c-4084-1e4b-0d27-6fc3e58c69bc

hzn-03.mars.lab.adms.local      naa.6000c295700766964171dbfd54255a52    VSAN UUID: 52db9dab-6d58-f114-8c5c-5588eeda8d74

hzn-03.mars.lab.adms.local      naa.6000c2991f6d62712a8c393848841262    VSAN UUID: 52008055-88dd-1a4a-d388-63d61263e375

hzn-03.mars.lab.adms.local      naa.6000c29e9d03e2ab4bbde77477ad9f71    VSAN UUID: 52f2f60d-af30-1359-97ae-88d0e7cab6f7

hzn-03.mars.lab.adms.local      naa.6000c29fca64a36cef628aaf428bbdcd    VSAN UUID: 52d6ff5e-e98b-ff67-25fd-815472c9fca1

hzn-03.mars.lab.adms.local      naa.6000c29fee3d835eb5cca5ce7e117bd1    VSAN UUID: 5248d763-e44e-223c-ccbc-83cac14a3ca3

hzn-04.mars.lab.adms.local      naa.6000c2921933a76b4c545ab5c8ea6710    VSAN UUID: 522cc47a-c25d-c60f-07ea-c387518831a4

hzn-04.mars.lab.adms.local      naa.6000c2947e8abc8a4f3f651eb8cb6ce2    VSAN UUID: 528ed5b2-e6f7-eee4-b687-6315c1d4731b

hzn-04.mars.lab.adms.local      naa.6000c295c0e832e7855cc78e0e8cd8ea    VSAN UUID: 52b8d8af-bdeb-d4ca-e46d-45f8feffee99

hzn-04.mars.lab.adms.local      naa.6000c296f949776eebdbb76717075006    VSAN UUID: 52c56620-f893-9267-9731-bed3fb50767f

hzn-04.mars.lab.adms.local      naa.6000c299eff81a866827d27d84dd9625    VSAN UUID: 52ed7a36-867f-3bf8-d3a9-d6ce09cec33b

hzn-04.mars.lab.adms.local      naa.6000c29e0a44a71c92cf0657e2d1883b    VSAN UUID: 52119754-66b0-c1cd-dd73-2b08c47bf163

hzn-04.mars.lab.adms.local      naa.6000c29e9cb97148c403df1828723e62    VSAN UUID: 52b7da24-ee49-c806-2550-25b12584c0e1

As you can see, the devices are present with the appropriate UUIDs, but the cmmds database is unaware of the relationship (hope I am using the proper terminology).

Any one have any ideas as to how I can get the VMs disks objects to be marked as ACTIVE and aware of its policy?

7 Replies
TheBobkin
Champion
Champion

Hello marsherian​,

I would start by checking the state of hosts hzn-03 and hzn-04 and their disks:

Check (from any host) that they are not in a Decom state (e.g. decomState: 6) as per CMMDS:

# cmmds-tool find -t NODE_DECOM_STATE -f json

Check that all of their disk-groups are in CMMDS and functioning normally:

# esxcli vsan storage list

RVC usually will indicate this also:

> vsan.disks_stats <pathToCluster>

I have noted in nested labs that disk-groups not being mounted on boot due to insufficient memory heap can occur, you should see this in the boot.gz/vmkernel.log.

Are there any healthy active components on those same disks?

Bob

Reply
0 Kudos
marsherian
Enthusiast
Enthusiast

TheBobkin​ thanks for the pointers,

The disks aren't in DECOM, but there are no other objects are present in the two nested hosts:

hzn-03 NODE_DECOM_STATE

[root@hzn-03:~] cmmds-tool find -t NODE_DECOM_STATE -f json

{

"entries":

[

{

   "uuid": "594d4721-e1d2-a466-5f38-005056b34157",

   "owner": "594d4721-e1d2-a466-5f38-005056b34157",

   "health": "Healthy",

   "revision": "7",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "594d4721-bfc1-b285-5465-005056b3d312",

   "owner": "594d4721-bfc1-b285-5465-005056b3d312",

   "health": "Healthy",

   "revision": "4",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "594d4782-6f02-c98b-6e2a-005056b3fb13",

   "owner": "594d4782-6f02-c98b-6e2a-005056b3fb13",

   "health": "Healthy",

   "revision": "7",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "594d46f6-5ef7-3e93-1b2d-005056b3b80e",

   "owner": "594d46f6-5ef7-3e93-1b2d-005056b3b80e",

   "health": "Healthy",

   "revision": "2",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "594d4721-e99e-92bf-0db0-005056b3beea",

   "owner": "594d4721-e99e-92bf-0db0-005056b3beea",

   "health": "Healthy",

   "revision": "4",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "594d488f-4f6b-48e2-9fee-005056b30d27",

   "owner": "594d488f-4f6b-48e2-9fee-005056b30d27",

   "health": "Healthy",

   "revision": "6",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

]

}

hzn-04 NODE_DECOM_STATE

cmmds-tool find -t NODE_DECOM_STATE -f json

{

"entries":

[

{

   "uuid": "594d4721-e1d2-a466-5f38-005056b34157",

   "owner": "594d4721-e1d2-a466-5f38-005056b34157",

   "health": "Healthy",

   "revision": "7",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "594d4721-bfc1-b285-5465-005056b3d312",

   "owner": "594d4721-bfc1-b285-5465-005056b3d312",

   "health": "Healthy",

   "revision": "4",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "594d4782-6f02-c98b-6e2a-005056b3fb13",

   "owner": "594d4782-6f02-c98b-6e2a-005056b3fb13",

   "health": "Healthy",

   "revision": "7",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "594d46f6-5ef7-3e93-1b2d-005056b3b80e",

   "owner": "594d46f6-5ef7-3e93-1b2d-005056b3b80e",

   "health": "Healthy",

   "revision": "2",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "594d4721-e99e-92bf-0db0-005056b3beea",

   "owner": "594d4721-e99e-92bf-0db0-005056b3beea",

   "health": "Healthy",

   "revision": "4",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

,{

   "uuid": "594d488f-4f6b-48e2-9fee-005056b30d27",

   "owner": "594d488f-4f6b-48e2-9fee-005056b30d27",

   "health": "Healthy",

   "revision": "6",

   "type": "NODE_DECOM_STATE",

   "flag": "2",

   "minHostVersion": "0",

   "md5sum": "3c2593056659ee3c9e97039a3eefea8e",

   "valueLen": "80",

   "content": {"decomState": 0, "decomJobType": 0, "decomJobUuid": "00000000-0000-0000-0000-000000000000", "progress": 0, "affObjList": [ ], "errorCode": 0, "updateNum": 0, "majorVersion": 0},

   "errorStr": "(null)"

}

]

}

One item that may be of interest, there are no stats for the vmdks for the broken VM,

hzn-vc VMDK Stats

vsan.vmdk_stats ~h4 ~h4/vms/hzn-vc.mars.lab.adms.local/

2018-03-29 08:44:07 +0000: Fetching general information about cluster

2018-03-29 08:44:08 +0000: Fetching general information about VMs

2018-03-29 08:44:08 +0000: Fetching information about vSAN objects

2018-03-29 08:44:09 +0000: Fetching vSAN stats

2018-03-29 08:44:09 +0000: Done fetching info, drawing table

+----------------------------+-----------+---------------+-----------+---------+----------+

|                            |         Disk Capacity (in GB)         | Read Cache (in GB) |

+----------------------------+-----------+---------------+-----------+---------+----------+

| Disk Name                  | Disk Size | Used Capacity | Data Size | Used    | Reserved |

+----------------------------+-----------+---------------+-----------+---------+----------+

| hzn-vc.mars.lab.adms.local |           |               |           |         |          |

+----------------------------+-----------+---------------+-----------+---------+----------+

hzn-psc VMDK Stats

vsan.vmdk_stats ~h1 ~h1/vms/hzn-psc.mars.lab.adms.local/

2018-03-29 08:45:35 +0000: Fetching general information about cluster

2018-03-29 08:45:36 +0000: Fetching general information about VMs

2018-03-29 08:45:36 +0000: Fetching information about vSAN objects

2018-03-29 08:45:37 +0000: Fetching vSAN stats

2018-03-29 08:45:38 +0000: Done fetching info, drawing table

2018-03-29 08:45:38 +0000: Warning: No RC stats for 72 components

+------------------------------------------------------------------------------------------+-----------+---------------+-----------+---------+----------+

|                                                                                          |         Disk Capacity (in GB)         | Read Cache (in GB) |

+------------------------------------------------------------------------------------------+-----------+---------------+-----------+---------+----------+

| Disk Name                                                                                | Disk Size | Used Capacity | Data Size | Used    | Reserved |

+------------------------------------------------------------------------------------------+-----------+---------------+-----------+---------+----------+

| hzn-psc.mars.lab.adms.local                                                              |           |               |           |         |          |

| [vsanDatastore] 09db275a-83a4-9a93-8002-005056b302d8/hzn-psc.mars.lab.adms.local.vmdk    | 12.0      | 9.0 (0.8x)    | 9.0       | 0.0     | 0.0      |

| [vsanDatastore] 09db275a-83a4-9a93-8002-005056b302d8/hzn-psc.mars.lab.adms.local_1.vmdk  | 1.8       | 1.8 (1.0x)    | 1.8       | 0.0     | 0.0      |

| [vsanDatastore] 09db275a-83a4-9a93-8002-005056b302d8/hzn-psc.mars.lab.adms.local_2.vmdk  | 10.0      | 10.0 (1.0x)   | 10.0      | 0.0     | 0.0      |

| [vsanDatastore] 09db275a-83a4-9a93-8002-005056b302d8/hzn-psc.mars.lab.adms.local_3.vmdk  | 10.0      | 3.6 (0.4x)    | 3.6       | 0.0     | 0.0      |

| [vsanDatastore] 09db275a-83a4-9a93-8002-005056b302d8/hzn-psc.mars.lab.adms.local_4.vmdk  | 10.0      | 10.0 (1.0x)   | 10.0      | 0.0     | 0.0      |

| [vsanDatastore] 09db275a-83a4-9a93-8002-005056b302d8/hzn-psc.mars.lab.adms.local_5.vmdk  | 10.0      | 0.6 (0.1x)    | 0.6       | 0.0     | 0.0      |

| [vsanDatastore] 09db275a-83a4-9a93-8002-005056b302d8/hzn-psc.mars.lab.adms.local_6.vmdk  | 0.0       | 0.0 (1.2x)    | 0.0       | 0.0     | 0.0      |

| [vsanDatastore] 09db275a-83a4-9a93-8002-005056b302d8/hzn-psc.mars.lab.adms.local_7.vmdk  | 0.0       | 0.0 (1.2x)    | 0.0       | 0.0     | 0.0      |

| [vsanDatastore] 09db275a-83a4-9a93-8002-005056b302d8/hzn-psc.mars.lab.adms.local_8.vmdk  | 0.0       | 0.0 (1.2x)    | 0.0       | 0.0     | 0.0      |

| [vsanDatastore] 09db275a-83a4-9a93-8002-005056b302d8/hzn-psc.mars.lab.adms.local_9.vmdk  | 0.0       | 0.0 (1.2x)    | 0.0       | 0.0     | 0.0      |

| [vsanDatastore] 09db275a-83a4-9a93-8002-005056b302d8/hzn-psc.mars.lab.adms.local_10.vmdk | 0.0       | 0.0 (1.2x)    | 0.0       | 0.0     | 0.0      |

| [vsanDatastore] 09db275a-83a4-9a93-8002-005056b302d8/hzn-psc.mars.lab.adms.local_11.vmdk | 0.0       | 0.0 (1.2x)    | 0.0       | 0.0     | 0.0      |

+------------------------------------------------------------------------------------------+-----------+---------------+-----------+---------+----------+

I went and searched all of the cluster members for each of the objects, three hosts have data for the first disk of the VM, but nothing  else:

objtool output for each VMDK and each host

[root@hzn-01:~] awk '/RW/{uuid=gensub(/"|vsan:\/\//,"","g",$NF);print "/usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u "uuid}'  /vmfs/volumes/vsanDatastore/hzn-vc.mars.lab.adms.local/hzn-vc.mars.lab.adms.local_*.vmdk | sh -xs

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7aa5265a-a876-986f-8495-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ea5265a-2d99-7f71-e7c3-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ea5265a-6525-4da8-dd7b-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7aa5265a-7413-35a6-a1df-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7aa5265a-099e-04da-a3c3-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ba5265a-9a3e-480d-453e-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ba5265a-13a3-d947-10fa-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ca5265a-d9ef-5b75-c0d3-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ca5265a-f991-ddba-7c6e-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7da5265a-3dc6-e9fa-93a9-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7da5265a-ca35-b837-5f35-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

[root@hzn-02:~] awk '/RW/{uuid=gensub(/"|vsan:\/\//,"","g",$NF);print "/usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u "uuid}'  /vmfs/volumes/vsanDatastore/hzn-vc.mars.lab.adms.local/hzn-vc.mars.lab.adms.local_*.vmdk | sh -xs

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7aa5265a-a876-986f-8495-005056b392af

Object Attributes --

UUID:7aa5265a-a876-986f-8495-005056b392af

Object type:vsan

Object size:1882193920

User friendly name:(null)

HA metadata:(null)

Allocation type:Thin

Policy:((\"stripeWidth\" i6) (\"cacheReservation\" i0) (\"proportionalCapacity\" i0) (\"hostFailuresToTolerate\" i0) (\"forceProvisioning\" i0) (\"spbmProfileId\" \"aa6d5a82-1c88-45da-85d3-3d74b91a5bad\") (\"spbmProfileGenerationNumber\" l+1) (\"spbmProfileName\" \"vSAN Default Storage Policy\"))

Object class: vdisk

Object capabilities: NONE

Object path: /vmfs/volumes/vsan:5237ef43c7d0883f-7faba8c926063f45/6aa5265a-1ea1-5ca9-422e-005056b392af/hzn-vc.mars.lab.adms.local_1.vmdk

Group uuid: 6aa5265a-1ea1-5ca9-422e-005056b392af

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ea5265a-2d99-7f71-e7c3-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ea5265a-6525-4da8-dd7b-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7aa5265a-7413-35a6-a1df-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7aa5265a-099e-04da-a3c3-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ba5265a-9a3e-480d-453e-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ba5265a-13a3-d947-10fa-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ca5265a-d9ef-5b75-c0d3-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ca5265a-f991-ddba-7c6e-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7da5265a-3dc6-e9fa-93a9-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7da5265a-ca35-b837-5f35-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

[root@hzn-02:~]

[root@hzn-03:~] awk '/RW/{uuid=gensub(/"|vsan:\/\//,"","g",$NF);print "/usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u "uuid}'  /vmfs/volumes/vsanDatastore/hzn-vc.mars.lab.adms.local/hzn-vc.mars.lab.adms.local_*.vmdk | sh -xs

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7aa5265a-a876-986f-8495-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ea5265a-2d99-7f71-e7c3-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ea5265a-6525-4da8-dd7b-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7aa5265a-7413-35a6-a1df-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7aa5265a-099e-04da-a3c3-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ba5265a-9a3e-480d-453e-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ba5265a-13a3-d947-10fa-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ca5265a-d9ef-5b75-c0d3-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ca5265a-f991-ddba-7c6e-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7da5265a-3dc6-e9fa-93a9-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7da5265a-ca35-b837-5f35-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

[root@hzn-03:~]

[root@hzn-04:~] awk '/RW/{uuid=gensub(/"|vsan:\/\//,"","g",$NF);print "/usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u "uuid}'  /vmfs/volumes/vsanDatastore/hzn-vc.mars.lab.adms.local/hzn-vc.mars.lab.adms.local_*.vmdk | sh -xs

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7aa5265a-a876-986f-8495-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ea5265a-2d99-7f71-e7c3-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ea5265a-6525-4da8-dd7b-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7aa5265a-7413-35a6-a1df-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7aa5265a-099e-04da-a3c3-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ba5265a-9a3e-480d-453e-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ba5265a-13a3-d947-10fa-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ca5265a-d9ef-5b75-c0d3-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ca5265a-f991-ddba-7c6e-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7da5265a-3dc6-e9fa-93a9-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7da5265a-ca35-b837-5f35-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

[root@hzn-05:~] awk '/RW/{uuid=gensub(/"|vsan:\/\//,"","g",$NF);print "/usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u "uuid}'  /vmfs/volumes/vsanDatastore/hzn-vc.mars.lab.adms.local/hzn-vc.mars.lab.adms.local_*.vmdk | sh -xs

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7aa5265a-a876-986f-8495-005056b392af

Object Attributes --

UUID:7aa5265a-a876-986f-8495-005056b392af

Object type:vsan

Object size:1882193920

User friendly name:(null)

HA metadata:(null)

Allocation type:Thin

Policy:((\"stripeWidth\" i6) (\"cacheReservation\" i0) (\"proportionalCapacity\" i0) (\"hostFailuresToTolerate\" i0) (\"forceProvisioning\" i0) (\"spbmProfileId\" \"aa6d5a82-1c88-45da-85d3-3d74b91a5bad\") (\"spbmProfileGenerationNumber\" l+1) (\"spbmProfileName\" \"vSAN Default Storage Policy\"))

Object class: vdisk

Object capabilities: NONE

Object path: /vmfs/volumes/vsan:5237ef43c7d0883f-7faba8c926063f45/6aa5265a-1ea1-5ca9-422e-005056b392af/hzn-vc.mars.lab.adms.local_1.vmdk

Group uuid: 6aa5265a-1ea1-5ca9-422e-005056b392af

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ea5265a-2d99-7f71-e7c3-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ea5265a-6525-4da8-dd7b-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7aa5265a-7413-35a6-a1df-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7aa5265a-099e-04da-a3c3-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ba5265a-9a3e-480d-453e-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ba5265a-13a3-d947-10fa-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ca5265a-d9ef-5b75-c0d3-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ca5265a-f991-ddba-7c6e-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7da5265a-3dc6-e9fa-93a9-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7da5265a-ca35-b837-5f35-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

[root@hzn-06:~] awk '/RW/{uuid=gensub(/"|vsan:\/\//,"","g",$NF);print "/usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u "uuid}'  /vmfs/volumes/vsanDatastore/hzn-vc.mars.lab.adms.local/hzn-vc.mars.lab.adms.local_*.vmdk | sh -xs

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7aa5265a-a876-986f-8495-005056b392af

Object Attributes --

UUID:7aa5265a-a876-986f-8495-005056b392af

Object type:vsan

Object size:1882193920

User friendly name:(null)

HA metadata:(null)

Allocation type:Thin

Policy:((\"stripeWidth\" i6) (\"cacheReservation\" i0) (\"proportionalCapacity\" i0) (\"hostFailuresToTolerate\" i0) (\"forceProvisioning\" i0) (\"spbmProfileId\" \"aa6d5a82-1c88-45da-85d3-3d74b91a5bad\") (\"spbmProfileGenerationNumber\" l+1) (\"spbmProfileName\" \"vSAN Default Storage Policy\"))

Object class: vdisk

Object capabilities: NONE

Object path: /vmfs/volumes/vsan:5237ef43c7d0883f-7faba8c926063f45/6aa5265a-1ea1-5ca9-422e-005056b392af/hzn-vc.mars.lab.adms.local_1.vmdk

Group uuid: 6aa5265a-1ea1-5ca9-422e-005056b392af

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ea5265a-2d99-7f71-e7c3-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ea5265a-6525-4da8-dd7b-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7aa5265a-7413-35a6-a1df-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7aa5265a-099e-04da-a3c3-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ba5265a-9a3e-480d-453e-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ba5265a-13a3-d947-10fa-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ca5265a-d9ef-5b75-c0d3-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7ca5265a-f991-ddba-7c6e-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7da5265a-3dc6-e9fa-93a9-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

+ /usr/lib/vmware/osfs/bin/objtool getAttr --bypassDom -u 7da5265a-ca35-b837-5f35-005056b392af

Failed to find a local component

Failed to find a local component to read from

object getAttr error: Failure

[root@hzn-06:~]

It looks like the data just isn't there anymore.

Reply
0 Kudos
TheBobkin
Champion
Champion

Hello marsherian

"The disks aren't in DECOM,"

Well actually this is the node decom states but yes the disks are not available for component placement if the host is in decomState 6.

As an aside, CMMDS is per cluster, not per node thus why you have the same output from running this on hzn-03 and hzn-04 with both showing entries for all nodes in the cluster.

"One item that may be of interest, there are no stats for the vmdks for the broken VM"

This is expected - if the Object is inaccessible then you can't see its stats.

"It looks like the data just isn't there anymore."

As I said in my last comment, there are only really 3 feasible reasons why all components on a particular host would be inaccessible:

1. Node is in a decom state (not the case here but to be sure for sure you can refresh this using #esxcli vsan maintenancemode cancel).

2. Host is network partitioned from the other cluster members (easily verified by checking the membership count using # esxcli vsan cluster get).

3. Disk-group(s) unhealthy, not in CMMDS, not mounted at boot - I reckon this is the most likely cause here and is easily checked using a variety of methods (e.g. Proactive VM creation test, attempting to create Objects that require 5/6 Fault Domains (e.g. a R1, FTT=2 Object or a R5/6 FTT=2 Object), checking using #vdq #esxcli vsan storage list, observing 0 IOs via #esxtop (U or X)).

Bob

marsherian
Enthusiast
Enthusiast

Thanks for clarifying.... understanding who, what, when, and how the objects are stored is a dark art when one is peeling back the layers of the onion... very easy to get the wrong end of the stick.

Cluster network partitioning was the initial cause of the problem, all hosts-vms are present and connectable now.

Alas, the VM creation tests, etc are not possible since the missing VM is the vCenter VM.

I've been running all of the commands per-host to see if there is some discrepancy in their opinion of the state.

Oddly, vsan.cmmds_find from rvc (connected to each host individually) Shows the DOM_OBJECTS as healthy for each of the disks. It's a rather lengthy file so I've attached it below. The owner denoted for most objects is hzn-02 and hzn-05 for the remainder:

$ awk '/Healthy/{print $8}' ~/hzn-vc-cmmds_find-perHost.txt  | sort | uniq -c  | sort -nr

     92 594d4721-e1d2-a466-5f38-005056b34157

     23 hzn-02.mars.lab.adms.local

      8 594d4782-6f02-c98b-6e2a-005056b3fb13

      2 hzn-05.mars.lab.adms.local

Reply
0 Kudos
TheBobkin
Champion
Champion

Hello marsherian​,

The only Storage Policy appearing here is 'Default Virtual SAN Storage Policy' that has been modified to be FTT=0 and Stripe-Width=6 - what this means is that each DOM-OBJECT is striped over 6 LSOM-OBJECTs (AKA components), inability to access even one of these components will result in the DOM-OBJECT being inaccessible.

Of the output you posted, there are currently 30 state:5 (ACTIVE) components and 330 state:6 (ABSENT) components.

For every DOM-OBJECT there is a CONFIG_STATUS entry that informs of the current availability of the DOM-OBJECT, in your case this is not just the vCenter/PSC vmdk Objects but numerous others that are inaccessible. The term 'Healthy' here is actually a reference to the state of the CONFIG_STATUS itself, not of the Objects it describes.

In the output you posted there are only 4 Objects which are state: 7 and thus healthy, the rest are 12 and 14s which mean the Objects are inaccessible and non-recoverable in their current state:

Line 151: | 2 | CONFIG_STATUS | 79a5265a-a5b9-ec3f-8b4f-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>12, "CSN"=>199, "SCSN"=>311}                              |

Line 250: | 3 | CONFIG_STATUS | 7aa5265a-a876-986f-8495-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>7, "CSN"=>221, "SCSN"=>5673}                              |

Line 341: | 2 | CONFIG_STATUS | 7ea5265a-2d99-7f71-e7c3-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>14, "CSN"=>210, "SCSN"=>1961}                             |

Line 432: | 2 | CONFIG_STATUS | 7ea5265a-6525-4da8-dd7b-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>14, "CSN"=>244, "SCSN"=>356}                              |

Line 524: | 2 | CONFIG_STATUS | 7aa5265a-7413-35a6-a1df-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>14, "CSN"=>153, "SCSN"=>167}                              |

Line 617: | 2 | CONFIG_STATUS | 7aa5265a-099e-04da-a3c3-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>12, "CSN"=>161, "SCSN"=>171}                              |

Line 709: | 2 | CONFIG_STATUS | 7ba5265a-9a3e-480d-453e-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>14, "CSN"=>204, "SCSN"=>484}                              |

Line 799: | 2 | CONFIG_STATUS | 7ba5265a-13a3-d947-10fa-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>12, "CSN"=>148, "SCSN"=>159}                              |

Line 891: | 2 | CONFIG_STATUS | 7ca5265a-d9ef-5b75-c0d3-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>12, "CSN"=>154, "SCSN"=>163}                              |

Line 982: | 2 | CONFIG_STATUS | 7ca5265a-f991-ddba-7c6e-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>12, "CSN"=>140, "SCSN"=>148}                              |

Line 1073: | 2 | CONFIG_STATUS | 7da5265a-3dc6-e9fa-93a9-005056b392af | hzn-05.mars.lab.adms.local | Healthy | {"state"=>12, "CSN"=>105, "SCSN"=>116}                              |

Line 1165: | 2 | CONFIG_STATUS | 7da5265a-ca35-b837-5f35-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>12, "CSN"=>167, "SCSN"=>177}                              |

Line 1257: | 2 | CONFIG_STATUS | 79a5265a-a5b9-ec3f-8b4f-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>12, "CSN"=>199, "SCSN"=>311}                              |

Line 1356: | 3 | CONFIG_STATUS | 7aa5265a-a876-986f-8495-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>7, "CSN"=>221, "SCSN"=>5673}                              |

Line 1447: | 2 | CONFIG_STATUS | 7ea5265a-2d99-7f71-e7c3-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>14, "CSN"=>210, "SCSN"=>1961}                             |

Line 1538: | 2 | CONFIG_STATUS | 7ea5265a-6525-4da8-dd7b-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>14, "CSN"=>244, "SCSN"=>356}                              |

Line 1630: | 2 | CONFIG_STATUS | 7aa5265a-7413-35a6-a1df-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>14, "CSN"=>153, "SCSN"=>167}                              |

Line 1723: | 2 | CONFIG_STATUS | 7aa5265a-099e-04da-a3c3-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>12, "CSN"=>161, "SCSN"=>171}                              |

Line 1815: | 2 | CONFIG_STATUS | 7ba5265a-9a3e-480d-453e-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>14, "CSN"=>204, "SCSN"=>484}                              |

Line 1905: | 2 | CONFIG_STATUS | 7ba5265a-13a3-d947-10fa-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>12, "CSN"=>148, "SCSN"=>159}                              |

Line 1997: | 2 | CONFIG_STATUS | 7ca5265a-d9ef-5b75-c0d3-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>12, "CSN"=>154, "SCSN"=>163}                              |

Line 2088: | 2 | CONFIG_STATUS | 7ca5265a-f991-ddba-7c6e-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>12, "CSN"=>140, "SCSN"=>148}                              |

Line 2179: | 2 | CONFIG_STATUS | 7da5265a-3dc6-e9fa-93a9-005056b392af | 594d4782-6f02-c98b-6e2a-005056b3fb13 | Healthy | {"state"=>12, "CSN"=>105, "SCSN"=>116}                              |

Line 2271: | 2 | CONFIG_STATUS | 7da5265a-ca35-b837-5f35-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>12, "CSN"=>167, "SCSN"=>177}                              |

Line 2363: | 2 | CONFIG_STATUS | 79a5265a-a5b9-ec3f-8b4f-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>12, "CSN"=>199, "SCSN"=>311}                              |

Line 2462: | 3 | CONFIG_STATUS | 7aa5265a-a876-986f-8495-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>7, "CSN"=>221, "SCSN"=>5673}                              |

Line 2553: | 2 | CONFIG_STATUS | 7ea5265a-2d99-7f71-e7c3-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>14, "CSN"=>210, "SCSN"=>1961}                             |

Line 2644: | 2 | CONFIG_STATUS | 7ea5265a-6525-4da8-dd7b-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>14, "CSN"=>244, "SCSN"=>356}                              |

Line 2736: | 2 | CONFIG_STATUS | 7aa5265a-7413-35a6-a1df-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>14, "CSN"=>153, "SCSN"=>167}                              |

Line 2829: | 2 | CONFIG_STATUS | 7aa5265a-099e-04da-a3c3-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>12, "CSN"=>161, "SCSN"=>171}                              |

Line 2921: | 2 | CONFIG_STATUS | 7ba5265a-9a3e-480d-453e-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>14, "CSN"=>204, "SCSN"=>484}                              |

Line 3011: | 2 | CONFIG_STATUS | 7ba5265a-13a3-d947-10fa-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>12, "CSN"=>148, "SCSN"=>159}                              |

Line 3103: | 2 | CONFIG_STATUS | 7ca5265a-d9ef-5b75-c0d3-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>12, "CSN"=>154, "SCSN"=>163}                              |

Line 3194: | 2 | CONFIG_STATUS | 7ca5265a-f991-ddba-7c6e-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>12, "CSN"=>140, "SCSN"=>148}                              |

Line 3285: | 2 | CONFIG_STATUS | 7da5265a-3dc6-e9fa-93a9-005056b392af | 594d4782-6f02-c98b-6e2a-005056b3fb13 | Healthy | {"state"=>12, "CSN"=>105, "SCSN"=>116}                              |

Line 3377: | 2 | CONFIG_STATUS | 7da5265a-ca35-b837-5f35-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>12, "CSN"=>167, "SCSN"=>177}                              |

Line 3469: | 2 | CONFIG_STATUS | 79a5265a-a5b9-ec3f-8b4f-005056b392af | hzn-02.mars.lab.adms.local | Healthy | {"state"=>12, "CSN"=>199, "SCSN"=>311}                              |

Line 3568: | 3 | CONFIG_STATUS | 7aa5265a-a876-986f-8495-005056b392af | hzn-02.mars.lab.adms.local | Healthy | {"state"=>7, "CSN"=>221, "SCSN"=>5673}                              |

Line 3659: | 2 | CONFIG_STATUS | 7ea5265a-2d99-7f71-e7c3-005056b392af | hzn-02.mars.lab.adms.local | Healthy | {"state"=>14, "CSN"=>210, "SCSN"=>1961}                             |

Line 3750: | 2 | CONFIG_STATUS | 7ea5265a-6525-4da8-dd7b-005056b392af | hzn-02.mars.lab.adms.local | Healthy | {"state"=>14, "CSN"=>244, "SCSN"=>356}                              |

Line 3842: | 2 | CONFIG_STATUS | 7aa5265a-7413-35a6-a1df-005056b392af | hzn-02.mars.lab.adms.local | Healthy | {"state"=>14, "CSN"=>153, "SCSN"=>167}                              |

Line 3935: | 2 | CONFIG_STATUS | 7aa5265a-099e-04da-a3c3-005056b392af | hzn-02.mars.lab.adms.local | Healthy | {"state"=>12, "CSN"=>161, "SCSN"=>171}                              |

Line 4027: | 2 | CONFIG_STATUS | 7ba5265a-9a3e-480d-453e-005056b392af | hzn-02.mars.lab.adms.local | Healthy | {"state"=>14, "CSN"=>204, "SCSN"=>484}                              |

Line 4117: | 2 | CONFIG_STATUS | 7ba5265a-13a3-d947-10fa-005056b392af | hzn-02.mars.lab.adms.local | Healthy | {"state"=>12, "CSN"=>148, "SCSN"=>159}                              |

Line 4209: | 2 | CONFIG_STATUS | 7ca5265a-d9ef-5b75-c0d3-005056b392af | hzn-02.mars.lab.adms.local | Healthy | {"state"=>12, "CSN"=>154, "SCSN"=>163}                              |

Line 4300: | 2 | CONFIG_STATUS | 7ca5265a-f991-ddba-7c6e-005056b392af | hzn-02.mars.lab.adms.local | Healthy | {"state"=>12, "CSN"=>140, "SCSN"=>148}                              |

Line 4391: | 2 | CONFIG_STATUS | 7da5265a-3dc6-e9fa-93a9-005056b392af | 594d4782-6f02-c98b-6e2a-005056b3fb13 | Healthy | {"state"=>12, "CSN"=>105, "SCSN"=>116}                              |

Line 4483: | 2 | CONFIG_STATUS | 7da5265a-ca35-b837-5f35-005056b392af | hzn-02.mars.lab.adms.local | Healthy | {"state"=>12, "CSN"=>167, "SCSN"=>177}                              |

Line 4575: | 2 | CONFIG_STATUS | 79a5265a-a5b9-ec3f-8b4f-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>12, "CSN"=>199, "SCSN"=>311}                              |

Line 4674: | 3 | CONFIG_STATUS | 7aa5265a-a876-986f-8495-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>7, "CSN"=>221, "SCSN"=>5673}                              |

Line 4765: | 2 | CONFIG_STATUS | 7ea5265a-2d99-7f71-e7c3-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>14, "CSN"=>210, "SCSN"=>1961}                             |

Line 4856: | 2 | CONFIG_STATUS | 7ea5265a-6525-4da8-dd7b-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>14, "CSN"=>244, "SCSN"=>356}                              |

Line 4948: | 2 | CONFIG_STATUS | 7aa5265a-7413-35a6-a1df-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>14, "CSN"=>153, "SCSN"=>167}                              |

Line 5041: | 2 | CONFIG_STATUS | 7aa5265a-099e-04da-a3c3-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>12, "CSN"=>161, "SCSN"=>171}                              |

Line 5133: | 2 | CONFIG_STATUS | 7ba5265a-9a3e-480d-453e-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>14, "CSN"=>204, "SCSN"=>484}                              |

Line 5223: | 2 | CONFIG_STATUS | 7ba5265a-13a3-d947-10fa-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>12, "CSN"=>148, "SCSN"=>159}                              |

Line 5315: | 2 | CONFIG_STATUS | 7ca5265a-d9ef-5b75-c0d3-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>12, "CSN"=>154, "SCSN"=>163}                              |

Line 5406: | 2 | CONFIG_STATUS | 7ca5265a-f991-ddba-7c6e-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>12, "CSN"=>140, "SCSN"=>148}                              |

Line 5497: | 2 | CONFIG_STATUS | 7da5265a-3dc6-e9fa-93a9-005056b392af | 594d4782-6f02-c98b-6e2a-005056b3fb13 | Healthy | {"state"=>12, "CSN"=>105, "SCSN"=>116}                              |

Line 5589: | 2 | CONFIG_STATUS | 7da5265a-ca35-b837-5f35-005056b392af | 594d4721-e1d2-a466-5f38-005056b34157 | Healthy | {"state"=>12, "CSN"=>167, "SCSN"=>177}                      

"Cluster network partitioning was the initial cause of the problem, all hosts-vms are present and connectable now."

How are you determining that they are now clustered?

Do all hosts state that cluster membership = 6 when checked using # esxcli vsan cluster get?

Is the cluster revision count incrementing? (indicating membership flapping)

Is this a vSAN 6.6 cluster and using Unicast or Multicast?

Is traffic being passed both ways via the vSAN-enabled vmk on each node?

[Host1]# vmkping -I vmkX <IP of vSAN-VMK on Host2>

I note that there are only 3 nodes in this case that have state:5 components (and as a result seemingly accessible Objects):

"faultDomainId"=>"594d4721-e1d2-a466-5f38-005056b34157",

"faultDomainId"=>"594d488f-4f6b-48e2-9fee-005056b30d27

"faultDomainId"=>"594d4782-6f02-c98b-6e2a-005056b3fb13

This would indicate these hosts can communicate with one another but not the others.

"Alas, the VM creation tests, etc are not possible since the missing VM is the vCenter VM"

Sorry there, lost track of your initial point that vC was down - posting RVC output made my head assume it was up (yes, using another vC)

Bob

Reply
0 Kudos
marsherian
Enthusiast
Enthusiast

First of all, thank you very much... you've educated me greatly on the internals of vSAN and cleared a lot up!

My policy choices in this environment were made with performance, and not resiliency in mind when we were short on space in the parent vSAN... when we expanded, I never adjusted!

I really appreciate you explanation of the CSN states, and that the report status of "Healthy" is a bit misleading. Might you be able to point me at docs on the state definitions?

How are you determining that they are now clustered?

->via esxcli vsan cluster get.

Do all hosts state that cluster membership = 6 when checked using # esxcli vsan cluster get?

-> They do indeed.

Is the cluster revision count incrementing? (indicating membership flapping)

-> It was whilst I was debugging the comms between hosts. (The maintenance we'd done was re-wiring the network, and I failed to properly set the MTU on the NSX network where these nested environments sit, which lead to the flappy connectivity in the nested environment.)

Is this a vSAN 6.6 cluster and using Unicast or Multicast?

6.6.1 in Unicast.

Is traffic being passed both ways via the vSAN-enabled vmk on each node?

I have validated full daisy-chains of ping. #[hostn] vmking -I xmkX [vsan-vmknic-host(n+{1,6})]

I think my poor striping and FTT choices months ago have bitten me, obviously. Why the data should have gone AWOL on the VMDK of the ESXi hosts in the cluster is a bit concerning though.

I'd never have that FTT/striping in a real environment, but understanding how this broke will assist in advising customers on proper policies in real life.

Again, thank you so much for your participation in this, it's been invaluable.

Reply
0 Kudos
TheBobkin
Champion
Champion

Hello marsherian​,

"First of all, thank you very much... you've educated me greatly on the internals of vSAN and cleared a lot up!"

Cheers - this is fairly similar stuff to my day-job (GSS) but I care enough about this product to help out in my own time where possible.

"I really appreciate you explanation of the CSN states, and that the report status of "Healthy" is a bit misleading. Might you be able to point me at docs on the state definitions?"

I don't believe there are any (official) external documentation relating to these nor CMMDS-Tool in general, likely due to the potential for misuse that could result in people breaking their data.

The basic rule of thumb for DOM-Object availability/recoverability is: even number=bad, odd-number=good (13 being a gray-area here as this means ACTIVE data component but no quorum, so creating a working  Object from this requires some minor hackery).

While cluster membership showing all members indicates these *are* clustered, can you double-check the Unicast agent list on all hosts and ensure all of them have all other hosts entries correct? (and marked as Unicast=1)

"I think my poor striping and FTT choices months ago have bitten me, obviously. Why the data should have gone AWOL on the VMDK of the ESXi hosts in the cluster is a bit concerning though."

vSAN can get a bit 'quirky' when you nest it, thus why this is unsupported. I have seen some odd things in nested labs but never had any issues with Data availability despite my best efforts to break it.

What are the 'disks' of these nested hosts backed by? VMDK Objects on another vSAN? If so - are these Objects healthy?

I will PM you a few last-ditch efforts that might help here.

Bob

Reply
0 Kudos