JeremeyWise
Enthusiast
Enthusiast

vSAN - Update ESXi Configuration - Won't clear "vCenter state is authoratative"

Cluster after move and repair of VSAN has error of "Update ESXI Configuration"   for update ESXI Configuration.  All three hosts note : 

 

Last Update by VC

Different VC (60f584a0-1d04-3c42-154b-a0423f377a7e)

 

Running wizard for vCenter to take over ownership of vSAN completes but never clears error.

 

My assumption is the UUID "60f584a0-1d04-3c42-154b-a0423f377a7e"  was the UUID of the temp vCenter I used to repair the cluster.  But not sure how to get it to let go and revert under this vCenter.   Typical vCenter lack of detail logging to further root cause things 😛

 

 


Nerd needing coffee
0 Kudos
2 Replies
JeremeyWise
Enthusiast
Enthusiast

<poke on this thread>

 

Been working on other fires.. back to this topic:

 

esxi_vCenter_not_Authoratative.png

Back working on this project. 

 

vCenter state is authoritativeSILENCE ALERT
UPDATE ESXI CONFIGURATION

 

Run update :  event task list shows success but hosts still listed as needing to be set to authoritative.

 

tail /var/ syslog.log

2021-09-08T20:33:21Z backup.sh[2381243]: Creating ConfigStore Backup
2021-09-08T20:33:21Z configStoreBackup: ConfigStore backup completed successfully
2021-09-08T20:33:21Z configStoreBackup: ConfigStore backup completed with rc = 101
2021-09-08T20:33:21Z configStoreBackup: ConfigStore backup completed successfully
2021-09-08T20:33:21Z configStoreBackup: ConfigStore backup completed with rc = 101
2021-09-08T20:33:22.575Z ConfigStore[2381360]: Log for ConfigStore version=1.0 build=build-17867351 option=Release
2021-09-08T20:33:22.575Z ConfigStore[2381360]: Could not expand environment variable HOME.
2021-09-08T20:33:22.575Z ConfigStore[2381360]: Could not expand environment variable HOME.
2021-09-08T20:33:22.575Z ConfigStore[2381360]: DictionaryLoad: Cannot open file "/usr/lib/vmware/config": No such file or directory.
2021-09-08T20:33:22.575Z ConfigStore[2381360]: DictionaryLoad: Cannot open file "~/.vmware/config": No such file or directory.
2021-09-08T20:33:22.575Z ConfigStore[2381360]: DictionaryLoad: Cannot open file "~/.vmware/preferences": No such file or directory.
2021-09-08T20:33:22.575Z ConfigStore[2381360]: Switching to VMware syslog extensions
2021-09-08T20:33:23Z backup.sh[2381243]: Locking esx.conf
2021-09-08T20:33:23Z backup.sh[2381243]: Creating archive
2021-09-08T20:33:23Z backup.sh[2381243]: Unlocking esx.conf
2021-09-08T20:33:24Z backup.sh[2381243]: Using key ID 5c4c03bf-c118-48e3-a03c-1d34080191a3 to encrypt

[root@thor:~] tail -f /var/log/vmkernel.log

2021-09-08T20:59:00.173Z cpu25:2098895 opID=1afa4246)World: 11986: VC opID 11232485-W773-069e maps to vmkernel opID 1afa4246
2021-09-08T20:59:00.173Z cpu25:2098895 opID=1afa4246)Config: 716: "ClomRepairDelay" = 60, Old Value: 60, (Status: 0x0)
2021-09-08T20:59:00.189Z cpu25:2098895 opID=1afa4246)Config: 716: "DOMOwnerForceWarmCache" = 0, Old Value: 0, (Status: 0x0)
2021-09-08T20:59:00.204Z cpu25:2098895 opID=1afa4246)Config: 716: "SwapThickProvisionDisabled" = 1, Old Value: 1, (Status: 0x0)
2021-09-08T20:59:00.212Z cpu25:2098895 opID=1afa4246)Config: 716: "goto11" = 0, Old Value: 0, (Status: 0x0)
2021-09-08T20:59:00.213Z cpu25:2098895 opID=1afa4246)Config: 716: "ClomBgProRebalanceEnabled" = 1, Old Value: 1, (Status: 0x0)
2021-09-08T20:59:00.220Z cpu25:2098895 opID=1afa4246)Config: 716: "ClomBgProRebalanceThreshold" = 30, Old Value: 30, (Status: 0x0)
2021-09-08T20:59:00.229Z cpu25:2098895 opID=1afa4246)Config: 716: "HostFailureThresholdState" = 0, Old Value: 0, (Status: 0x0)
2021-09-08T20:59:00.233Z cpu25:2098895 opID=1afa4246)Config: 716: "InternalOpThresholdState" = 0, Old Value: 0, (Status: 0x0)
2021-09-08T20:59:00.239Z cpu25:2098895 opID=1afa4246)RDT: RDTVSISetEnableRdma:2519: Rdma already disabled. Nothing to do.
2021-09-08T20:59:00.242Z cpu25:2098895 opID=1afa4246)Config: 716: "DedupScope" = 0, Old Value: 0, (Status: 0x0)
2021-09-08T20:59:00.250Z cpu25:2098895 opID=1afa4246)Config: 716: "GuestUnmap" = 0, Old Value: 0, (Status: 0x0)
2021-09-08T20:59:00.253Z cpu25:2098895 opID=1afa4246)Config: 716: "DomCompResyncThrottle" = 0, Old Value: 0, (Status: 0x0)
2021-09-08T20:59:00.565Z cpu26:2098902 opID=a511cdc6)World: 11986: VC opID 112324a5-06a5 maps to vmkernel opID a511cdc6
2021-09-08T20:59:00.565Z cpu26:2098902 opID=a511cdc6)RDT: RDTVSIGetSubClusterSecCfgMode:4774: Current security mode 0, state 0
2021-09-08T20:59:07.272Z cpu23:2097247)------------ ------------ ------------ ------------ ------------ ------------------------------
2021-09-08T20:59:07.272Z cpu23:2097247) min,KB max,KB minLimit,KB eMin,KB rMinPeak,KB name
2021-09-08T20:59:07.272Z cpu23:2097247)------------ ------------ ------------ ------------ ------------ ------------------------------
2021-09-08T20:59:07.272Z cpu23:2097247) 204800 204800 -1 204800 204800 host/vim/vmvisor/config-file-tracker
2021-09-08T20:59:07.272Z cpu23:2097247)------------ ------------ ------------ ------------ ------------ ------------------------------
2021-09-08T20:59:07.272Z cpu23:2097247) 0 -1 -1 1092 72312 python.2383871
2021-09-08T20:59:07.272Z cpu23:2097247) 0 -1 -1 132 132 uwWorldStore.2383871
2021-09-08T20:59:07.272Z cpu23:2097247) 0 -1 -1 136 136 worldGroup.2383871
2021-09-08T20:59:07.272Z cpu23:2097247) 0 -1 -1 0 70692 uw.2383871
2021-09-08T20:59:07.272Z cpu23:2097247) 0 -1 -1 136 136 vsiHeap.2383871
2021-09-08T20:59:07.272Z cpu23:2097247) 0 -1 -1 264 792 pt.2383871
2021-09-08T20:59:07.272Z cpu23:2097247) 0 -1 -1 288 288 cartelheap.2383871
2021-09-08T20:59:07.272Z cpu23:2097247) 0 -1 -1 0 0 uwshmempt.2383871
2021-09-08T20:59:07.272Z cpu23:2097247) 0 -1 -1 136 136 uwAsyncRemapHeap.2383871
2021-09-08T20:59:07.272Z cpu23:2097247)------------ ------------ ------------ ------------ ------------ ------------------------------
2021-09-08T20:59:09.018Z cpu14:2098904 opID=aaad51e0)World: 11986: VC opID 112324fa-06b4 maps to vmkernel opID aaad51e0
2021-09-08T20:59:09.018Z cpu14:2098904 opID=aaad51e0)RDT: RDTVSIGetSubClusterSecCfgMode:4774: Current security mode 0, state 0

<<< not looking like much >>>

 

Those are the outputs from log files from one server while I run vCenter upgrade


Nerd needing coffee
0 Kudos
JeremeyWise
Enthusiast
Enthusiast

< Update> 

 

I put another new 1TB SSD into the server to see if it was something with the disk

 

Task Name
 Add disks to the vSAN cluster
Status
 A general system error occurred: Failed to reserve disk t10.ATA_____WDC__WDS100T2B0B2D00YS70_________________203612801631________ with exception: Failed to reserve disk t10.ATA_____WDC__WDS100T2B0B2D00YS70_________________203612801631________ with exception: Reserve failed with error code: -1
Initiator
 com.vmware.vsan.health
 
 
I tried to track down from other postings..  almost like the disk has some leftovers from previous vSAN on it and just can't figure out how to wipe disk 
 
 
[root@odin:~] esxcfg-scsidevs -c |grep 183533804564
t10.ATA_____WDC_WDS100T2B0B2D00YS70__________________183533804564________ Direct-Access /vmfs/devices/disks/t10.ATA_____WDC_WDS100T2B0B2D00YS70__________________183533804564________ 953869MB HPP Local ATA Disk (t10.ATA_____WDC_WDS100T2B0B2D00YS70__________________183533804564________)
[root@odin:~] esxcfg-scsidevs -ld t10.ATA_____WDC_WDS100T2B0B2D00YS70__________________183533804564________
t10.ATA_____WDC_WDS100T2B0B2D00YS70__________________183533804564________
Device Type: Direct-Access
Size: 953869 MB
Display Name: Local ATA Disk (t10.ATA_____WDC_WDS100T2B0B2D00YS70__________________183533804564________)
Multipath Plugin: HPP
Console Device: /vmfs/devices/disks/t10.ATA_____WDC_WDS100T2B0B2D00YS70__________________183533804564________
Devfs Path: /vmfs/devices/disks/t10.ATA_____WDC_WDS100T2B0B2D00YS70__________________183533804564________
Vendor: ATA Model: WDC WDS100T2B0B- Revis: 90WD
SCSI Level: 5 Is Pseudo: false Status: on
Is RDM Capable: false Is Removable: false
Is Local: true Is SSD: true
Other Names:
vml.01000000003138333533333830343536342020202020202020574443205744
VAAI Status: unsupported
[root@odin:~] partedUtil getptbl /vmfs/devices/disks/t10.ATA_____WDC_WDS100T2B0B2D00YS70__________________183533804564________
msdos
121601 255 63 1953525168

[root@odin:~] esxcfg-mpath -ld t10.ATA_____WDC_WDS100T2B0B2D00YS70__________________183533804564________
sata.vmhba0-sata.0:2-t10.ATA_____WDC_WDS100T2B0B2D00YS70__________________183533804564________
Runtime Name: vmhba0:C0:T2:L0
Device: t10.ATA_____WDC_WDS100T2B0B2D00YS70__________________183533804564________
Device Display Name: Local ATA Disk (t10.ATA_____WDC_WDS100T2B0B2D00YS70__________________183533804564________)
Adapter: vmhba0 Channel: 0 Target: 2 LUN: 0
Adapter Identifier: sata.vmhba0
Target Identifier: sata.0:2
Plugin: HPP
State: active
Transport: sata

[root@odin:~] dd if=/dev/zero of=/vmfs/devices/disks/t10.ATA_____WDC_WDS100T2B0B2D00YS70__________________183533804564________

dd: can't open '/vmfs/devices/disks/t10.ATA_____WDC_WDS100T2B0B2D00YS70__________________183533804564________': Function not implemented
[root@odin:~]


But .. back to esxi hamstrug from doing lower level wipefs or dd etc.. to remove data from disk
 
 
I have another server .. exact same motherboard, RAID controller,  firmware,  disk drives,  which is working without issue.  So it has to be some kind of configuration delta.
 
 
 

Nerd needing coffee
0 Kudos