Hi, my vsphere-csi-driver has a problem with volume metadata updating. But it looks like problem with first try and another one is working well. The first try ends with this error: CNS: Failed to update disk metadata:N5Vmomi5Fault11SystemError9ExceptionE
First try:
2021-08-09T07:41:59.773Z info vsanvcmgmtd[35922] [vSAN@6876 sub=AdapterServer opId=098314c7] Invoking 'updateVolumeMetadata' on 'cns-volume-manager', session '5284b747-25c2-9185-3b4b-0e72757eda1a'
2021-08-09T07:41:59.773Z info vsanvcmgmtd[35922] [vSAN@6876 sub=FcdService opId=098314c7] CNS: RetrieveVStorageObjectInfo disk begin for volumeId: 76369948-665b-4e23-9e59-59cffaa9d965
2021-08-09T07:41:59.809Z info vsanvcmgmtd[35922] [vSAN@6876 sub=FcdService opId=098314c7] CNS: RetrieveVStorageObjectInfo end
2021-08-09T07:41:59.815Z verbose vsanvcmgmtd[35922] [vSAN@6876 sub=PyBackedMO opId=098314c7] Enter vim.cns.VolumeManager.updateVolumeMetadata, Pending: 2 (5284b747-25c2-9185-3b4b-0e72757eda1a)
2021-08-09T07:41:59.815Z info vsanvcmgmtd[35922] [vSAN@6876 sub=CnsVolMgr opId=098314c7] CNS: UpdateVolumeMetadata with spec: (vim.cns.VolumeMetadataUpdateSpec) [
--> (vim.cns.VolumeMetadataUpdateSpec) {
--> volumeId = (vim.cns.VolumeId) {
--> id = "76369948-665b-4e23-9e59-59cffaa9d965"
--> },
--> metadata = (vim.cns.VolumeMetadata) {
--> containerCluster = (vim.cns.ContainerCluster) {
--> clusterType = "KUBERNETES",
--> clusterId = "K8S-DEV",
--> vSphereUser = "VSPHERE.LOCAL\k8s-dev",
--> clusterFlavor = "VANILLA"
--> },
--> entityMetadata = (vim.cns.EntityMetadata) [
--> (vim.cns.KubernetesEntityMetadata) {
--> entityName = "postgres",
--> labels = (vim.KeyValue) [
--> (vim.KeyValue) {
--> key = "app",
--> value = "postgres"
--> },
--> (vim.KeyValue) {
--> key = "app.kubernetes.io/managed-by",
--> value = "Helm"
--> },
--> (vim.KeyValue) {
--> key = "chart",
--> value = "postgres-12"
--> },
--> (vim.KeyValue) {
--> key = "heritage",
--> value = "Helm"
--> },
--> (vim.KeyValue) {
--> key = "release",
--> value = "radim"
--> }
--> ],
--> clusterId = "K8S-DEV",
--> entityType = "PERSISTENT_VOLUME_CLAIM",
--> namespace = "default",
--> referredEntity = (vim.cns.KubernetesEntityReference) [
--> (vim.cns.KubernetesEntityReference) {
--> entityType = "PERSISTENT_VOLUME",
--> entityName = "pvc-f14d2028-fbe4-4ce7-aedd-0faf8d81db6e",
--> clusterId = "K8S-DEV",
--> }
--> ],
--> }
--> ],
--> containerClusterArray = (vim.cns.ContainerCluster) [
--> (vim.cns.ContainerCluster) {
--
2021-08-09T07:41:59.821Z info vsanvcmgmtd[35922] [vSAN@6876 sub=CnsTask opId=098314c7] A com.vmware.cns.tasks.updatevolume task is created: task-1171277
2021-08-09T07:41:59.821Z verbose vsanvcmgmtd[35922] [vSAN@6876 sub=PyBackedMO opId=098314c7] Exit vim.cns.VolumeManager.updateVolumeMetadata (5 ms)
2021-08-09T07:41:59.821Z info vsanvcmgmtd[35922] [vSAN@6876 sub=VsanTaskSvc opId=098314c7] ADD task 'task-1171277', total: 4
2021-08-09T07:41:59.821Z info vsanvcmgmtd[35922] [vSAN@6876 sub=AdapterServer opId=098314c7] Finished 'updateVolumeMetadata' on 'cns-volume-manager' (48 ms): done
2021-08-09T07:41:59.828Z info vsanvcmgmtd[09401] [vSAN@6876 sub=CnsVolMgr opId=098314c7] CNS: Updating volume task started
2021-08-09T07:41:59.828Z info vsanvcmgmtd[09401] [vSAN@6876 sub=FcdService opId=098314c7] CNS: UpdateVStorageObjectMetadata disk begin for volumeId: 76369948-665b-4e23-9e59-59cffaa9d965
2021-08-09T07:42:27.439Z error vsanvcmgmtd[09401] [vSAN@6876 sub=FcdService opId=098314c7] CNS: Failed to update disk metadata:N5Vmomi5Fault11SystemError9ExceptionE(Fault cause: vmodl.fault.SystemError
--> )
--> [context]zKq7AVECAAAAAH15AwESdnNhbnZjbWdtdGQAAOW7LmxpYnZtYWNvcmUuc28AAL3bGQDV+BgAt2MWAZVkGGxpYnZtb21pLnNvAAEpaBgC2RIPX2Nucy5zbwACkxYPAmUwDgKVxg0CjAEKAqeaCgIpqQoABLsiABIRIwCAqSoDh38AbGlicHRocmVhZC5zby4wAAS/NQ9saWJjLnNvLjYA[/context]
2021-08-09T07:42:27.440Z error vsanvcmgmtd[09401] [vSAN@6876 sub=BlockService opId=098314c7] Updating metadata for volume with id 76369948-665b-4e23-9e59-59cffaa9d965 failed with error:N3cns12CnsExceptionE(CNS: Failed to update disk metadata:Fault cause: vmodl.fault.SystemError
--> )
--> [context]zKq7AVECAAAAAH15AwEPdnNhbnZjbWdtdGQAAOW7LmxpYnZtYWNvcmUuc28AAL3bGQDV+BgAQGUWATUwC19jbnMuc28AAVMyDgGVxg0BjAEKAaeaCgEpqQoABLsiABIRIwCAqSoCh38AbGlicHRocmVhZC5zby4wAAO/NQ9saWJjLnNvLjYA[/context]
2021-08-09T07:42:27.440Z info vsanvcmgmtd[09401] [vSAN@6876 sub=CnsTask opId=098314c7] Update volume completed: task-1171277
And second try:
2021-08-09T07:42:03.638Z info vsanvcmgmtd[35922] [vSAN@6876 sub=AdapterServer opId=098314c8] Invoking 'updateVolumeMetadata' on 'cns-volume-manager', session '5284b747-25c2-9185-3b4b-0e72757eda1a'
2021-08-09T07:42:03.638Z info vsanvcmgmtd[35922] [vSAN@6876 sub=FcdService opId=098314c8] CNS: RetrieveVStorageObjectInfo disk begin for volumeId: 76369948-665b-4e23-9e59-59cffaa9d965
2021-08-09T07:42:03.727Z info vsanvcmgmtd[35922] [vSAN@6876 sub=FcdService opId=098314c8] CNS: RetrieveVStorageObjectInfo end
2021-08-09T07:42:03.732Z verbose vsanvcmgmtd[35922] [vSAN@6876 sub=PyBackedMO opId=098314c8] Enter vim.cns.VolumeManager.updateVolumeMetadata, Pending: 2 (5284b747-25c2-9185-3b4b-0e72757eda1a)
2021-08-09T07:42:03.732Z info vsanvcmgmtd[35922] [vSAN@6876 sub=CnsVolMgr opId=098314c8] CNS: UpdateVolumeMetadata with spec: (vim.cns.VolumeMetadataUpdateSpec) [
--> (vim.cns.VolumeMetadataUpdateSpec) {
--> volumeId = (vim.cns.VolumeId) {
--> id = "76369948-665b-4e23-9e59-59cffaa9d965"
--> },
--> metadata = (vim.cns.VolumeMetadata) {
--> containerCluster = (vim.cns.ContainerCluster) {
--> clusterType = "KUBERNETES",
--> clusterId = "K8S-DEV",
--> vSphereUser = "VSPHERE.LOCAL\k8s-dev",
--> clusterFlavor = "VANILLA"
--> },
--> entityMetadata = (vim.cns.EntityMetadata) [
--> (vim.cns.KubernetesEntityMetadata) {
--> entityName = "postgres-0",
--> clusterId = "K8S-DEV",
--> entityType = "POD",
--> namespace = "default",
--> referredEntity = (vim.cns.KubernetesEntityReference) [
--> (vim.cns.KubernetesEntityReference) {
--> entityType = "PERSISTENT_VOLUME_CLAIM",
--> entityName = "postgres",
--> namespace = "default",
--> clusterId = "K8S-DEV"
--> }
--> ],
--> }
--> ],
--> containerClusterArray = (vim.cns.ContainerCluster) [
--> (vim.cns.ContainerCluster) {
--> clusterType = "KUBERNETES",
--> clusterId = "K8S-DEV",
--> vSphereUser = "k8s-dev@vsphere.local",
--> clusterFlavor = "VANILLA"
--> }
--> ]
--> }
--> }
--> ]
2021-08-09T07:42:03.738Z info vsanvcmgmtd[35922] [vSAN@6876 sub=CnsTask opId=098314c8] A com.vmware.cns.tasks.updatevolume task is created: task-1171278
2021-08-09T07:42:03.738Z verbose vsanvcmgmtd[35922] [vSAN@6876 sub=PyBackedMO opId=098314c8] Exit vim.cns.VolumeManager.updateVolumeMetadata (6 ms)
2021-08-09T07:42:03.738Z info vsanvcmgmtd[35922] [vSAN@6876 sub=VsanTaskSvc opId=098314c8] ADD task 'task-1171278', total: 5
2021-08-09T07:42:03.738Z info vsanvcmgmtd[35922] [vSAN@6876 sub=AdapterServer opId=098314c8] Finished 'updateVolumeMetadata' on 'cns-volume-manager' (100 ms): done
2021-08-09T07:42:27.451Z info vsanvcmgmtd[09401] [vSAN@6876 sub=CnsVolMgr opId=098314c8] CNS: Updating volume task started
2021-08-09T07:42:27.451Z info vsanvcmgmtd[09401] [vSAN@6876 sub=FcdService opId=098314c8] CNS: UpdateVStorageObjectMetadata disk begin for volumeId: 76369948-665b-4e23-9e59-59cffaa9d965
2021-08-09T07:42:29.593Z info vsanvcmgmtd[09401] [vSAN@6876 sub=FcdService opId=098314c8] CNS: UpdateVStorageObjectMetadata end
2021-08-09T07:42:29.598Z info vsanvcmgmtd[09401] [vSAN@6876 sub=CnsTask opId=098314c8] Update volume completed: task-1171278