VMware Cloud Community
Tom31415
Enthusiast
Enthusiast

CNS: Failed to update disk metadata

Hi, my vsphere-csi-driver has a problem with volume metadata updating. But it looks like problem with first try and another one is working well. The first try ends with this error: CNS: Failed to update disk metadata:N5Vmomi5Fault11SystemError9ExceptionE

First try:

2021-08-09T07:41:59.773Z info vsanvcmgmtd[35922] [vSAN@6876 sub=AdapterServer opId=098314c7] Invoking 'updateVolumeMetadata' on 'cns-volume-manager', session '5284b747-25c2-9185-3b4b-0e72757eda1a'
2021-08-09T07:41:59.773Z info vsanvcmgmtd[35922] [vSAN@6876 sub=FcdService opId=098314c7] CNS: RetrieveVStorageObjectInfo disk begin for volumeId: 76369948-665b-4e23-9e59-59cffaa9d965
2021-08-09T07:41:59.809Z info vsanvcmgmtd[35922] [vSAN@6876 sub=FcdService opId=098314c7] CNS: RetrieveVStorageObjectInfo end
2021-08-09T07:41:59.815Z verbose vsanvcmgmtd[35922] [vSAN@6876 sub=PyBackedMO opId=098314c7]  Enter vim.cns.VolumeManager.updateVolumeMetadata, Pending: 2 (5284b747-25c2-9185-3b4b-0e72757eda1a)
2021-08-09T07:41:59.815Z info vsanvcmgmtd[35922] [vSAN@6876 sub=CnsVolMgr opId=098314c7] CNS: UpdateVolumeMetadata with spec: (vim.cns.VolumeMetadataUpdateSpec) [
-->    (vim.cns.VolumeMetadataUpdateSpec) {
-->       volumeId = (vim.cns.VolumeId) {
-->          id = "76369948-665b-4e23-9e59-59cffaa9d965"
-->       }, 
-->       metadata = (vim.cns.VolumeMetadata) {
-->          containerCluster = (vim.cns.ContainerCluster) {
-->             clusterType = "KUBERNETES", 
-->             clusterId = "K8S-DEV", 
-->             vSphereUser = "VSPHERE.LOCAL\k8s-dev", 
-->             clusterFlavor = "VANILLA"
-->          }, 
-->          entityMetadata = (vim.cns.EntityMetadata) [
-->             (vim.cns.KubernetesEntityMetadata) {
-->                entityName = "postgres", 
-->                labels = (vim.KeyValue) [
-->                   (vim.KeyValue) {
-->                      key = "app", 
-->                      value = "postgres"
-->                   }, 
-->                   (vim.KeyValue) {
-->                      key = "app.kubernetes.io/managed-by", 
-->                      value = "Helm"
-->                   }, 
-->                   (vim.KeyValue) {
-->                      key = "chart", 
-->                      value = "postgres-12"
-->                   }, 
-->                   (vim.KeyValue) {
-->                      key = "heritage", 
-->                      value = "Helm"
-->                   }, 
-->                   (vim.KeyValue) {
-->                      key = "release", 
-->                      value = "radim"
-->                   }
-->                ], 
-->                clusterId = "K8S-DEV", 
-->                entityType = "PERSISTENT_VOLUME_CLAIM", 
-->                namespace = "default", 
-->                referredEntity = (vim.cns.KubernetesEntityReference) [
-->                   (vim.cns.KubernetesEntityReference) {
-->                      entityType = "PERSISTENT_VOLUME", 
-->                      entityName = "pvc-f14d2028-fbe4-4ce7-aedd-0faf8d81db6e", 
-->                      clusterId = "K8S-DEV", 
-->                   }
-->                ], 
-->             }
-->          ], 
-->          containerClusterArray = (vim.cns.ContainerCluster) [
-->             (vim.cns.ContainerCluster) {
--
2021-08-09T07:41:59.821Z info vsanvcmgmtd[35922] [vSAN@6876 sub=CnsTask opId=098314c7] A com.vmware.cns.tasks.updatevolume task is created: task-1171277
2021-08-09T07:41:59.821Z verbose vsanvcmgmtd[35922] [vSAN@6876 sub=PyBackedMO opId=098314c7]  Exit  vim.cns.VolumeManager.updateVolumeMetadata (5 ms)
2021-08-09T07:41:59.821Z info vsanvcmgmtd[35922] [vSAN@6876 sub=VsanTaskSvc opId=098314c7] ADD task 'task-1171277', total: 4
2021-08-09T07:41:59.821Z info vsanvcmgmtd[35922] [vSAN@6876 sub=AdapterServer opId=098314c7] Finished 'updateVolumeMetadata' on 'cns-volume-manager' (48 ms): done
2021-08-09T07:41:59.828Z info vsanvcmgmtd[09401] [vSAN@6876 sub=CnsVolMgr opId=098314c7] CNS: Updating volume task started
2021-08-09T07:41:59.828Z info vsanvcmgmtd[09401] [vSAN@6876 sub=FcdService opId=098314c7] CNS: UpdateVStorageObjectMetadata disk begin for volumeId: 76369948-665b-4e23-9e59-59cffaa9d965
2021-08-09T07:42:27.439Z error vsanvcmgmtd[09401] [vSAN@6876 sub=FcdService opId=098314c7] CNS: Failed to update disk metadata:N5Vmomi5Fault11SystemError9ExceptionE(Fault cause: vmodl.fault.SystemError
--> )
--> [context]zKq7AVECAAAAAH15AwESdnNhbnZjbWdtdGQAAOW7LmxpYnZtYWNvcmUuc28AAL3bGQDV+BgAt2MWAZVkGGxpYnZtb21pLnNvAAEpaBgC2RIPX2Nucy5zbwACkxYPAmUwDgKVxg0CjAEKAqeaCgIpqQoABLsiABIRIwCAqSoDh38AbGlicHRocmVhZC5zby4wAAS/NQ9saWJjLnNvLjYA[/context]
2021-08-09T07:42:27.440Z error vsanvcmgmtd[09401] [vSAN@6876 sub=BlockService opId=098314c7] Updating metadata for volume with id 76369948-665b-4e23-9e59-59cffaa9d965 failed with error:N3cns12CnsExceptionE(CNS: Failed to update disk metadata:Fault cause: vmodl.fault.SystemError
--> )
--> [context]zKq7AVECAAAAAH15AwEPdnNhbnZjbWdtdGQAAOW7LmxpYnZtYWNvcmUuc28AAL3bGQDV+BgAQGUWATUwC19jbnMuc28AAVMyDgGVxg0BjAEKAaeaCgEpqQoABLsiABIRIwCAqSoCh38AbGlicHRocmVhZC5zby4wAAO/NQ9saWJjLnNvLjYA[/context]
2021-08-09T07:42:27.440Z info vsanvcmgmtd[09401] [vSAN@6876 sub=CnsTask opId=098314c7] Update volume completed: task-1171277

And second try:

2021-08-09T07:42:03.638Z info vsanvcmgmtd[35922] [vSAN@6876 sub=AdapterServer opId=098314c8] Invoking 'updateVolumeMetadata' on 'cns-volume-manager', session '5284b747-25c2-9185-3b4b-0e72757eda1a'
2021-08-09T07:42:03.638Z info vsanvcmgmtd[35922] [vSAN@6876 sub=FcdService opId=098314c8] CNS: RetrieveVStorageObjectInfo disk begin for volumeId: 76369948-665b-4e23-9e59-59cffaa9d965
2021-08-09T07:42:03.727Z info vsanvcmgmtd[35922] [vSAN@6876 sub=FcdService opId=098314c8] CNS: RetrieveVStorageObjectInfo end
2021-08-09T07:42:03.732Z verbose vsanvcmgmtd[35922] [vSAN@6876 sub=PyBackedMO opId=098314c8]  Enter vim.cns.VolumeManager.updateVolumeMetadata, Pending: 2 (5284b747-25c2-9185-3b4b-0e72757eda1a)
2021-08-09T07:42:03.732Z info vsanvcmgmtd[35922] [vSAN@6876 sub=CnsVolMgr opId=098314c8] CNS: UpdateVolumeMetadata with spec: (vim.cns.VolumeMetadataUpdateSpec) [
-->    (vim.cns.VolumeMetadataUpdateSpec) {
-->       volumeId = (vim.cns.VolumeId) {
-->          id = "76369948-665b-4e23-9e59-59cffaa9d965"
-->       }, 
-->       metadata = (vim.cns.VolumeMetadata) {
-->          containerCluster = (vim.cns.ContainerCluster) {
-->             clusterType = "KUBERNETES", 
-->             clusterId = "K8S-DEV", 
-->             vSphereUser = "VSPHERE.LOCAL\k8s-dev", 
-->             clusterFlavor = "VANILLA"
-->          }, 
-->          entityMetadata = (vim.cns.EntityMetadata) [
-->             (vim.cns.KubernetesEntityMetadata) {
-->                entityName = "postgres-0", 
-->                clusterId = "K8S-DEV", 
-->                entityType = "POD", 
-->                namespace = "default", 
-->                referredEntity = (vim.cns.KubernetesEntityReference) [
-->                   (vim.cns.KubernetesEntityReference) {
-->                      entityType = "PERSISTENT_VOLUME_CLAIM", 
-->                      entityName = "postgres", 
-->                      namespace = "default", 
-->                      clusterId = "K8S-DEV"
-->                   }
-->                ], 
-->             }
-->          ], 
-->          containerClusterArray = (vim.cns.ContainerCluster) [
-->             (vim.cns.ContainerCluster) {
-->                clusterType = "KUBERNETES", 
-->                clusterId = "K8S-DEV", 
-->                vSphereUser = "k8s-dev@vsphere.local", 
-->                clusterFlavor = "VANILLA"
-->             }
-->          ]
-->       }
-->    }
--> ]
2021-08-09T07:42:03.738Z info vsanvcmgmtd[35922] [vSAN@6876 sub=CnsTask opId=098314c8] A com.vmware.cns.tasks.updatevolume task is created: task-1171278
2021-08-09T07:42:03.738Z verbose vsanvcmgmtd[35922] [vSAN@6876 sub=PyBackedMO opId=098314c8]  Exit  vim.cns.VolumeManager.updateVolumeMetadata (6 ms)
2021-08-09T07:42:03.738Z info vsanvcmgmtd[35922] [vSAN@6876 sub=VsanTaskSvc opId=098314c8] ADD task 'task-1171278', total: 5
2021-08-09T07:42:03.738Z info vsanvcmgmtd[35922] [vSAN@6876 sub=AdapterServer opId=098314c8] Finished 'updateVolumeMetadata' on 'cns-volume-manager' (100 ms): done
2021-08-09T07:42:27.451Z info vsanvcmgmtd[09401] [vSAN@6876 sub=CnsVolMgr opId=098314c8] CNS: Updating volume task started
2021-08-09T07:42:27.451Z info vsanvcmgmtd[09401] [vSAN@6876 sub=FcdService opId=098314c8] CNS: UpdateVStorageObjectMetadata disk begin for volumeId: 76369948-665b-4e23-9e59-59cffaa9d965
2021-08-09T07:42:29.593Z info vsanvcmgmtd[09401] [vSAN@6876 sub=FcdService opId=098314c8] CNS: UpdateVStorageObjectMetadata end
2021-08-09T07:42:29.598Z info vsanvcmgmtd[09401] [vSAN@6876 sub=CnsTask opId=098314c8] Update volume completed: task-1171278

 

0 Kudos
0 Replies