VMware Cloud Community

vSphere with Tanzu - Persistent Volume issue : ServerFaultCode: NotAuthenticated

  • Tried creating a pv and pvc, it worked successfully

k get pv,pvc

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM              STORAGECLASS   REASON   AGE

persistentvolume/pvc-122a109a-0ced-4601-b271-09a0ca43d18f   9Gi        RWO            Delete           Bound    default/myclaim2   dt-vsan                 71m

persistentvolume/pvc-8b1b8cc0-9ded-49a2-9217-6d7f90a01fd6   9Gi        RWO            Delete           Bound    default/myclaim3   dt-vmfs                 60m

persistentvolume/pvc-e78fb5fa-bdad-4245-a7e0-954c98d89990   12Gi       RWO            Delete           Bound    default/myclaim5   dt-vmfs                 19m


NAME                             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE

persistentvolumeclaim/myclaim2   Bound    pvc-122a109a-0ced-4601-b271-09a0ca43d18f   9Gi        RWO            dt-vsan        71m

persistentvolumeclaim/myclaim3   Bound    pvc-8b1b8cc0-9ded-49a2-9217-6d7f90a01fd6   9Gi        RWO            dt-vmfs        60m

persistentvolumeclaim/myclaim5   Bound    pvc-e78fb5fa-bdad-4245-a7e0-954c98d89990   12Gi       RWO            dt-vmfs        19m



  • Tried creating a pod and it is failing during volume attachment to a worker node.


  Type     Reason              Age                From                                                   Message

  ----     ------              ----               ----                                                   -------

  Normal   Scheduled           <unknown>                                                                 Successfully assigned default/nginx3 to tkgs-cluster-2-workers-ksm4n-c599d66cf-j5g8t

  Warning  FailedAttachVolume  16m (x9 over 18m)  attachdetach-controller                                AttachVolume.Attach failed for volume "pvc-e78fb5fa-bdad-4245-a7e0-954c98d89990" : rpc error: code = Internal desc = observed Error: "ServerFaultCode: NotAuthenticated" is set on the volume "f5dc08e7-520d-48b7-8278-c426bd3f2f9b-e78fb5fa-bdad-4245-a7e0-954c98d89990" on virtualmachine "tkgs-cluster-2-workers-ksm4n-c599d66cf-j5g8t"

  Warning  FailedMount         47s (x8 over 16m)  kubelet, tkgs-cluster-2-workers-ksm4n-c599d66cf-j5g8t  Unable to attach or mount volumes: unmounted volumes=[myclaim5], unattached volumes=[myclaim5 default-token-hns22]: timed out waiting for the condition

  Warning  FailedAttachVolume  17s (x8 over 14m)  attachdetach-controller                                AttachVolume.Attach failed for volume "pvc-e78fb5fa-bdad-4245-a7e0-954c98d89990" : rpc error: code = Internal desc = Watch on virtualmachine "tkgs-cluster-2-workers-ksm4n-c599d66cf-j5g8t" timed out



  • When tried to see the volumeattachment, we see timeout error.


  Attacher:   csi.vsphere.vmware.com

  Node Name:  tkgs-cluster-2-workers-ksm4n-c599d66cf-j5g8t


    Persistent Volume Name:  pvc-e78fb5fa-bdad-4245-a7e0-954c98d89990


  Attach Error:

    Message:  rpc error: code = Internal desc = Watch on virtualmachine "tkgs-cluster-2-workers-ksm4n-c599d66cf-j5g8t" timed out

    Time:     2021-12-14T09:19:00Z

  Attached:   false


Anyone faced this?

Labels (2)
Tags (2)
0 Kudos
3 Replies

This issue was fixed after regenerating the workload storage user password. Workload storage user that gets created as part of Supervisor cluster enablement, it's credentials was not working somehow, so regenerated the password and then the issue was resolved.

Tags (1)
0 Kudos
VMware Employee
VMware Employee

Hey Dinesh, what do you mean by "regenerating the workload storage user password"?!

0 Kudos

Login to one of the SupervisorControlPlaneVMs and run:

kubectl rollout restart deployment vsphere-csi-controller -n vmware-system-csi

this will regenerate Workload Storage User password which is stored in vsphere-config-secret in the same namespace.

This resolved my issue with PVC not bounding to PV, which was failing to be created due to Not Authenticated error.

In my case I was creating Tanzu cluster with additional storage for containerd.

0 Kudos