14 Replies Latest reply on Jan 22, 2018 5:55 AM by marnow

    vSAN - Datastore capacity shows 0 after last patch

    marnow Novice

      Hello,

       

      Recently patched to VMware ESXi, 6.5.0, 7388607, the capacity view no longer works! I have to refresh the Summary View to pull the numbers but after another vCenter refresh info disappear...

       

      Also I'm getting this message: VMware vSAN in cluster XXX in datacenter YYY does not have capacity.

       

      Veeam is unable to backup the VMs.

        • 1. Re: vSAN - Datastore capacity shows 0 after last patch
          Great_White_Tec Expert
          vExpertVMware Employees

          Hi Marnow,

           

          What is the version of your vCenter? Did you upgrade/patch vCenter to a higher/same level build as your hosts prior to patching the hosts?

           

          Your vCenter should be at build number 7312210

          • 2. Re: vSAN - Datastore capacity shows 0 after last patch
            marnow Novice

            Yes, I patched the vCenter as well. I'm running 6.5.0.13000 Build Number 7312210

            • 3. Re: vSAN - Datastore capacity shows 0 after last patch
              Great_White_Tec Expert
              VMware EmployeesvExpert

              I've seen this is the past, and usually the resolution is a vCenter restart.

               

              If you use RVC, do you see the correct info?

               

              localhost/<datacenter>/datastores> show <vsanDatastore>

               

              • 4. Re: vSAN - Datastore capacity shows 0 after last patch
                TheBobkin Virtuoso
                VMware EmployeesvExpert

                Hello marnow,

                 

                 

                Welcome to vSAN Community.

                 

                Determine where is seeing 0B datastore - whether it is an issue with the web-client, the vCenter or the vSAN cluster side:

                - Check via CLI on a host with #df -h and that the hosts are clustered #esxcli vsan cluster get

                - Check from rvc like GreatWhiteTec said.

                - Check with UI client instead of Flash client (https://FQDN-or-IP-Address-of-VC/UI).

                - Check which necessary vCenter services are running or not running (service-control --status --all in vCSA CLI)

                 

                Is this currently a production cluster and/or have any dependencies on vCenter? If not and the hosts see normal datastore size then try restarting the vCenter services or rebooting the vC.

                VMware Knowledge Base

                 

                 

                Bob

                • 5. Re: vSAN - Datastore capacity shows 0 after last patch
                  marnow Novice

                  Nope, I see 0 GB

                   

                  /localhost/OU-UTS-VSAN/datastores> show 1

                  path: /localhost/OU-UTS-VSAN/datastores/vsanDatastore

                  type: vsan

                  url: ds:///vmfs/volumes/vsan:5208d5c6f81944d9-8520459c09710020/

                  multipleHostAccess: true

                  capacity: 0.00GB

                  free space: 0.00GB

                  /localhost/OU-UTS-VSAN/datastores>

                  • 6. Re: vSAN - Datastore capacity shows 0 after last patch
                    Great_White_Tec Expert
                    VMware EmployeesvExpert

                    From vCenter UI. Click vCenter server (top) > Configure > Storage Provider. You should have 1 provider per vSAN host, what is the status for each of them?

                    • 7. Re: vSAN - Datastore capacity shows 0 after last patch
                      marnow Novice

                      All Storage Providers are Online.

                       

                      I see the issue in both versions of vCenter (Flash and HTML5), when I execute df-h on hosts, the numbers are correct:

                       

                      vsan       203.8T  22.8T    181.0T  11% /vmfs/volumes/vsanDatastore

                       

                      All services are up and running:

                       

                      root@vc6 [ ~ ]# service-control --status --all

                      Running:

                      applmgmt lwsmd pschealth vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-cm vmware-content-library vmware-eam vmware-perfcharts vmware-psc-client vmware-rhttpproxy vmware-sca vmware-sps vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-updatemgr vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-client vsphere-ui

                      Stopped:

                      vmcam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-rbd-watchdog vmware-vcha

                       

                      I've attempted to reboot the vCenter, it failed with storage related issues, called support, we run

                       

                      fsck -fy /dev/mapper/log_vg-log

                       

                      VMware Knowledge Base

                       

                      ....some errors have been fixed and started normally.

                      • 8. Re: vSAN - Datastore capacity shows 0 after last patch
                        TheBobkin Virtuoso
                        VMware EmployeesvExpert

                        Hello marnow,

                         

                         

                        Possibly a long shot but check timezones and licensing as others have seen these resolve what appears to be similar issues:

                        https://communities.vmware.com/thread/552827

                         

                        Check NTP sync between hosts and vCenter as this can cause too many issues to count.

                         

                        Are other vSAN elements displaying properly such as Disk Management and vSAN Health?

                         

                         

                        Bob

                        • 9. Re: vSAN - Datastore capacity shows 0 after last patch
                          marnow Novice

                          I think, it has something to do with one of the host being partitioned from the vSAN datastore...I rebooted the questionable host and the values were back while the host was offline...

                           

                          When I execute df -h....i get

                           

                          vsan         0.0B   0.0B      0.0B   0% /vmfs/volumes/vsanDatastore

                           

                          There is no network issue, I verified that....should I leave and join the host to the vSAN cluster to fix this?

                          • 10. Re: vSAN - Datastore capacity shows 0 after last patch
                            TheBobkin Virtuoso
                            vExpertVMware Employees

                            Hello marnow,

                             

                             

                            Where are you checking df -h from when this shows as 0B?

                            A partitioned host will show 0B with df -h, this is expected.

                             

                            Check that that host is participating in the cluster:

                            # esxcli vsan cluster get

                            If it is showing as cluster membership of 1 then it is partitioned from the rest of the cluster.

                            The above also tells if it is Unicast Mode enabled or not.

                             

                            Check vSAN networking is configured correctly:

                            # esxcli vsan network list

                             

                            Check ping between the vSAN configured vmk on one host to the others e.g.:

                            # vmkping -I vmkX 192.168.X.X

                            (get the IP and vmk# of the vSAN interface from GUI or #esxcfg-vmknic -l)

                             

                            Check that hosts Unicast lists are fully populated with all other nodes in the clusters info:

                            # esxcli vsan unicastagent list

                             

                            Make sure the host is not in Maintenance Mode or vSAN Decom state:

                            # esxcli vsan maintenancemode cancel

                             

                            If all of the above is fine then try leave and rejoin cluster:

                            # esxcli vsan cluster leave

                            # esxcli vsan cluster join -u <Sub-cluster UUID from another cluster member>

                             

                            What build version did you upgrade the cluster from, was it using Unicast and are all the disks upgraded to on-disk format v5?

                             

                            Health check GUI will help a lot with determining many issues including much of the above (Cluster > Monitor > vSAN > Health).

                             

                             

                            Bob

                            • 11. Re: vSAN - Datastore capacity shows 0 after last patch
                              marnow Novice

                              Currently working with support, we've removed the host from the cluster and the error disappeared....they are also investigating the "double unicast entries for witness" issue on each host in the cluster...

                               

                              NodeUuid                              IsWitness  Supports Unicast  IP Address       Port  Iface Name

                              ------------------------------------  ---------  ----------------  --------------  -----  ----------

                              5968d583-11ca-ff62-db19-246e963c74a8          0              true  192.168.250.10  12321

                              5968db7c-1580-f42a-1ae3-246e963e7fd8          0              true  192.168.250.12  12321

                              5968dec6-6963-8be6-36ca-246e963e7450          0              true  192.168.250.13  12321

                              5968f36f-ff76-8748-4eb3-246e963e71f0          0              true  192.168.250.23  12321

                              5968f251-5ed4-942b-04f2-246e963e1a08          0              true  192.168.250.21  12321

                              5968f315-bddc-30d6-2e03-246e963e1948          0              true  192.168.250.22  12321

                              5968f057-fd42-e63e-14b9-246e963e19d8          0              true  192.168.250.20  12321

                              5922e807-d875-58b0-064f-f4e9d494b820          1              true  10.8.120.21     12321

                              00000000-0000-0000-0000-000000000000          1              true  10.8.120.21     12321

                               

                              They said it is not normal, vSAN Heath Check is reporting "vSAN cluster configuration consistency" error....

                               

                              • 12. Re: vSAN - Datastore capacity shows 0 after last patch
                                Great_White_Tec Expert
                                vExpertVMware Employees

                                Is this a Stretched Cluster? (doesn't look like)

                                 

                                Were any of those hosts used for a 2-node of SC configuration in the past?

                                • 13. Re: vSAN - Datastore capacity shows 0 after last patch
                                  marnow Novice

                                  Yes, it is a stretched cluster 4+4+1 (witness). All fresh install, it stared after we removed host from the cluster.

                                  • 14. Re: vSAN - Datastore capacity shows 0 after last patch
                                    marnow Novice

                                    We had been able successfully join the host to the cluster, network sync error disappeared.