No VVol space after 3par failover

[cheese2]

We have a 3PAR 7400 SAN providing storage to our VMware system.  It's been running VVols quite successfully for several months, plus a couple of conventional luns with vmfs data stores on them.  Recently, one of the two controller nodes failed, and it did this so smoothly we didn't notice for a couple of days (this is dev data so it's not monitored very closely).  However the VVol data store now reports 0 bytes free space so we can't provision any new VMs or migrate anything to it.  Existing VMs continue to work just fine, and the 3par console reports more than 52TB available.

The vasa provider shows as being online.  I refreshed the certificate which changed nothing.  Eventually, after moving running VMs off to the vmfs data stores, I deleted the VVol data store and the vasa provider, ran vasareset on the 3par, and recreated both.  Same problem.  On the VVol data store Summary page it reports the capacities of various CPGs under Storage Capability Profiles.  The Used Space values are plausible but the Free Space values are preposterous - many are 0.00 B and others are long negative numbers like -2,388,941,340,672.00 B.

Any idea how we recover this?  I had been quite pleased with the VVol performance and was about to recommend we use them in production.  The 3par is running OS 3.2.2 MU4 and our vCenter is 6.5, although the hosts are still on 6.0u2.  It will be some time until we are able to replace the dead node and we cant reboot the remaining one until then.

[esiebert7625]

Double check the time sync in your environment, your ESXi hosts, vCenter Server and 3PAR array should all have their clocks in sync. Being out of sync is one one cause of the behavior you are seeing.

[cheese2]

Ah now I think you're on to something here.  Both the remaining 3par controller and the vCSA are referenced to the same NTP server and are in sync, however upon closer inspection the vCSA doesn't appear to have a time zone set.  The field is blank and the time comes up as UTC, whereas the 3par is set to Europe/London which is currently UTC+1 for the summer.  I don't seem to be able to update the vCSA time zone - I can pick whatever I want but the field remains blank and the time never changes from UTC.  Odd that this hasn't caused any issues with AD authentication.

I have gone into the shell of the vCSA and found that /etc/localtime links to itself.  I did an rm /etc/localtime followed by ln - s /usr/share/zoneinfo/Europe/London localtime and the time is now correct (and I can also change it from the GUI now) however nothing has changed even after deleting and recreating both the vasa provider and the vvol datastore again.  I also get the error "vSphere HA failed to create a configuration vVol for this datastore and so will not be able to protect virtual machines on the datastore until the problem is resolved.  Error: (vim.fault.CannotCreateFile) {" which is consistent with not being able to write anything.