Thin-provisioned LUN capacity Exceeded / The operation is not allowed in the current state of the datastore.

[jrmunday]

Hi All,

Has anyone seen this behavior before or could you help explain why this happening so that I can take the appropriate remedial action.

Basic summary of Infrastructure;

• vSphere 5.0 U2 with Ent Plus licences
• 30x ESXi 5.0 hosts - build 1254542
• HDS SAN - FC block storage (multiple 2TB LUNs) - SDRS cluster
• NetApp MetroCluster - NFS storage (multiple 5TB volumes) - SDRS cluster

We are currently tight on block storage capacity and are in the process of provisioning additional NFS storage - thin provisioning is widely use on both. On the HDS datastores I have the "Thin-provisioned LUN capacity Exceeded" alarm triggered, but there is still free space available on the datastore.

Yesterday I had the situation where I was unable to add any additional disks (even a small 10GB disk) or storage vMotion VM's between HDS datastores (except for 1 or 2 datastores out of 14) - NFS was unaffected. The error that I got in both cases was "The operation is not allowed in the current state of the datastore." The only way that I could resolve this was to restart the vCenter services. The first time everything worked for a few hours, and the the second time I restarted vCenter server completely.

Since then I have added an additional 2TB LUN to the SDRS cluster, and still have the "Thin-provisioned LUN capacity Exceeded" alarm triggered - but am able to storage vMotion to all HDS datastores. Capacity is currently as follows;

Name               FreeSpaceMB    CapacityMB

----               -----------    ----------

xxxxxxxxxxxxx      411980         2047744

xxxxxxxxxxxxx      391602         2047744

xxxxxxxxxxxxx      402413         2047744

xxxxxxxxxxxxx      427012         2047744

xxxxxxxxxxxxx      501918         2047744

xxxxxxxxxxxxx      394800         2047744

xxxxxxxxxxxxx      397538         2047744

xxxxxxxxxxxxx      398614         2047744

xxxxxxxxxxxxx      459313         2047744

xxxxxxxxxxxxx      306059         2047744

xxxxxxxxxxxxx      378631         2047744

xxxxxxxxxxxxx      392283         2047744

xxxxxxxxxxxxx      382561         2047744

xxxxxxxxxxxxx      396588         2047744

xxxxxxxxxxxxx      302924         2047744

So the questions are;

1. Has anyone seen this before or did vCenter just get into a wobble?
2. Do I need to be concerned about anything major?
3. Are there any other areas that I should check?

Thanks,

Jon

[mikejroberts]

I have the exact same issue on 5.0 U2.  I am pretty sure it is a bug with SDRS clusters and I haven't found a way around it other than restarting the vCenter service.  I have seen it several times and the latest was on a LUN that still had 630GB out of 800GB free on a VMAX 20K array.  I have not had time to open a support case, but maybe someone with VMware will chime in and tell what is wrong.

[jrmunday]

Thanks for the feedback Mike, I will open a SR with VMware.

[jrmunday]

VMware have told me this is a problem with the HDS VASA storage provider ... I'm dubious, but we have a telco with VMware and Hitachi next week - let's see how this pans out?

[mikejroberts]

Then it must also be a problem with the EMC VASA provider.  That answer sounds suspect to me since I am on EMC and having the same issue you are having on HDS.

[jrmunday]

Yes, that's what I thought too

I checked the compatibility list and the HDS VASA provider (including the exact versions I'm using) are all fully supported by VMware ... will keep you posted.

[PavanKG]

Since you said that all these are thin provisioned luns .Did you check the luns at the storage end for any overcommitment ....please do check that first

[mikejroberts]

My LUNs are fully provisioned/committed on the array.  The VMs are thin in most cases.  My latest alert was on a fairly new LUN that was less than 25% utilized.

[MDuff43]

I was wondering if you got a solution to this issue.  We have EMC VMAX storage as well and are having this same issue.

[jeremywisneski]

EMC has a KB regarding this:  https://support.emc.com/kb/87287

For those who don't have access to the link above:

Issue	VASA does not allow addition of a VM to the datastore. Thin provisioned LUN capacity exceeded. After installing the VASA Provider getting error when trying to update datastore. Error message: the operation is not allowed in the current state of the datastore
Environment	Feature: vStorage API for Storage Awareness (VASA) Product: VMware ESXi5. SW: VSphere 5.0
Cause	With the thin-provisioning alert by default the datastore will deny any operations if it is more than 80% full.  This is a hard-coded limitation and cannot be changed.  However, in this instance the customer was unable to manage the datastore (create VM, migrate VM to it, etc.) even after disabling/clearing the alert.  Appears something was stuck in vSphere, needed the services to be restarted. Investigating with Engineering.
Resolution	Disable the thin-provisioning alert, clear any remaining alerts from the datastores and restart the vSphere services.