Hierarchy of native clones of file name.vmdk is too deep

[barnette08]

I am just kicking the tires on this new 5.1.2 environment and doing some testing to get it up and running, but I am seeing some really odd storage related issues while trying to do some basic tasks in the GUI.  When I try to provision more than 1 VM in a vApp I get an error stating insufficient permissions or datastore has exceeded its file count limit, and when I attempt to publish things from an organization into that organization's catalog I get an error stating the hierarchy of native clones of file vmdk is too deep.

The storage array gives r/w and root access to the datastores , so I really feel like permissions are not the issue.  Does anyone have thoughts on this?

While trying to provision more than 1 VM in a vApp:

Insufficient permissions or datastore has exceeded its file count limit.

vCenter Server task (moref: task-2102) failed in vCenter Server 'dccvcdvcprd01' (1c495c36-1f5b-4322-bc19-b77b8526b898).

- Unable to access file [dccb-nfsprd01] Win7_Test-1 (f1f78f06-57fb-4dd7-b524-7325f6b79929)

While attempting to upload to catalog:

Failure occurred while trying to access file [dccb-nfsprd01] Red Hat Enterprise Linux 6.4 x86 (4221568e-58b5-48e2-b3c6-bb3e995a58be)/Red Hat Enterprise Linux 6.4 x86 (4221568e-58b5-48e2-b3c6-bb3e995a58be).vmdk.

vCenter Server task (moref: task-3967) failed in vCenter Server 'dccvcdvcprd01' (1c495c36-1f5b-4322-bc19-b77b8526b898).

- The hierarchy of native clones of file [dccb-nfsprd01] Red Hat Enterprise Linux 6.4 x86 (4221568e-58b5-48e2-b3c6-bb3e995a58be)/Red Hat Enterprise Linux 6.4 x86 (4221568e-58b5-48e2-b3c6-bb3e995a58be).vmdk is too deep.

Thanks in advance!

[IamTHEvilONE]

if you are doing linked clones, then vCenter is the one performing the cloning task.  vCloud Director is just making the API call to request the action.

If you see the same/similar error state in vCenter itself, then try to work it from that angle.

Also, can you shorten your virtual machine name?  I might not be the name of the VM, but the length of name.  If you add the same VM twice to a vApp ... it'll be vmname and vmname-1 ... but in your case the first instance would seem short enough.  the latter one seems a little long ... but nothing I haven't seen before.

However, I would state that this is a native linked clone ... what type of storage is this?  is this NFS with VAAI enabled ... it's possible to limit the chain depth on the storage server when using Native linked clones.

[barnette08]

this is NFS with VAAI enabled, but the chain length on all VMs in the vApp that is being published is only 2.  I tried shortening the name and here is what I am getting now:

One or more exceptions have occurred

- Failure occurred while trying to access file [dccb-nfsprd01] Win7-5 (5d3ef229-2673-4643-bf16-c5e9e7fe98f5)/Win7-5 (5d3ef229-2673-4643-bf16-c5e9e7fe98f5).vmdk.

vCenter Server task (moref: task-4418) failed in vCenter Server 'dccvcdvcprd01' (1c495c36-1f5b-4322-bc19-b77b8526b898).

- The hierarchy of native clones of file [dccb-nfsprd01] Win7-5 (5d3ef229-2673-4643-bf16-c5e9e7fe98f5)/Win7-5 (5d3ef229-2673-4643-bf16-c5e9e7fe98f5).vmdk is too deep.

- Failure occurred while trying to access file [dccb-nfsprd01] Win2k12-7 (ef115f15-d164-47e2-81fa-9ec82949d98c)/Win2k12-7 (ef115f15-d164-47e2-81fa-9ec82949d98c).vmdk.

vCenter Server task (moref: task-4422) failed in vCenter Server 'dccvcdvcprd01' (1c495c36-1f5b-4322-bc19-b77b8526b898).

- The hierarchy of native clones of file [dccb-nfsprd01] Win2k12-7 (ef115f15-d164-47e2-81fa-9ec82949d98c)/Win2k12-7 (ef115f15-d164-47e2-81fa-9ec82949d98c).vmdk is too deep.

- Failure occurred while trying to access file [dccb-nfsprd01] RHEL4 (42b63df0-1e72-4cc2-8722-026380fd6387)/RHEL4 (42b63df0-1e72-4cc2-8722-026380fd6387).vmdk.

vCenter Server task (moref: task-4423) failed in vCenter Server 'dccvcdvcprd01' (1c495c36-1f5b-4322-bc19-b77b8526b898).

- The hierarchy of native clones of file [dccb-nfsprd01] RHEL4 (42b63df0-1e72-4cc2-8722-026380fd6387)/RHEL4 (42b63df0-1e72-4cc2-8722-026380fd6387).vmdk is too deep.

- Failure occurred while trying to access file [dccb-nfsprd01] LAAS2 (f74788bf-dd02-4bbb-8971-12860ceb2592)/LAAS2 (f74788bf-dd02-4bbb-8971-12860ceb2592).vmdk.

vCenter Server task (moref: task-4419) failed in vCenter Server 'dccvcdvcprd01' (1c495c36-1f5b-4322-bc19-b77b8526b898).

- The hierarchy of native clones of file [dccb-nfsprd01] LAAS2 (f74788bf-dd02-4bbb-8971-12860ceb2592)/LAAS2 (f74788bf-dd02-4bbb-8971-12860ceb2592).vmdk is too deep.

- Failure occurred while trying to access file [dccb-nfsprd01] Win2k8-6 (52d37435-1cf8-4e4f-85a3-83367ed51a4c)/Win2k8-6 (52d37435-1cf8-4e4f-85a3-83367ed51...

I might try disabling the dccb-nfsprd01 datastore and see if the issue continues on dccb-nfsprd02, which is another datastore in the PvDC.  Any other suspicions that just jump right out? This seems somewhat bizarre to me.

[barnette08]

Update:

When disabling the dccb-nfsprd01 datastore and forcing all clones operations to the dccb-nfsprd02 datastore, everything seems to work fine.  So what I ended up doing was deleting and recreating the new NFS filesystem and export.  On the newly recreate and mounted datastore, I still get the above errors when attempting to publish to the catalog.

Also when disabling VAAI in vCD the error goes away and the task succeeds. Does anything have any thoughts as to why VAAI might be acting this way?

[barnette08]

Solved, Final Update:

After looking at vCenter and trying to determine the differences between the 2 different datastores, I noticed that SIOC was not enabled on the datastore in question.  I did test deployments after re-enabling VAAI and SIOC being enabled seemed to clear things up.

As of this afternoon and no configuration changes this issue has returned.  Eventhough, this VM had a chain length of 2, I still attempted to consolidate it and got the following error:

null
vCenter Server task (moref: task-5459) failed in vCenter Server 'dccvcdvcprd01' (1c495c36-1f5b-4322-bc19-b77b8526b898).
- The specified delta disk format 'nativeFormat' is not supported.

Beyond that, The hierarchy of native clones of file [dccb-nfsprd01] RHEL_6.4_x64 (fa56cc2f-0c09-43c0-93d7-f126cd8b94a2)/RHEL_6.4_x64 (fa56cc2f-0c09-43c0-93d7-f126cd8b94a2).vmdk is too deep error has returned!

[barnette08 · Accepted solution]

Solved:

This turned out to be an array level issue related to VAAI.  We are running R31 code on our VNX storage array and 'VAAI nested clone support' didn't come into play until R32/INYO for VNX.  I have temporarily switched to FC storage since doing an upgrade right now is not an option for us, but I did some testing by mounting some NFS datastores from an INYO array to these hosts and the same tasks succeeded without issue (very quickly I might add).

[IamTHEvilONE]

Good to know .... If I had a way of reasonably summarizing this, I'd like to write an article about it .... I just need to know what the heck to write (I'm not a storage guy).

[barnette08]

Yea, the more I think about it, it makes sense though...If you look at the error and if the array doesn't support 'nested clones' with VAAI, anything beyond the first link of the chain is "too deep", or at least thats the way I interpret it now.

One side affect of this that hasn't exactly been cleaned up from an NFS perspective is the error that is received when attempting to run a consolidate.  This happens on both the R31/R32 revision arrays, but not on the FC datastores.

- The specified delta disk format 'nativeFormat' is not supported.

I will try to get you some details for your write-up from some of the internal guys here in unified storage and send them your way.

[IamTHEvilONE]

the nativeFormat message sounds like it was created with VAAI enabled, but then VAAI was disabled.  so you have a native linked clone you are trying to work with, when in non-native mode.

[IamTHEvilONE]

Also "- The specified delta disk format 'nativeFormat' is not supported." message will happen if you attempt to issue a 'consolidate' action on a Native VAAI based linked clone because it's not supported.