App Volume VMDK is not attached to a client (Failed to add disk 'scsi0:2')

[lookandfeel]

I have a vSphere cluster with vSAN datastore and VMFS datastores.
In the past, instant clone desktops were provisioned in the same vSAN datastore. This is still running fine.

For capacity reasons, I want to provision the instant clones on newly created VMFS datastores (cp-templates, cp-replica etc also). The AppStacks still should be on vSAN datastore.

Always when a user logs in to the desktops, during the AppStack mount process I get these entries in the System Messages log in AppVolume:

vSphere operation failed: Failed to add disk 'scsi0:2'
Declined all 2 attachments for "User<xxx>" on "Machine <xxx>"
Timed out when loading assigned volumes, please retry or contact your administrator.

I checked the permissions in vSphere already, and the configured user account has Global permissions and the Administrator Role is configured for all objects.

Testing the same behavior with NFS (v3) datastore, the AppStack mounting works as expected and fine.
Testing the same behavior with Local ESXi datastore, it results in the same issue as with shared VMFS datastore.

Does somebody have an idea whats going wrong here?

I attached a screenshot also.

Regards,

Andre

[da2125]

Anyone have a resolution for this, I am seeing the same error.

[lookandfeel]

Hi,

I am still troubleshooting on this. Do you have more than one App Volumes Managers installed in one vCenter, which are independently from each other?

[da2125]

We have multiple managers behind a load balance address.

The agent Manager address points towards this load balance address.

The load balancers then forward the traffic to one of these Managers based on least traffic

[lookandfeel]

Hi,

thats seems a valid solution, because all App Volumes Managers are in a kind of HA cluster, and share the same information/database I think?

Can you reproduce this issue with local VMFS datastore also?

And can you test this behavior with NFS?

Thats the point I am wondering. NFS is working fine. VMFS (shared or local) is not working.

[Ray_handels]

Have you set Mount Local in the Machine Manager settings page?

It seems as if Appvolumes cannot find the Appstack on the storage you are trying to attach it to. And these appstacks are read only? Otherwise it cannot attach it. You could try and check the .vmdk settings file to see if the value ddb.deletable=false, this blocks the disk file from being deleted. If it does not have this setting it cannot attach the Appstack.

When you look in vSphere what does it say there when you cannot attach the appstack?

[lookandfeel]

Hi,

I tested the "mount on host" option also.

The appstacks are still located on a vSAN datastore. The desktops are provisioned on VMFS.

In the vmdk of the appstack is no ddb.deletable entry. Adding that with option "false" also does not change the result. The issue still persists.

I attached a screenshot (combined the UI from App Volumes Manager and the vSphere Client UI).

[Ray_handels]

I would need to have a closer look as I don't have all the info available. Sorry but can't really help you any further than this.

[cacer33]

I'm having this issue too. 

[2022-06-13 17:13:33 UTC] P4276R5180RA ERROR RvSphere: vSphere operation failed: Invalid configuration for device '0'.

[2022-06-13 17:13:33 UTC] P4276R5180RA ERROR Cvo: Declined all 1 attachments for "

[cacer33]

Did you get this fixed?

[da2125]

We've sent VMWare many logs but so far no resolution.

Curious: Do you have any orphaned machines in a 'Discovered virtual machine' folder in any of your vCenters?

Does the endpoints System EventViewer having an issue contain:

EventID: 51;  Message: An error was detected on device \Device\Harddisk*\DR* during a paging operation, or...

EventID: 3; Message: Filter Manager failed to attach to volume '\Device\Harddisk*\DR*'. This volume will be unavailable for filtering until a reboot. The final status was 0xc03a001c.

[MarkusNemeth]

Hi all,

any solution for this by now?

We have the exact same Problem. AppStacks are placed on vSAN DS and Instant Clones on ISCSI VMFS DS.

AppStack Attachment for Instant Clones that are placed on the vSAN DS works fine. Attachments for Instant clones placed on the VMFS DS does not work.

I also have a test environment with 3 VMFS datastores. In this test environment it doesn't matter on which Datastore the instant clones are located or the appstacks, mounting the appstacks always works.

Is it at all possible to mount AppStacks from a vSAN Datastore to Instant Clones outside this vSAN Datastore?

[chriskoch99]

Also seeing this.  VMs and VMDKs are on the same vSAN datastore in a VxRail cluster.  Only happens occasionally, otherwise hundreds of successful logins daily.   Same errors in the AV system logs when it happens.   Curious if anyone has figured anything out.