App Volumes 2.13.1 Manager1 registry entry missing

[avang82]

We've recently upgraded to App Volumes 2.13.1 and have noticed on random occasions that appstacks to do not attach to instant clone VMs.  We're not presented with any errors at login either.  If you attempt to attach an appstack immediately from the App Volumes Manager, it never does get attached.  We have found that for some reason the registry entry for Manager1 is removed.  Reverting to a previous version of the App Volumes agent on the VMs appear to correct the issue for the time being.

Has anybody else experienced this and if there has been a fix?  I currently have a support request open with VMware and awaiting an answer as well.

[Sravan_k]

Hi Avang82,

mean while, can you please provide me screen shot of your app volume parameters under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\svservice\Parameters

Thank you,

Vkmr.

[avang82]

Here's what our parameters are.  The screenshot is from a VM with appstacks successfully attached.  When App Volumes does fail to attach appstacks, the only difference is Manager1 entry is missing.

I did receive a response from VMware that this appears to be a known bug with NGVC timing out because Manager1 is missing.  A workaround/hotpatch is still in the works.

[Gaurav_Baghla]

Having multiple registries are fine but do you have a LB for this so you would have just 1 entry there. If not a DNS RR would also help.

[avang82]

Yes, we use a load balancer and only have an entry for Manager1.  Manager2 and Manager3 were added for testing purposes.  However that still doesn't fix the problem with Manager1 being deleted.

[rdonohue]

I've got this same issue, and working with support to troubleshoot.

I'm seeing other odd behavior that may or may not be related, but curious if you've got the same thing on yours.

I'm trying to get this working in an instant clone pool. One issue that I'm seeing when this fails, that I don't see on any other machines in my environment is that the %logonserver% variable is not getting set in windows. You can check this by typing "echo %logonserver%" in command prompt. It should output one of your DC's. You can also type "systeminfo" in command prompt and there should be a line titled logonserver. This should also show one of your DC's. For me it just says N/A.

Also, even though active directory is seeing the login (you can check the "lastlogon" of the user in AD) App Volume Manager does not see it and only shows the Last Login time of the user when it successfully attached the volumes. Even if I hit the "Sync" button on the Directory>Users page, it just shows the last login time that worked correctly.

So I'm not sure if maybe this is some type of communication issue between the Manager and AD, or possibly a replication issue in AD? The fact that it also removes this registry entry seems like it's also a communication issue between the vm and the manager too. The common denominator seems to be the Manager. It seems to just stop talking to the vm and AD.

*edit*

I missed the part where you mentioned this being a known bug. I'm letting support tech I'm working with know, since he doesn't seem to be aware of it. Did they give you any reference number or a kb related to the bug? I don't see this particular issue listed as a known issue on anything public.

[avang82]

Sorry I don't have anything to reference this bug to.  I was just told that it was a bug and that my case has been added to the bug report.  I'd be notified once a patch or fix was available. 

As for the logonserver variable, I am not experiencing that problem.  However, App Volume Manager does not recognize my last login time like what you've also seen.  But I think that is due to Manager1 being deleted from the registry.  The missing registry entry essentially kills the communication between the Manager and Agent.

[rdonohue]

Support is saying this issue will be resolved in 2.14. The workaround is to add a local gpo to just add that registry entry back at startup. I'm testing now.

[Sravan_k]

Thanks for sharing the screenshot of parameters,

try the below workaround [may work, but please give a try]:

use AVM load balance address rather than using different managers address with the combination of app volume agent version 2.12.1 

Please provide your feedback on it

Thank you,

Vkmr.

[rdonohue]

Adding the registry entry back during startup does not work. It seems to be deleting that entry at some point after. Just to add a note, we are not using load balancers in my environment so only have the Manager1 entry and no others. So it's not an issue related to the load balancing. Support has told me that another workaround is to downgrade the vmware view agent on the golden image. I've asked them what specific version to go back to, waiting to hear back.

*edit*

We have also rolled back to App Volume agent version 2.12.1, which had no effect.

[avang82]

I had actually thought about adding the registry entry with a gpo as well but figured it would fail due to the timing when its applied and when App Volumes removes the entry. 

In my environment, we're currently on Manager 2.13.1 and Agent 2.11.  Agent 2.11 is being used because I was told this issue was also occurring in version 2.12.1.

[Perttu]

Hi avang82​

Are you using 'use network from parent image' selection or do you have explicitly selected networks from the list? If latter does it make any difference to switch the network to the same that the snapshot has, create a new snapshot if necessary.

[avang82]

We are actually using both methods.  We only have a few parent images that basically have the OS and appstack all applications.  For pools that use the same VLAN as the parent, "Parent VM Network Selected" is chosen.  We specify the VLAN for other pools that are different from the parent image.

[Perttu]

Same here and I myself can now confirm that this didn't help here either. It looked promising from the first push as the list of online entities from App Volumes manager finally matched to what Horizon had provisioned but later I found these broken machines again. Continuing with the support and I cannot even roll back to 2.11 due to KB mismatch: 2.11 Agent installer requires KB3033929, but that's superseded by another update then and I have no idea how to fix this.

Should we return to linked clones.

[rdonohue]

After a lot of screwing around I've got this working. I've got more testing to do but it seems to be working consistently at this point. Here's how I'm set up.

App Volumes Manager - 2.13.1

App Volumes Agent - 2.13.1

Horizon Agent - 7.0.3

Switch agent/manager communication to use HTTP instead of HTTPS.

I'm not sure yet if it's the horizon agent downgrade or the switch to http that fixed it. I'll report back once I've played around with them both. Obviously running everything over http is not ideal, so not a great fix if that's the only way to get it working.

[avang82]

Perttu - You can bypass the requirement by using the silent install instead.  Just run the following command.

"App Volumes Agent.msi" /qn MANAGER_ADDR=<Manager_FQDN/IP> MANAGER_PORT=<port> REBOOT=ReallySuppress

rdonohue - Let me know how testing goes with the use of HTTP.  Here's my current setup.

App Volumes Manager - 2.13.2

App Volumes Agent 2.11

Horizon Agent 7.3.1

I was provided with App Volumes Hot Patch 2.13.2 which corrected another bug found in 2.13.1.  Apparently if any VMs were in an offline state in App Volumes Manager at the time of the 2.13.1 upgrade, appstacks could no longer be attached to the VM.  The only work around would be to change the complete naming scheme of VMs or downgrade back to 2.12.1.

[Perttu]

Thanks for the tip avang82​. Okay now I'm seeing this 'Timed out waiting on NGVC after 150 seconds' also with 2.11 agent. What next?

Here is our setup:

Horizon 7.3.2

AppVolumes Manager 2.12.1

UEM 9.2.0 NoAD

Windows 7 Ent. SP1 with December patches in.

Open questions.

- This started quite recently.

- No changes done apart from Windows patches.

- 6 cp-replicas in cluster, does this an effect on NGVC? If I understood correctly this service is the 'Instant Clone Composer'.

[avang82]

I've noticed it happening on Agent 2.11 as well.  The only thing I have not tried yet is reverting both Manager and Agent to a previous version.

[avang82]

Just got off the phone with VMware support.  Apparently Manager1 registry entry being deleted has been a known bug since App Volumes 2.11.  The occurrences have just become more frequent with 2.12. and 2.13.