Unstable environment with UIA+profile writable volumes, appstacks, UEM

[PFS]

Hey guys,

We really want to get on board with this JMP style, and are giving it a really good go, but it seems that no matter what we try, something is always broken.

As the title suggests, we are using UIA+Profile writable volumes, around 10 Appstacks (2 of which are heavily used - Adobe Acrobat and Java for ADP), and UEM.

Our App Volumes Agent and manager is 2.14.2, but we've had issues with every upgrade so far (and solved some), and UEM version 9.4.

Our issue is mostly with App Volumes Appstacks. Our Adobe Appstack constantly breaks with the error of "Adobe Application Manager, required to start your trial, is missing or damaged."

Previously, when a user had any Appstack assigned to them, upon a recompose, their Office would fail to license correctly and even in 2.10, it would completely fail to open.

This seemed to be resolved with the combination of upgrading to 2.12 and enabling shared computer activation for Office.

We are using dedicated-assignment linked clones which do not refresh at logoff, as well as an instant clones and finally, dedicated standalone vcenter computers, but the issues happen on all of them, with the potential exception of vcenter standalone computers. I know that AppVolumes is designed to work in a non-persistent environment, but even those have the same issues.

Now I'm testing with Writable Volumes UIA only + UEM profile redirection, but of course, the start menu doesn't work. We had the same issue with Writable Volumes UIA + Profile, and I was able to get it working by excluding a specific file path in snapvol.cfg (by the way, why does so much have to be done on my end for it to work normally)

I have submitted many SRs for VMware, but it seems that it will work for a while and then suddenly someone will stop working and I'll have to reopen the SR and then we won't really get anywhere again. It feels like I'm chasing my own tail here.

I know this is a lot for one discussion, but my biggest concern is that I don't feel like I'm doing anything out of the ordinary or necessarily wrong, and it is strange that I'm doing so many things to try to stable the environment and NOTHING is consistently working. The only real thing I've fixed with AppVolumes is the Microsoft Office activation.

But the Adobe appstack - COME ON. What is going on here? It's like it will work one day, and then the next day, for no reason at all, filed go missing from C:\Programdata\Adobe. Why would files remove themselves with no instigation? It works for the first time on provisioning and then I'll notice that in another session some files disappear or maybe the permissions on the folders becomes corrupt.

Is there a whole side to this I'm not understanding? I've heard success stories from large businesses with App Volumes + JMP, so what am I doing wrong? Someone please help, this is out of control.

[pchapman]

It sounds like your problems are related to writable volumes.  They are an absolute nightmare to use in practice.

"But the Adobe appstack - COME ON. What is going on here? It's like it will work one day, and then the next day, for no reason at all, filed go missing from C:\Programdata\Adobe. Why would files remove themselves with no instigation? It works for the first time on provisioning and then I'll notice that in another session some files disappear or maybe the permissions on the folders becomes corrupt."

This sounds to me like something is getting corrupted on the writable volume.  Have you mounted the writable volume on a machine without the appvol agent on it to investigate?

I think you really need to consider getting rid of writable volumes, even if it means changing the architecture

[GTO455]

As far as a UIA+Profile, we have been told that we shouldn't use them (think roaming profiles) and to stick with the UIA only with UEM profile redirection (no AppData), which we have done. Supposedly, UIA+Profile can create havoc with Windows upgrades and updates. We have enough problems already, so if I can eliminate one ahead of time, I will.

Also had the same problem with the Start Menu that you are having. Used this KB to fix it , but it was still flaky after the fix, (there was a significant delay when you clicked on the Start Menu before it loaded), but it worked.

I finally found a thread that pointed the issue to the Windows 10 Default Template in the OSOT being the problem. I changed over to using the Login VSI template for optimization and that fixed the problem completely. No more delays and the Start Menu now loads normally.

I can completely relate to your pain, you fix one thing and it creates three others. It's not even close to being consistent. Nothing but problems with App Volumes.  It's killing my login times, and I have only 1 App stack and a writable loading for every user.

I have a thread on here that has gone unanswered that involves a firewall command that runs during the writable mount that takes 50 seconds to run in my environment.  From what I can understand from it, it runs as an elevated cmd process that;

1. Deletes the current Windows Firewall config file it (AppVol) creates (See #3),

2. Export the current firewall config, and then

3. imports the firewall config it just deleted.

It does this EVERY TIME a writable loads. Whut? Why?

The other problem I'm having is if delete my writable and have it recreate at login, the first and second login time is great with the new writable, and then it gets progressively worse with subsequent logins until I delete it and recreate it again.

Hope this helps.

Chuck

[Ray_handels]

With Windows 10 it is quite hard to work with the writable I must conclude to that. We are still using it with UIA + Profile though but it takes quite some effort to get it to work for about 90%.

One thing I noticed though. If you use writable volumes you need to refresh the machines after use, otherwise the machine will still have some files and other crap in the VDI machine that is being imported into the newly attached writable.So make sure to refresh after use!!

We are using dedicated-assignment linked clones which do not refresh at logoff, as well as an instant clones and finally, dedicated standalone vcenter computers, but the issues happen on all of them, with the potential exception of vcenter standalone computers. I know that AppVolumes is designed to work in a non-persistent environment, but even those have the same issues.

[sjesse]

I think the +Profile part is what really kills things. Even vmware's internal team won't use the profile part because it causes alot of problems. Watch the vmworld video if you haven't

VMworld On-Demand Video Library

I asked about our writable problems at the end, and they said once they switched to the uia only template alot of the problems went away.

[PFS]

Apologies, I didn't clarify that we don't use writable volumes for linked clones (for those, we use a Persistent Data Disk). We only use writable volumes for the instant clones.

@ray_handels

[PFS]

You've hit the head of the nail with what we're experiencing. Can delete writable volume and recreate it - it works perfectly for maybe one or two days.

How does this even happen?

I'll have to check out that OSOT setting you mentioned. While testing with the writable volume uia only, it had the same start menu delay you mentioned.

So you don't specify UEM to redirect the AppData folder? Why not? Does this not cause issues?

[PFS]

Thanks for your reply.

I'd like to App Volumes for instant clones but the trouble is getting it to work properly. I'm also testing with UEM.

What do you mean mount the writable on a machine without the agent installed? I didn't even know that was possible. How do I do that and then what would I do after

[GTO455]

You can open your writable within the VM it is mounted to. Just open File Explorer and type C:\SnapVolumesTemp and the writable will open up. It's just a hidden directory in Windows.

You can also open up your writable in 7Zip, and I find it easier to read this way. Copy the writable vmdk file (the 10GB file not the metadata file) to a different location (i.e., your local hard drive) and install 7Zip on your PC. With 7Zip installed, just right click on the vmdk file and select Open with 7Zip. Very cool!

BTW- Using the LoginVSI Template in OSOT does make login times a bit slower, but at least you have a snappy Start Menu. Trade-offs I guess!

[GTO455]

When I redirected the APpData folder in my Dev environment, all sorts of weird things started happening; Desktop icons went away, folders stopped redirecting, etc. So I made the decision not to use it in Production.

It is essentially a roaming profile at that point. Even the Vmware Technician suggested we not use AppData redirection in UEM.

[Ray_handels]

You can open your writable within the VM it is mounted to. Just open File Explorer and type C:\SnapVolumesTemp and the writable will open up. It's just a hidden directory in Windows.

Yes you technically can but it wont do you any good.

Because the Appvolumes filter driver is active it will show you all folders located on the C drive of the machine and it wont be representative of was is actually on the machine.

Just create a basic machine without agent, go to edit machine, add disk, browse the datastore and go to cloudvolumes\writable and add the disk you want. If it does not connect persistent either attach it non persistent (you cannot change anything) or change the ddb.deletable flag of the vmdk... Just search the web for that one .

Do change settings in the writable at your own risk. It could make the writable non usable anymore.

[PFS]

Thanks for the help, I'm just not sure what to look for when the disk is mounted

[Ray_handels]

It will mostly be the info in the svroot folder, these are the files stored in your writable.

You will also find a snapvol.dat file on the root of the writable. When you load that as a Hive in regedit you can read throguh it.

I find that most of the times you need to look into the metadata\FS hyve. This should be a one on one with the files located in the svroot as it registers what folders are being created in the writable (this is both for UIA + Profile and UIA only). Most of the time you will find some #DELETED# values inthere. What this does is mark folders as deleted.

So for example if you have a C:\Temp folder in your GI and you remove it while writable is active it will create a value in the hyve with a temp folder on C in which it creates a #DELETED# value and will not show the C:\Temp folder anymore.

Sometimes this technique also masks important folders and they will then stop working as (AFAIK) is user triggered, Is system tries to create a folder it won't create it and thus throw some ugly errors.

[HappierIT_TSmit]

Outlook ost and Search Index and Writeable Volumes

I ended up only using writable volumes for *.OST files and the windows.edb file (windows search).

Everything else is handled through UEM.

If i could have used Profile + UIA for everything I would have, it would have been a lot less config work for instant clones.

I ended up creating a custom snapvol.cfg that only captures c:\outlookdata\OST H/T App Volumes and OST files. | Age Roskam

I also capture c:\ProgramData\Microsoft\Search\Data\Applications\Windows and run the scripts referenced here VMware Knowledge Base

you cannot use %userprofile% or %username% variables in the snapvol.cfg hence the relocation of the OST file.

the most stable I've had the writables in my environment is to only virtualize the two paths

scope=global

type=writable

################################################################

# File system

################################################################

virtualize=\ProgramData\Microsoft\Search\Data\Applications\Windows

virtualize=\OUTLOOKDATA\OST

################################################################

# Registry

################################################################

################################################################

# File system exclusions

################################################################

################################################################

# Registry exclusions

################################################################

################################################################

# Process exclusions

################################################################

################################################################

# 64-Bit OS exclusions

################################################################

# This should always be the last line in the policy

os=any

[mchadwick19]

@HappierIT_TSmith

That SnapVol.cfg seems to stabilize my environment as well for the sole purpose of storing my Outlook data file.

Also drastically improves my login times which is great!

[Skocza]

Hi TSmith,

May please ask if this, your solution below reg. snapvol.cfg configured to capture only OST and Search Indexing is still relevant or has this been all fixed with already released last version of AppVolume (2.16) and UEM 9.7) ? I am not reffering to clean up the sanpvol.cfg as that is still messed up with a lot of stuff that we don't need to capture, so can be done, but only reffering to that restriction for Writabel such as configured paths in it (  virtualize=\OUTLOOKDATA), is that still needed to capture the OST file correctly? Does not last version of UEM handle it correclty even without that path specified in snapvol.cgf for writable?

BTW. when I set it up the way you have for writable volume and then configured UEM (9.7), based on Age Roskam Create a custom writeable volume for specifically the Outlook OST and Search Indexes | Age Roskam  , it didn't stored the OST in that specified path as for Age, si it that we use user rights only for the user instead of Admin rights (and that is the reason why he doing that restriction and you as well, the admin rights and restrict users to install nothing on writable)?

Thanks in advance for your answer!

[sappomannoz]

Hi,

if you have some kind of Microsoft subscription you should be able to use FSLogix for free. It's a great solution for profile persistency and more, and it actually works!

[rcscott44]

Sounds like I am in the minority, but I have had the exact opposite experience with Writable UIA + Profile and UEM.  When we used UIA+ Profile volumes only, our logon times were sub-30 seconds.  When I added on UEM, our logon times jumped 100-150%.  I tried using UIA only writables, but some of the specialized apps we use would not maintain persistence for user experience on UEM only.  I tried every method of UEM app profiling and could not capture all the relevant app data. 

We have reverted to UIA + Profile and dialed back UEM to capture user setting for only the most relevant apps and map printers, network drives, and shortcuts.  We still need to delete and reissue a writable volume about 2x per month per 100 users.  But, UEM allows the reconfigure to happen in about 5 minutes and almost no loss of user settings (just those specialized apps)

We deployed UEM at Horizon 7.5, UEM 9.4 and AppVol 2.14.  I have since upgraded to H7.7, UEM 9.6, and AV 2.15.  No noticeable performance difference.  We also run Win10 LTSC 2019 desktops with Office 2016 installed on the Master Image.  I used the Image prep tool to scan the base image, but manually made the changes it recommended (About 75%) Allowing the tool to make changes causes issues later in production.  Also, we had to create Desktop Pool-Specific AppVolume Provision machines to get any App Stack that has additional Office Apps (like Visio or Project)  That has to do with AD-Based Activation and how Office handles it.  The Provisoning PC needs to be a clone of the Golden Image that has the core Office installation in order to successfully attach and use the AppStack of Visio.  I can't find it, but I believe Adobe has a similar requirement in order to work correctly.

-Bob

[Ray_handels]

rcscott44​ your not the only one. We are also using it like that. We didn't use UEM at all and when we do needed to recreate the writables everyone would lose all info.

We stayed on the full writable, only add managed apps we provide in UEM. We do see a reset of a writable of about 2 or 3 a week (of over 2000) so that's not that much to be honest.

We do use the remove from profile option from UEM. It gives you a bit more options to restore user settings from UEM, otherwise they are stored in the writable and removing stuff there is tedious to say the least..

When looking at logintimes and stuff that happens during login most of the time Appvolumes is not to blame for the slow logon times. Thing is that stuff that you load during logon with UEM is fully penalized to the logon time (not the Flex stuff off course). Appvolumes only loads writable info before releasing logon which in our case mostly takes up about 3 to 5 seconds. After that the W10 logon proces starts, it will be a bit slower due to Appvolumes also merging info during that period but that's just a case of having enough resources on your VDI machine.