Hot Shot
Hot Shot

Office 365 Slow when ANY appstack attached

So we've been looking into some performance issues we have with Microsoft Office and it turns out it is App Volumes causing it.  First I'll explain the issue.  If you are using a Microsoft Office product the first thing you will notice is launching an Office app i.e. Microsoft Word it will take 2-3 times longer if you have ANY appstacks attached to the VM.  If this was the only problem I could get over it, as a 6 second launch time as opposed to a 2 second launch time isn't killer, rather just a bit annoying, but the real kicker is even once the application has launched certain things are very slow within the application and users can't stand it.  An example of this, when you have Microsoft Word open, open a blank word document and go to insert and select Object.  It will bring up this window:
pastedImage_0.png

If you have no appstacks attached this will happen in 1-2 seconds.  If you attach ANY appstacks at all this will take 10-15 seconds.  This isn't the only thing that happens slowly within Office, but it is the easiest to reproduce so I put it on here. 

The real issue here is that this isn't just if you have Microsoft Office in an appstack, this happens anytime you attach any appstack to your vm.  You can literally have a machine with Microsoft Office installed and everything will work fine, and while you are sitting there go to the app volumes manager and deploy an appstack with just notepad++ for instance immediately to your vm.  Now if you try to launch Word/Excel whatever it is significantly slower and doing certain things (as in the example above) is ridiculously slow.  Basically using app volumes at all makes Microsoft Office painfully slow at points. 

This happens if you are using Windows 7 or Windows 10, and App Volumes agents 2.13.0 to 2.14.0 at the minimum as I've tested all of those. Anyone have any ideas as to a way to fix this or help with this?

0 Kudos
31 Replies
Expert
Expert

I do not have a solution to the issue you are experiencing however our environment has the same problem. I've seen the same behavior as you with removing all the appstacks significantly speeds up Office in addition to the OS as a whole.

I did play around with resetting the policies with svservce.exe and that resolves the issues as well but appstacks don't function. With that in mind, I assume its something caused by a policy or its just the effect of having a file system filter driver active.

What back-end storage are you using for your VMs? We use vsan in a hybrid configuration. I'm not sure if that has something to do with it.

0 Kudos
Hot Shot
Hot Shot

I've also had a report of a very similar sounding issue - it doesn't matter what version of Office is in use; the app is sluggish in general. The same site is also experiencing very long logon times when an Office appstack is been attached.


I have a suspicion that the logon time is more to do with anti-virus based on what their onsite IT guys have found with testing, so that may be a separate thing entirely.

Unfortunately, I've not come across any solution to this either. Have you opened a case with vmware yet?

0 Kudos
Hot Shot
Hot Shot

We are using XIO for our storage so it definitely isn't a storage issue.  Our OS isn't laggy really, but there are a few things that I've noticed are bad.  Event Viewer tends to be really slow, and if you try to run procmon it is almost unusable if you have an appstack attached so it isn't just Office, but Office seems particularly bad and is used by all of our users.

0 Kudos
Leadership
Leadership

I reported the insert object delay awhile ago and I'm not sure there is much you can do, the way I shortened it was installing every possible feature of office, but 10-15 seconds is what I still see. If you use something like procmon you see it running through HKEY_CLASSES_ROOT I beleive repeatedly, the it goes through the redirected registry keys as well, they are the registry keys that have snap in the name.Its taking longer because its looking at multiple copies of this registry key I believe. I think its getting better, in 2.10, at least for use I saw it freeze for 5 minutes after clicking the button.

0 Kudos
Hot Shot
Hot Shot

This may be a bit premature, but it looks like the slowdown in our case was down to the inclusion of the spectre/ meltdown patches in the recent MS rollups. We're testing the registry entries mentioned in this thread at the moment:

Slow logins? This might help...

So far, in test at least, this has made a huge impact with Office & logon times - Office is now usable again, and logon times have gone down. May be worth checking if your gold image has one of the rollups mentioned. The blog below seems up to date with these:

https://blog.barkly.com/meltdown-spectre-patches-list-windows-update-help#windows-updates

0 Kudos
Hot Shot
Hot Shot

Yeah, I think the issue is with the registry too,  I guess at least it isn't 5 minutes and is down to like 15 seconds, but that is still pretty bad especially since it doesn't matter if Office is in and appstack or not but rather it happens if you attach any appstack at all.

0 Kudos
Hot Shot
Hot Shot

alsmk2, I tried these registry settings and they do actually help on launch of office products.  Mine launch in 2-3 seconds if I do those keys rather than about 5-6 seconds so that is nice.  That is if security will actually allow me to keep those in place.  As for the objects button within office these changes do help a bit.  It now takes about 6-10 seconds instead of 10-15 so I guess that is good, but probably just masking the issue with better performance in general.

0 Kudos
Virtuoso
Virtuoso

Is any of you guys using VSphere 6.5??

0 Kudos
Hot Shot
Hot Shot

We are using vSphere 6.5 U1

0 Kudos
Virtuoso
Virtuoso

Why I'm asking is because we installed ESX 6.5 on our new hosts (among a bunch of other stuff) and eventually found out that this really hurts performance. The VDI machines from time to time just seem to freeze up without any good reason. No issues in CPU, memory or IOPS.

When we eventually downgraded to 6.0 everything went back to normal..

0 Kudos
Leadership
Leadership

I went from vcenter 5.5 horizon 6.2 and app volumes 2.10 and went to vcenter 6.5 horizon 7.4 and appvolumes 2.12. The only realy issues I ran into is the instant clone issue in horizon, and a few specific app volumes issues that the next versions of app volumes fix. I use the msi version of office 2016 and the delay for the insert object has been the same since we first saw it in office 2013. Performance wise we haven't seen any difference, but our horizon usage is a lot smaller then the environments that some of you run,.

0 Kudos
Expert
Expert

We're running vsphere 6.0 u3. Three months ago we had a serve performance issue after applying the Windows 10 spectre/meltdown patches to our gold images. I was able to resolve that by disabling that on the esxi host and everything returned to normal.

0 Kudos
Hot Shot
Hot Shot

We have no performance issues really other than app volumes related.  In fact you can have Office on a persistent VM with normal performance, and attach a blank appstack and Office performance will drop significantly.  This is obviously and app volumes issue. 

The original spectre meltdown fixes did cause us some performance issues early but vmware pulled them before we ever got to deploying them to anything other than our test environment.  They then came back like 2 months later with patches for esxi that worked better.  We also at that point had put in patched versions of our firmware on the hosts so maybe that helped.  Basically the only performance issue we had was an overall 5-10% hit but definitely not specific to certain apps like Office.  We are actually testing this threads suggestions out as well like poster alsmk2 said:
Slow logins? This might help...

This seems to fix any of the 5-10% hit we had and does in fact help Office work a bit faster, but the problem is the performance still noticeably degrades as soon as you attach an appstack.  Basically anything that queries the registry becomes very slow.  When you click Object in Office it queries the registry for applications that have Object references for Office.  It looks like because app volumes just makes duplicates of the entire HKLM for each appstack attached regardless of what is in it then that query takes significantly longer because it has to query so much more.

0 Kudos
Contributor
Contributor

I have exactly the same problem.

Did you find a solution ?

0 Kudos
Hot Shot
Hot Shot

No there is no solution as of right now.  The suggestion above by alsmk2 to somewhat disable the spectre patches helps quite a bit but otherwise there is no actual app volumes solution as of yet that I'm aware of.  I've had a ticket open for like 3 months with VMware and apparently their stance is that this is acceptable behavior from app volumes.

0 Kudos
Hot Shot
Hot Shot

We've began to slowly back away from App Volumes with new deployments now for the very reason that vmware seem to think App Volumes in its current state is acceptable. The theory of it is amazing, but the execution is beyond poor. Every minor patch fixes one problem, but adds even more issues (usually with some older problems magically re-appearing too).

We're moving down the route of moving apps back into the gold image and relying on good de-duping storage to claw back the storage benefits seen by app volumes. It's a far simpler and much more reliable solution.

With the sites that persist with App Volumes, without a doubt, 99% of the time, we can trace a seemingly random issue back to AV doing something totally stupid; whether that is cocking up View provisioning by continually interrupting composer actions, to causing painful logon times because a company has the audacity to try and use O365/ 2016 (how dare they!).

After several years of AV I've gone from loving it to absolutely loathing it.

View - Unbeatable

vRealize - Awesome

UEM - Amazing

App Volumes - Steaming turd

0 Kudos
Enthusiast
Enthusiast

Any changes in latest 2.13.3 on 2.14.2 versions of app volumes ?

Everything in GI is a no no, thinapps are bad joke and App stacks has whole set of own issues.

0 Kudos
Virtuoso
Virtuoso

Just want to join into the party without the ESX version being as issue.

We see the exact same problem when starting Word or inserting the object in word as the OP said.

The thing is that it is only the first time you start up Word. If you already have word started up (for us it is mostly Word that bugs us) it is rather quick. What we see and which is reproducible every time is that after you start word and you try to edit the opened document (even if it is a new blank document) you have to wait for 5 seconds before you are able to start using Word.

What we found out is that it seems as if Word, when starting up, is running through all appstacks to check and see if it can find specific information. I even saw it looking through appstacks and trying to find mu profile information which will never ever be in an appstack....

If anyone has any resolution please post. And I would suggest raising a call with VMWare about it..This should be fixed, period..

0 Kudos
Hot Shot
Hot Shot

We too are having similar issues.  We have Office 2016 installed directly on our master image however we use AppStacks for other applications.  We are getting complaints about Office (MS Word and Outlook) being slow or not responding.

In our case, everything was fine on AppVol 2.12, but we started seeing this issue after upgrading to 2.14.2.

0 Kudos