Microsoft's Aggressive EOL Timeline and Horizon Full Clone Issues

[familylifetech]

We have been using Horizon for 150 users with full clones for a few years now but we are starting to second guess this decision in light of Microsoft's aggressive end of life cycle for Windows 10.

Our users require the use of full clones. To meet common security practices, we have to ensure our VM's have the latest security updates so we cannot ignore the EOL for Windows 10 and let them no longer receive updates.

We have been using Pro licenses and we have 1809 VM's being deployed to new users currently. The majority of our users still have 1803 so they will have to change by this November. The pro license of 1809 will be obsolete in May 2020 so their EOL in short as well. Our users are not going to be happy with us giving them new machines regularly and having them re-install their programs every 6 months or less. We are looking into changing the licenses on our 1803 and 1809 machines to Enterprise instead of Pro which will delay this concern out until November 2020 and May 2021 thus give us some breathing room. It looks like this could be done by just updating the license number, so they will be able to keep their current VM and be oblivious to the change.

NOTE: We attempted to try in-place upgrading for 1803 to 1809 but we are running into random VM freezing so it looks like fresh OS builds are our only reliable option.

The problem I am now looking at is build 1903 is setup to expire in December of 2020. THIS IS NUTS!!! Is Microsoft no longer allowing Enterprise licenses more time? VMware doesn't even have support for 1903 yet and nobody I have talked to has a clue as to when they will. Even if we get support for 1903 by the end of the year, this leaves us less than a year for our users to use it before they have to get a new machine and we start all over again. In all practicality, our users would be needing to get new machines and re-install everything every 6 months (based on delays for VMware to support the latest OS and getting everyone changed).

Is anyone else out there using full clones and come up with a reasonable way to work through these issues?

[dolzhenkom]

In this case, it sounds like the issue isn't with Microsoft's aggressive life cycle, but with the fact that you're having issues with in-place upgrades. I would recommend troubleshooting why those aren't working, as that should be a fully supported feature for full clones and makes a lot more sense than reimaging the VMs every 6 months.

You should also decide on a schedule for testing and putting into production the latest versions of Windows 10. In my organization, we start testing the newest Enterprise release as soon as it comes out, then deploy it once the next release comes out and skip that release. In other words, we started testing 1809 when it was first released, and are now deploying it to the enterprise with the release of 1903, which we will skip. The next OS we will test is 1909 or its equivalent. This gives us roughly a 1 year cushion to upgrade Windows 10 before the release we're upgrading from is out of support; we started upgrading 1709 to 1809 in May 2019 with the release of 1903, and 1709 Enterprise is end of life in April of 2020.

[cbaptiste]

I am planning on using SCCM to manage my Windows 10 persistent desktops. Not until I get everything the way I envision it I am not removing them off Windows 7. My recommendation is to do the same. Treat the full clones as desktops the same way you have your physicals. Try to deploy the user's applications through SCCM as well. Easier to reinstall if needed. One more thing I would suggest is to leverage a solution like FSLogix or App Volumes writable volumes to maintain end users applications that are not on SCCM and their AppData folder. I hear great things regarding FSlogix. Never used it myself. I have used writable volumes before and it sucked. I do not think it is worth the headache. I recommended VMware to redesign it using vhd disk instead rather than vmdk. That way it can live on a shared storage, less IOPs, easier to back up, and also easier on the login, etc... Last I checked they thought it was a good idea. We shall see.

[familylifetech]

Thank you for the input. Some things to note though.

We did a very thorough test of upgrading Win10 VM's and everything looked good. Even my personal upgraded VM is still working fine. I don't live in it daily but it works every time I want in it. Once we did our first live user batch (30 VM's) we started seeing issues with many of them having their VM randomly freeze, crash programs and/or the network adapter stops passing traffic. For some people, it happened multiple times in the same day and others after several days. We have found no real pattern yet. We can only power off and on to recover them because restarting just gets a black unresponsive screen. It acts like the VM's resources are shot but none of the logs or inspection tools show that happening. We can hit Ctrl-Alt-Del to get the blue windows response screen but even Task Manager will not load. It is a brutal issue to troubleshoot as it is random and seems to only occur after heavy usage and not on all computers. We have moved a couple of the impacted users to new VM's and I have signed into their old VM to see if we can reproduce their issue, but nothing to report after 2 days. There is nothing like looking for a disappearing needle in a haystack.

Are you doing in place upgrades in your environment then or giving users new VM's? We reached out to VMware support but the response we got is that it is not suggested to do in place upgrades but instead new images. I don't blame them...this is a beast of an issue to try to find a resolution to. If your curious of the upgrade steps we took, I can provide you the details.

Sorry for the long detailed view but it helps me layout the timeline....

Isn't your approach going to be a problem with the 1909 EOL schedule? The Enterprise EOL for 1903 is the same as the Pro license which is not what Microsoft has done in the past. (https://support.microsoft.com/en-us/help/13853/windows-lifecycle-fact-sheet) I would think the following could be possible dates and concerns.

- Lets assume 1909 is released Nov 2019 and EOL is May 2021 (based on what 1903 looks like and assuming the same Microsoft aggressive 18 month EOL approach)

- I am guessing that VMware will begin supporting 1909 six months after release so on May 2020 you can begin testing it. (hopefully it isn't required that the Horizon version be updated but that is a possibility and adds more prep time)

- Spend 1 month prepping, testing, etc. so the environment is ready to give new 1909 machines Jun 2020.

- You can now have existing users get a new VM and re-install programs or (if it has no problems) you touch each VM and do an in place upgrade to 1909...starting in June 2020.

Even if you got all of your VM's upgraded to 1909 in 1 months time (which would be an ambitious goal for our 150 VMs), they are only going to be supported and get windows updates for 10-11 months (until May 2021). Since you are skipping every other release, you would be skipping version "2003" and target supporting version "2009". We assume it comes out on Nov 2020 and it takes VMware 6 months to support it so you can begin testing it on May 2021. Now we have a problem because all of your current users are on 1909 and there EOL is in the same month. Your VM's would not be getting updates until you get them on a newer version. And then the problem continues year after year.

This would be a problem because Microsoft has moved starting with 1903 to an 18 month support window, regardless if you use Enterprise licensing. I hope this isn't really the long term plan and they give us more time on these releases with Enterprise.

[familylifetech]

We are using persona management and folder redirect so our users My documents folders will follow them if we assign them a new computer. We use Desktop Authority as our software deployment tool and we have been very pleased with it. We have it deploying VBS scripts to different things like adding shortcuts and other actions as well.

The main issue we have with both our physical and VDI VM's is how our organization treats these computers. People still expect to be full Admins and they install a plethora of different programs that they "need for work". This makes for moving to linked clones and thinapp a problem. Our long term plan is to move users away from Admin access and use a tool like Thycotic to allow for elevating permissions for approved installers and applications. If we do this first with physical machines, we might get an idea of what programs we need to then try to setup in thinapp and maybe we can move to linked clone persistent disk.

[dolzhenkom]

I think there is some confusion on Microsoft's lifecycle for Windows 10 Enterprise. March feature updates (the ones we skip) are only serviced for 18 months, but September feature updates (1709, 1809, etc.) are serviced for 30 months. This is listed as such in the Windows 10 lifecycle page you linked to (https://support.microsoft.com/en-us/help/13853/windows-lifecycle-fact-sheet), and is the change they made last year specifically to accommodate enterprises that need more time for testing... although with Microsoft, anything can change.

So, assuming Microsoft's schedule and numbering convention remains intact (based on prior SAC releases, this is likely to shift by a few months):

• 3/2020: 1909 is released. We begin testing it internally.
• 9/2020: 2003 is released. At this point, 1909 should have been supported by Horizon for several months. Testing wraps up and production deployment begins.
• 3/2021: 2009 is released. We begin testing it internally, all of production should be on 1909 by now.
• 9/2021: 2013 is released. At this point, 2009 should have been supported by Horizon for several months. Testing wraps up and production deployment begins.
• 3/2022: 2019 is released. We begin testing it internally, all of production should be on 2009 by now.
• 9/2022: Support for 1909 ends, but we've already been on 2009 for 6-12 months.

I don't believe VMware provides SLAs for how quickly they'll certify support for new releases of Windows 10, but they do hint at a quick turnaround time in this KB article: VMware Knowledge Base

Shortly after a new SAC is released, VMware will qualify the new Windows 10 SAC release for the most recently released Horizon 7.x Server and Agent.

[familylifetech]

Smack!!! (The sound of my hand slapping and sliding down my face) :smileylaugh:

I missed that very important detail in the Enterprise September feature updates section going for 30 months instead of 18 which helps. Have you been doing in-place upgrades for your full clones or do you give your users new VM's each time you refresh?

Here are my tweaks to the dates you listed and what I will be reviewing as a possible approach to the refresh schedule also. You listed March but I think you meant September release dates as they are the ones that last 30 months. I rounded down to the nearest month to simplify. The only drawback is that a new staff member could get a VM just before we release the next OS version so we would want to delay making that user upgrade their OS build for a year which means just before the VM will be EOL. This assumes that we will be giving the user a new VM and not doing an in-place upgrade. If we can do in-place upgrade and the user is "none the wiser", it is less of an issue to wait on the timing. We don't want the user to have to redo their programs more often than once a year.

• 04/2019: Start deploying 1809 (currently doing)
• 04/2019-03/2020: Update or give new 1809 VMs to all 1803 users. (from oldest to newest)
• 04/2019: 1903 released = Skip
• 10/2019: 1909 is released and as soon as VMware supports it, test.
• 10/2019: 1703 EOL
• 04/2020: (or sooner) Start deploying 1909
• 04/2020-03/2021: Update or give new 1909 VMs to all 1809 users. (from oldest to newest)
• 04/2020: 2003 released = Skip
• 04/2020: 1709 EOL
• 10/2020: 2009 is released and as soon as VMware supports it, test.
• 10/2020: 1803 EOL
• 04/2021: (or sooner) Start deploying 2009
• 04/2021-03/2022: Update or give new 2009 VMs to all 1909 users. (from oldest to newest)
• 04/2021: 2103 released = Skip
• 04/2021: 1809 EOL
• 10/2021: 2109 is released and as soon as VMware supports it, test.
• 10/2021: 1903 EOL
• 04/2022: (or sooner) Start deploying 2109
• 04/2022-03/2023: Update or give new 2109 VMs to all 2009 users. (from oldest to newest)
• 04/2022: 2203 released = Skip
• 04/2022: 1909 EOL

[dolzhenkom]

Ah good catch, I shifted the dates by 6 months, my mistake. I think most of your concerns will be rectified once you figure out a solution to your issues with in-place upgrades. In our environment, we are using Instant Clones that were originally created from a 1709 Enterprise image, and I have migrated several pools onto 1809 successfully by performing an in-place upgrade on the gold VMs. There were a few minor optimizations I scripted and deployed after the upgrade, such as re-running OSOT since the in-place upgrade undoes many of those optimizations. Aside from that, it was pretty much as simple as patching the VM and testing to ensure everything still works with the new release. Granted, performing in-place upgrades on hundreds/thousands of full clones is more likely to result in failures than doing the same on a handful of parent images, but those should be the exception and treated as one-offs.

Depending on your environment and business needs, I can see you being able to deploy new Windows releases pretty rapidly, maybe within 2 months:

• Internal IT testing for 2 weeks
• Snapshot and upgrade a heterogeneous sample of machines, wait 1 month to identify and resolve issues reported by users
• Phased rollout to remaining VMs over 2 weeks

How you do this is of course totally up to you, but through proper planning and by using in-place upgrades, Microsoft's lifecycle shouldn't be a concern.

[familylifetech]

Our process is well documented so we can build a new VM template from a new ISO relatively quickly. OSOT is definitely an important step and inevitably we find some new feature that we have to edit a registry or GPO to resolve. I'm not too worried about the testing of new images and that turn around. Since we use full clones and not linked or instant clones, we get to touch every VM for an in place upgrade. We don't get to re-compose to do upgrades, they are done one at a time. This sadly is driven by the need for our users to be able to install anything they want on the VM.

I am still banging my head against a wall trying to find out what is causing Windows 1803 to 1809 upgrades to behave poorly. Nothing like trying to find a needle in a disappearing haystack. I may need to create a new post to see if anyone has any good ideas.

[vmstack]

You may have to check the compatibility matrix and see if there is any fix available for the 1809 issue.  For us, the workaround given here worked   -   VMware Knowledge Base

[familylifetech]

We ran into that one when testing and we believe we have addressed it. Note that if you are still using 7.5, you will have to continually change your VM's hpet to True manually when provisioning new VMs. We ended up moving to 7.8 and then all of our new VM's default to True. During our in place upgrading steps we set the value to true for each VM that moved to 1809.

Our current problem doesn't react the same way. The agent is always available but it has the appearance of explorer crashing or becoming unresponsive. I have a VM that I clicked on file explorer hours before it actually brought it up, yet the CPU and memory seem fine based on systeminfo and vsphere. It is bizarre

[cbaptiste]

Personally, first thing first. I would drop Persona management and migrate to User Environment Manager. Secondly, adopt FSLogix, which I am pretty sure is free for your organization. Third, get rid of full clones unless the users are okay with starting from scratch every year or so.

Here's my reasoning:
1) Because agents must be installed in a certain order, it makes it extremely hard to upgrade VMware environments as you will have to revisit each desktops, uninistall, and re-install the agents. That's really not ideal. We have in our organization a full clones environment that was built 6 years ago give or take and it is still running on the same vCenter and the same Horizon View version. We have been migrating them to a newer environment but we certainty can not upgrade. I have spoken to multiple colleagues at VMware and none of them have ever suggested going with full clones.
2) Microsoft crazy upgrade cycle. With each version sometime things are completely incompatible to the point where they break. Example: In 1703, 1803, 1809 the start menu and appx are completely different. I am running two VDI environments simultaneously. One on 1803 and the other on 1809. I just found that the UEM start menu persistent stops working for both as soon as I deployed 1809. Now I need to figure out what to do about it since my 1803 environment is already full blown production and 1809 is scheduled to go into production in 2 weeks.

3) FSLogix allows you to use instant clones and persists your end user's data as if they were on a full clones. Their VHD drives map at logon. The users can install their apps as well. So you can upgrade your instant clones as you wish and it would not make a different since the user's data lives on a separate drive than the operation system.

[familylifetech]

Thanks for the feedback. I am going to read up on FSlogix and see what all it can offer. Because of the requirement to allow users to install programs of their choosing, this has always brought us back to the need for full clones. If we can find a reliable, relatively low maintenance method of providing instant or linked clones AND the users get the same experience of a full clone, that would be amazing but sadly this has been a pipe dream so far. Due to the vast number of apps users install, trying to turn them all into ThinApps has been unrealistic. Thanks for the suggestions.

[cbaptiste]

Give it a try and report your findings. I plan on giving it a spin soon. Just been reading mostly and Microsoft seems to have high hopes. I have a few colleagues at use to be AppSense and they said the way FsLogix works with the filter drivers to integrate with the OS, even they couldn't replicate it so well. I must say AppSense is pretty great suite with a great and smart team behind it. I value their opinion.