To resolve the issue I disabled the app readiness service. Which did the trick, but then if someone tried to click the cog off the start menu nothing would happen. Somehow having that service disabled broke the settings metro app. We found that if you right-clicked and selected personalize it would work for the session (probably a bit different as you're using linked clones). However that wasn't a great work around for a lot of our users, so we implemented redeploying the appx package for it.