I am not entirely convinced masking would fix the problem. Because if the problem lies in how Ryzen handles 16-bit/32-bit instruction requests from a VM that causes the Windows 98 VM installation to crash, any kind of masking won't fix the problem.
I have managed to install Windows 98 Second Edition on Workstation Pro 12.5.x, Intel Skylake CPU without having to do anything special. The only additional step was to download Creative Ensoniq PCI drivers to get the audio working.
See the screenshot below that I managed to partially fool CPU-Z in the Windows 98 VM to think that it is using an AMD CPU instead of an Intel Skylake. I used the CPUID for AMD K6 processor (family 5 model 6 stepping 2). Note that it is partial as you'd see in the "Instructions" CPU-Z still thinks it has x86-64 (K6 is supposed to be 32-bit only) as well as other technologies that came several years after K6. It includes an Intel only technology (TSX). It also sees MMX although I think the way Intel CPUs and AMD CPUs show the MMX capabilities are in different registers/bits. Only MMX and possibly 3DNow! prefetching might have an impact to Windows 98.
I don't know either if AMD masking methods are the same for Intel CPUs so I am not sure if it would have any effect.
If you want to give it a try, add the following lines to the vmx configuration file before the installation. The brand string isn't really important. I just find it weird if I didn't change it, CPU-Z still shows a Skylake string in the "Specification" while showing an AMD logo.
cpuid.1.eax = "0000:0000:0000:0000:0000:0101:0110:0010"
cpuid.brandString = "AMD-K6tm w/ multimedia extensions"
I'll try that later on and see what happens, I take it I'm supposed to add the string at the bottom of the .txt document containing the .vmx file.
Nope, didn't work. Thanks for the info though.
I tried submitting a support request to VMware but for some reason when I get to "Select the Product with the Associated Issue" nothing is listed.
I sincerely hope VMware becomes aware of this issue and should release an update for Workstation regarding Ryzen CPUs.
I've found a solution. You need to go into Safe Mode, then System Properties > Performance > File System > Troubleshooting > Disable all 32-bit protected mode disk drivers.
A side effect of this is that CD/DVD support is lost. But that's easily fixed by enabling the DOS ones: https://www.betaarchive.com/forum/viewtopic.php?t=12175
Performance isn't reduced by any significant amount, and Windows 98 seems to work without issues.