I'm trying to boot a Mac VM actually running in VMware Workstation Pro 17.01.
I tried two avenues:
first I powered it on to firmware and then create a boot option into recovery mode. I tried a few options among the <.diagnosis> options available, but none worked. (It booted into normal OS, no matter what changes I made).
Second I modified the vmx file to include there a directive to boot into recovery (one line at the end of the file). In this case, the MacOS VM falls into a loop at the time of boot and remains there until powered off. (a message is displayed, but the loop is too fast and it can't be read).
It is a violation of Apple's licensing to run their OS on non-Apple hardware. We cannot help you here or VMware could get into legal trouble.
Also, the only way of even attempting to run MacOS on Workstation is to use an unauthorized crack program to alter the Workstation program. So you've come to a vendor's site admitting you hacked their software and are asking for help?!?