Additional information: Have attempted to capture what is happening with procmon, but everytime I do that, it works fine. As if the system is reading the EXE file too fast usually, while when procmon is running it slightly slows the system down and then runs as it should. Tried to turn off the Windows Defender aswell as excluding area and so on, but no change.
Will continue to troubleshoot, but could be this version is just too experimental on 2016.
I think I found the issue, waiting for VMWare support to let me know if they are able to replicate this issue.
The folders that I had the executable in is locked down to only 2 groups being allowed to execute the files. However, Authenticated Users had the access to list content, just not run the files. For some reason when the list permission is there it fails to start the execuatble correctly, as if it doesn't check the other groups before giving the error. So when I set this, even though I have other groups which grants execute permissions it fails with the error message: The system cannot find the file specified.
If I remove the Authenticated Users permission, the other groups that already have the execute permissions seems to kick in and it works fine. Alternatively I could also add the permissions Read extended attributes and Read permissions to Authenticated Users and it also would work. But List Folder alone creates a strange issue on Server 2016 starting Thinapp 5.2.2 executable.
Support not able to replicate the issue, not sure if we run the exact same security and user environments, but this has solved our problem so marking it as solved.