Hi Team/Guys/Experts.
Quick question on how can I make a specific user to only have the access to SSH and the others cant do on a specific host only.
Have you tried these steps ?
https://www.ostechnix.com/allow-deny-ssh-access-particular-user-group-linux/
Please test it on a test server before you do it on production server. I have not tried on ESXi so far, however the file is there and you can give a try.