Hello,
How to allow access for a specific user to be the Only user that have SSH to ESXi 6.5 ?
Maybe the Normal Lockdown Mode will help you here: Lockdown Mode Behavior
You could specify the user(s) as exception users and if they have admin privileges on the ESXi host they can login via SSH. Or you specify them in the advanced option "DCUI.Access".
Probably the exact same way you do it on *NIX environments: edit /etc/ssh/sshd_config with the AllowUsers parameter.