Not sure if you've figured it out but if you intend to restrict access, then you need to assign it some role within the vSphere Inventory ... or else these users will NOT have any permissions and by default, the vSphere UI will not allow logins that have no permissions (this is default behavior)