My Setup: ESXi 5 free edition. Windows 2008 R2 Server as Domain Controller with a Domain like this: "subdomain.contoso.com". There is no domain just like "contoso.com".
The ESXi is a member of of said "subdomain.contoso.com". It joined the domain without a hitch. In the "Permissions" tab, I can add users and groups from that domain and add permissions to them. In the permissions dialog, I only see "contoso" as domain name.
Here is the actual issue:
When I am logged into a PC as user "contoso\esxiadmin" and try connecting to the ESXi server with the vSphere client by checking the box "Log in with Windows Session Credentials" (Sorry, I do not use the English client, so I kinds have to guess what the translation would be), I get an error that I entered an incorrect username or password. I get the same error if I log into windows as "subdomain.contoso.com\esxiadmin".
If I try to manually enter the username and password in the form "contoso\esxiadmin" and the correct password, I get the same error.
If I use the form "subdomain.contoso.com\esxiadmin" and the correct password, I can login. However, I would like to be able to login with the Windows credentials to save some typing. Unfortunalely, I cannot create a "contoso.com" domain as an easy fix 
.
To make things even weirder, the events log on the esxi host shows that the user "contoso\esxiadmin" logged in and then immediately logged out again. The timestamp for both events (logged in and looged out) is the same down to the second.
Any ideas or suggestions would be kindly appreciated.