no SSH access after upgrade to vSphere 4.1

[Erik_Zandboer]

Hi all,

Just upgraded my test environment to vSphere 4.1. All appears to work, but when I tried to login using putty (ssh), I got "access denied" on the user account especially created for that purpose. After logging in on the host directly using the VI client, I can see the user "vmware" sitting there. "Grant shell access to this user" is marked. I tried to reset the password, use a more complex password, created another user (vmware2) with shell access enabled. Nothing helps. I login using putty, and keep getting "access denied", just as if the account is not granted access via SSH.

I have no easy option for now to login on the console directly, so I cannot enable root access for the time being as well.

Anyone seen this?

Visit my blog at http://www.vmdamentals.com

[ScottBentley]

I may be way off the mark, but could the firewall have been re enabled as part of the upgrade?

[DSTAVERT]

Have you checked to make sure the SSH service has been started.

[Erik_Zandboer]

I must admit I did not check the firewall yet. But SSH server is enabled already... I assume sshd is already running, because port 22 does reply if I telnet to the host to port 22. It replies with "SSH-2.0-OpenSSH_4.3"...

Visit my blog at http://www.vmdamentals.com

[ScottBentley]

If it is responding on port 22 I guess that rules out the firewall theory as well

[Erik_Zandboer]

Got a change to login to the console directly. sshd was indeed running. I enabled PermitRootLogin, restarted the sshd daemon. Now I can at least putty into the hosts again, but still I cannot gain ssh access using the user account. So yes I can do my thing, but no real solution yet.

Visit my blog at http://www.vmdamentals.com

[Erik_Zandboer]

Tested some more. I activated the new Authentication Service. And that works. My AD account can login via SSH. But I want a local user to be able to SSH in as well (because not being able to login using a local user would imply that when the AD is down, so is my ability to SSH in).

Looking in the /var/log/secure file, I see the following entries when I try to login:

Jul 14 08:47:48 Terrance sshd[1243]: Failed password for vmware from 172.31.0.183 port 52527 ssh2

Jul 14 08:47:51 Terrance sshd[1243]: pam_access(sshd:auth): access denied for user `vmware' from `172.31.0.183'

Jul 14 08:47:52 Terrance sshd[1243]: pam_unix(system-auth-generic:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=172.31.0.183 user=vmware

I checked /etc/passwd and /etc/shadow . They both show the vmware user (and do not show them when I remove the user). I even tried to remove the /home/vmware folder and have it recreated when I recreate the vmware user. All to no avail; the user vmware still cannot SSH in?!?!

Visit my blog at http://www.vmdamentals.com

[Erik_Zandboer]

Just upgraded a completely different environment to vSphere 4.1 . Same problem. I am starting to think this is by design?

Visit my blog at http://www.vmdamentals.com

[schepp]

Does ssh -v or ssh -vvv tell you more about why access is denied?

Did you try to create a user as root on the command line via useradd?

Did you check if the shell of the user vmware in /etc/passwd was set to /bin/bash or something and not /sbin/nologin?

Regards

[Erik_Zandboer]

I tried to ssh -vvv from one host to the other. Part of the output of this:

<......>
debug1: Unspecified GSS failure. Minor code may provide more information
Unknown code krb5 195

debug1: Unspecified GSS failure. Minor code may provide more information
Unknown code krb5 195

debug1: Unspecified GSS failure. Minor code may provide more information
Unknown code krb5 195

debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /root/.ssh/identity
debug3: no such identity: /root/.ssh/identity
debug1: Trying private key: /root/.ssh/id_rsa
debug3: no such identity: /root/.ssh/id_rsa
debug1: Trying private key: /root/.ssh/id_dsa
debug3: no such identity: /root/.ssh/id_dsa
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
vmware@192.168.0.34's password:
debug3: packet_send2: adding 64 (len 60 padlen 4 extra_pad 64)
debug2: we sent a password packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-with-mic,password
Permission denied, please try again.
vmware@192.168.0.34's password:

Scary stuff... I double checked the /etc/passwd file. Shell is set to /bin/bash.

I wonder if anyone who upgraded from 4.0 to 4.1 does NOT get this problem? (Since two upgrades up till now reflected the same problem)?



Visit my blog at http://www.vmdamentals.com

[schepp]

I'll do some upgrades today and post my upgrade-horrors here

[PuliSukumar · Accepted solution]

Hi All,

This is the design change in ESX 4.1. As per new design only Power Users for Service Console and Administrators in VMkernel can log-in to console using ssh. Users without these privilages can not log-in to Service Console.

The same is captured in Documentaion. Please check the "Note" in "Post Upgrade Considerations" section in Upgrade Guide (http://www.vmware.com/pdf/vsphere4/r41/vsp_41_upgrade_guide.pdf).

Snip from the Document:-

&lt;snip&gt;

NOTE After upgrading to ESX 4.1, only the Administrator user has access to the service console. To grant

service console access to other users after the upgrade, consider granting the Administrator permissions to

other users.

&lt;/snip&gt;

[Erik_Zandboer]

@PuliSukumar: You hit the nail on the head. Still a very strange "note" to make considering the "administrators" part. Why not tell exactly what is needed. I added the vmware user to the "root" group and now the vmware user can login again using ssh.

Thanks all for your time and effort on this!

Visit my blog at http://www.vmdamentals.com

[cblomart]

There is a limitation with /etc/security/access.conf

I'll have to get a look on service console users...

Cédric

[degustator]

I'm also very confused with the change. Earlier the best practice was to have permitrootlogin=no, SSH with non-privileged user account then use su. Now if we are supposed to SSH directly with administrator account it seems to me quite similar to having permitrootlogin=yes. Why has the best practiece changed?

[hawk1278]

I have been testing out ESX 4.1 in my lab. I did some digging around on the Service Console and noticed that in /etc/security/access.conf only root, vpxuser, and vslauser were the only accounts that had login access. The last entry of -:ALL:ALL was keeping others from logging in to the SC via direct console or ssh. To allow access for a specific user you can do +:username:ALL or for everyone to have SC access to +:ALL:ALL. To setup the access.conf with more granualarity do a man on access.conf.

[cblomart]

For all the test i've done, modification to /etc/security/access.conf are not retained after reboot.

So either add a script to reapply the needed changes or...

Rely on existing settings.

[alvinswim]

actually if you login directly to the host via the vSphere Client, just add your COS user as an administrator on the permissions tab, and that should save it.

you won't need a script after that.

[Spad]

I'm also very confused with the change. Earlier the best practice was to have permitrootlogin=no, SSH with non-privileged user account then use su. Now if we are supposed to SSH directly with administrator account it seems to me quite similar to having permitrootlogin=yes. Why has the best practiece changed?

I'm just as confused; the whole point of having a non-privileged user account for SSH access is so that if a 3rd party app, script or person that uses it gets compromised it can't do any real damage to your host, but if you now have to make them all Administrators anyway, you might as well just permit root login and be done with it.

[cblomart]

The best practice now is to limit console to VMware stuff. No extra apps. For now Hardware monitoring is "supported" but i have the feeling it will be done with specific vmware package in the future.