no SSH access after upgrade to vSphere 4.1

[Erik_Zandboer]

Hi all,

Just upgraded my test environment to vSphere 4.1. All appears to work, but when I tried to login using putty (ssh), I got "access denied" on the user account especially created for that purpose. After logging in on the host directly using the VI client, I can see the user "vmware" sitting there. "Grant shell access to this user" is marked. I tried to reset the password, use a more complex password, created another user (vmware2) with shell access enabled. Nothing helps. I login using putty, and keep getting "access denied", just as if the account is not granted access via SSH.

I have no easy option for now to login on the console directly, so I cannot enable root access for the time being as well.

Anyone seen this?

Visit my blog at http://www.vmdamentals.com

[timmp]

Yes this is frustrating. I encountered this issue and only now did I see this post.

I did try both a clean install of 4.1 and an upgrade from 4.0 U1 or U2 to 4.1 using esxupdate from cmd line. I encountered the same issue. I tried opening fw ports and enabling the "active directory kerberos" option and that did not work. While I understand VMware's intention to allow AD integration through the VI Client I never thought that "feature" would break an existing useful setup using kerberos to connect to an AD DC. I too do not allow ssh root login. This now creates a rather unuseful setup since I login via an AD account to authenticate and then su to root. This is yet another "enhancement" to force people off ESX and on to ESXi.

I am not going to allow ssh root access and do not want to use the authenticate users feature for AD. This blows!

[drosala]

Wow, this is crap! I don't register my unix machines in AD, so the domain name isn't recognized. I am very surprised at how Windows centric VMWare has become, considering that the base product ESX is Linux. Come on VMWare use your heads, and not limit us to one OS! Time to start looking closer at Xen.

[Erik_Zandboer]

You do not NEED to add your ESX nodes to your AD. In fact, I would vote against that in the first place. If you have a serious problem and your AD is down, you would no longer be able to log into your nodes. You can simply create another user, and add that user to the 'root' group. It works, but it is not preferable security-wise.

VMware has been focussing on Windows. That is simply because the largest part of the VMs running on VMware are Windows machines. They simply follow the majority of their customers.

And the prime reason for me to reply: ESX is definitely not Linux based. It's a beast on its own. This misconception was introduced by the ESX installs that use a Redhat installer.

Visit my blog at http://www.vmdamentals.com

[cblomart]

I'm not worrying about the availability of our AD as it is not a single server and is fairly resillient.

In case of problem with AD we still have root access trought out of band.

An integration with AD is interessant to avoid accounts definition multiplication on large infrastructures. Managing root and Administrator accounts can be a security hell not to talk about extra "standard" local users.

ESX is certainly not linux but management access has always been based on a linuxish base.

Before it was based on a Redhat and now on a busybox but it still has a prompt, basic commands and interpreters (python). Simply to make it accessible to administrators. How by the way should not mingle with it, i recon. There is a middle bettween locking the "hypervisor" and allowing management.

If i understand it, i nevertheless find this "locking syndrom" annoying:

-no deployement tool integration (Altiris Agent)

-not standard hardware monitoring (Dell OpenManage, HP System Management Homepage, ...)

-no integrated authentication for Virtualization admin and troubleshooting (it is helpfull to be able to stop or restart a VM when the host is not communicating with vCenter anymore).

-...

(we're talking about tens to a hundred esx(i) hosts and the equivalent of physical servers which have these tools)

Cédric

[drosala]

Not only does "ESX installs that use a Redhat installer", but ESX needs linux to boot. So you cannot say there is a "misconception" that it is Linux based.

Quoted from,

<div>Up through the current ESX version 4.1, a Linux kernel is started first,^{[[5]|http://en.wikipedia.org/wiki/VMware_ESX#cite_note-esxbootvideo-4]} and is used to load a variety of specialized virtualization components, including VMware's 'vmkernel' component.

-And-

ESX uses a Linux kernel to load additional code: often referred to by VMware, Inc. as the "vmkernel".

</div></div>

Without Linux ESX just wouldn't run.

Also as to the numbers of guests running on vmware... I again think you are a little misguided in your opinion. We have just about even numbers between Windows and Linux, and it would be short sighted to develop for one OS or even focus for one OS in this day of getting the most ROI. If you are going to treat them as red hair stepchildern, then companies will go elsewhere... i.e. Xen.

[Erik_Zandboer]

But it is a misconception. First of all, ESX does not use Linux after boot. So the hypervisor works completely on its own. And that is what counts: You do not want an operating system (any operating system!) inbetween.

Secondly ESX does not really need, but uses Linux in order to boot. That is just something from the early years of ESX... They needed to get something booting on a server. So why not use something already available? Linux was a logical choice. They could have used any other "open" operating system such as... ehm... Well that's probably why they choose Linux

My opinion as to how many VMs of which OS-type run on VMware environments is completely irrelevant... VMware states (or at least stated in the past) that over 90% of running VMs are Windows based. So that is what they primarily aim at. And I understand that. They choose to aim at a platform that most of their customers use... If you don't follow the majority of your customers when decision making, you'd be doing something wrong.

But make no mistake, I loved ESX 2.5's web interface, and hated to see it go in ESX3.0. From that time on it was "Windows required". So yes, I think VMware might have overshot the target. You could still please Windows users with any virtual appliance delivering a web interface that covers it all... But I guess you also have to look at how fast you can develop a tool that works... Given the knowledge of your dev teams.

So as you see, there is much much more to it than just thinking they like to piss off Linux users... And they don't if you look at the vast amount of (Linux) OSses supported!

Visit my blog at http://www.vmdamentals.com

[Cruicer]

Found a good little work around on this, worked on several of my upgrades that I just completed...

One of the changes that VMWARE did was to utilize the Pluggable Authenication Module (PAM), earlier version this was set to "no", to change it, it is pretty easy.

Going to need to get access to the console either by enabling root logon for a bit or by a iLO session or dare I say it, stand in front of the host with a keyboard and a monitor...

Use your choice of a editor and modify the /etc/ssh/sshd_config

Look for this section

# Authentication:

#LoginGraceTime 2m
PermitRootLogin no    <-------- change this to "yes" to enable root login
#StrictModes yes
#MaxAuthTries 6

Then this section for PAM

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication mechanism.
# Depending on your PAM configuration, this may bypass the setting of
# PasswordAuthentication, PermitEmptyPasswords, and
# "PermitRootLogin without-password". If you just want the PAM account and
# session checks to run without PAM authentication, then enable this but set
# ChallengeResponseAuthentication=no
#UsePAM no
UsePAM yes

By the new default in ESX 4.1, UsePAM is set to yes, change this to:

UsePAM no
#UsePAM yes

Save it restart the sshd service and you are good to go with a SSH session.  Ohh yea, don't forget to set the PermitRootLogin back to no

[EdMard]

I had the same problem when I upgraded from 4.0 to ESX 4.1 but only on one host. I have two hosts and I upgraded them exactly the same way through UM. One host came up fine and I was able to log in locally with my account (PermitRootlLogin is always set to no for our environment). The second server upgraded fine as well but when I tried to login locally with my account, I had the "access denied" issue.

I checked the esxcfg-firewall and everything else but nothing worked. I logged in as root to the local machine and checked out the ssh_config and sshd_config were were the same. I did eventually locate the source of the problem which was the /etc/group file. I noticed that on the server that I was still able to log in to, my user was member of the root group whereas the other server did not have my user listed. I added my user back to /etc/group and voila! it worked:

Login in as root and add your user to the /etc/group file:

vi /etc/group:

root:x:0:root,user1,user2,user3,etc

Save the file:

:wq

Login using your favorite SSH client.

Ed

[cblomart]

Is the change retained after reboot. I might be pessimist but i doubt it.

I just seen that you could do persistent changes by adding the necessary files to the oem.tgz file on the Hypervisor1 partition.

ie:

1. tar -zcvf oem.tgz /etc/group

2. cp oem.tgz /vmfs/volumes/Hypervisor1/

Take care... i doubt this is supported by vmware...

Disclaimer<http://www.minfin.fgov.be/disclaimer.htm

[EdMard]

Sorry for the delay. I finally had a chance to reboot my servers last weekened after patching and I was still able to SSH after the reboot. No other changes were required.

[ScottBentley]

I am currently away from the office, If you have an urgent query or support request please contact technical@cbs-it.co.uk or telephone 0116 2643700

Scott Bentley

MCITP MCSA, MCSE Security & Messaging Specialist

Technical Manager

Central Business Services

Technical DDI : 0116 264 3700

E-mail : scott.bentley@cbs-osn.co.uk

Telephone : 0116 277 8111

Facsimile : 0116 278 8332

Internet : http://www.cbs-osn.co.uk<http://www.cbs-osn.co.uk/>;

VOIP : SIP/8111@asterisk.cbs-osn.co.uk

[Cruicer]

I will be out of the office beginning on Friday 5/13, and will not be returning until Monday 5/16, I will return your email when I return to the office.