vcsa 6.7: Cannot login as root even after changing the password

[stehega]

Hello all,

I am unable to login to port 5480 as the root user (Unable to authenticate user). Also unable to login via ssh. I have already run the procedure to reset the root password but it is still not working. I can login as administrator@vsphere.local and sudo su to become root.n  I can change the password there using paswd but still cannot login

I have created a new SuperAdmin user and set the password and still unable to login using that account.

root@photon-machine [ /var/lib/sso-user ]# tail -f /var/log/vmware/applmgmt/applmgmt.log
2023-09-04T15:39:44.150 [25349]DEBUG:vmware.vherd.transport.authentication_manager:HTTP METHOD b'POST'
2023-09-04T15:39:44.151 [25349]DEBUG:vmware.vherd.transport.authentication_local:authenticateRequest: RPCPath = system.listMethods
2023-09-04T15:39:44.152 [25349]DEBUG:vmware.vherd.transport.authentication_local:authenticateRequest: using Authorization header
2023-09-04T15:39:44.152 [25349]DEBUG:vmware.vherd.transport.authentication:No cookie in request
2023-09-04T15:39:44.152 [25349]DEBUG:vmware.vherd.transport.authentication:Verifying credentials against Linux PAM of user: root
2023-09-04T15:39:44.168 [25349]DEBUG:vmware.vherd.transport.authentication:pam_tally2.get b'Login Failures Latest failure From\nroot 0 \n'
b''
0
2023-09-04T15:39:44.169 [25349]DEBUG:vmware.vherd.transport.authentication:User root login tally 0
2023-09-04T15:39:44.171 [25349]DEBUG:vmware.vherd.transport.authentication:Linux credential authentication FAILURE for user: root
2023-09-04T15:39:44.171 [25349]DEBUG:vmware.vherd.transport.authentication:pam_tally2 reset flag: --reset=0
2023-09-04T15:39:44.185 [25349]DEBUG:vmware.vherd.transport.authentication:pam_tally2.reset b'Login Failures Latest failure From\nroot 0 \n'
b''
0
2023-09-04T15:39:44.186 [25349]DEBUG:vmware.vherd.transport.authentication_manager:HTTP METHOD b'POST'
2023-09-04T15:39:44.188 [25349]DEBUG:vmware.vherd.transport.authentication_local:authenticateRequest: RPCPath = system.listMethods
2023-09-04T15:39:44.188 [25349]DEBUG:vmware.vherd.transport.authentication_local:authenticateRequest: using Authorization header
2023-09-04T15:39:44.188 [25349]DEBUG:vmware.vherd.transport.authentication:No cookie in request
2023-09-04T15:39:44.189 [25349]DEBUG:vmware.vherd.transport.authentication:Verifying credentials against Linux PAM of user: root
2023-09-04T15:39:44.202 [25349]DEBUG:vmware.vherd.transport.authentication:pam_tally2.get b'Login Failures Latest failure From\nroot 0 \n'
b''
0
2023-09-04T15:39:44.203 [25349]DEBUG:vmware.vherd.transport.authentication:User root login tally 0
2023-09-04T15:39:44.207 [25349]DEBUG:vmware.vherd.transport.authentication:Linux credential authentication FAILURE for user: root
2023-09-04T15:39:44.208 [25349]DEBUG:vmware.vherd.transport.authentication:pam_tally2 reset flag: --reset=0
2023-09-04T15:39:44.225 [25349]DEBUG:vmware.vherd.transport.authentication:pam_tally2.reset b'Login Failures Latest failure From\nroot 0 \n'
b''
0
2023-09-04T15:39:44.226 [25349]INFO:twisted:"127.0.0.1" - - [04/Sep/2023:15:39:43 +0000] "POST /applmgmt/RPC2 HTTP/1.1" 500 443 "-" "Python-xmlrpc/3.5"
2023-09-04T15:39:44.229 [25349]DEBUG:vmware.vherd.transport.authentication_manager:HTTP METHOD b'POST'
2023-09-04T15:39:44.231 [25349]DEBUG:vmware.vherd.transport.authentication_local:authenticateRequest: RPCPath = system.listMethods
2023-09-04T15:39:44.231 [25349]DEBUG:vmware.vherd.transport.authentication_local:authenticateRequest: using Authorization header
2023-09-04T15:39:44.231 [25349]DEBUG:vmware.vherd.transport.authentication:No cookie in request
2023-09-04T15:39:44.232 [25349]DEBUG:vmware.vherd.transport.authentication:Verifying credentials against Linux PAM of user: root
2023-09-04T15:39:44.246 [25349]DEBUG:vmware.vherd.transport.authentication:pam_tally2.get b'Login Failures Latest failure From\nroot 0 \n'
b''
0
2023-09-04T15:39:44.247 [25349]DEBUG:vmware.vherd.transport.authentication:User root login tally 0
2023-09-04T15:39:44.249 [25349]DEBUG:vmware.vherd.transport.authentication:Linux credential authentication FAILURE for user: root
2023-09-04T15:39:44.249 [25349]DEBUG:vmware.vherd.transport.authentication:pam_tally2 reset flag: --reset=0
2023-09-04T15:39:44.261 [25349]DEBUG:vmware.vherd.transport.authentication:pam_tally2.reset b'Login Failures Latest failure From\nroot 0 \n'
b''
0
2023-09-04T15:39:44.262 [25349]DEBUG:vmware.vherd.transport.authentication_manager:HTTP METHOD b'POST'
2023-09-04T15:39:44.264 [25349]DEBUG:vmware.vherd.transport.authentication_local:authenticateRequest: RPCPath = system.listMethods
2023-09-04T15:39:44.264 [25349]DEBUG:vmware.vherd.transport.authentication_local:authenticateRequest: using Authorization header
2023-09-04T15:39:44.264 [25349]DEBUG:vmware.vherd.transport.authentication:No cookie in request
2023-09-04T15:39:44.264 [25349]DEBUG:vmware.vherd.transport.authentication:Verifying credentials against Linux PAM of user: root
2023-09-04T15:39:44.276 [25349]DEBUG:vmware.vherd.transport.authentication:pam_tally2.get b'Login Failures Latest failure From\nroot 0 \n'
b''
0
2023-09-04T15:39:44.277 [25349]DEBUG:vmware.vherd.transport.authentication:User root login tally 0
2023-09-04T15:39:44.279 [25349]DEBUG:vmware.vherd.transport.authentication:Linux credential authentication FAILURE for user: root
2023-09-04T15:39:44.279 [25349]DEBUG:vmware.vherd.transport.authentication:pam_tally2 reset flag: --reset=0
2023-09-04T15:39:44.293 [25349]DEBUG:vmware.vherd.transport.authentication:pam_tally2.reset b'Login Failures Latest failure From\nroot 0 \n'
b''
0
2023-09-04T15:39:44.294 [25349]INFO:twisted:"127.0.0.1" - - [04/Sep/2023:15:39:43 +0000] "POST /applmgmt/RPC2 HTTP/1.1" 500 443 "-" "Python-xmlrpc/3.5"
2023-09-04T15:39:45.188 [25349]INFO:vmware.vherd.transport.ssh_access_collector:Feature switch: VCSA_SSH_ACCESS_EVENT disabled.
2023-09-04T15:39:50.187 [25349]INFO:vmware.vherd.transport.ssh_access_collector:Feature switch: VCSA_SSH_ACCESS_EVENT disabled.

The applmgmt service is running:

root@photon-machine [ /var/lib/sso-user ]# service-control --status applmgmt
Running:
applmgmt
root@photon-machine [ /var/lib/sso-user ]#

root@photon-machine [ /var/lib/sso-user ]# pam_tally2 --user root
Login Failures Latest failure From
root 0
root@photon-machine [

Command> localaccounts.user.list

5:
Passwordstatus: valid
Status: enabled
Email: ''
Role: superAdmin
Username: root
Fullname: root

Any ideas would be greatly appreciated! 

Stephen

[MerlevedeN]

Hi Stephan,

maybe this post can help you with the reset: http://vcloud-lab.com/entries/vcenter-server/resetting-root-password-in-vmware-vcenter-server-applia...

[battybishop]

and this is the official VMware KB Resetting root password in vCenter Server Appliance 6.5 / 6.7 / 7.x (2147144) (vmware.com)

[Sachchidanand]

Please see the following KBs helps you getting the desired result:

https://kb.vmware.com/s/article/50113586

https://kb.vmware.com/s/article/59344

Regards,

Sachchidanand

[stehega]

Hi MerlevedeN,

I had reset the password already using this procedure but it didn't resolve the issue.

Thanks!

Stephen

[stehega]

Hi Sachchidanand,

Unfortunately neither of these solutions results in allowing me to log in as root successfully.

The only difference I see in the /etc/passwd file is it set as follows:

root:x:0:0:root:/root:/bin/appliancesh

I changed it to what it shows in the KB but it didn't work.  Still unavble to log in as root even after changing the password.  I can login as administrator and sudo su to become root and also change the password but it doesn't allow me to login to the :5480 page or ssh in.

[Sachchidanand]

For now, you can use the same administrator password for VAMI interface (:5480) to login.

Regards,

Sachchidanand

[stehega]

Hi Sachchidanand,

My actual issue is having to upgrade the VCSA.  The two storage systems it manages cannot be changed once it has been tied to a vCenter. I would have to reinstall the storage systems.

Regards,

Stephen

[Sachchidanand]

Just login to the VAMI interface using administrator account and from the update page you can update VCSA to the desired version.

Regards,

Sachchidanand

[Sachchidanand]

Forget to mention, take snapshot before doing any update activity.

Regards,

Sachchidanand

[stehega]

Hi Sachchidanand,

I need to upgrade this VCSA from 6.7 to 7.0 and I was mounting the ISO to perform the upgrade.  I am not aware of a process to do the upgrade through VAMI.

Thanks again,

Stephen

[Sachchidanand]

Please go through the following KB, it will help you understand updating VCSA using VAMI interface

https://kb.vmware.com/s/article/84322

Regards,

Sachchidanand

[stehega]

Hi Sachchidanand,

Each of the procedures in that KB require the root password of the current VCSA.  As I am unable to connect using the root account any of these procedures will fail.

Regards,

Stephen

[battybishop]

It may be best in the long term to build a new vCenter and then move clusters/hosts across, I know this could be a major task but you will also benefit from not bringing across a lot of old legacy information patches etc. from the old VCSA. 

[stehega]

Hi Batty,

Unfortunately the VCSA manages two storage systems, which once managed by the vCenter cannot be changed.  I would have to reinstall the two storage systems as well.

I cannot understand how the root password is getting changed successfully but not allowed to access the VCSA. Nor how I can create new SuperAdmin accounts and still not be able to log in using those accounts.

Thanks,

Stephen

[CallistoJag]

Can you reboot the VCSA? Is there a monitoring tool that might be causing the root user to get locked out with too many failed attempts? If this happens even changing the pw will not help, but a reboot might give you a window to log in before it locks out again however the fact it shows no failed attempts suggests this is not the case  

[Sachchidanand]

You can login to VAMI using administrator account and in that case root password is not required. please see this specific line:

For vSphere 6.7 and 7.0, vSphere SSO Administrator Credentials  "administrator@vsphere.local" can be used for patching appliance and root password will not be required but it has to be valid (NOT Expired).

I think you have valid root password and it's not expired, although you are not able to login.

Regards,

Sachchidanand

[stehega]

The issue is resolved.  I asked my colleague if we could open an SR with VMware support and he had a look at the system.

He found the following:

root@photon-machine [ ~ ]# diff /etc/pam.d/system-auth*
6c6
< auth required pam_tally2.so file=/var/log/tallylog deny=3 onerr=fail even_deny_root unlock_time=86400 root_unlock_time=300
---
> #auth required pam_tally2.so file=/var/log/tallylog deny=3 onerr=fail even_deny_root unlock_time=86400 root_unlock_time=300
root@photon-machine [ ~ ]#

Someone had been trying to add security to the system and locked root out...!

Thanks all for your help on this issue.

Stephen