VMware Cloud Community
Trobinox
Enthusiast
Enthusiast

"Remote access for ESXi local user account 'root' has been locked for 120 seconds..."

Hello everyone

I got the following message every hour:

"Remote access for ESXi local user account 'root'  has been locked for 120 seconds..."

I found a lot of information how to figure this out:

Security.AccountLockFailures. Maximum number of failed login attempts before a user’s account is locked. Zero disables account locking.

Security.AccountUnlockTime. Number of seconds that a user is locked out.


But I didn't get out where to find the source IP who is trying to access our ESXi hosts. Isn't it in /var/log/auth.log ?

Thank you for your input.

16 Replies
RAJ_RAJ
Expert
Expert

HI ,

Are you using any management tools on your environment  .

This issue typically comes if any third party monitoring software that is unable to login and locks out the root user.

Use any monitoring tools slike Orion SolarWinds or TCPDump  to identify product/source which is causing the issue.

RAJESH RADHAKRISHNAN VCA -DCV/WM/Cloud,VCP 5 - DCV/DT/CLOUD, ,VCP6-DCV, EMCISA,EMCSA,MCTS,MCPS,BCFA https://ae.linkedin.com/in/rajesh-radhakrishnan-76269335 Mark my post as "helpful" or "correct" if I've helped resolve or answered your query!
Reply
0 Kudos
Trobinox
Enthusiast
Enthusiast

It's a customer environment, so I don't know exactly what they have for monitoring tools. That is why I'm asking for the file who contains the source IP. and such failed login logs.

Reply
0 Kudos
erikverbruggen
Hot Shot
Hot Shot

The log file you are looking for is /var/log/auth.log.

More information can be found in this KB article, https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=20048...

Reply
0 Kudos
RAJ_RAJ
Expert
Expert

Hi ,

Try below

Login to the ESXi console and ran this command

# less /var/log/auth.log

You can see below example when I given wrong password second time it triggered error message

2017-03-02T08:30:57Z sshd[751607]: Connection from 192.168.1.50 port 55000

2017-03-02T08:31:02Z sshd[751607]: Accepted keyboard-interactive/pam for root from 10.20.2.17 port 55000 ssh2

2017-03-02T08:31:02Z sshd[751607]: pam_unix(sshd:session): session opened for user root by (uid=0)

2017-03-02T08:31:02Z sshd[751610]: Session opened for 'root' on /dev/char/pty/t0

2017-03-02T08:31:34Z sshd[751644]: Connection from 192.168.1.50 port 55075

2017-03-02T08:31:39Z sshd[751645]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.20.2.17  user=root

2017-03-02T08:31:41Z sshd[751644]: error: PAM: Authentication failure for root from 192.168.1.50

RAJESH RADHAKRISHNAN VCA -DCV/WM/Cloud,VCP 5 - DCV/DT/CLOUD, ,VCP6-DCV, EMCISA,EMCSA,MCTS,MCPS,BCFA https://ae.linkedin.com/in/rajesh-radhakrishnan-76269335 Mark my post as "helpful" or "correct" if I've helped resolve or answered your query!
Enelass
Contributor
Contributor

I happened to have to same issues on all my ESXi hosts roughly one month after Upgrading from ESXi 5.5 to ESXi 6.5.

I upgraded using Updating Manager and went from a Lenovo ESXi image for my IBM servers to a VMware generic ESXi image.

/var/log/syslog.log revealed a vendor script repeatedly running and failing.

Revoking the script from scheduled tasks (cron) fixed the issue.

You may want to have a look at your cron jobs with vi:

$ vi vi /var/spool/cron/crontabs/root

ShRootLock.png

Reply
0 Kudos
msripada
Virtuoso
Virtuoso

You can check the event which gives you the IP address and also there is a new feature in 6.0 which has default account lockout. Pls find below URL for more info https://www.v-front.de/2015/04/watch-out-esxi-60-introduces-root.html?m=1

Thanks,

MS

Reply
0 Kudos
cfizz34vmware
Enthusiast
Enthusiast

I am too having this this issue but just one of many hosts.  I am not sure if it is a DELL plugin or a SOLARWINDS plugin and since it does not tell me the IP I cant' find what is causing it.  I checked the ESXi shell and it is not enabled.

Reply
0 Kudos
msripada
Virtuoso
Virtuoso

check the vobd.log in the ESXi host after taking putty session and it should give the ip address or check the hostd.log which gives the login failure with ip address

Thanks,

MS

Reply
0 Kudos
cfizz34vmware
Enthusiast
Enthusiast

I keep seeing this...

I keep seeing this and getting lockouts

2017-10-16T07:51:40Z ^T: pam_unix(openwsman:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=  user=root

2017-10-16T07:51:43Z ^T: pam_tally2(openwsman:auth): user root (0) tally 7869, deny 5

2017-10-16T07:51:43Z addVob[1256326]: Could not expand environment variable HOME.

2017-10-16T07:51:43Z addVob[1256326]: Could not expand environment variable HOME.

2017-10-16T07:51:43Z addVob[1256326]: DictionaryLoad: Cannot open file "/usr/lib/vmware/config": No such file or directory.

2017-10-16T07:51:43Z addVob[1256326]: DictionaryLoad: Cannot open file "~/.vmware/config": No such file or directory.

2017-10-16T07:51:43Z addVob[1256326]: DictionaryLoad: Cannot open file "~/.vmware/preferences": No such file or directory.

Reply
0 Kudos
vmjoe
Enthusiast
Enthusiast

I have the same problem, and can't find any log that contains the SOURCE of that failed logins! :smileyshocked:

vobd.log only contains the message "Remote access for ESXi local user account 'root' has been locked for 120 seconds" over and over, but no log file seems to have the source IP... how is it possible to find the problem?!

Reply
0 Kudos
msripada
Virtuoso
Virtuoso

Try rhttpproxy or hostd log for information same time as the vodb states"Remote access for ESXi local user account 'root' has been locked for 120 seconds... rhttpproxy should give you IP address

Thanks,

MS

Reply
0 Kudos
SmilePeng
Contributor
Contributor

Hello

      I have the same problem.Do you find a solution to the problem ? Please tell me the solution,thank you.

Reply
0 Kudos
afragop72
Contributor
Contributor

Hi all,

I have the same issue and I can't figure out what is going...

It happened to me to a fresh installation of ESXi 6.5 U2 - no monitoring tools etc. to the server, just configured an external syslog (Log Insight) to collect logs.

Any further ideas?

Thx

T.

Reply
0 Kudos
plz1
Contributor
Contributor

I had the same issue.

I was able to find the IP address via vCenter > click host > Monitor > Tasks & Events > Events.  Each failed logon would show the warning you see on the main summary screen, but also the failed logon attempt like "root@<source IP address>.  In my case, I wasn't familiar with the IP I was seeing (likely legacy stuff, this popped up after a root password change), so I modified the firewall settings for SSH and vSphere Web Access to only trust the IP ranges I actually trust.  900 seconds later, good to go, warnings cleared.

IP redacted below, but you get the idea.  I believe this was added to vSphere 6.x, FWIW.

pastedImage_1.png

Reply
0 Kudos
iranna_totagi
Contributor
Contributor

Can I get a solution for this:

ESXi root password is getting locked frequently

518965_518965.pngpastedImage_1.png

ESXi root password is getting locked frequently

I found that no machine/agent is used to authenticate ESXi server:

I rebooted ESXi several times.

Same issue.

Earlier the ESXi version was:

VMware ESXi, 6.5.0, 8294253

I even upgraded ESXi to patch but still seeing same issues:

VMware ESXi 6.5.0 build-16576891, Update 3

[root@btp01esx16:/var/log] pam_tally2 --user root

Login           Failures Latest failure     From

root              250    09/14/20 05:29:57  unknown

[root@btp01esx16:/var/log] pam_tally2 --user root --reset

Login           Failures Latest failure     From

root              250    09/14/20 05:29:57  unknown

I see following lines are continuosly recorded on /var/log/auth.log

2020-09-13T04:49:32Z sshd[71851]: rekeyed inbound cipher

2020-09-13T05:49:32Z sshd[71851]: rekeyed outbound cipher

2020-09-13T05:49:32Z sshd[71851]: rekeyed inbound cipher

2020-09-13T06:49:33Z sshd[71851]: rekeyed outbound cipher

2020-09-13T06:49:33Z sshd[71851]: rekeyed inbound cipher

2020-09-13T07:49:33Z sshd[71851]: rekeyed outbound cipher

2020-09-13T07:49:34Z sshd[71851]: rekeyed inbound cipher

2020-09-13T08:49:34Z sshd[71851]: rekeyed outbound cipher

2020-09-13T08:49:34Z sshd[71851]: rekeyed inbound cipher

2020-09-13T09:49:35Z sshd[71851]: rekeyed outbound cipher

2020-09-13T09:49:35Z sshd[71851]: rekeyed inbound cipher

2020-09-13T10:49:35Z sshd[71851]: rekeyed outbound cipher

2020-09-13T10:49:36Z sshd[71851]: rekeyed inbound cipher

2020-09-13T11:49:36Z sshd[71851]: rekeyed outbound cipher

2020-09-13T11:49:36Z sshd[71851]: rekeyed inbound cipher

2020-09-13T12:49:37Z sshd[71851]: rekeyed outbound cipher

2020-09-13T12:49:37Z sshd[71851]: rekeyed inbound cipher

2020-09-13T13:49:37Z sshd[71851]: rekeyed outbound cipher

2020-09-13T13:49:38Z sshd[71851]: rekeyed inbound cipher

2020-09-13T14:49:38Z sshd[71851]: rekeyed outbound cipher

2020-09-13T14:49:38Z sshd[71851]: rekeyed inbound cipher

2020-09-13T15:49:39Z sshd[71851]: rekeyed outbound cipher

2020-09-13T15:49:39Z sshd[71851]: rekeyed inbound cipher

2020-09-13T16:49:40Z sshd[71851]: rekeyed outbound cipher

2020-09-13T16:49:40Z sshd[71851]: rekeyed inbound cipher

2020-09-13T17:49:40Z sshd[71851]: rekeyed outbound cipher

2020-09-13T17:49:41Z sshd[71851]: rekeyed inbound cipher

2020-09-13T18:49:41Z sshd[71851]: rekeyed outbound cipher

2020-09-13T18:49:41Z sshd[71851]: rekeyed inbound cipher

2020-09-13T19:49:42Z sshd[71851]: rekeyed outbound cipher

2020-09-13T19:49:42Z sshd[71851]: rekeyed inbound cipher

2020-09-13T20:49:42Z sshd[71851]: rekeyed outbound cipher

2020-09-13T20:49:42Z sshd[71851]: rekeyed inbound cipher

2020-09-13T21:49:43Z sshd[71851]: rekeyed outbound cipher

2020-09-13T21:49:43Z sshd[71851]: rekeyed inbound cipher

2020-09-13T22:49:44Z sshd[71851]: rekeyed outbound cipher

2020-09-13T22:49:44Z sshd[71851]: rekeyed inbound cipher

2020-09-13T23:49:44Z sshd[71851]: rekeyed outbound cipher

2020-09-13T23:49:45Z sshd[71851]: rekeyed inbound cipher

2020-09-14T00:49:45Z sshd[71851]: rekeyed outbound cipher

2020-09-14T00:49:45Z sshd[71851]: rekeyed inbound cipher

2020-09-14T01:49:46Z sshd[71851]: rekeyed outbound cipher

2020-09-14T01:49:46Z sshd[71851]: rekeyed inbound cipher

2020-09-14T02:49:46Z sshd[71851]: rekeyed outbound cipher

2020-09-14T02:49:47Z sshd[71851]: rekeyed inbound cipher

2020-09-14T03:49:47Z sshd[71851]: rekeyed outbound cipher

2020-09-14T03:49:47Z sshd[71851]: rekeyed inbound cipher

2020-09-14T04:49:48Z sshd[71851]: rekeyed outbound cipher

2020-09-14T04:49:48Z sshd[71851]: rekeyed inbound cipher

Reply
0 Kudos
WongaD
Contributor
Contributor

It could be possible a remote machine is trying to access the host. If you are using a monitoring tool such as vCenter, enable a lockdown of the host so that you can only access the host via vCenter portal or the local host. Below is the steps to do so in vCenter 6.5: 

1. Select the host 

2. Click on Configure

3. Under System, select Security Profile

4. Under Lockdown Mode, select Edit

5. Choose Normal and ok

Reply
0 Kudos