vRA 7.6 Appliance Failed Authentication Attempts

[kwrobert]

Hello All,

Running a distributed vRA 7.6 lab here with 3 appliances. My lab was forcibly shutdown without warning (power outage) the other day and I am now seeing lots of failed login attempts to the root account on my appliances. Here is the relevant section of the /var/log/messages:

```

2019-08-23T21:21:41.332649+00:00 vra02 vami /opt/vmware/share/htdocs/service/cafe/config-page.py[16484]: info Executing shell command... ['/usr/sbin/vami', 'host-addresses']

2019-08-23T21:21:42.002571+00:00 vra02 vami /opt/vmware/share/htdocs/service/cafe/config-page.py[16484]: info Processing request

2019-08-23T21:21:42.002606+00:00 vra02 vami /opt/vmware/share/htdocs/service/cafe/config-page.py[16484]: info Authenticating with sfcb server.

2019-08-23T21:21:42.002614+00:00 vra02 vami /opt/vmware/share/htdocs/service/cafe/config-page.py[16484]: info user:root

2019-08-23T21:21:42.149287+00:00 vra02 vami-sfcbd: pam_unix(vami-sfcb:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=  user=root

2019-08-23T21:21:46.340061+00:00 vra02 vami /opt/vmware/share/htdocs/service/cafe/config-page.py[16484]: info Unable to authenticate with given username and password

Authentication error, username: root

```

If I `tail -f /var/log/messages` on any of my appliances, I keep seeing that section scroll by repeatedly. Before anyone panics, I don't believe is trying to brute force my root account. I had my network admin check the firewall and there isn't any traffic traversing the firewall (which functions as my router) so this traffic is coming from somewhere within my environment. However, I'm having a hard time determining who is the culprit. I've shutdown all the Management Agents running on the IaaS boxes in my environment, and that seems to have reduced the number of failed login attempts.

Tried rebooting the appliances, but that didn't fix the problem. If you look in the directory shown above in the log messages, it's filled with a bunch of Python code running the VAMI web interface. I've tracked down which top level function is throwing this error, but haven't jumped all the way down the rabbit hole on that front. I don't know why requests are repeatedly being made to this webpage with the wrong password, or who/what is making the requests, but if anyone has any ideas or could point me in the right troubleshooting direction it would be much appreciated! My hunch is that this is a local process to each appliance trying to authenticate, but I could be totally wrong.

Thanks all!

[kwrobert · Accepted solution]

Welp, found the problem.

I had some tabs open to the VAMI page of all of my appliances in a long forgotten browser window. As soon as I closed those the failed login attempts stopped and I have yet to see any more since I closed them. I feel dumb for not checking this, all the breadcrumbs were there when looking through those Python files. The fact that I spent as much time as I did debugging this today to discover old browser tabs to be the problem is a bit aggravating.

Still, extremely strange behavior for a VAMI page with a timed out login to run in an endless loop attempting to authenticate with an expired password token, continuously fail, and eventually lock everybody out of the root account. If I hadn't set up passwordless public key authentication to SSH into the root account on all my appliances this would have been a nightmare to deal with. Maybe I brought things to a weird state having as many browser tabs open as I did, but this seems like a bug somewhere that should probably be fixed.

---------------------------------------------------------------------------------------------------------

Was it helpful? Let us know by completing this short survey here.