I'm a bit of a VMWare noob and I've taken over responsibility for some struggling, out-of-warranty, sadly underpowered ESX servers and trying to learn as much as I can. I'm an IT professional and have been for 20 years so I'm not a TOTAL noob, just not really familiar with vcenter and none of the troubleshooting I can find in the KB's/etc is helping me.
First to give a complete picture of my situation - the first symptom of my problems was that suddenly, although nothing had changed to my knowledge, the vsphere client was suddenly unable to log into the vcenter using my usual administrator password. I could (and can still) log into each individual ESX host as root.
I also found that http://(address of vcenter) was failing with the seemingly common error:
503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x000055ed2fdb3820] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)
So when I went to start troubleshooting this upon following the KB articles I noticed that I cannot log in to the vcenter as root. I don't know the password, and I suspect it's expired anyway..
So what I did was connect using the vSphere client to the ESX host which the vcenter itself runs on. For purposes of this discussion we'll say the vcenter runs on 192.168.0.10 and the host is 192.168.0.11. I connected to .11 using the vsphere client and from there I could access the console of the vcenter VM itself.
I was able to restart the machine, and get into the GRUB bootloader which I had the password for, add the init=/bin/bash and unlock/reset the root password.
At this time I also discovered the filesystem was full due to a bunch of .bz2 files in /var/log. I cleaned up about 5G of those old files by deleting them while I was at it. Verified there is now space on the machine in df/du command output.
Rebooted the machine and it boots up to the console screen where it asks me to push F2 to "customize my system" or F12 to restart.
I press F2, and try to log in as root, using the password I just set moments ago, and it fails. Incorrect user/password. Not sure it matters but I should note it takes awhile to fail - it's not instant, but maybe 20 seconds go by before it fails me.
The only thing I can apparently do is get back to the GRUB loader and add init=/bin/bash and get back into single user but I cannot get the darn thing to accept my root password after hitting F2.
Am I doing something wrong? Where should I look to troubleshoot?
