We are a small school with a small tech staff and no VMware experts, so I really hope someone can help us out.
Our VCSA 6.0 was doing fine but recently rebooted after a power failure. I realized something was wrong when I got the following when trying to connect with my browser:
503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x7fa482aaf3a0] _serverNamespace = /vsphere-client _isRedirect = false _port = 9090). Our backups are also failing with a 503 error.
Here are some notes of interest that I typed while watching the server reboot:
Init SMTP port (sendmail) failed
Waiting for VMware Service Control AGent... OK
Starting VMware vPostgres: Starting VMware Inventory Service...
Waiting for VMware inventory service...
Warning: VMware inventory service may have failed to start
sed: couldn't flush /etc/vmware-vpx//sedGKgW32: No space left on device
sed: couldn't flush /etc/vmware-vpx//sedFjrhvy: No space left on device
vmware-vpxd: VC SSL Certificate does not exist, it will be generated by vpxd
Waiting for the embedded database to start up...........failed
Executing pre-startup scripts...
vmware-vpxd: Starting vpxd by administrative request.
success
vmware-vpxd: Waiting for vpxd to start listening for requests on 8089
Waiting for vpxd to initialize....
VPXD is not running. Cannot collect live core dump....
Warning: VMware ESX Agent manager may have failed to start
Warning: Vmware vsphere profile-driven storage service MHFTS
I've dealt with issues of /storage/log running out of space in the past, and I suspect that some element of the server's storage is either full or needs fsck. However I can't get to bash to check these things. Typing "shell.set --enabled True" from the appliance shell results in "Unknown command: 'shell.set'". Trying to enable bash from the server console results in "DCUI aborted by exception". I also tried booting directly to bash from GRUB, but when I tried to fsck /dev/mapper/log_vg-log, which I'd read about somewhere, I was told that checking a mounted volume could cause severe damage. I umounted it, but then I couldn't figure out how to to fsck it.
I've reached the limit of my knowledge and would appreciate any assistance. Thank you!
I met the same problem today. It's because of "/" running out of space.
1. You can use a Linux ISO image (RHEL, Suse... etc.) as a boot CD/DVD, and then reboot the vCenter virtual machine from the CD/DVD
2. Select "Rescue System" (or something like that)
3. If you use rhel to boot the VM, the original "/" will be mounted at /mnt/sysimage (I'm not sure about Suse, but you can find it any way)
4. Use "df -H" to find if "/mnt/sysimage" is 100% full.
5. Use "du -sh" to find which directory/file occupys most of the space. If my case, I found /mnt/sysimage/var/log/audit/audit.log and clear it.
6. Reboot the vCenter. It's ok.
Good luck.
I "think" that log rotation failed and because of that, the hard disk for logs (and maybe other things) are full, and because of that, strange things happen like this.
I would suggest to do the following check:
Are you able to log into the console?
If so, can you please do a
df -H
and share the result with us?
Please keep in mind that there is a possibility to redeploy/reinstall the VCSA....
Sorry, did not read the last part
So, it looks like full disks...
I will think about other options!
I met the same problem today. It's because of "/" running out of space.
1. You can use a Linux ISO image (RHEL, Suse... etc.) as a boot CD/DVD, and then reboot the vCenter virtual machine from the CD/DVD
2. Select "Rescue System" (or something like that)
3. If you use rhel to boot the VM, the original "/" will be mounted at /mnt/sysimage (I'm not sure about Suse, but you can find it any way)
4. Use "df -H" to find if "/mnt/sysimage" is 100% full.
5. Use "du -sh" to find which directory/file occupys most of the space. If my case, I found /mnt/sysimage/var/log/audit/audit.log and clear it.
6. Reboot the vCenter. It's ok.
Good luck.
That was what I needed! I deleted the audit.log file, rebooted, and the services all started. I also needed to deal with the log disk at 100% capacity. For anyone else who may have this problem I used these articles:
vCenter Appliance root Partition 100% full due to Audit.log files not being rotated (2149278) | VMwa...
/storage/log directory is full in vCenter Server Appliance 6.0 (2143565) | VMware KB
Thanks very much for your help.