VCSA 6.0 boots but services do not start

[VMschool]

We are a small school with a small tech staff and no VMware experts, so I really hope someone can help us out.

Our VCSA 6.0 was doing fine but recently rebooted after a power failure.  I realized something was wrong when I got the following when trying to connect with my browser:

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x7fa482aaf3a0] _serverNamespace = /vsphere-client _isRedirect = false _port = 9090).  Our backups are also failing with a 503 error.

Here are some notes of interest that I typed while watching the server reboot:

Init SMTP port (sendmail) failed

Waiting for VMware Service Control AGent... OK

Starting VMware vPostgres: Starting VMware Inventory Service...

Waiting for VMware inventory service...

Warning: VMware inventory service may have failed to start

sed: couldn't flush /etc/vmware-vpx//sedGKgW32: No space left on device

sed: couldn't flush /etc/vmware-vpx//sedFjrhvy: No space left on device

vmware-vpxd: VC SSL Certificate does not exist, it will be generated by vpxd

Waiting for the embedded database to start up...........failed

Executing pre-startup scripts...

vmware-vpxd: Starting vpxd by administrative request.

success

vmware-vpxd: Waiting for vpxd to start listening for requests on 8089

Waiting for vpxd to initialize....

VPXD is not running.  Cannot collect live core dump....

Warning: VMware ESX Agent manager may have failed to start

Warning: Vmware vsphere profile-driven storage service MHFTS

I've dealt with issues of /storage/log running out of space in the past, and I suspect that some element of the server's storage is either full or needs fsck.  However I can't get to bash to check these things.  Typing "shell.set --enabled True" from the appliance shell results in "Unknown command: 'shell.set'".  Trying to enable bash from the server console results in "DCUI aborted by exception".  I also tried booting directly to bash from GRUB, but when I tried to fsck /dev/mapper/log_vg-log, which I'd read about somewhere, I was told that checking a mounted volume could cause severe damage.  I umounted it, but then I couldn't figure out how to to fsck it.

I've reached the limit of my knowledge and would appreciate any assistance.  Thank you!

[johandijkstra]

I "think" that log rotation failed and because of that, the hard disk for logs (and maybe other things) are full, and because of that, strange things happen like this.

I would suggest to do the following check:

Are you able to log into the console?

If so, can you please do a

df -H

and share the result with us?

Please keep in mind that there is a possibility to redeploy/reinstall the VCSA....

[johandijkstra]

Sorry, did not read the last part

So, it looks like full disks...

I will think about other options!

[Cityforest · Accepted solution]

I met the same problem today. It's because of "/" running out of space.

1. You can use a Linux ISO image (RHEL, Suse... etc.) as a boot CD/DVD, and then reboot the vCenter virtual machine from the CD/DVD

2. Select "Rescue System" (or something like that)

3. If you use rhel to boot the VM, the original "/" will be mounted at /mnt/sysimage (I'm not sure about Suse, but you can find it any way)

4. Use "df -H" to find if "/mnt/sysimage" is 100% full.

5. Use "du -sh" to find which directory/file occupys most of the space. If my case, I found /mnt/sysimage/var/log/audit/audit.log and clear it.

6. Reboot the vCenter. It's ok.

Good luck.

[VMschool]

That was what I needed!  I deleted the audit.log file, rebooted, and the services all started.  I also needed to deal with the log disk at 100% capacity.  For anyone else who may have this problem I used these articles:

vCenter Appliance root Partition 100% full due to Audit.log files not being rotated (2149278) | VMwa...

/storage/log directory is full in vCenter Server Appliance 6.0 (2143565) | VMware KB

Increasing the disk space for the VMware vCenter Server Appliance in vSphere 6.0 (2126276) | VMware ...

Thanks very much for your help.