hostd not starting

[hemaphore]

Hi All,

I am really hoping someone can help me here.

The / filesystem filled up on our ESX 3.0.2 host which currently runs 6 production VMs. As a result the hostd died.

I freed up the space (it was a giant .iso file in someone's /home dir) but now everytime the hostd starts up the following information appears in the hostd.log file and the hostd crashes:

Application error: vmodl.fault.SystemError

Backtrace generated:

eip 0x74177e

eip 0x698369

eip 0x6524b5

eip 0x816fc9c

eip 0x81520db

eip 0x8296b28

eip 0x823d2f1

eip 0x823bb44

eip 0x8238297

eip 0x82c4753

eip 0x82d1586

eip 0x82f94ad

eip 0x8305de4

eip 0x6aea79a

eip 0x8090e51

I know there is a thread with a similar error that talks about replacing the services.xml file, however I've verified that the file is identical to the ones running on other servers in our cluster.

Does anyone have any ideas as to what could be wrong?

If I need to supply more info please let me know.

Thanks very much in advance.

[Texiwill]

Hello,

You may wish to also delete the files in the directory /var/core, as this is where core files are stored and hostd seems to be crashing regularly. That will also help your disk space issues.

Run 'service mgmt-vmware restart'

Review the /var/log/vmware/hostd.log to look for any errors in the start of hostd. THat is where I would start. Then also look for updates. I had a hostd crash due to a bad firewall rule added by a third party product.

Best regards,
Edward L. Haletky
VMware Communities User Moderator
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
Blue Gears and SearchVMware Pro Blogs -- Top Virtualization Security Links -- Virtualization Security Round Table Podcast

[Cooldude09]

whats the output of df -h & du -h /. Also try to delete files under /var/log which are not useful.

Regards

Anil

Save the planet, Go Green

if you found my answer to be useful, feel free to mark it as Helpful or Correct.

[hemaphore]

Thanks all for the suggestions.

We ended up contacting VMware support and the technician was able to determine that because the filesystem was full, the esx.conf file no longer existed. So he ended up recreating the esx.conf file.

Here's what he did (from the .bash_history):

cp /boot/initrd-2.4.21-47.0.1.ELvmnix.img /tmp/

cd /tmp/

gunzip -dc initrd-2.4.21-47.0.1.ELvmnix.img > initrd.unziped

mkdir initrd

mount -o loop initrd.unziped initrd

cd /tmp/initrd/etc/vmware/

cp esx.conf /etc/vmware

cd /tmp/

umount /tmp/initrd

esxcfg-boot -b

less /etc/vmware/esx.conf

service mgmt-vmware restart

These steps recreated our esx.conf file and the hostd was able to start.