ESX Host not booting (Major issue!!!)

[Eire09]

Hi All,

I'm having a major problem with one of my ESX hosts. I rebooted the host & now it is not coming back up.

I've tried to boot the server in VMware Server mode, VMware Server Debug Mode & Service Console Only Troubleshooting Mode bit still no joy.

It looks like the issue has something to do with storage I've mounted in /etc/fstab but I cannot get into this directory to take out the lines that are mounting the storage. Any suggestions for this?

I'm getting the following back in the console

MKDIR: Cannot create directory '/var/lock' Read-Only file system

Touch: Creating '/var/locks/subsys/vmware': No such file or directory

----

---

Then when I boot through Troubleshooting mode I get the following:

Remounting root filesystem in read-write mode: : line 17 in /etc/fstab is bad

Mount: Mount point ext3 does not exist

Service Console

*Mount Point password: ***** does not exist*

Mount Point does not exist

Mount Point ext3 does not exist

Mount Point 0 does not exist

Can anyone point me in the right direction to get this resolved?

Would appreciate any help you can offer guys.

Thanks in advance.

[Datto]

One option would be to boot the box with a Knoppix boot CD and use the troubleshooting tools from that CD to access the disk where the fstab file is at -- http://www.knoppix.com.

Another option would be when Grub shows up on screen showing you your boot options at the beginning of the boot process, highlight the Troubleshooting boot line and hit E for edit. Put a 1 following a spacee at the end of the Kernel line (usually the 2nd line that shows up when editing the Grub entry) and then hit the B key to boot that entry with the 1 at the end of the 2nd line (Kernel line). Don't know whether that will do it or not since the fstab is having a problem but it might allow you to get to the fstab file and edit it.

Datto

[Troy_Clavell]

Are your VM's on local storage? If not, I say reboot, blow away they local array and start from scratch. A rebuild may be a lot easier than trying to troubleshoot.

[menjur]

Hi,

is the ESX installation local or on your SAN, if its local you can remove all connetions and try it again.

otherwise its indeed the best option if your vm are not local to rebuild the machine.

[Eire09]

Thanks Datto.

I was able to get into the file /etc/fstab however it is opening in Read only mode!! Do you know how I change this so I can write to this file?

ESX is installed locally & all the VM's are on shared storage.

Thanks for your replies guys.

[menjur]

You can use the chmod command for this.

see the link for help

[Eire09]

Thanks for that.

However I'm still having no luck.

I enter chmod go+rw fstab & I got the following message Read Only File System

So it looks like the File System is read only. Any idea how I change this?

Thanks again

[menjur]

try this chmod 755 fstab

but do you really want to do this reinstallation of the ESX is so quicker and just so save.

here a link of understanding de fstab file http://www.tuxfiles.org/linuxhelp/fstab.html

[Datto]

Just wondering -- did you use a Knoppix CD to boot the ESX server or did you access the ESX disk in some other manner?

Datto

[Eire09]

First, I accessed the disk through the grub menu & entering through SIngle User Mode.

I was able to access /etc/fstab but unfortunately I couldn't save changes to fstab becuase it is read only.

I'm now using the knoppix cd but cannot seem access /etc/fstab on the hard disk.

Any further suggestions?

Pedro

[Datto]

Recent versions of Knoppix put an icon on the desktop for each partition that Knoppix can see -- see if there are icons for your ESX host partitions showing up on the desktop. If so, right click on the icon and see if there is a choice for changing the partition to Read/Write (or unchecking Read-Only). If so, choose that option, open a command prompt, navagate to your fstab for editing. Here's a link that describes that approach: http://lists.evolt.org/archive/Week-of-Mon-20030929/149435.html

Here are other links to use that show how to mount the appropriate ESX partition for editing:

-- see the section on rescuing a non-booting LInux system

Datto

[Datto]

Here's a step-by-step on how to use Knoppix to rescue a non-bootable ESX host that has a messed-up fstab file.

1) Download the Knoppix 6.01 ISO from http://www.knoppix.com (make sure to get the English version since there's an English (en) as well as a German (de) version, assuming you're an English speaking person).

2) Burn the Knoppix 6.01 ISO to a CD.

3) Boot the ESX host from the Knoppix 6.01 CD -- take the defaults on the bootup and within a minute or two a graphical desktop will show up on-screen.

4) In the lower left corner of the screen after the bootup are Quick Launch icons. Click on the Quick Launch icon for terminal emulator. A command box appears on screen.

5) Change to root by typing su - in the command box (no password is needed). Note that is su space dash as the command.

6) Type the command fdisk -l (that's fdisk space dash ell) to see the partitions on your hard drive. If you're using an IDE drive for your ESX host boot drive, the partition you're likely looking for is called /dev/hda2. If you're using a SCSI drive as your ESX host boot drive, the partition you're likely looking for is /dev/sda2. Make sure the appropriate partition is listed on the resulting display from the command fdisk -l

7) Check to make sure Knoppix put in a mount directory for your hard drive partition by navigating to the /mnt directory with the command cd /mnt and then with the command ls -l see if a directory for hda2 or sda2 was automatically created by Knoppix. If not, create the directory after navigating to the /mnt directory with the command mkdir hda2 for IDE boot dirves or mkdir sda2 if you have SCSI boot drives on your ESX host.

😎 For IDE boot drives on your ESX host, type the command mount -o rw /dev/hda2 /mnt/hda2 For SCSI boot drives on your ESX host, type the command mount -o rw /dev/sda2 /mnt/sda2 you should be returned to the prompt after executing those commands.

9) Navigate to the /mnt/hda2 directory for IDE boot drives or /mnt/sda2 for ESX hosts with SCSI boot hard drives and perform the command ls -l and you should see the directory structure on your ESX host hard drive.

10) Navigate to the /etc directory with the command cd /etc and use the built-in nano editor to load up your fstab file with the command nano fstab and edit your fstab file as needed. Save the file back to the hard drive.

11) In the command box type umount /dev/hda2 to unmount your ESX host partition. Note this is umount, not unmount.

12) Then type the command reboot in the command prompt. When prompted on-screen to remove the CD boot media, do so and hit return (note you'll have 2 minutes to do this before Knoppix will automatically start the reboot of your ESX HOST). Your ESX host will start up using the edits you inserted into the fstab file via Knoppix.

Datto