Read-only mode on all file systems

[antipov]

After failure of one of hard disks in RAID 1, all file systems have passed in a read-only mode.

Disk have replaced, has passed rebuild, but file systems are still in read-only mode.

Management Service fails to start with error "/var/lock....." is in read-only mode.

How to convert file systems to read-write mode without reboot system?

ESX 3.5 update 4

[AndreTheGiant · Accepted solution]

Use this commands:

mount -o remount -o ro /

fsck -f /dev/XXXX

where XXXX is the / device (you can see the name with df)

Andre

**if you found this or any other answer useful please consider allocating points for helpful or correct answers

[antipov]

Thanks, a lot. After run fsck command its find some errors and fix it. But requires reboot. This esx server configured as HA cluster node and some virtuals machines continue to work on it. How we can migrate this VMs to another node in the same cluster? We shutdown one VM and can't find ways to run it on another nodes. Help us, please.

It is normal behavior of ESX server, when all file systems mounts as "read only" when RAID1 array lost one hard drive?

[AndreTheGiant]

How we can migrate this VMs to another node in the same cluster? We shutdown one VM and can't find ways to run it on another nodes.

I suppose that you do not have VMotion feature.

If you have VC, shutdown VM, use migrate to move the VM to the other ESX (but keep the file to the same location).

Then power on.

If you do not have VC, shutdown VM, go to the second ESX, browse datastore, enter in VM folder, and add to inventory the .vmx file.

Then power on.

Andre

**if you found this or any other answer useful please consider allocating points for helpful or correct answers

[AndreTheGiant]

It is normal behavior of ESX server, when all file systems mounts as "read only" when RAID1 array lost one hard drive?

Is not usually, but may happen.

Anyway this problem does not impact the VM. You can temporally leave the ESX with this error, and reboot when you can.

Andre

**if you found this or any other answer useful please consider allocating points for helpful or correct answers

[antipov]

We have VMotion feature and VC, but when the disk fails, that cluster node comes to "disconnected" state in VC, and we cannot migrate any VM from that node. And we cannot start VMware Management Service to connect host to the VC or to connect directly to the host via VMware Infrastructure Client. How we can migrate powered off VMs to another host in same cluster from the command prompt?

One more question. IF the problem host after reboot will not start how we can power on VMs to another node?

When we try to add to inventory the .vmx file VC say that this machine is already exist, because VMs stored at the shared LUN.

[AndreTheGiant]

IF the problem host after reboot will not start how we can power on VMs to another node?

When we try to add to inventory the .vmx file VC say that this machine is already exist, because VMs stored at the shared LUN.

If you have VC you have first to remove the "old" VM from inventory.

Andre

**if you found this or any other answer useful please consider allocating points for helpful or correct answers

[kjb007]

You can't really migrate them off using vmotion, since that task requires writing to logs it does not have access to any longer. As Andre pointed out, you will have to remove the old vm from inventory, and then add it back in to the cluster. It will show as being powered down, and you can't start it. You will then have to go into the vm, and shut it down from within the vm. As soon as it powers off, you can hit power on in the vCenter, and start it up.

-KjB

VMware vExpert

[antipov]

Thank You. We solve our issue. Tell me please, how we can prevent this ESX failure (read-only file systems)? We have ability to configure RAID 5, RAID 0 and RAID 1+0 on the server. This array contain only ESX operating system. All VMs is located on the SAN storage.

[kjb007]

Read-only filesystem is a result of disconnection or latency in storage. You have to make sure your controllers are not saturated, and are not full high utilization. The read-only business is to protect against corruption as the disks were inaccessible for a time. Typically, it has to be disconnected for almost 1 minute before they go read-only. Make sure there's no communication problem to your storage.

-KjB

VMware vExpert