Red Hat Partition Becoming Readonly Frequently

[Aneesh801]

Hi All,

I have done a P2V of a Redhat 5 server. After the migration the /opt partition of the server is becoming read only frequently. I have done an sVmotion of the virtual machine for checking the specific lun is having any issue, but no luck. The issue still continues. Anyone have any idea on this.

[marcelo_soares]

Check 2 things: First the OS. This KB http://kb.vmware.com/kb/51306 shows how some Linux versions have this kind of problem.

Also, check your storage access on your ESX. It may be degraded. Check this tutorial: http://kb.vmware.com/kb/1008205

[Aneesh801]

Thank you Marcelo.

Also the users are saying that the file servers are very slow after making it to virtual. Had done all the best practices from vmware end. We are using Dell Compellent storage. Some physical servers in that storage is also facing slowness. How can i have a detailed check on the storage from the vmware end. Please let me provide some useful commands for that.

[marcelo_soares]

Check this KB: http://kb.vmware.com/kb/1008205

Check if you are having high latency times on DAVG (and low - near zero - on KAVG). If yes, yo may be having some kind of issue on your storage.

[Aneesh801]

Ya Marcelo,  i am getting very high values on DAVG and value nearly zero on KAVG. But the DAVG value increases periodically and will remain there only for few seconds and comes back to a value between 6 & 10. I have attached a screen shot. Please have a check.

[marcelo_soares]

Too high. You may be facing some SAN issues. You can check the log /var/log/vmkernel (for ESX and ESXi 5) or for ESXi 4.x, perform the command: cat /var/log/messages | grep vmkernel | less to check the same log. Check if you are not having SCSI errors.

[Aneesh801]

Marcelo,

I am getting some errors like the one which is pasted below from vmkernel.log.

2012-04-05T18:59:01.279Z cpu18:4114)ScsiDeviceIO: 2316: Cmd(0x412441803ec0) 0x12, CmdSN 0xe5489 to dev "naa.600508e00000000006bc6ddd369bae0d" failed H:0x0 D:0x2 P:0x0 V
2012-04-05T19:04:01.272Z cpu14:4110)ScsiDeviceIO: 2316: Cmd(0x4124413fb080) 0x12, CmdSN 0xe550d to dev "naa.600508e00000000006bc6ddd369bae0d" failed H:0x0 D:0x2 P:0x0 V
2012-04-05T19:09:01.267Z cpu18:4114)ScsiDeviceIO: 2316: Cmd(0x412441f4e640) 0x12, CmdSN 0xe5587 to dev "naa.600508e00000000006bc6ddd369bae0d" failed H:0x0 D:0x2 P:0x0 V

Is this realted to storage latency.

[marcelo_soares]

Yes. Check the KB http://kb.vmware.com/kb/1030381, the error you is having is the d:0x2:

VMK_SCSI_DEVICE_CHECK_CONDITION = 0x2

vmkernel:  76:23:45:36.239 cpu11:22687)NMP: nmp_CompleteCommandForPath: Command  0x28 (0x41000b10f000) to NMP device  "naa.600601601f70190016361881f3b4de11" failed on physical path  "vmhba3:C0:T2:L10" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x4 0x3.

This status is returned when a command fails for a specific reason.  When a CHECK CONDITION is received, the ESX storage stack will send out a  SCSI command 0x3 (REQUEST SENSE) in order to get the SCSI sense data  (Sense Key, Additional Sense Code, ASC Qualifier, and other bits). The  sense data is listed after "Valid sense data" in the order of Sense Key,  Additional Sense Code, and ASC Qualifier. When decoding Additional  Sense Code/ASC Qualifier bits, they must be read together.

You will need to send the errors to your storage vendor for analysis. 99.9% sure you are having some kind of misconfiguration of physical problem.

[Aneesh801]

Thank you very much Marcelo. I will contact the storage vendor and will get back to you with the updates.

[Aneesh801]

Hi Marcelo,

Data progression was happening in the storage and the admin disabled it.  Also i have moved the vms having issue to a tier1 storage. Seems these two steps fixed the issue. Thanks for your assistance in this.

Aneesh

[marcelo_soares]

Great news, and thanks for replying back with the solution!