We have an incident where a there is a disk failure in our central stroage on EVA5000 which cause the filesystems on some of the RedHat AS4 based VM hosts to read only, and below is an example of log entry we saw in a number of ESX servers (running VM Infrastructure 3.02) which last for about 20 seconds,
May 12 06:02:19 xxx vmkernel: 8:23:05:11.951 cpu0:1028)LinSCSI: 2610: Forcing device status from SDSTAT_GOOD to SDSTAT_BUSY
We've opened a case with VMware and they said that this is because ESX lost communication with the storage which is normal behaviour. We then open up a case with HP on the EVA side, and they said the disk outage should NOT affect the server so nothing wrong at it's end except replace the disk.
As we do not see this happen on the Windows or Solaris hosts, I am wondering why this will impact the Linux hosts? and why it do not recover when ESX restored the storage connection? Our only way to recover is to reboot the VM hosts. Has anyone experience this as well? Is there anything we can do to prevent this from happening again?
