SCSI reservation conflicts (alerting)

[vmproteau]

We had a bad ESX Host HBA flake out and start flooding the fiber with errors. Over time this started to negatively affect all Hosts and associated VMs sharing the Datastores this Host was accessing. The result was delayed disk access requests throughout the cluster.

• From a Host perspective there were various anomolies in Virtual Center pointing to an issue.

• From a Windows guest perspective, we started seeing disk(11) and symmpi(15) errors in the system event log (attached).

We alert for various things throughout the environment however; this particular event started at night and wasn't noticed until morning.

Questions:

1. What are people using to alert them of excess SCSI reservation conflicts. Is there a commonly used tool that can monitor ESX logs for errors?

2. If I can't do that, I've considered alerting for the 11 and 15 system event errors on the guest.

3. Lastly, events don't apper to be generated regularly on the guest in the event of a complete disconnect from SAN. I'm sure there are anomalies but, I'm looking for something consistent and relatively immediate. I've been experimenting but, haven't found anything on the guest that can be used in this scenerio so, I'm also wanting to find a way to alert for this.

[Texiwill · Accepted solution]

Hello,

1. What are people using to alert them of excess SCSI reservation conflicts. Is there a commonly used tool that can monitor ESX logs for errors?

At the moment it is role your own. I personally use logcheck which emails me reports when there are issues. It removes 90% of the junk. I give a recipe to follow in my book.

1. If I can't do that, I've considered alerting for the 11 and 15 system event errors on the guest.

That is also an approach to take.

1. Lastly, events don't apper to be generated regularly on the guest in the event of a complete disconnect from SAN. I'm sure there are anomalies but, I'm looking for something consistent and relatively immediate. I've been experimenting but, haven't found anything on the guest that can be used in this scenerio so, I'm also wanting to find a way to alert for this.

The logfile has in it 'SCSI RESERVATION CONFLICT' all in caps so it is pretty noticeable.

Best regards,
Edward L. Haletky
VMware Communities User Moderator
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
Blue Gears and SearchVMware Pro Blogs -- Top Virtualization Security Links -- Virtualization Security Round Table Podcast