VMware Communities: Message List - WARNING: SCSI: 5519: Failing I/O due to too many reservation conflicts

Re: WARNING: SCSI: 5519: Failing I/O due to too many reservation conflicts

Texiwill — Tue, 19 Jun 2007 14:46:06 GMT

Hello,

VMFS metadata activities occur when you:

update the path/name, access/modification/create times, permissions/ownerships, size of a file on the VMFS
Files grow in 15MB chunks usually.
vMotion, clone, remove, migrate a file on the VMFS
open/close files on the VMFS

Effectively everytime you touch one of the VM files you buy a lock. This is why most VMDKs are monolithic. You do not get locks for changing the contents of an already opened file. But the open produces a short term lock, as does the close. And only if a file grows greater than the existing bounds do you produce locks.

-delta.vmdk files are a case where the file is constantly growing past its boundaries. Every 15MBs requires a new lock to change the size of the file.

The more simultaneous LUN activities the more chance of failures do to SCSI reservation conflicts. Reducing the number of LUN activities solves the problem. For example, 8 simultaneous vMotions on the same LUN will produce SCSI reservation conflicts. Yet 8 simultaneous vMotions using 8 different LUNs will not.

Best regards,
Edward

Re: WARNING: SCSI: 5519: Failing I/O due to too many reservation conflicts

Muddy — Tue, 19 Jun 2007 07:19:53 GMT

How do I check the LUN Metadata activities? I am seeing the same errors intermittantly with my HP BL685c servers and EVA5000. It is error free for a week then starts showing these errors intermittantly. I am not creating a VM or VMotion at the time. Errors logged at night during backups as well as during the day. Tried setting the port speed and no difference. Does anyone have a view on if the Insite Manager agents could be causing it?

Re: WARNING: SCSI: 5519: Failing I/O due to too many reservation conflicts

Texiwill — Mon, 18 Jun 2007 12:35:14 GMT

Hello,

SCSI Reservation conflicts are do to performing to many VMFS metadata updates simultaneously. Depending on your SAN/iSCSI server the limit on simultaneous metadata updates is severely limited before you start to through SCSI conflict messages.

First things I would do: Verify the firmware level of your SAN, Fabric Switches, and HBAs are all at the desired level. For example, if you have a Hitachi SAN there was a firmware upgrade required so that the reservations responded much faster and therefore limited the amount of conflicts. Without this minimum version, conflicts would happen constantly.

Second thing I would do, is verify any 3rd party agents, or in house written scripts that run on your ESX hosts on a regular basis. If for example, you constantly 'touch' the files on your SAN to make changes to times, ownership, or groups, and you are doing this simultaneously on ALL your ESX servers, you could easily throw 1000s if not millions of SCSI Reservation conflicts.

Are you touching (file metadata changes), creating, removing, vmotioning, backing up (create and delete) VMs on the same LUN simultaneously? The limit on the number of metadata updates is on a per LUN basis.

I would take your LUN metadata activities and pare them down to only 1 per LUN at any given time and slowly increase that number until you see reservations conflicts. That will give you the upper limit that you can do per LUN.

Best regards,
Edward

Re: WARNING: SCSI: 5519: Failing I/O due to too many reservation conflicts

falsehope — Fri, 15 Jun 2007 17:37:04 GMT

foo,
Any update???

Re: WARNING: SCSI: 5519: Failing I/O due to too many reservation conflicts

foofighter26 — Mon, 30 Apr 2007 09:51:25 GMT

We are still having these issues does anyone have any suggestions?

We have a support request open with VM but this is going very slowly!

regards,

Paul

Re: WARNING: SCSI: 5519: Failing I/O due to too many reservation conflicts

wtreutz — Mon, 16 Apr 2007 15:00:00 GMT

Have you Informations, what HP preferred??
I know, some Storage-Vendors preferred fix seetings for Port-Speed too. I preferred fix settings for both to all FC-Target-Devices, if possible.

regards
Werner

Re: WARNING: SCSI: 5519: Failing I/O due to too many reservation conflicts

foofighter26 — Mon, 16 Apr 2007 14:55:21 GMT

All are set to f-type only apart from the truncation between switches which is e-type.

Also Portspeed is set to auto at the moment.

Re: WARNING: SCSI: 5519: Failing I/O due to too many reservation conflicts

wtreutz — Mon, 16 Apr 2007 14:41:29 GMT

How are your FC-Switch-Port settings, and in which szzenario do you have the errors.

I had seen in /var/log/vmkernel something like
vmkernel: 0:00:44:23.107 cpu2:1036)WARNING: SCSI: 7916: status SCSI reservation conflict, rstatus #c0de01 for vmhba1:0:0. residual R 919, CR 0, ER 3
vmkernel: 0:00:44:23.107 cpu2:1036)WARNING: FS3: 4008: Reservation error: SCSI reservation conflict
vmkernel: 0:00:44:23.393 cpu2:1034)SCSI: vm 1034: 5509: Sync CR at 0
vmkernel: 0:00:44:23.393 cpu2:1034)WARNING: SCSI: 5519: Failing I/O due to too many reservation conflicts
after FC-Storage failover. For us it was reproducible the settings on the FC-Switch-Ports. We must set the PortSpeed fix and set the PortType to F-Type only (no E-Type and no L-Type)

WARNING: SCSI: 5519: Failing I/O due to too many reservation conflicts

foofighter26 — Mon, 16 Apr 2007 14:10:38 GMT

WARNING: SCSI: 5519: Failing I/O due to too many reservation conflicts in vmkernel logs. We are currently having an issue where 3 blades changed to not responsive and the vms on them were powered on but unresponsive. The datastore on the SAN's scsi reservation had changed from none to regular which caused the datastore to become inaccessible. We rebooted the blades which appeared to change the reservation on the SAN back to none then we were able to power on the vms with the exception of one blade where we had to rescan the datastore to get the datastore back. Equipment: HP EVA 8000 HP BL20P G4 Blades Fibre Switch is 4GB