VMware Communities > VMTN > VMware Infrastructure > VI: ESX 3.5 > Discussions
Up to Discussions in VI: ESX 3.5

This Question is Not Answered

1 "correct" answer available (10 pts) 1 "helpful" answer available (6 pts)
8 Replies Last post: Jun 19, 2007 7:46 AM by Texiwill
Reply

WARNING: SCSI: 5519: Failing I/O due to too many reservation conflicts

Apr 16, 2007 7:10 AM

Click to view foofighter26's profile Hot Shot foofighter26 147 posts since
Dec 19, 2006
WARNING: SCSI: 5519: Failing I/O due to too many reservation conflicts in vmkernel logs. We are currently having an issue where 3 blades changed to not responsive and the vms on them were powered on but unresponsive. The datastore on the SAN's scsi reservation had changed from none to regular which caused the datastore to become inaccessible. We rebooted the blades which appeared to change the reservation on the SAN back to none then we were able to power on the vms with the exception of one blade where we had to rescan the datastore to get the datastore back. Equipment: HP EVA 8000 HP BL20P G4 Blades Fibre Switch is 4GB
Reply Re: WARNING: SCSI: 5519: Failing I/O due to too many reservation conflicts Apr 16, 2007 7:41 AM
Click to view wtreutz's profile Enthusiast wtreutz 81 posts since
Oct 7, 2004
How are your FC-Switch-Port settings, and in which szzenario do you have the errors.

I had seen in /var/log/vmkernel something like
vmkernel: 0:00:44:23.107 cpu2:1036)WARNING: SCSI: 7916: status SCSI reservation conflict, rstatus #c0de01 for vmhba1:0:0. residual R 919, CR 0, ER 3
vmkernel: 0:00:44:23.107 cpu2:1036)WARNING: FS3: 4008: Reservation error: SCSI reservation conflict
vmkernel: 0:00:44:23.393 cpu2:1034)SCSI: vm 1034: 5509: Sync CR at 0
vmkernel: 0:00:44:23.393 cpu2:1034)WARNING: SCSI: 5519: Failing I/O due to too many reservation conflicts
after FC-Storage failover. For us it was reproducible the settings on the FC-Switch-Ports. We must set the PortSpeed fix and set the PortType to F-Type only (no E-Type and no L-Type)
Reply Re: WARNING: SCSI: 5519: Failing I/O due to too many reservation conflicts Apr 16, 2007 7:55 AM
in response to: wtreutz
Click to view foofighter26's profile Hot Shot foofighter26 147 posts since
Dec 19, 2006
All are set to f-type only apart from the truncation between switches which is e-type.

Also Portspeed is set to auto at the moment.
Reply Re: WARNING: SCSI: 5519: Failing I/O due to too many reservation conflicts Apr 16, 2007 8:00 AM
in response to: foofighter26
Click to view wtreutz's profile Enthusiast wtreutz 81 posts since
Oct 7, 2004
Have you Informations, what HP preferred??
I know, some Storage-Vendors preferred fix seetings for Port-Speed too. I preferred fix settings for both to all FC-Target-Devices, if possible.

regards
Werner
Reply Re: WARNING: SCSI: 5519: Failing I/O due to too many reservation conflicts Apr 30, 2007 2:51 AM
Click to view foofighter26's profile Hot Shot foofighter26 147 posts since
Dec 19, 2006
We are still having these issues does anyone have any suggestions?

We have a support request open with VM but this is going very slowly! :)

regards,

Paul
Reply Re: WARNING: SCSI: 5519: Failing I/O due to too many reservation conflicts Jun 15, 2007 10:37 AM
in response to: foofighter26
Click to view falsehope's profile Novice falsehope 11 posts since
Jun 1, 2007
foo,
Any update???
Reply Re: WARNING: SCSI: 5519: Failing I/O due to too many reservation conflicts Jun 18, 2007 5:35 AM
in response to: falsehope
Click to view Texiwill's profile Champion Texiwill 4,719 posts since
Jan 13, 2004
Hello,

SCSI Reservation conflicts are do to performing to many VMFS metadata updates simultaneously. Depending on your SAN/iSCSI server the limit on simultaneous metadata updates is severely limited before you start to through SCSI conflict messages.

First things I would do: Verify the firmware level of your SAN, Fabric Switches, and HBAs are all at the desired level. For example, if you have a Hitachi SAN there was a firmware upgrade required so that the reservations responded much faster and therefore limited the amount of conflicts. Without this minimum version, conflicts would happen constantly.

Second thing I would do, is verify any 3rd party agents, or in house written scripts that run on your ESX hosts on a regular basis. If for example, you constantly 'touch' the files on your SAN to make changes to times, ownership, or groups, and you are doing this simultaneously on ALL your ESX servers, you could easily throw 1000s if not millions of SCSI Reservation conflicts.

Are you touching (file metadata changes), creating, removing, vmotioning, backing up (create and delete) VMs on the same LUN simultaneously? The limit on the number of metadata updates is on a per LUN basis.

I would take your LUN metadata activities and pare them down to only 1 per LUN at any given time and slowly increase that number until you see reservations conflicts. That will give you the upper limit that you can do per LUN.

Best regards,
Edward
Reply Re: WARNING: SCSI: 5519: Failing I/O due to too many reservation conflicts Jun 19, 2007 12:19 AM
Click to view Muddy's profile Lurker Muddy 1 posts since
Jan 9, 2007
How do I check the LUN Metadata activities? I am seeing the same errors intermittantly with my HP BL685c servers and EVA5000. It is error free for a week then starts showing these errors intermittantly. I am not creating a VM or VMotion at the time. Errors logged at night during backups as well as during the day. Tried setting the port speed and no difference. Does anyone have a view on if the Insite Manager agents could be causing it?
Reply Re: WARNING: SCSI: 5519: Failing I/O due to too many reservation conflicts Jun 19, 2007 7:46 AM
in response to: Muddy
Click to view Texiwill's profile Champion Texiwill 4,719 posts since
Jan 13, 2004
Hello,

VMFS metadata activities occur when you:

update the path/name, access/modification/create times, permissions/ownerships, size of a file on the VMFS
Files grow in 15MB chunks usually.
vMotion, clone, remove, migrate a file on the VMFS
open/close files on the VMFS

Effectively everytime you touch one of the VM files you buy a lock. This is why most VMDKs are monolithic. You do not get locks for changing the contents of an already opened file. But the open produces a short term lock, as does the close. And only if a file grows greater than the existing bounds do you produce locks.

-delta.vmdk files are a case where the file is constantly growing past its boundaries. Every 15MBs requires a new lock to change the size of the file.

The more simultaneous LUN activities the more chance of failures do to SCSI reservation conflicts. Reducing the number of LUN activities solves the problem. For example, 8 simultaneous vMotions on the same LUN will produce SCSI reservation conflicts. Yet 8 simultaneous vMotions using 8 different LUNs will not.

Best regards,
Edward
Actions

Incoming Links

Up to Discussions in VI: ESX 3.5