VMware

This Question is Not Answered

1 "correct" answer available (10 pts) 1 "helpful" answer available (6 pts)
8 Replies Last post: Jun 19, 2007 7:46 AM by Texiwill  

WARNING: SCSI: 5519: Failing I/O due to too many reservation conflicts posted: Apr 16, 2007 7:10 AM

Click to view foofighter26's profile Hot Shot 170 posts since
Dec 19, 2006
WARNING: SCSI: 5519: Failing I/O due to too many reservation conflicts in vmkernel logs. We are currently having an issue where 3 blades changed to not responsive and the vms on them were powered on but unresponsive. The datastore on the SAN's scsi reservation had changed from none to regular which caused the datastore to become inaccessible. We rebooted the blades which appeared to change the reservation on the SAN back to none then we were able to power on the vms with the exception of one blade where we had to rescan the datastore to get the datastore back. Equipment: HP EVA 8000 HP BL20P G4 Blades Fibre Switch is 4GB
Click to view wtreutz's profile Enthusiast 82 posts since
Oct 7, 2004
How are your FC-Switch-Port settings, and in which szzenario do you have the errors.

I had seen in /var/log/vmkernel something like
vmkernel: 0:00:44:23.107 cpu2:1036)WARNING: SCSI: 7916: status SCSI reservation conflict, rstatus #c0de01 for vmhba1:0:0. residual R 919, CR 0, ER 3
vmkernel: 0:00:44:23.107 cpu2:1036)WARNING: FS3: 4008: Reservation error: SCSI reservation conflict
vmkernel: 0:00:44:23.393 cpu2:1034)SCSI: vm 1034: 5509: Sync CR at 0
vmkernel: 0:00:44:23.393 cpu2:1034)WARNING: SCSI: 5519: Failing I/O due to too many reservation conflicts
after FC-Storage failover. For us it was reproducible the settings on the FC-Switch-Ports. We must set the PortSpeed fix and set the PortType to F-Type only (no E-Type and no L-Type)
Click to view wtreutz's profile Enthusiast 82 posts since
Oct 7, 2004
Have you Informations, what HP preferred??
I know, some Storage-Vendors preferred fix seetings for Port-Speed too. I preferred fix settings for both to all FC-Target-Devices, if possible.

regards
Werner
Click to view falsehope's profile Novice 14 posts since
Jun 1, 2007
foo,
Any update???
Click to view Texiwill's profile Guru 10,205 posts since
Jan 13, 2004
Hello,

SCSI Reservation conflicts are do to performing to many VMFS metadata updates simultaneously. Depending on your SAN/iSCSI server the limit on simultaneous metadata updates is severely limited before you start to through SCSI conflict messages.

First things I would do: Verify the firmware level of your SAN, Fabric Switches, and HBAs are all at the desired level. For example, if you have a Hitachi SAN there was a firmware upgrade required so that the reservations responded much faster and therefore limited the amount of conflicts. Without this minimum version, conflicts would happen constantly.

Second thing I would do, is verify any 3rd party agents, or in house written scripts that run on your ESX hosts on a regular basis. If for example, you constantly 'touch' the files on your SAN to make changes to times, ownership, or groups, and you are doing this simultaneously on ALL your ESX servers, you could easily throw 1000s if not millions of SCSI Reservation conflicts.

Are you touching (file metadata changes), creating, removing, vmotioning, backing up (create and delete) VMs on the same LUN simultaneously? The limit on the number of metadata updates is on a per LUN basis.

I would take your LUN metadata activities and pare them down to only 1 per LUN at any given time and slowly increase that number until you see reservations conflicts. That will give you the upper limit that you can do per LUN.

Best regards,
Edward
Click to view Muddy's profile Lurker 1 posts since
Jan 9, 2007
How do I check the LUN Metadata activities? I am seeing the same errors intermittantly with my HP BL685c servers and EVA5000. It is error free for a week then starts showing these errors intermittantly. I am not creating a VM or VMotion at the time. Errors logged at night during backups as well as during the day. Tried setting the port speed and no difference. Does anyone have a view on if the Insite Manager agents could be causing it?
Click to view Texiwill's profile Guru 10,205 posts since
Jan 13, 2004
Hello,

VMFS metadata activities occur when you:

update the path/name, access/modification/create times, permissions/ownerships, size of a file on the VMFS
Files grow in 15MB chunks usually.
vMotion, clone, remove, migrate a file on the VMFS
open/close files on the VMFS

Effectively everytime you touch one of the VM files you buy a lock. This is why most VMDKs are monolithic. You do not get locks for changing the contents of an already opened file. But the open produces a short term lock, as does the close. And only if a file grows greater than the existing bounds do you produce locks.

-delta.vmdk files are a case where the file is constantly growing past its boundaries. Every 15MBs requires a new lock to change the size of the file.

The more simultaneous LUN activities the more chance of failures do to SCSI reservation conflicts. Reducing the number of LUN activities solves the problem. For example, 8 simultaneous vMotions on the same LUN will produce SCSI reservation conflicts. Yet 8 simultaneous vMotions using 8 different LUNs will not.

Best regards,
Edward

VMware Developer

SDKs, APIs, Videos, Learn and much more in the Developer community.

Learn More

Developer Sample Code

Increase your developer productivity with VMware API sample code.

Learn More

VMworld Sessions & Labs

Online access to the latest VMworld Sessions & Labs and online services.

Learn more

Purchase PSO Credits Online

Purchase credits to redeem training and consulting services online.

Buy Now

Community Hardware Software

View reported configurations or report your own.

Learn More

VMware vSphere

Come witness the next giant leap in virtualization.

Register Today

Communities