Failed to detach lun

[ashwinsancheti]

During the unmount operation of VMFS datastore we perform following steps.

• UNmount the vmfs volume (UnmountVMFSVolume())
• Detech the scsi lun. (DetachSsciLun())

Detach scsi lun operation is getting failed with “cannot change host configuration” with following error in hostd and vmkernel logs.

This is very critical issue for us.

Corresponding logs.

1. VMkernel.log

2015-10-12T21:29:26.088Z cpu7:34586 opID=e7de38b1)World: 14299: VC opID 49cf6a36-9d maps to vmkernel opID e7de38b1

2015-10-12T21:29:26.088Z cpu7:34586 opID=e7de38b1)LVM: 13173: File system '[PURE_40GB_DS_GX_BACKUP_165_2_159, 561c25ff-1e85d11c-9e1a-0026b9529af6]' (LV snap-2fa22fd0-549af13e-228dc311-e67d-0026b9529af6) un-mounted.

2015-10-12T21:29:36.752Z cpu5:36977081 opID=cee81ea4)World: 14299: VC opID 3a07085f-3e maps to vmkernel opID cee81ea4

2015-10-12T21:29:36.752Z cpu5:36977081 opID=cee81ea4)WARNING: ScsiDevice: 7509: Detach failed for device :naa.624a9370c9cfbacd653c81f20002cca9. Exceeded the number of devices that can be detached, please cleanup stale detach entries.

2015-10-12T21:29:40.031Z cpu7:36977081)World: 14299: VC opID hostd-b7e4 maps to vmkernel opID 87de59b4

1. Hostd.log

VmFileSystem: uuid:549af13e-4c537d9e-e170-0026b9529af6, Label:PURE_40GB_DS,logicalDevice:549af13e-228dc311-e67d-0026b9529af6, headExtent:naa.624a9370c9cfbacd653c81f2000115f1:1

VmFileSystem: Fs Info from UnresolvedVmfsVolume

VmFileSystem: uuid:561c25ff-1e85d11c-9e1a-0026b9529af6, Label:PURE_40GB_DS_GX_BACKUP_165_2_159,logicalDevice:snap-2fa22fd0-549af13e-228dc311-e67d-0026b9529af6,headExtent:naa.624a9370c9cfbacd653c81f20002cca9:1

2015-10-12T21:29:36.752Z [28880B70 error 'Hostsvc' opID=3a07085f-3e user=vpxuser] Failed to Set Device Status OFF for lun with

--> Uuid 02000b0000624a9370c9cfbacd653c81f20002cca9466c61736841: Sysinfo error on operation returned status : Limit exceeded. Please see the VMkernel log for detailed error information

2015-10-12T21:29:36.753Z [28880B70 info 'Default' opID=3a07085f-3e user=vpxuser] AdapterServer caught exception: vim.fault.HostConfigFault

2015-10-12T21:29:36.753Z [28880B70 info 'Vimsvc.TaskManager' opID=3a07085f-3e user=vpxuser] Task Completed : haTask-ha-host-vim.host.StorageSystem.detachScsiLun-42295874 Status error

2015-10-12T21:29:36.753Z [28880B70 info 'Solo.Vmomi' opID=3a07085f-3e user=vpxuser] Activation [N5Vmomi10ActivationE:0x27a62a90] : Invoke done [detachScsiLun] on [vim.host.StorageSystem:storageSystem]

2015-10-12T21:29:36.753Z [28880B70 verbose 'Solo.Vmomi' opID=3a07085f-3e user=vpxuser] Arg lunUuid:

--> "02000b0000624a9370c9cfbacd653c81f20002cca9466c61736841"

2015-10-12T21:29:36.753Z [28880B70 info 'Solo.Vmomi' opID=3a07085f-3e user=vpxuser] Throw vim.fault.HostConfigFault

2015-10-12T21:29:36.753Z [FFDEE920 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {

-->    dynamicType = <unset>,

-->    key = 668759416,

-->    chainId = 0,

-->    createdTime = "1970-01-01T00:00:00Z",

-->    userName = "",

-->    datacenter = (vim.event.DatacenterEventArgument) null,

-->    computeResource = (vim.event.ComputeResourceEventArgument) null,

-->    host = (vim.event.HostEventArgument) {

-->       dynamicType = <unset>,

-->       name = "sg-esx7.commvault.com",

-->       host = 'vim.HostSystem:ha-host',

-->    },

-->    vm = (vim.event.VmEventArgument) null,

-->    ds = (vim.event.DatastoreEventArgument) null,

-->    net = (vim.event.NetworkEventArgument) null,

-->    dvs = (vim.event.DvsEventArgument) null,

-->    fullFormattedMessage = <unset>,

-->    changeTag = <unset>,

-->    eventTypeId = "esx.problem.scsi.device.detach.failed",

-->    severity = <unset>,

-->    message = <unset>,

-->    arguments = (vmodl.KeyAnyValue) [

--> (vmodl.KeyAnyValue) {

--> dynamicType = <unset>,

--> key = "1",

--> value = "naa.624a9370c9cfbacd653c81f20002cca9",

-->       }

-->    ],

-->    objectId = "ha-eventmgr",

-->    objectType = "vim.HostSystem",

-->    objectName = <unset>,

-->    fault = (vmodl.MethodFault) null,

--> }

2015-10-12T21:29:36.753Z [28880B70 info 'Solo.Vmomi' opID=3a07085f-3e user=vpxuser] Result:

--> (vim.fault.HostConfigFault) {

-->    dynamicType = <unset>,

-->    faultCause = (vmodl.MethodFault) null,

-->    msg = "",

--> }

2015-10-12T21:29:36.754Z [FFDEE920 info 'Vimsvc.ha-eventmgr'] Event 23121 : Detach failed for device :naa.624a9370c9cfbacd653c81f20002cca9. Exceeded the number of devices that can be detached, please cleanup stale detach entries.

[greco827]

Check to see if you have other storage devices that are detached but not permanently removed.

If you want the device to permanently decommission from an ESXi host, manually remove the NAA entries from the host configuration:

1. To list the permanently detached devices, run this command:
   
   # esxcli storage core device detached list
   
   You see output similar to:
   
   Device UID                            State
   ------------------------------------  -----
   naa.50060160c46036df50060160c46036df  off
   naa.6006016094602800c8e3e1c5d3c8e011  off
   
   
2. To permanently remove the device configuration information from the system, run this command:
   
   # esxcli storage core device detached remove -d NAA_ID
   
   For example:
   
   # esxcli storage core device detached remove -d naa.50060160c46036df50060160c46036df

VMware KB: Unmounting a LUN or detaching a datastore/storage device from multiple VMware ESXi 5.x/6....

[KWirz]

I had the same problem. It occurs if you frequently mount snapshots. This error occurs around the time you have 512 old entries. You can remove them one at a time as mentioned above. Here is a script to remove all the stale entries:

esxcli storage core device detached list|grep -E "^naa|^eui"|awk '{print $1}'|while read i;do echo "Removing device \"$i\" from the detached list ...";esxcli storage core device detached remove -d "$i";sleep 1;done