VMware Cloud Community
Cougar281
Enthusiast
Enthusiast

Host dropping iSCSI connection

I've been having issues for a while with storage becoming unavailable which typically only happened if I put a heavy load on it. Typically, the only time it would show up is in the rare event of a power outage if I was scrambling to try and get a bunch of guests to suspend, it would start out ok but then after a little bit, it would come to a grinding halt and everything would end up crashing anyway. I was originally thinking it was a FreeNAS issue, as I never had issues like this when I was using a Promise vTrak, and I've never had issues with Dell Powervault, EqualLogic or Compellent, but when I had a similar occurrence today, I found event logs on one of my hosts that suggest that it may actually be a host issue. The system is a Dell Poweredge FX2s chassis with two FC630's as VMWare hosts running 7.0.3.01200 and one FC630 running FreeNAS 11.1u7, with two SAS HBAs passed through to it connecting that compute module to external SAS enclosures. The switching is handled by a pair of FN410S IO Aggregators in full switch mode, with all iSCSI switching contained within those modules. 

To do some testing, I set up another FreeNAS server on an R620 and connected that to another external enclosure and provisioned storage on it, then moved some 'less essential' machines to it. After really beating it up by storage vmotioning eleven machines, when I attempted to move another one in the opposite direction (from the LUN I moved the other machines to to the one they came from), it failed. I then noticed that one of the two HBAs was missing the LUNs from the test FreeNAS, and when I tried to browse what was the target LUN, it couldn't be accessed. 

I put the host in maintenance mode and initiated a reboot, but it hung on shutdown forcing me to reboot it using the iDRAC. Once it came back, all LUNs were present on all paths. When I looked in the logs, I found a lot of entries pertaining to the qfle3i driver.

 

"Description","Type","Date Time","Task","Target","User","Event Type ID"
"Migrating VMware vCenter Server 7.0 off host","Information","01/31/2023, 12:28:16 PM","","VMware vCenter Server 7.0","********","vim.event.VmEmigratingEvent"
"Started to enter maintenance mode. Waiting for virtual machines to shut down, suspend, or migrate","Information","01/31/2023, 12:28:13 PM","","esxi1.domain.local","********","vim.event.EnteringMaintenanceModeEvent"
"Task: Enter maintenance mode","Information","01/31/2023, 12:28:13 PM","Enter maintenance mode","esxi1.domain.local","********","vim.event.TaskEvent"
"Event burst of 'com.vmware.vc.StatelessAlarmTriggeredEvent' started","Warning","01/31/2023, 12:28:12 PM","","esxi1.domain.local","","com.vmware.vc.EventBurstStartedEvent"
"Alarm 'Host error' on esxi1.domain.local triggered by event 15446001 'Issue detected on esxi1.domain.local in DC1: qfle3i:vmhba70: qfle3i_process_ofld_cmpl:2603 unable to allocate iSCSI context resources
 (2023-01-31T18:28:02.695Z cpu20:2098174)'","Information","01/31/2023, 12:28:12 PM","","esxi1.domain.local","","com.vmware.vc.StatelessAlarmTriggeredEvent"
"Alarm 'Host error' on esxi1.domain.local triggered by event 15446000 'Issue detected on esxi1.domain.local in DC1: cnic_qfle3_iscsi_ofld1:2386:(0000:01:00.3): Error: iscsi_conn=0x100 max_iscsi_conn=0x100
 (2023-01-31T18:28:02.695Z cpu20:2098174)'","Information","01/31/2023, 12:28:12 PM","","esxi1.domain.local","","com.vmware.vc.StatelessAlarmTriggeredEvent"
"Alarm 'Host error' on esxi1.domain.local triggered by event 15445999 'Issue detected on esxi1.domain.local in DC1: qfle3i:vmhba70: qfle3i_process_ofld_cmpl:2603 unable to allocate iSCSI context resources
 (2023-01-31T18:28:01.938Z cpu20:2098174)'","Information","01/31/2023, 12:28:12 PM","","esxi1.domain.local","","com.vmware.vc.StatelessAlarmTriggeredEvent"
"Alarm 'Host error' on esxi1.domain.local triggered by event 15445998 'Issue detected on esxi1.domain.local in DC1: cnic_qfle3_iscsi_ofld1:2386:(0000:01:00.3): Error: iscsi_conn=0x100 max_iscsi_conn=0x100
 (2023-01-31T18:28:01.938Z cpu20:2098174)'","Information","01/31/2023, 12:28:12 PM","","esxi1.domain.local","","com.vmware.vc.StatelessAlarmTriggeredEvent"
"Alarm 'Host error' on esxi1.domain.local triggered by event 15445997 'Issue detected on esxi1.domain.local in DC1: qfle3i:vmhba70: qfle3i_process_ofld_cmpl:2603 unable to allocate iSCSI context resources
 (2023-01-31T18:27:59.674Z cpu10:2098174)'","Information","01/31/2023, 12:28:12 PM","","esxi1.domain.local","","com.vmware.vc.StatelessAlarmTriggeredEvent"
"Alarm 'Host error' on esxi1.domain.local triggered by event 15445996 'Issue detected on esxi1.domain.local in DC1: cnic_qfle3_iscsi_ofld1:2386:(0000:01:00.3): Error: iscsi_conn=0x100 max_iscsi_conn=0x100
 (2023-01-31T18:27:59.674Z cpu10:2098174)'","Information","01/31/2023, 12:28:12 PM","","esxi1.domain.local","","com.vmware.vc.StatelessAlarmTriggeredEvent"
"Alarm 'Host error' on esxi1.domain.local triggered by event 15445995 'Issue detected on esxi1.domain.local in DC1: qfle3i:vmhba70: qfle3i_process_ofld_cmpl:2603 unable to allocate iSCSI context resources
 (2023-01-31T18:27:58.919Z cpu20:2098174)'","Information","01/31/2023, 12:28:12 PM","","esxi1.domain.local","","com.vmware.vc.StatelessAlarmTriggeredEvent"
"Alarm 'Host error' on esxi1.domain.local triggered by event 15445994 'Issue detected on esxi1.domain.local in DC1: cnic_qfle3_iscsi_ofld1:2386:(0000:01:00.3): Error: iscsi_conn=0x100 max_iscsi_conn=0x100
 (2023-01-31T18:27:58.919Z cpu20:2098174)'","Information","01/31/2023, 12:28:12 PM","","esxi1.domain.local","","com.vmware.vc.StatelessAlarmTriggeredEvent"
"Alarm 'Host error' on esxi1.domain.local triggered by event 15445993 'Issue detected on esxi1.domain.local in DC1: qfle3i:vmhba70: qfle3i_process_ofld_cmpl:2603 unable to allocate iSCSI context resources
 (2023-01-31T18:27:56.657Z cpu0:2098174)'","Information","01/31/2023, 12:28:12 PM","","esxi1.domain.local","","com.vmware.vc.StatelessAlarmTriggeredEvent"
"Alarm 'Host error' on esxi1.domain.local triggered by event 15445992 'Issue detected on esxi1.domain.local in DC1: cnic_qfle3_iscsi_ofld1:2386:(0000:01:00.3): Error: iscsi_conn=0x100 max_iscsi_conn=0x100
 (2023-01-31T18:27:56.656Z cpu0:2098174)'","Information","01/31/2023, 12:28:12 PM","","esxi1.domain.local","","com.vmware.vc.StatelessAlarmTriggeredEvent"
"Alarm 'Host error' on esxi1.domain.local triggered by event 15445991 'Issue detected on esxi1.domain.local in DC1: qfle3i:vmhba70: qfle3i_process_ofld_cmpl:2603 unable to allocate iSCSI context resources
 (2023-01-31T18:27:55.904Z cpu10:2098174)'","Information","01/31/2023, 12:28:12 PM","","esxi1.domain.local","","com.vmware.vc.StatelessAlarmTriggeredEvent"
"Alarm 'Host error' on esxi1.domain.local triggered by event 15445990 'Issue detected on esxi1.domain.local in DC1: cnic_qfle3_iscsi_ofld1:2386:(0000:01:00.3): Error: iscsi_conn=0x100 max_iscsi_conn=0x100
 (2023-01-31T18:27:55.904Z cpu10:2098174)'","Information","01/31/2023, 12:28:12 PM","","esxi1.domain.local","","com.vmware.vc.StatelessAlarmTriggeredEvent"
"Alarm 'Host error' on esxi1.domain.local triggered by event 15445989 'Issue detected on esxi1.domain.local in DC1: qfle3i:vmhba70: qfle3i_process_ofld_cmpl:2603 unable to allocate iSCSI context resources
 (2023-01-31T18:27:53.648Z cpu0:2098174)'","Information","01/31/2023, 12:28:12 PM","","esxi1.domain.local","","com.vmware.vc.StatelessAlarmTriggeredEvent"
"Alarm 'Host error' on esxi1.domain.local triggered by event 15445988 'Issue detected on esxi1.domain.local in DC1: cnic_qfle3_iscsi_ofld1:2386:(0000:01:00.3): Error: iscsi_conn=0x100 max_iscsi_conn=0x100
 (2023-01-31T18:27:53.648Z cpu0:2098174)'","Information","01/31/2023, 12:28:12 PM","","esxi1.domain.local","","com.vmware.vc.StatelessAlarmTriggeredEvent"
"Alarm 'Host error' on esxi1.domain.local triggered by event 15445987 'Issue detected on esxi1.domain.local in DC1: qfle3i:vmhba70: qfle3i_process_ofld_cmpl:2603 unable to allocate iSCSI context resources
 (2023-01-31T18:27:52.895Z cpu0:2098174)'","Information","01/31/2023, 12:28:12 PM","","esxi1.domain.local","","com.vmware.vc.StatelessAlarmTriggeredEvent"
"Alarm 'Host error' on esxi1.domain.local triggered by event 15445986 'Issue detected on esxi1.domain.local in DC1: cnic_qfle3_iscsi_ofld1:2386:(0000:01:00.3): Error: iscsi_conn=0x100 max_iscsi_conn=0x100
 (2023-01-31T18:27:52.895Z cpu0:2098174)'","Information","01/31/2023, 12:28:12 PM","","esxi1.domain.local","","com.vmware.vc.StatelessAlarmTriggeredEvent"
"Alarm 'Host error' on esxi1.domain.local triggered by event 15445985 'Issue detected on esxi1.domain.local in DC1: qfle3i:vmhba70: qfle3i_process_ofld_cmpl:2603 unable to allocate iSCSI context resources
 (2023-01-31T18:27:50.633Z cpu17:2098174)'","Information","01/31/2023, 12:28:12 PM","","esxi1.domain.local","","com.vmware.vc.StatelessAlarmTriggeredEvent"
"Alarm 'Host error' on esxi1.domain.local triggered by event 15445984 'Issue detected on esxi1.domain.local in DC1: cnic_qfle3_iscsi_ofld1:2386:(0000:01:00.3): Error: iscsi_conn=0x100 max_iscsi_conn=0x100
 (2023-01-31T18:27:50.633Z cpu17:2098174)'","Information","01/31/2023, 12:28:12 PM","","esxi1.domain.local","","com.vmware.vc.StatelessAlarmTriggeredEvent"
"Alarm 'Host error' on esxi1.domain.local triggered by event 15445983 'Issue detected on esxi1.domain.local in DC1: qfle3i:vmhba70: qfle3i_process_ofld_cmpl:2603 unable to allocate iSCSI context resources
 (2023-01-31T18:27:49.877Z cpu8:2098174)'","Information","01/31/2023, 12:28:12 PM","","esxi1.domain.local","","com.vmware.vc.StatelessAlarmTriggeredEvent"
"Alarm 'Host error' on esxi1.domain.local triggered by event 15445982 'Issue detected on esxi1.domain.local in DC1: cnic_qfle3_iscsi_ofld1:2386:(0000:01:00.3): Error: iscsi_conn=0x100 max_iscsi_conn=0x100
 (2023-01-31T18:27:49.877Z cpu8:2098174)'","Information","01/31/2023, 12:28:12 PM","","esxi1.domain.local","","com.vmware.vc.StatelessAlarmTriggeredEvent"
"Alarm 'Host error' on esxi1.domain.local triggered by event 15445981 'Issue detected on esxi1.domain.local in DC1: qfle3i:vmhba70: qfle3i_process_ofld_cmpl:2603 unable to allocate iSCSI context resources
 (2023-01-31T18:27:47.611Z cpu8:2098174)'","Information","01/31/2023, 12:28:12 PM","","esxi1.domain.local","","com.vmware.vc.StatelessAlarmTriggeredEvent"
"Alarm 'Host error' on esxi1.domain.local triggered by event 15445980 'Issue detected on esxi1.domain.local in DC1: cnic_qfle3_iscsi_ofld1:2386:(0000:01:00.3): Error: iscsi_conn=0x100 max_iscsi_conn=0x100
 (2023-01-31T18:27:47.611Z cpu8:2098174)'","Information","01/31/2023, 12:28:12 PM","","esxi1.domain.local","","com.vmware.vc.StatelessAlarmTriggeredEvent"
"Alarm 'Host error' on esxi1.domain.local triggered by event 15445979 'Issue detected on esxi1.domain.local in DC1: qfle3i:vmhba70: qfle3i_process_ofld_cmpl:2603 unable to allocate iSCSI context resources
 (2023-01-31T18:27:46.858Z cpu8:2098174)'","Information","01/31/2023, 12:28:12 PM","","esxi1.domain.local","","com.vmware.vc.StatelessAlarmTriggeredEvent"
"Alarm 'Host error' on esxi1.domain.local triggered by event 15445978 'Issue detected on esxi1.domain.local in DC1: cnic_qfle3_iscsi_ofld1:2386:(0000:01:00.3): Error: iscsi_conn=0x100 max_iscsi_conn=0x100
 (2023-01-31T18:27:46.858Z cpu8:2098174)'","Information","01/31/2023, 12:28:12 PM","","esxi1.domain.local","","com.vmware.vc.StatelessAlarmTriggeredEvent"
"Alarm 'Host error' on esxi1.domain.local triggered by event 15445977 'Issue detected on esxi1.domain.local in DC1: qfle3i:vmhba70: qfle3i_process_ofld_cmpl:2603 unable to allocate iSCSI context resources
 (2023-01-31T18:27:44.598Z cpu14:2098174)'","Information","01/31/2023, 12:28:12 PM","","esxi1.domain.local","","com.vmware.vc.StatelessAlarmTriggeredEvent"
"Alarm 'Host error' on esxi1.domain.local triggered by event 15445976 'Issue detected on esxi1.domain.local in DC1: cnic_qfle3_iscsi_ofld1:2386:(0000:01:00.3): Error: iscsi_conn=0x100 max_iscsi_conn=0x100
 (2023-01-31T18:27:44.598Z cpu14:2098174)'","Information","01/31/2023, 12:28:12 PM","","esxi1.domain.local","","com.vmware.vc.StatelessAlarmTriggeredEvent"
"Alarm 'Host error' on esxi1.domain.local triggered by event 15445975 'Issue detected on esxi1.domain.local in DC1: qfle3i:vmhba70: qfle3i_process_ofld_cmpl:2603 unable to allocate iSCSI context resources
 (2023-01-31T18:27:43.844Z cpu14:2098174)'","Information","01/31/2023, 12:28:12 PM","","esxi1.domain.local","","com.vmware.vc.StatelessAlarmTriggeredEvent"
"Alarm 'Host error' on esxi1.domain.local triggered by event 15445974 'Issue detected on esxi1.domain.local in DC1: cnic_qfle3_iscsi_ofld1:2386:(0000:01:00.3): Error: iscsi_conn=0x100 max_iscsi_conn=0x100
 (2023-01-31T18:27:43.844Z cpu14:2098174)'","Information","01/31/2023, 12:28:12 PM","","esxi1.domain.local","","com.vmware.vc.StatelessAlarmTriggeredEvent"
"Alarm 'Host error' on esxi1.domain.local triggered by event 15445973 'Issue detected on esxi1.domain.local in DC1: qfle3i:vmhba70: qfle3i_process_ofld_cmpl:2603 unable to allocate iSCSI context resources
 (2023-01-31T18:27:41.582Z cpu0:2098174)'","Information","01/31/2023, 12:28:12 PM","","esxi1.domain.local","","com.vmware.vc.StatelessAlarmTriggeredEvent"
"Alarm 'Host error' on esxi1.domain.local triggered by event 15445972 'Issue detected on esxi1.domain.local in DC1: cnic_qfle3_iscsi_ofld1:2386:(0000:01:00.3): Error: iscsi_conn=0x100 max_iscsi_conn=0x100
 (2023-01-31T18:27:41.582Z cpu0:2098174)'","Information","01/31/2023, 12:28:12 PM","","esxi1.domain.local","","com.vmware.vc.StatelessAlarmTriggeredEvent"
"Event burst of 'vim.event.GeneralHostWarningEvent' started","Warning","01/31/2023, 12:28:11 PM","","esxi1.domain.local","","com.vmware.vc.EventBurstStartedEvent"
"User dcui@127.0.0.1 logged in as VMware-client/6.5.0","Information","01/31/2023, 12:28:10 PM","","esxi1.domain.local","dcui","vim.event.UserLoginSessionEvent"
"User dcui@127.0.0.1 logged in as VMware-client/6.5.0","Information","01/31/2023, 12:28:09 PM","","esxi1.domain.local","dcui","vim.event.UserLoginSessionEvent"
"User dcui@127.0.0.1 logged in as VMware-client/6.5.0","Information","01/31/2023, 12:28:09 PM","","esxi1.domain.local","dcui","vim.event.UserLoginSessionEvent"
"qfle3i:vmhba70: qfle3i_process_ofld_cmpl:2603 unable to allocate iSCSI context resources
 (2023-01-31T18:28:02.695Z cpu20:2098174)","Warning","01/31/2023, 12:28:02 PM","","esxi1.domain.local","","vim.event.GeneralHostWarningEvent"
"cnic_qfle3_iscsi_ofld1:2386:(0000:01:00.3): Error: iscsi_conn=0x100 max_iscsi_conn=0x100
 (2023-01-31T18:28:02.695Z cpu20:2098174)","Warning","01/31/2023, 12:28:02 PM","","esxi1.domain.local","","vim.event.GeneralHostWarningEvent"
"qfle3i:vmhba70: qfle3i_process_ofld_cmpl:2603 unable to allocate iSCSI context resources
 (2023-01-31T18:28:01.938Z cpu20:2098174)","Warning","01/31/2023, 12:28:01 PM","","esxi1.domain.local","","vim.event.GeneralHostWarningEvent"
"cnic_qfle3_iscsi_ofld1:2386:(0000:01:00.3): Error: iscsi_conn=0x100 max_iscsi_conn=0x100
 (2023-01-31T18:28:01.938Z cpu20:2098174)","Warning","01/31/2023, 12:28:01 PM","","esxi1.domain.local","","vim.event.GeneralHostWarningEvent"
"qfle3i:vmhba70: qfle3i_process_ofld_cmpl:2603 unable to allocate iSCSI context resources
 (2023-01-31T18:27:59.674Z cpu10:2098174)","Warning","01/31/2023, 12:27:59 PM","","esxi1.domain.local","","vim.event.GeneralHostWarningEvent"
"cnic_qfle3_iscsi_ofld1:2386:(0000:01:00.3): Error: iscsi_conn=0x100 max_iscsi_conn=0x100
 (2023-01-31T18:27:59.674Z cpu10:2098174)","Warning","01/31/2023, 12:27:59 PM","","esxi1.domain.local","","vim.event.GeneralHostWarningEvent"
"qfle3i:vmhba70: qfle3i_process_ofld_cmpl:2603 unable to allocate iSCSI context resources
 (2023-01-31T18:27:58.919Z cpu20:2098174)","Warning","01/31/2023, 12:27:58 PM","","esxi1.domain.local","","vim.event.GeneralHostWarningEvent"
"cnic_qfle3_iscsi_ofld1:2386:(0000:01:00.3): Error: iscsi_conn=0x100 max_iscsi_conn=0x100
 (2023-01-31T18:27:58.919Z cpu20:2098174)","Warning","01/31/2023, 12:27:58 PM","","esxi1.domain.local","","vim.event.GeneralHostWarningEvent"
"qfle3i:vmhba70: qfle3i_process_ofld_cmpl:2603 unable to allocate iSCSI context resources
 (2023-01-31T18:27:56.657Z cpu0:2098174)","Warning","01/31/2023, 12:27:56 PM","","esxi1.domain.local","","vim.event.GeneralHostWarningEvent"
"cnic_qfle3_iscsi_ofld1:2386:(0000:01:00.3): Error: iscsi_conn=0x100 max_iscsi_conn=0x100
 (2023-01-31T18:27:56.656Z cpu0:2098174)","Warning","01/31/2023, 12:27:56 PM","","esxi1.domain.local","","vim.event.GeneralHostWarningEvent"
"qfle3i:vmhba70: qfle3i_process_ofld_cmpl:2603 unable to allocate iSCSI context resources
 (2023-01-31T18:27:55.904Z cpu10:2098174)","Warning","01/31/2023, 12:27:55 PM","","esxi1.domain.local","","vim.event.GeneralHostWarningEvent"
"cnic_qfle3_iscsi_ofld1:2386:(0000:01:00.3): Error: iscsi_conn=0x100 max_iscsi_conn=0x100
 (2023-01-31T18:27:55.904Z cpu10:2098174)","Warning","01/31/2023, 12:27:55 PM","","esxi1.domain.local","","vim.event.GeneralHostWarningEvent"
"qfle3i:vmhba70: qfle3i_process_ofld_cmpl:2603 unable to allocate iSCSI context resources
 (2023-01-31T18:27:53.648Z cpu0:2098174)","Warning","01/31/2023, 12:27:53 PM","","esxi1.domain.local","","vim.event.GeneralHostWarningEvent"
"cnic_qfle3_iscsi_ofld1:2386:(0000:01:00.3): Error: iscsi_conn=0x100 max_iscsi_conn=0x100
 (2023-01-31T18:27:53.648Z cpu0:2098174)","Warning","01/31/2023, 12:27:53 PM","","esxi1.domain.local","","vim.event.GeneralHostWarningEvent"
"qfle3i:vmhba70: qfle3i_process_ofld_cmpl:2603 unable to allocate iSCSI context resources
 (2023-01-31T18:27:52.895Z cpu0:2098174)","Warning","01/31/2023, 12:27:52 PM","","esxi1.domain.local","","vim.event.GeneralHostWarningEvent"
"cnic_qfle3_iscsi_ofld1:2386:(0000:01:00.3): Error: iscsi_conn=0x100 max_iscsi_conn=0x100
 (2023-01-31T18:27:52.895Z cpu0:2098174)","Warning","01/31/2023, 12:27:52 PM","","esxi1.domain.local","","vim.event.GeneralHostWarningEvent"
"qfle3i:vmhba70: qfle3i_process_ofld_cmpl:2603 unable to allocate iSCSI context resources
 (2023-01-31T18:27:50.633Z cpu17:2098174)","Warning","01/31/2023, 12:27:50 PM","","esxi1.domain.local","","vim.event.GeneralHostWarningEvent"
"cnic_qfle3_iscsi_ofld1:2386:(0000:01:00.3): Error: iscsi_conn=0x100 max_iscsi_conn=0x100
 (2023-01-31T18:27:50.633Z cpu17:2098174)","Warning","01/31/2023, 12:27:50 PM","","esxi1.domain.local","","vim.event.GeneralHostWarningEvent"
"qfle3i:vmhba70: qfle3i_process_ofld_cmpl:2603 unable to allocate iSCSI context resources
 (2023-01-31T18:27:49.877Z cpu8:2098174)","Warning","01/31/2023, 12:27:49 PM","","esxi1.domain.local","","vim.event.GeneralHostWarningEvent"
"cnic_qfle3_iscsi_ofld1:2386:(0000:01:00.3): Error: iscsi_conn=0x100 max_iscsi_conn=0x100
 (2023-01-31T18:27:49.877Z cpu8:2098174)","Warning","01/31/2023, 12:27:49 PM","","esxi1.domain.local","","vim.event.GeneralHostWarningEvent"
"qfle3i:vmhba70: qfle3i_process_ofld_cmpl:2603 unable to allocate iSCSI context resources
 (2023-01-31T18:27:47.611Z cpu8:2098174)","Warning","01/31/2023, 12:27:47 PM","","esxi1.domain.local","","vim.event.GeneralHostWarningEvent"
"cnic_qfle3_iscsi_ofld1:2386:(0000:01:00.3): Error: iscsi_conn=0x100 max_iscsi_conn=0x100
 (2023-01-31T18:27:47.611Z cpu8:2098174)","Warning","01/31/2023, 12:27:47 PM","","esxi1.domain.local","","vim.event.GeneralHostWarningEvent"
"qfle3i:vmhba70: qfle3i_process_ofld_cmpl:2603 unable to allocate iSCSI context resources
 (2023-01-31T18:27:46.858Z cpu8:2098174)","Warning","01/31/2023, 12:27:46 PM","","esxi1.domain.local","","vim.event.GeneralHostWarningEvent"
"cnic_qfle3_iscsi_ofld1:2386:(0000:01:00.3): Error: iscsi_conn=0x100 max_iscsi_conn=0x100
 (2023-01-31T18:27:46.858Z cpu8:2098174)","Warning","01/31/2023, 12:27:46 PM","","esxi1.domain.local","","vim.event.GeneralHostWarningEvent"
"qfle3i:vmhba70: qfle3i_process_ofld_cmpl:2603 unable to allocate iSCSI context resources
 (2023-01-31T18:27:44.598Z cpu14:2098174)","Warning","01/31/2023, 12:27:44 PM","","esxi1.domain.local","","vim.event.GeneralHostWarningEvent"
"cnic_qfle3_iscsi_ofld1:2386:(0000:01:00.3): Error: iscsi_conn=0x100 max_iscsi_conn=0x100
 (2023-01-31T18:27:44.598Z cpu14:2098174)","Warning","01/31/2023, 12:27:44 PM","","esxi1.domain.local","","vim.event.GeneralHostWarningEvent"
"qfle3i:vmhba70: qfle3i_process_ofld_cmpl:2603 unable to allocate iSCSI context resources
 (2023-01-31T18:27:43.844Z cpu14:2098174)","Warning","01/31/2023, 12:27:43 PM","","esxi1.domain.local","","vim.event.GeneralHostWarningEvent"
"cnic_qfle3_iscsi_ofld1:2386:(0000:01:00.3): Error: iscsi_conn=0x100 max_iscsi_conn=0x100
 (2023-01-31T18:27:43.844Z cpu14:2098174)","Warning","01/31/2023, 12:27:43 PM","","esxi1.domain.local","","vim.event.GeneralHostWarningEvent"
"qfle3i:vmhba70: qfle3i_process_ofld_cmpl:2603 unable to allocate iSCSI context resources
 (2023-01-31T18:27:41.582Z cpu0:2098174)","Warning","01/31/2023, 12:27:41 PM","","esxi1.domain.local","","vim.event.GeneralHostWarningEvent"
"cnic_qfle3_iscsi_ofld1:2386:(0000:01:00.3): Error: iscsi_conn=0x100 max_iscsi_conn=0x100
 (2023-01-31T18:27:41.582Z cpu0:2098174)","Warning","01/31/2023, 12:27:41 PM","","esxi1.domain.local","","vim.event.GeneralHostWarningEvent"
"Event burst of 'vim.event.GeneralHostWarningEvent' ended","Information","01/31/2023, 12:28:05 PM","","esxi1.domain.local","","com.vmware.vc.EventBurstEndedEvent"
"Bursting event 'vim.event.GeneralHostWarningEvent' occurred 288 times since Tuesday, January 31, 2023 6:24:04 PM UTC","Information","01/31/2023, 12:28:05 PM","","esxi1.domain.local","","com.vmware.vc.EventBurstCompressedEvent"
"Event burst of 'com.vmware.vc.StatelessAlarmTriggeredEvent' ended","Information","01/31/2023, 12:28:05 PM","","esxi1.domain.local","","com.vmware.vc.EventBurstEndedEvent"
"Bursting event 'com.vmware.vc.StatelessAlarmTriggeredEvent' occurred 288 times since Tuesday, January 31, 2023 6:24:41 PM UTC","Information","01/31/2023, 12:28:05 PM","","esxi1.domain.local","","com.vmware.vc.EventBurstCompressedEvent"
"Alarm 'Host error': an SNMP trap for entity esxi1.domain.local was sent","Information","01/31/2023, 12:26:41 PM","","esxi1.domain.local","","vim.event.AlarmSnmpCompletedEvent"
"Alarm 'Host error' on esxi1.domain.local triggered an action","Information","01/31/2023, 12:26:41 PM","","esxi1.domain.local","","vim.event.AlarmActionTriggeredEvent"
"Login to iSCSI target iqn.2005-10.org.freenas2.ctl:vmware2-b-tgt on vmhba70 @ vmk2 failed. The iSCSI initiator could not establish a network connection to the target.","Error","01/31/2023, 12:26:06 PM","","esxi1.domain.local","","esx.problem.storage.iscsi.target.connect.error"
"Login to iSCSI target iqn.2005-10.org.freenas2.ctl:vmware1-b-tgt on vmhba70 @ vmk2 failed. The iSCSI initiator could not establish a network connection to the target.","Error","01/31/2023, 12:26:06 PM","","esxi1.domain.local","","esx.problem.storage.iscsi.target.connect.error"
"Login to iSCSI target iqn.2005-10.org.freenas.ctl:vmware2 on vmhba70 @ vmk2 failed. The iSCSI initiator could not establish a network connection to the target.","Error","01/31/2023, 12:26:06 PM","","esxi1.domain.local","","esx.problem.storage.iscsi.target.connect.error"

 

 

I searched around for some of the errors, but didn't come up with anything overly helpful. 

Does anyone have any ideas as to what might be causing this? Are the 'native' qfle3 drivers as bad and unstable as they used to be?

 

 

Reply
0 Kudos
6 Replies
maksym007
Expert
Expert

As always start with checking iDRAC/ iLO if there some erros visible. 
Check with esxcli hba adapters if they are online. 
Check driver and firmware versions

Reply
0 Kudos
Cougar281
Enthusiast
Enthusiast

No errors anywhere in the logs that I can find, The HBAs are online and when the one unit dropped off, the hosts connection to my main FreeNAS system was still up, so one HBA showed five devices and five paths, while the 'problem' controller showed three devices and three paths. Also, the 'problem' controller was (and is) replying to pings from the test FreeNAS box.

Firmware is current (15.35.06) and the driver version is (should be) the version that's baked into the Dell customized 7.0.3 build 20842708 (1.4.35.0-1OEM.700.1.0.15843807).

 

Tags (1)
Reply
0 Kudos
maksym007
Expert
Expert

Do you have a direct connection I mean from Host to NAS or switch is between them? 

Reply
0 Kudos
Cougar281
Enthusiast
Enthusiast

Just the FN410S IO aggregators (basically a fancy name for a switch) are in between the FreeNAS hosts and the VMWare hosts. With regards to the FC630 hosts, everything is connected via QP 10Gb NDCs via the backplane, and the R620 test FreeNAS machine has one connection to each FN410S via Dell direct attach SFP+ twinax cables.

Reply
0 Kudos
Cougar281
Enthusiast
Enthusiast

I'm beginning to think that I'm barking up the wrong tree and the problem actually IS with FreeNAS after all and the errors in VMware were a red herring caused by FreeNAS having gone sideways. After rebooting the host, I left it in maintenance mode for a while, and I tried to move a few machines off the test FreeNAS machine, and they failed with a message saying something along the lines of 'unable to complete due to host status' suggesting that the host was in maintenance mode, which the one the machines were running on obviously was not in. I later discovered that even though the host was reporting 5/5 for the devices/paths, when I tried to browse the datastore, it still wasn't behaving. 

I took the host out of maintenance mode and.... nothing moved over to it like usually happens. Trying to manually move (both storage and compute) machines resulted in a similar error. Then I bounced the test FreeNAS machine and pretty much as soon as it went down, DRS started moving machines over to the host I had rebooted earlier. After the reboot I can now browse the datastores on the test FreeNAS machine, and other functions are working as expected. It seems like FreeNAS may actually be what's taking a walk, causing those errors on the ESXi host and really annoying both hosts.

Reply
0 Kudos
maksym007
Expert
Expert

Glad that ESXi hosts were Ok from very beginning. 

Reply
0 Kudos