ql_ima.log file

[daboud]

Hello Everyone.

I've discovered that a log file named ql_ima.log is growing over time.  Eventually, it gets big enough to fill the /tmp directory.  This of course causes problems with any other process that needs to write to the /tmp directory.  I've tried turning off verbose logging with "esxcfg-module --verbose 0 qla4xxx" which I think probably works for that module, but don't know if the ql_ima.log file belongs to that module.  There is a ql_ima_sdm.log file that seems to have a limit on size configured somewhere because the ql_ima_sdm.log_old file exists (always at 4.8MB).  Of note is that a "ql_ima.log_old" file does NOT exist.

Does anyone know how I can disable this ql_ima.log file completely or at least cap it at a few MBs?  I don't believe I've been able to find the right person at VMware Technical Support that knows anything about this file or it's configuration.  If anybody knows anything, I would greatly appreciate it.

ESXi 5.5 build 1331820

Thanks.

[SteveAntle]

Hello,

I am having the same issue on all of my ESXi hosts:

Product: VMware ESXi

Version: 5.5.0

Build: Releasebuild-1474528

Update: 0

The two files in question are:

188154205 May 30 11:25 ql_ima.log

    3195746 May 30 11:25 ql_ima_sdm.log

Located in /tmp directory.

I am having issues now with VDP backups - one by one the host management interfaces are crashing during a VDP backup process  and the hosts disconnect from vCenter Server. This only started happening since the /tmp folder started to fill up (related? not sure, but /tmp having 0% free space is an issue for sure).

Help anyone?

[daboud]

Steve.

I did get an answer from VMware.  You need to update the Qlogic driver.  I downloaded qla4xxx-634.55.28.0-1682448.zip and extracted it to get the off-line bundle file.  I then uploaded it to a datastore and ran these steps to fix the issue.  After I was finished, the ql_ima.log and accompanying files were gone (deleted or moved I don’t know).  But anyway, this fixed the issue of the /tmp directory filling up and causing me problems.

Place host in Maintenance Mode

Navigate to datastore directory

# cd /vmfs/volumes/<Datastore name>

List the files

# ls

Copy the offline bundle to /var/log/vmware

# cp qla4xxx-634.55.28.0-offline_bundle-1682448.zip /var/log/vmware

Install the driver

# esxcli software vib install -d /var/log/vmware/qla4xxx-634.55.28.0-offline_bundle-1682448.zip

Reboot

# reboot

Mark

[daboud]

Steve.

I did get an answer from VMware. You need to update the Qlogic driver. I downloaded qla4xxx-634.55.28.0-1682448.zip and extracted it to get the off-line bundle file. I then uploaded it to a datastore and ran these steps to fix the issue. After I was finished, the ql_ima.log and accompanying files were gone (deleted or moved I don’t know). But anyway, this fixed the issue of the /tmp directory filling up and causing me problems.

Place host in Maintenance Mode

Navigate to datastore directory

1. cd /vmfs/volumes/<Datastore name>

List the files

1. ls

Copy the offline bundle to /var/log/vmware

1. cp qla4xxx-634.55.28.0-offline_bundle-1682448.zip /var/log/vmware

Install the driver

1. esxcli software vib install -d /var/log/vmware/qla4xxx-634.55.28.0-offline_bundle-1682448.zip

Reboot

1. reboot

Mark

[SteveAntle]

Mark,

Thanks for the reply, the drive update worked like a charm!

Steve

[MatthewCEvans]

I had a number of hosts where the log files were being created in the /tmp directory but a number that didn't have the logs, even though everything was patched to the same version and same updates.  Turned out that these log files are also related to the QLogic iSCSI IMA Plug-IN.  To turn off logging from the QLogic iSCSI IMA Plug-IN, delete this file: /etc/ql_ima.cfg.  This information is from this QLogic PDF: http://filedownloads.qlogic.com/files/driver/79829/release_iscsi_adapter_driver_vmware_esx_esxi_41x.....

[HippoCoar]

Mark,

Thank you so much for this. We didn't even realize we had a problem with our /tmp directory until we tried to run a data collection tool that used that directory to run the script. Our /tmp directories were completely full before this ran.

[krktech]

I have this issue on a few hosts and many hosts that don't have the issue, all with similar setup, vSphere 5.5 and the same QLogic version. ( ima-qla4xxx  500.2.01.31-1vmw.0.3.040121).   So why are only a few hosts getting the ql_ima.log files created and filling up /tmp?   The only thing I can find is that the hosts with the issue have iSCSI connections to Nimble storage (hosts with only iSCSI to NetApp don't have the log files created).  

Thanks to Mark for the fix.

[Darkschneidr]

Interesting... I have HP servers with iscsi connections to Nimble as well, and I'm seeing it.

[doubletwist13]

Add us to the list - we are connected to a Nimble storage and have this issue. We will try the driver update and see if that helps.

[musc_county_is]

We recently had this issue, which caused a host to basically become unusable and needed to be physically powered off.

We are running ESXi 5.5.0 2718055 on Dell R720 hosts and connect to an EqualLogic SAN.

When the server was back online, the VMware support engineer that we worked with deleted the ql_ima.log and ql_ima.sdm.log files in the /tmp directory and mentioned that we needed to reboot the hosts "regularly" to help prevent this form happening, but did not offer any resolution or mention the solution that Mark posted.

It would be nice to know if that solution would work for our systems or if there is something else that should be done.

After running the host has been running for 3.5 days, this is what we show for file sizes:

7695511 Aug 31 13:49 ql_ima.log

2946749 Aug 31 13:49 ql_ima_sdm.log

5000027 Aug 31 07:13 ql_ima_sdm.log_old

[systimax]

Same problem with us and same scenario. HP servers connected to a Nimble SAN.

Do you guys suggest installing the drivers from Qlogic instead of updating from HPs embedded ESX driver repository or delete the CFG file that does the logging?

[Darkschneidr]

What I wound up doing was to push out a new baseline esxi image that I knew didn't include that driver, which solved the issue for me. I don't have qlogic hardware in my hosts.

[BonnoB]

Just had contact with Nimble support. Indeed, for us Nimble users it is the same.

Host in maintenance mode.

ssh into host

delete /tmp/ql_ima* log files,

delete /etc/ql_ima.cfg file (file existence is responsible for creating log files)

update driver (download it from MyVMware, instructions see other mesage)

reboot host

Exit maintenance mode