Strieckel
Contributor
Contributor

/tmp/mili2d.log writes my tmp folder full which course an esxi outage

Hi Gent`s

I hope you could help me further or pointing into the right direction.

We had last week an ESXi Server outage which was produced as the /TMP folder was full.

I notice now that there is still writing an mili2d.log, which has filled the /TMP folder.

As far as I could find out this log file will be produced by our network cards.

My question is how could I stop writing the log file and how could I solve the error which are reported in the log file...see below.

Many thanks for your help.

Br

Jörg

Fri May 22 02:57:28 2015,14995682,   ERROR:could not open device node

Fri May 22 02:57:28 2015,14995682,   ERROR:ioctl_get_chipinfo: ioctl_get_chipinfo failed

Fri May 22 02:57:28 2015,14995682,   ERROR:enumerate_be_nics: ioctl_get_chipinfo failed with status = 1

Fri May 22 02:57:28 2015,14995682,   CRITICAL:NO BE CARD FOUND

Fri May 22 02:57:28 2015,15585519,   ERROR:could not open device node

Fri May 22 02:57:28 2015,15585519,   ERROR:ioctl_get_chipinfo: ioctl_get_chipinfo failed

Fri May 22 02:57:28 2015,15585519,   ERROR:enumerate_be_nics: ioctl_get_chipinfo failed with status = 1

Fri May 22 02:57:28 2015,15585519,   CRITICAL:NO BE CARD FOUND

Fri May 22 02:58:45 2015,15585762,   ERROR:could not open device node

Fri May 22 02:58:45 2015,15585762,   ERROR:ioctl_get_chipinfo: ioctl_get_chipinfo failed

Fri May 22 02:58:45 2015,15585762,   ERROR:enumerate_be_nics: ioctl_get_chipinfo failed with status = 1

Fri May 22 02:58:45 2015,15585762,   CRITICAL:NO BE CARD FOUND

0 Kudos
16 Replies
cykVM
Expert
Expert

The mili2d.log is from the MILI management daemon for Emulex HBAs. So maybe there's a problem with your Emulex card?

See also this discussion with the mili2d.log filling up: /tmp running out of Space - vSphere 5 - LOG FILE ERROR: log.cpp(610) : No CIMPLE_HOME env var define...

Same could happen with RHEL: /tmp running out of Space due to mili2d.log file - Red Hat Customer Portal

Guess the daemon needs to be configured or even uninstalled for that to stop.

If your host is an HP you may also read: Emulex - HP Enterprise Business Community

Strieckel
Contributor
Contributor

@cykVM

Many thanks for the quick response...will check the links.

Br

Joerg

0 Kudos
Strieckel
Contributor
Contributor

Hmmm...the links did not give me an answer.

The redhat link is not allowed to access as I have no service account for redhat.

The strange thing is,  it seems tobe produced by an emulex product, but I can not find a emulex hardware installed in my server. HBA / NIC`s are Qlogic...

How could I find out which process driver is writing the mili2d.log, sorry to ask but I am not really familar with Linux commands.

Br

Joerg

0 Kudos
cykVM
Expert
Expert

Qlogic and Emulex once were the same company, so it might be that Qlogic uses some driver parts of Emulex (or uses just the same filename for the logfile).

You may try to find the process writing to mili2d.log by: lsof | grep mili

Maybe you should check which drivers are actually installed on that host and thereafter check if any driver updates are available.

What's the server type/model and the type/model of the Qlogic cards?

0 Kudos
Strieckel
Contributor
Contributor

Hi cykVM

What you are writing is quit helpfull...many thanks.

lsof | grep mili   gives back "hostd-worker"


Hp dl 380p Gen8

Qlogic Corp HP NC523SFP 10GBE 2-port Ethernet Server Adapter

ISP2532-based 87Gb FC HBA


It`s possibly that there are new drivers availeble as HP has released a new service pack, but got not yet the time to update.


Br

Joerg


0 Kudos
cykVM
Expert
Expert

Hmm, the hostd-worker process can't be the culprit, must be some child process. I suspect some CIM provider being accidentally installed and that is constantly looking for an Emulex card, not finding it and logging that to the mili2d.log.But that's just a guess.

Was that host installed using the HP customized VMWare image?

Emulex calls it OneCommand Manager for VMWare - this is the managing piece of software for the Emulex cards. Here is a KB about that: VMware KB: Emulex CIM Provider Overview for ESXi 5.0

Are you maybe using Emulex cards in other hosts? Emulex states here Management that you need to install that OneCommand thing on every host if you use the plugin for vCenter. So maybe someone accidentally installed the CIM provider on that host (without an Emulex card)?

0 Kudos
Strieckel
Contributor
Contributor

Yes, it`s strange.

I have tried to search on the host for emulex but could not find anything.

In our esxi server farm are no emulex cards available, but I am not aware about server out of VMware...

Yep, we have installed the HP custom image.

0 Kudos
cykVM
Expert
Expert

Try this command on SSH/CLI:

esxcli system settings advanced list | grep CIM

this should list you the installed CIM providers. Anything with "Emulex" coming up?

For disabling a single or multiple providers you may follow this KB: VMware KB: Disabling an ESXi CIM provider when it fails or is unstable

0 Kudos
Strieckel
Contributor
Contributor

Nothing with Emulex to find?!

Do you have more ideas what I can check?

Happy weekend...

~ # esxcli system settings advanced list | grep CIM

   Path: /UserVars/CIMEnabled

   Description: Enable or disable the CIM service

   Path: /UserVars/CIMLogLevel

   Description: Set the log level of the CIM Service

   Path: /UserVars/CIMWatchdogInterval

   Description: Set the watchdog polling interval for the CIM Service

   Path: /UserVars/CIMvmw_hhrcwrapperProviderEnabled

   Description: Enable or disable the CIM vmw_hhrcwrapper provider

   Path: /UserVars/CIMvmw_pciProviderEnabled

   Description: Enable or disable the CIM vmw_pci provider

   Path: /UserVars/CIMvmw_iodmProviderProviderEnabled

   Description: Enable or disable the CIM vmw_iodmProvider provider

   Path: /UserVars/CIMvmw_hdrProviderEnabled

   Description: Enable or disable the CIM vmw_hdr provider

   Path: /UserVars/CIMvmw_kmoduleProviderEnabled

   Description: Enable or disable the CIM vmw_kmodule provider

   Path: /UserVars/CIMvmw_viProviderEnabled

   Description: Enable or disable the CIM vmw_vi provider

   Path: /UserVars/CIMvmw_sfcbrInteropProviderEnabled

   Description: Enable or disable the CIM vmw_sfcbrInterop provider

   Path: /UserVars/CIMvmw_swmgtProviderEnabled

   Description: Enable or disable the CIM vmw_swmgt provider

   Path: /UserVars/CIMvmw_hp-smx-providerProviderEnabled

   Description: Enable or disable the CIM vmw_hp-smx-provider provider

   Path: /UserVars/CIMvmw_hhrcwrapperProviderPollingInterval

0 Kudos
cykVM
Expert
Expert

Which version/build of VMWare are you running?

I would first do the firmware/BIOS updates from latest HP SPP, I guess.

Could you find any traces of a folder named 'ocmanager' in your filesystem?

From the log in your initial posting it looks for a "BE" HBA which should use either the be2net, be2iscsi or be3 driver.

Is your server still under warranty and you are able to open a support case with HP and/or VMWare?

Also try a:

esxcfg-scsidevs -a

on SSH/CLI.

Pretty stuck here myself...

0 Kudos
Strieckel
Contributor
Contributor

Hi cykVM

It`s an Vsphere 55u2 2456374 Build.

Hmmm, maybe the bnx2i driver which could course writting to the log file?

~ # esxcfg-scsidevs -a

vmhba0  hpsa              link-n/a  sas.50014380264bfec0                    (0:2                                                                                               :0.0) Hewlett-Packard Company Smart Array P420i

vmhba1  ata_piix          link-n/a  sata.vmhba1                             (0:0                                                                                               :31.2) Intel Corporation Patsburg 4 port SATA IDE Controller

vmhba2  qlnativefc        link-up   fc.5001438024250029:5001438024250028    (0:3                                                                                               6:0.0) QLogic Corp ISP2532-based 8Gb Fibre Channel to PCI Express HBA

vmhba3  qlnativefc        link-n/a  fc.500143802425002b:500143802425002a    (0:3                                                                                               6:0.1) QLogic Corp ISP2532-based 8Gb Fibre Channel to PCI Express HBA

vmhba4  qlnativefc        link-up   fc.5001438024250009:5001438024250008    (0:3                                                                                               9:0.0) QLogic Corp ISP2532-based 8Gb Fibre Channel to PCI Express HBA

vmhba5  qlnativefc        link-n/a  fc.500143802425000b:500143802425000a    (0:3                                                                                               9:0.1) QLogic Corp ISP2532-based 8Gb Fibre Channel to PCI Express HBA

vmhba32 bnx2i             unbound   iscsi.vmhba32                           Broa                                                                                               dcom iSCSI Adapter

vmhba33 bnx2i             unbound   iscsi.vmhba33                           Broa                                                                                               dcom iSCSI Adapter

vmhba34 ata_piix          link-n/a  sata.vmhba34                            (0:0                                                                                               :31.2) Intel Corporation Patsburg 4 port SATA IDE Controller

~ #

There is no directory "ocmanager" to find in my entire system.

I guess as well to update first with latest HPP SPP, otherwise I will open an support case on Vmware.

Many thanks for your Support.

Br

Jörg

0 Kudos
cykVM
Expert
Expert

It's pretty unlikely (but not impossible) that the Broadcom iSCSI adapter is causing this. If possible you may just remove the actual adapter card.

Another thing you may try is to disable logging on the QLogic adapter, see VMware KB: Enabling or disabling verbose logging on QLogic and Emulex Host Bus Adapters

0 Kudos
Strieckel
Contributor
Contributor

Hmmm, when I look into vmkdevmgr.log I could find:

vmkdevmgr: DriverMap: Parsing map: /etc/vmware/driver.map.d/be2iscsi.map

vmkdevmgr: DriverMap: Parsing map: /etc/vmware/default.map.d/elxnet.map

Which still indicates something is running from emulex?!

0 Kudos
cykVM
Expert
Expert

Not necessarily, this is "only" the device manager checking for every supported device and thereafter loading the appropriate driver/kernel module if a device is found.

You may check the loaded kernel modules with the commands listed in above KB:

esxcfg-module --list

or to limit the output:

esxcfg-module --list | grep be2iscsi

esxcfg-module --list | grep elxnet

If anything comes up you may have to dig a little deeper why this driver/module is loaded.

0 Kudos
systimax
Contributor
Contributor

Did you ever resolve this? Same problem with HP hardware.

0 Kudos
Strieckel
Contributor
Contributor

Hi systimax

After deleteing all emulex VIB`s as I have no emulex hardware in use has the log file gone.

The following VIB`s I have deleted...

elxnet                         10.0.575.7-1OEM.550.0.0.1198611        Emulex           VMwareCertified   2013-03-16

ima-be2iscsi 4.6.261.0-1OEM.550.0.0.1198611 Emulex VMwareCertified   2013-03-16

lpfc 10.0.575.8-1OEM.550.0.0.1198611 Emulex VMwareCertified   2013-03-16

scsi-be2iscsi 4.6.261.0-1OEM.550.0.0.1198611 Emulex VMwareCertified   2013-03-16

Br

Strieckel

0 Kudos