I hope you could help me further or pointing into the right direction.
We had last week an ESXi Server outage which was produced as the /TMP folder was full.
I notice now that there is still writing an mili2d.log, which has filled the /TMP folder.
As far as I could find out this log file will be produced by our network cards.
My question is how could I stop writing the log file and how could I solve the error which are reported in the log file...see below.
Many thanks for your help.
Fri May 22 02:57:28 2015,14995682, ERROR:could not open device node
Fri May 22 02:57:28 2015,14995682, ERROR:ioctl_get_chipinfo: ioctl_get_chipinfo failed
Fri May 22 02:57:28 2015,14995682, ERROR:enumerate_be_nics: ioctl_get_chipinfo failed with status = 1
Fri May 22 02:57:28 2015,14995682, CRITICAL:NO BE CARD FOUND
Fri May 22 02:57:28 2015,15585519, ERROR:could not open device node
Fri May 22 02:57:28 2015,15585519, ERROR:ioctl_get_chipinfo: ioctl_get_chipinfo failed
Fri May 22 02:57:28 2015,15585519, ERROR:enumerate_be_nics: ioctl_get_chipinfo failed with status = 1
Fri May 22 02:57:28 2015,15585519, CRITICAL:NO BE CARD FOUND
Fri May 22 02:58:45 2015,15585762, ERROR:could not open device node
Fri May 22 02:58:45 2015,15585762, ERROR:ioctl_get_chipinfo: ioctl_get_chipinfo failed
Fri May 22 02:58:45 2015,15585762, ERROR:enumerate_be_nics: ioctl_get_chipinfo failed with status = 1
Fri May 22 02:58:45 2015,15585762, CRITICAL:NO BE CARD FOUND
The mili2d.log is from the MILI management daemon for Emulex HBAs. So maybe there's a problem with your Emulex card?
See also this discussion with the mili2d.log filling up: /tmp running out of Space - vSphere 5 - LOG FILE ERROR: log.cpp(610) : No CIMPLE_HOME env var define...
Same could happen with RHEL: /tmp running out of Space due to mili2d.log file - Red Hat Customer Portal
Guess the daemon needs to be configured or even uninstalled for that to stop.
If your host is an HP you may also read: Emulex - HP Enterprise Business Community
Hmmm...the links did not give me an answer.
The redhat link is not allowed to access as I have no service account for redhat.
The strange thing is, it seems tobe produced by an emulex product, but I can not find a emulex hardware installed in my server. HBA / NIC`s are Qlogic...
How could I find out which process driver is writing the mili2d.log, sorry to ask but I am not really familar with Linux commands.
Qlogic and Emulex once were the same company, so it might be that Qlogic uses some driver parts of Emulex (or uses just the same filename for the logfile).
You may try to find the process writing to mili2d.log by: lsof | grep mili
Maybe you should check which drivers are actually installed on that host and thereafter check if any driver updates are available.
What's the server type/model and the type/model of the Qlogic cards?
What you are writing is quit helpfull...many thanks.
lsof | grep mili gives back "hostd-worker"
Hp dl 380p Gen8
Qlogic Corp HP NC523SFP 10GBE 2-port Ethernet Server Adapter
ISP2532-based 87Gb FC HBA
It`s possibly that there are new drivers availeble as HP has released a new service pack, but got not yet the time to update.
Hmm, the hostd-worker process can't be the culprit, must be some child process. I suspect some CIM provider being accidentally installed and that is constantly looking for an Emulex card, not finding it and logging that to the mili2d.log.But that's just a guess.
Was that host installed using the HP customized VMWare image?
Emulex calls it OneCommand Manager for VMWare - this is the managing piece of software for the Emulex cards. Here is a KB about that: VMware KB: Emulex CIM Provider Overview for ESXi 5.0
Are you maybe using Emulex cards in other hosts? Emulex states here Management that you need to install that OneCommand thing on every host if you use the plugin for vCenter. So maybe someone accidentally installed the CIM provider on that host (without an Emulex card)?
Yes, it`s strange.
I have tried to search on the host for emulex but could not find anything.
In our esxi server farm are no emulex cards available, but I am not aware about server out of VMware...
Yep, we have installed the HP custom image.
Try this command on SSH/CLI:
esxcli system settings advanced list | grep CIM
this should list you the installed CIM providers. Anything with "Emulex" coming up?
For disabling a single or multiple providers you may follow this KB: VMware KB: Disabling an ESXi CIM provider when it fails or is unstable
Nothing with Emulex to find?!
Do you have more ideas what I can check?
~ # esxcli system settings advanced list | grep CIM
Description: Enable or disable the CIM service
Description: Set the log level of the CIM Service
Description: Set the watchdog polling interval for the CIM Service
Description: Enable or disable the CIM vmw_hhrcwrapper provider
Description: Enable or disable the CIM vmw_pci provider
Description: Enable or disable the CIM vmw_iodmProvider provider
Description: Enable or disable the CIM vmw_hdr provider
Description: Enable or disable the CIM vmw_kmodule provider
Description: Enable or disable the CIM vmw_vi provider
Description: Enable or disable the CIM vmw_sfcbrInterop provider
Description: Enable or disable the CIM vmw_swmgt provider
Description: Enable or disable the CIM vmw_hp-smx-provider provider
Which version/build of VMWare are you running?
I would first do the firmware/BIOS updates from latest HP SPP, I guess.
Could you find any traces of a folder named 'ocmanager' in your filesystem?
From the log in your initial posting it looks for a "BE" HBA which should use either the be2net, be2iscsi or be3 driver.
Is your server still under warranty and you are able to open a support case with HP and/or VMWare?
Also try a:
Pretty stuck here myself...
It`s an Vsphere 55u2 2456374 Build.
Hmmm, maybe the bnx2i driver which could course writting to the log file?
~ # esxcfg-scsidevs -a
vmhba0 hpsa link-n/a sas.50014380264bfec0 (0:2 :0.0) Hewlett-Packard Company Smart Array P420i
vmhba1 ata_piix link-n/a sata.vmhba1 (0:0 :31.2) Intel Corporation Patsburg 4 port SATA IDE Controller
vmhba2 qlnativefc link-up fc.5001438024250029:5001438024250028 (0:3 6:0.0) QLogic Corp ISP2532-based 8Gb Fibre Channel to PCI Express HBA
vmhba3 qlnativefc link-n/a fc.500143802425002b:500143802425002a (0:3 6:0.1) QLogic Corp ISP2532-based 8Gb Fibre Channel to PCI Express HBA
vmhba4 qlnativefc link-up fc.5001438024250009:5001438024250008 (0:3 9:0.0) QLogic Corp ISP2532-based 8Gb Fibre Channel to PCI Express HBA
vmhba5 qlnativefc link-n/a fc.500143802425000b:500143802425000a (0:3 9:0.1) QLogic Corp ISP2532-based 8Gb Fibre Channel to PCI Express HBA
vmhba32 bnx2i unbound iscsi.vmhba32 Broa dcom iSCSI Adapter
vmhba33 bnx2i unbound iscsi.vmhba33 Broa dcom iSCSI Adapter
vmhba34 ata_piix link-n/a sata.vmhba34 (0:0 :31.2) Intel Corporation Patsburg 4 port SATA IDE Controller
There is no directory "ocmanager" to find in my entire system.
I guess as well to update first with latest HPP SPP, otherwise I will open an support case on Vmware.
Many thanks for your Support.
It's pretty unlikely (but not impossible) that the Broadcom iSCSI adapter is causing this. If possible you may just remove the actual adapter card.
Another thing you may try is to disable logging on the QLogic adapter, see VMware KB: Enabling or disabling verbose logging on QLogic and Emulex Host Bus Adapters
Hmmm, when I look into vmkdevmgr.log I could find:
vmkdevmgr: DriverMap: Parsing map: /etc/vmware/driver.map.d/be2iscsi.map
vmkdevmgr: DriverMap: Parsing map: /etc/vmware/default.map.d/elxnet.map
Which still indicates something is running from emulex?!
Not necessarily, this is "only" the device manager checking for every supported device and thereafter loading the appropriate driver/kernel module if a device is found.
You may check the loaded kernel modules with the commands listed in above KB:
or to limit the output:
esxcfg-module --list | grep be2iscsi
esxcfg-module --list | grep elxnet
If anything comes up you may have to dig a little deeper why this driver/module is loaded.
After deleteing all emulex VIB`s as I have no emulex hardware in use has the log file gone.
The following VIB`s I have deleted...
elxnet 10.0.575.7-1OEM.5126.96.36.1998611 Emulex VMwareCertified 2013-03-16
ima-be2iscsi 4.6.261.0-1OEM.5188.8.131.528611 Emulex VMwareCertified 2013-03-16
lpfc 10.0.575.8-1OEM.5184.108.40.2068611 Emulex VMwareCertified 2013-03-16
scsi-be2iscsi 4.6.261.0-1OEM.5220.127.116.118611 Emulex VMwareCertified 2013-03-16