Issue with searching data and extracting fields

[pault17]

Hello,

I am having a bit of an issue with LI and the interactive analysis.

Basically what appears to have happened is I was importing my logs from my FTP server, i didn't like the format and fields so i changed it and now the data from my FTP logs is unusable. Any new data being ingested cannot be searched or have any fields extracted, but any data I had previously imported from a day or 2 ago appears to be working fine.

Anyone seen anything similar or have a solution? I cant seem to find a delete all button so that i can start with fresh data without having to set everything up again.

Cheers,

Paul

[admin]

If you changed the log format the existing extracted fields will not work as the new logs will have different pre and post context than what the fields expect.

I cant seem to find a delete all button so that i can start with fresh data without having to set everything up again. - What did you want to delete ; the extracted fields or all the data ingested so far? or both?

Thanks,

-Yogita.

[pault17]

That is the weird thing, existing extracted fields continue to work on old data and new existing fields that i create do not work on the new data. I believe its because the new data is not searchable for some reason, for example if i search for any words such as "IP" "Address" "192.168.1.100' no results are shown except from a few days ago even though I have data that should match from 5 minutes ago.

As for the delete all i figured if i could remove all the ingested data perhaps starting fresh would solve the issue, i tried removing all the custom extracted fields i created but that did not help.

[admin]

The new fields should work on the new data and the old fields will continue to work on the earlier data.

There is an unsupported way to delete all data but you don't need to delete the data to get extracted fields to work.

If you send me an email id I could schedule a quick webex and show you how to fix the extracted fields maybe?

Thanks.

[pault17 · Accepted solution]

For anyone else experiencing a similar issue where you are unable to get search results for a log file or 'Extract field' does not appear to work check the encoding of the log file.

Yogitap determined that my log file was encoded as UCS and it seems Log Insight requires ANSI or UTF.