Hi There,
Hopefully a simple question.
I have a server that has a scheduled task each day to download compressed log files (.gz) from a remote host and it is hoped we can ingest them into Log Insight. The local file server location is "synchronised" to the remote host - i.e. there are several months of log files at the remote host, and only new files are downloaded and added to the local file server.
My reading says that the Log Insight Agent cannot handle compressed gz files, and that Log Insight importer is a better option.
My question is if I told a task to re-import the entire log file directory each day, is it going to re-ingest all the data over and over again, or is there some sort of deduplication that will ensure duplicate log entries generated are ignored?
If there is no de-duplication, then I guess I will need to write some additional smarts into my scripts to only ingest the new files
Thanks
