<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:taxo="http://purl.org/rss/1.0/modules/taxonomy/" version="2.0">
  <channel>
    <title>topic Log Insight Importer behaviour when &amp;quot;re-importing&amp;quot; log files. in VMware Aria Operations for Logs Discussions</title>
    <link>https://communities.vmware.com/t5/VMware-Aria-Operations-for-Logs/Log-Insight-Importer-behaviour-when-quot-re-importing-quot-log/m-p/2213018#M1811</link>
    <description>&lt;HTML&gt;&lt;HEAD&gt;&lt;/HEAD&gt;&lt;BODY&gt;&lt;P&gt;Hi There,&lt;/P&gt;&lt;P&gt;&lt;/P&gt;&lt;P&gt;Hopefully a simple question.&lt;/P&gt;&lt;P&gt;&lt;/P&gt;&lt;P&gt;I have a server that has a scheduled task each day to download compressed log files (.gz) from a remote host and it is hoped we can ingest them into Log Insight. The local file server location is "synchronised" to the remote host - i.e. there are several months of log files at the remote host, and only new files are downloaded and added to the local file server.&lt;/P&gt;&lt;P&gt;&lt;/P&gt;&lt;P&gt;My reading says that the Log Insight Agent cannot handle compressed gz files, and that Log Insight importer is a better option.&lt;/P&gt;&lt;P&gt;&lt;/P&gt;&lt;P&gt;My question is if I told a task to re-import the entire log file directory each day, is it going to re-ingest all the data over and over again, or is there some sort of deduplication that will ensure duplicate log entries generated are ignored?&lt;/P&gt;&lt;P&gt;&lt;/P&gt;&lt;P&gt;If there is no de-duplication, then&amp;nbsp; I guess I will need to write some additional smarts into my scripts to only ingest the new files&lt;/P&gt;&lt;P&gt;&lt;/P&gt;&lt;P&gt;Thanks&lt;/P&gt;&lt;/BODY&gt;&lt;/HTML&gt;</description>
    <pubDate>Thu, 15 Aug 2019 03:54:33 GMT</pubDate>
    <dc:creator>Chage</dc:creator>
    <dc:date>2019-08-15T03:54:33Z</dc:date>
    <item>
      <title>Log Insight Importer behaviour when "re-importing" log files.</title>
      <link>https://communities.vmware.com/t5/VMware-Aria-Operations-for-Logs/Log-Insight-Importer-behaviour-when-quot-re-importing-quot-log/m-p/2213018#M1811</link>
      <description>&lt;HTML&gt;&lt;HEAD&gt;&lt;/HEAD&gt;&lt;BODY&gt;&lt;P&gt;Hi There,&lt;/P&gt;&lt;P&gt;&lt;/P&gt;&lt;P&gt;Hopefully a simple question.&lt;/P&gt;&lt;P&gt;&lt;/P&gt;&lt;P&gt;I have a server that has a scheduled task each day to download compressed log files (.gz) from a remote host and it is hoped we can ingest them into Log Insight. The local file server location is "synchronised" to the remote host - i.e. there are several months of log files at the remote host, and only new files are downloaded and added to the local file server.&lt;/P&gt;&lt;P&gt;&lt;/P&gt;&lt;P&gt;My reading says that the Log Insight Agent cannot handle compressed gz files, and that Log Insight importer is a better option.&lt;/P&gt;&lt;P&gt;&lt;/P&gt;&lt;P&gt;My question is if I told a task to re-import the entire log file directory each day, is it going to re-ingest all the data over and over again, or is there some sort of deduplication that will ensure duplicate log entries generated are ignored?&lt;/P&gt;&lt;P&gt;&lt;/P&gt;&lt;P&gt;If there is no de-duplication, then&amp;nbsp; I guess I will need to write some additional smarts into my scripts to only ingest the new files&lt;/P&gt;&lt;P&gt;&lt;/P&gt;&lt;P&gt;Thanks&lt;/P&gt;&lt;/BODY&gt;&lt;/HTML&gt;</description>
      <pubDate>Thu, 15 Aug 2019 03:54:33 GMT</pubDate>
      <guid>https://communities.vmware.com/t5/VMware-Aria-Operations-for-Logs/Log-Insight-Importer-behaviour-when-quot-re-importing-quot-log/m-p/2213018#M1811</guid>
      <dc:creator>Chage</dc:creator>
      <dc:date>2019-08-15T03:54:33Z</dc:date>
    </item>
  </channel>
</rss>

