Logstash – Process Same Log File (File Input) from Beginning/Start

Process Same Log File from Beginning in Logstash

Logstash – Process Same Log File (File Input) from Beginning/Start

     Many might have faced this problem while using Logstash to process same log file from the beginning. The problem is that when you kill/stop a Logstash (file input) process and start it again, the process will continue from where Logstash left previously. For example, consider a logstash instance processing a file containing 1000 lines, then you stop the instance when Logstash has processed 500 lines. If you start the instance again, the process will continue from 500th line even if you have configured start_position option in input.

input {
     file {
          #Other Options
          start_position => "beginning"
          #This will not make Logstash read the same file from start
     }
}

     The start_position option will just start the process from start of the file or end of the file.

     If you want to process same log file all over again from start while each time starting Logstash, use one of the following two methods.

Method 1

input { 
     file {
          sincedb_path => "/dev/null"
          start_position => "beginning"
     }
}

Method 2

     Delete .sincedb_*************** file from your system which is created by Logstash to store details about the files it processed. The .sincedb files are located in the home folder. Note that the file will be hidden and the path is given below. The disadvantage of this method is that, there will be multiple .sincedb files and you will not know which Logstash configuration belongs to which .sincedb file. So you will have to remove all the .sincedb file and thus you loose information about all the logs processed.

For Windows: C:\Users\user_name

For Linux: /home/user_name or /root(if Logstash is running as root user)

Related Links :

Web Application for Elasticsearch :
  1. ElasticTab – Elasticsearch to Excel Report (Web Application)
Elasticsearch Plugin:
  1. Elasticsearch Plugin To Generate (Save and E-Mail) Excel Reports
Elasticsearch:
  1. Execute Multiple Search Query in Elasticsearch
  2. Monitor Elasticsearch Servers with Shell Script - E-Mail Notification
  3. Execute Raw Elasticsearch Query using Transport Client – Java API
  4. Elasticsearch – Apply Nested Filter on Nested (Inner) Aggregation
  5. Execute Multiple Search Query in Elasticsearch
  6. Enable CORS to Send Cross Domain Request to Elasticsearch using AJAX
  7. Elasticsearch Java API – Get Index List
  8. Elasticsearch Java API – Get Alias List
  9. Elasticsearch Java API - Get Type List from given Index
  10. Elasticsearch Java API – Get Field List for a given Index and Type
  11. Elasticsearch Java API – Get Index Type List Mapping
  12. Elasticsearch – Use Script Filter/Conditon in Aggregation/Sub-Aggreagtion
  13. Elasticsearch – Compare/ScriptFilter/Condition on Two Fields using Script Filter – REST Query + Java API
  14. Elasticsearch - Date/Time(String)  Add/Subtract Duration - Days,Months,Years,Hours,Minutes,Seconds
Logstash:
  1. Logstash – Process Log File Once and Exit/Stop Logstash After Reading Log File Once
  2. Measure Logstash Performance using Metrics Filter – Issue/Error in Syntax (Unknown setting ‘message’ for stdout)
  3. Logstash – Process Same Log File (File Input) from Beginning/Start
  4. Create Custom Filter/Plugin to Emit New Events Manually in Logstash
Logstash and Elasticsearch:
  1. Query Elasticsearch Cluster in Filter Section when using Logstash
  2. Custom Elasticsearch Index Name/Type Name based on Events in Logstash
MongoDB and Elasticsearch:
  1. Import Data from Mongo DB to Elasticsearch using Elasticsearch River
 Read More...

[ YOU MAY ALSO LIKE ]

Leave a Reply