Logstash – Process Same Log File (File Input) from Beginning/Start

     Many might have faced this problem while using Logstash to process same log file from the beginning. The problem is that when you kill/stop a Logstash (file input) process and start it again, the process will continue from where Logstash left previously. For example, consider a logstash instance processing a file containing 1000 lines, then you stop the instance when Logstash has processed 500 lines. If you start the instance again, the process will continue from 500th line even if you have configured start_position option in input.

input {
     file {
          #Other Options
          start_position => "beginning"
          #This will not make Logstash read the same file from start

     The start_position option will just start the process from start of the file or end of the file.

     If you want to process same log file all over again from start while each time starting Logstash, use one of the following two methods.

Method 1

input { 
     file {
          sincedb_path => "/dev/null"
          start_position => "beginning"

Method 2

     Delete .sincedb_*************** file from your system which is created by Logstash to store details about the files it processed. The .sincedb files are located in the home folder. Note that the file will be hidden and the path is given below. The disadvantage of this method is that, there will be multiple .sincedb files and you will not know which Logstash configuration belongs to which .sincedb file. So you will have to remove all the .sincedb file and thus you loose information about all the logs processed.

For Windows: C:\Users\user_name

For Linux: /home/user_name or /root(if Logstash is running as root user)

Related Links :

Web Application for Elasticsearch :
  1. ElasticTab – Elasticsearch to Excel Report (Web Application)
Elasticsearch Plugin:
  1. Elasticsearch Plugin To Generate (Save and E-Mail) Excel Reports
  1. Execute Multiple Search Query in Elasticsearch
  2. Monitor Elasticsearch Servers with Shell Script - E-Mail Notification
  3. Execute Multiple Search Query in Elasticsearch
  4. Enable CORS to Send Cross Domain Request to Elasticsearch using AJAX
  5. Elasticsearch – Use Script Filter/Conditon in Aggregation/Sub-Aggreagtion
  6. Elasticsearch - Date/Time(String)  Add/Subtract Duration - Days,Months,Years,Hours,Minutes,Seconds
  7. Elasticsearch Geo-Shape Slow Indexing Performance - Solved
  8. Chrome Elasticsearch Sense Not Working – Solved
  1. Logstash – Process Log File Once and Exit/Stop Logstash After Reading Log File Once
  2. Measure Logstash Performance using Metrics Filter – Issue/Error in Syntax (Unknown setting ‘message’ for stdout)
  3. Logstash – Process Same Log File (File Input) from Beginning/Start
  4. Create Custom Filter/Plugin to Emit New Events Manually in Logstash
Logstash and Elasticsearch:
  1. Query Elasticsearch Cluster in Filter Section when using Logstash
  2. Custom Elasticsearch Index Name/Type Name based on Events in Logstash
MongoDB and Elasticsearch:
  1. Import Data from Mongo DB to Elasticsearch using Elasticsearch River

One thought on “Logstash – Process Same Log File (File Input) from Beginning/Start

Leave a Reply to Karthikeya Cancel reply