Many might have faced this problem while using Logstash to process same log file from the beginning. The problem is that when you kill/stop a Logstash (file input) process and start it again, the process will continue from where Logstash left previously. For example, consider a logstash instance processing a file containing 1000 lines, then you stop the instance when Logstash has processed 500 lines. If you start the instance again, the process will continue from 500th line even if you have configured start_position option in input.

input {
     file {
          #Other Options
          start_position => "beginning"
          #This will not make Logstash read the same file from start
     }
}

     The start_position option will just start the process from start of the file or end of the file.

     If you want to process same log file all over again from start while each time starting Logstash, use one of the following two methods.

Method 1

input { 
     file {
          sincedb_path => "/dev/null"
          start_position => "beginning"
     }
}

Method 2

     Delete .sincedb_*************** file from your system which is created by Logstash to store details about the files it processed. The .sincedb files are located in the home folder. Note that the file will be hidden and the path is given below. The disadvantage of this method is that, there will be multiple .sincedb files and you will not know which Logstash configuration belongs to which .sincedb file. So you will have to remove all the .sincedb file and thus you loose information about all the logs processed.

For Windows: C:\Users\user_name

For Linux: /home/user_name or /root(if Logstash is running as root user)

Related Links :