I have a scenario in Logstash where I want to process a log file in a path once in a day. In my case, a new log file will be added to a path which needs to be processed once and it takes roughly around 10 minutes to process the entire file. I do not want the Logstash process to be running round the clock to process just a small and single log file. Usually we can configure Logstash to read logs from a location using input file filter. This makes the Logstash process to run round the clock. There is a work around to make Logstash, to process the given file and exit from the process. i.e. finish the process.
./logstash agent -e 'input { stdin {} } output { ANY_OUTPUT_FILTER }' < /file_path/test.log
OR
./logstash agent -f /tmp/local/logstash.conf < /file_path/test.log
Logstash -e option let us give the Logstash configuration as inline text and -f let us give the configuration as file.
Related Links :
Hello Raghavendar,
Your method also does not work in my case. Can you provide some details
1. The logstash configuration file should not contain any input filter now (since we are providing the log file path on command line)?
2. Also, once i execute the above specified command the process, the process runs and exits in 15-20 seconds. Why?
Let me know in case you need any other information.
Consider file input filter which takes in file/folder as input. Logstash process will continuously tail the file for new content. If new content exist, Logstash will process it. This is quite normal. Consider a case where you have a file which has some 1000 lines and you are sure that no new content will be added to that file. Using the file input filter in this case will make Logstash process the file and the that process will just run continuously. The process wont get killed/stopped after it has processed the file. The normal use case can be used when you have content added to a file endlessly. You can use my approach if you have some file to process only once. For me, I had a requirement to process a folder. Each day a log file will be added to that folder. I need to process that log file only once since that file will not have content added. You got it now? Also there is no straight forward way to make Logstash to process a log file only once and exit using file input filter.
What is the problem which you are facing?
This post really helps me a lot, thanks!
This definitely worked for me after some trial and error. I’m using a custom config file ‘test.conf’ and had to define an input filter in the config file as follows:
input {
stdin {}
}
The command to read in my file ‘test.log’ via logstash then looked like: /opt/logstash/bin/logstash -f test.conf < test.log
After parsing the contents of the file, the logstash process shutdown successfully.
Before I started defining the input filter in the config file like above, the logstash process would startup and quickly shutdown without parsing the contents of the file, like ishita described.
The key here is not to put the word “agent” as a command line parameter to logstash if you don’t want it to run as a continuous service. In summary, logstash does support batch mode natively, all you have to do is use the stdin input plugin, pipe in the file, and don’t run it in agent mode.
This post is awesome, it worked for me.
All i have to do is use the below in my config file
input {
stdin {}
}
Then use logstash/bin/logstash -f test.conf < test.log
Where test.log is the file path that needs to be processed by logstash
Thank you so much for the article
I have a case where I am using log stash for reindexing huge data(big index to small monthly indexes). Input and output both have same elastic url. How to use workaround to exit logstash after reindexing finishes?