Category Archives: Programming

Measure Logstash Performance using Metrics Filter – Issue/Error in Syntax (Unknown setting ‘message’ for stdout)

Measure Logstash Performance using Metrics Filter - Issue/Error in Syntax (Unknown setting 'message' for stdout)

Measure Logstash Performance using Metrics Filter – Issue/Error in Syntax (Unknown setting ‘message’ for stdout)

Measure Logstash Performance using Metrics Filter - Issue/Error in Syntax (Unknown setting 'message' for stdout)

       If you want to measure the performance of Logstash, Metrics plugin is way to go. The syntax given in the documentation(Documentation Link) does not work which is given below. if “metric” in [tags] { stdout { message => “rate: %{events.rate_1m}” } }         The correct syntax with working Logstash… Read More »

Logstash – Process Same Log File (File Input) from Beginning/Start

Process Same Log File from Beginning in Logstash

Logstash – Process Same Log File (File Input) from Beginning/Start

Process Same Log File from Beginning in Logstash

     Many might have faced this problem while using Logstash to process same log file from the beginning. The problem is that when you kill/stop a Logstash (file input) process and start it again, the process will continue from where Logstash left previously. For example, consider a logstash instance processing a file containing 1000… Read More »

Query Elasticsearch Cluster in Filter Section when using Logstash

Query Elasticsearch Cluster in Filter Section when using Logstash

Query Elasticsearch Cluster in Filter Section when using Logstash

Query Elasticsearch Cluster in Filter Section when using Logstash

     This article details about on how to query Elasticsearch from Logstash when processing logs or when using any other inputs. input { file { } #Any other inputs } filter { grok { pattern => “%{NUMBER:attribute1} %{GREEDYDATA:attribute2}” } elasticsearch { hosts => [“HOSTNAME:9200/index_name/type_name”] query => ‘field_name.sub_field:%{attribute1}’ fields => [“attribute3_elasticsearch”, “attribute3_new”,”attribute4_elasticsearch”, “attribute4_new”] #We can… Read More »

Custom Elasticsearch Index Name/Type Name based on Events in Logstash

Custom Elasticsearch Index NameType Name based on Events in Logstash

Custom Elasticsearch Index Name/Type Name based on Events in Logstash

Custom Elasticsearch Index NameType Name based on Events in Logstash

     The article details on how to use custom Index/Type name when using Logstash for proessing logs and Elasticsearch as Output. input { file { } } filter { grok { pattern => “%{NUMBER:attribute1} %{GREEDYDATA:attribute2}” } ruby { code => ‘ event[“custom_index_name”] = event[“attribute1”] event[“custom_type_name”] = event[“attribute2”] ‘ #You can also set an field… Read More »

Create Custom Filter/Plugin to Emit New Events Manually in Logstash

Create Custom FilterPlugin to Emit New Events Manually in Logstash

Create Custom Filter/Plugin to Emit New Events Manually in Logstash

Create Custom FilterPlugin to Emit New Events Manually in Logstash

     This article details about on how to create new events using custom filters in Logstash. There is already a SPLIT filter available in Logstash which can be used to split a single event into multiple events based on a delimiter. But that will not suit us in all the cases. So below is… Read More »