Create Custom Filter/Plugin to Emit New Events Manually in Logstash

Create Custom FilterPlugin to Emit New Events Manually in Logstash

Create Custom Filter/Plugin to Emit New Events Manually in Logstash

     This article details about on how to create new events using custom filters in Logstash. There is already a SPLIT filter available in Logstash which can be used to split a single event into multiple events based on a delimiter. But that will not suit us in all the cases. So below is a sample Logstash configuration and a custom filter to create custom filter to emit events manually in Logstash. 

Logstash Configuration File

input {   
     	file {
     	}
}

filter {
     	grok {
          		pattern => "%{NUMBER:attribute1} %{GREEDYDATA:attribute2}" 		
     	}
     customfilter {}
}

output {
     	stdout { 
		          codec => rubydebug 
     	}
}

     In the filter section of the above configuration, we have GROK filter that reads and parses the input event into two fields namely attribute1 and attribute2. Then the control is moved to customfilter.

Ruby Custom Filter

require "logstash/filters/base"
require "logstash/namespace"
class LogStash::Filters::CustomFilter < LogStash::Filters::Base
config_name "customfilter"
milestone 1
public
def register
end

public
def filter(event)

     	#Read Events
     	attribute1_temp=event["attribute1"]
	     attribute2_temp=event["attribute2"]

     	#Your Business Logic

     	#Create New Event
     	custom_event=LogStash::Event.new()
     	custom_event["attribute1_modified"]=attribute1_temp
     	custom_event["attribute2_modified"]=attribute2_temp

     	#Emit New Event
     	yield custom_event

     	#Cancel the Main Event
     	event.cancel

end
end

     In custom filters, the parsed fields can be read using the syntax event[“attribute_name”]. Then we implement our own business logic and create new events based on the modified input fields. Thus we create a new event and assign custom fields as required and emit the event using the command yield event_name. Note that we can either emit the existing event(event in all the case) or emit the custom created event(custom_event in our case). Also make sure to cancel the existing event if you are emitting the custom_event. The main event which I refer is the event created by the Logstash. We can also use yield command in a loop to create any number of events as per the business requirement.

The customfilter is stored in path LOGSTASH_HOME/lib/logstash/filters/customfilter.rb.

Related Links :

Web Application for Elasticsearch :
  1. ElasticTab – Elasticsearch to Excel Report (Web Application)
Elasticsearch Plugin:
  1. Elasticsearch Plugin To Generate (Save and E-Mail) Excel Reports
Elasticsearch:
  1. Execute Multiple Search Query in Elasticsearch
  2. Monitor Elasticsearch Servers with Shell Script - E-Mail Notification
  3. Execute Raw Elasticsearch Query using Transport Client – Java API
  4. Elasticsearch – Apply Nested Filter on Nested (Inner) Aggregation
  5. Execute Multiple Search Query in Elasticsearch
  6. Enable CORS to Send Cross Domain Request to Elasticsearch using AJAX
  7. Elasticsearch Java API – Get Index List
  8. Elasticsearch Java API – Get Alias List
  9. Elasticsearch Java API - Get Type List from given Index
  10. Elasticsearch Java API – Get Field List for a given Index and Type
  11. Elasticsearch Java API – Get Index Type List Mapping
  12. Elasticsearch – Use Script Filter/Conditon in Aggregation/Sub-Aggreagtion
  13. Elasticsearch – Compare/ScriptFilter/Condition on Two Fields using Script Filter – REST Query + Java API
  14. Elasticsearch - Date/Time(String)  Add/Subtract Duration - Days,Months,Years,Hours,Minutes,Seconds
Logstash:
  1. Logstash – Process Log File Once and Exit/Stop Logstash After Reading Log File Once
  2. Measure Logstash Performance using Metrics Filter – Issue/Error in Syntax (Unknown setting ‘message’ for stdout)
  3. Logstash – Process Same Log File (File Input) from Beginning/Start
  4. Create Custom Filter/Plugin to Emit New Events Manually in Logstash
Logstash and Elasticsearch:
  1. Query Elasticsearch Cluster in Filter Section when using Logstash
  2. Custom Elasticsearch Index Name/Type Name based on Events in Logstash
MongoDB and Elasticsearch:
  1. Import Data from Mongo DB to Elasticsearch using Elasticsearch River
 Read More...

 

[ YOU MAY ALSO LIKE ]

Leave a Reply