This article details about on how to create new events using custom filters in Logstash. There is already a SPLIT filter available in Logstash which can be used to split a single event into multiple events based on a delimiter. But that will not suit us in all the cases. So below is a sample Logstash configuration and a custom filter to create custom filter to emit events manually in Logstash.
Logstash Configuration File
input { file { } } filter { grok { pattern => "%{NUMBER:attribute1} %{GREEDYDATA:attribute2}" } customfilter {} } output { stdout { codec => rubydebug } }
In the filter section of the above configuration, we have GROK filter that reads and parses the input event into two fields namely attribute1 and attribute2. Then the control is moved to customfilter.
Ruby Custom Filter
require "logstash/filters/base" require "logstash/namespace" class LogStash::Filters::CustomFilter < LogStash::Filters::Base config_name "customfilter" milestone 1 public def register end public def filter(event) #Read Events attribute1_temp=event["attribute1"] attribute2_temp=event["attribute2"] #Your Business Logic #Create New Event custom_event=LogStash::Event.new() custom_event["attribute1_modified"]=attribute1_temp custom_event["attribute2_modified"]=attribute2_temp #Emit New Event yield custom_event #Cancel the Main Event event.cancel end end
In custom filters, the parsed fields can be read using the syntax event[“attribute_name”]. Then we implement our own business logic and create new events based on the modified input fields. Thus we create a new event and assign custom fields as required and emit the event using the command yield event_name. Note that we can either emit the existing event(event in all the case) or emit the custom created event(custom_event in our case). Also make sure to cancel the existing event if you are emitting the custom_event. The main event which I refer is the event created by the Logstash. We can also use yield command in a loop to create any number of events as per the business requirement.
The customfilter is stored in path LOGSTASH_HOME/lib/logstash/filters/customfilter.rb.
Related Links :