Custom Elasticsearch Index Name/Type Name based on Events in Logstash

Custom Elasticsearch Index NameType Name based on Events in Logstash

Custom Elasticsearch Index Name/Type Name based on Events in Logstash

     The article details on how to use custom Index/Type name when using Logstash for proessing logs and Elasticsearch as Output.

input {   
     	file {
     	}
}

filter {
     	grok {
          		pattern => "%{NUMBER:attribute1} %{GREEDYDATA:attribute2}" 		
     	}

     ruby {
          		code => '				
               event["custom_index_name"] = event["attribute1"]				
               event["custom_type_name"]  = event["attribute2"]
                  '
          #You can also set an field or modify an existing field to suit your needs
     	}
}

output {
     elasticsearch { 
          		host => localhost
          		index => "index_name_%{custom_index_name}"
          index_type => "type_name_%{custom_type_name}"
          #You can also use index => "index_name_%{attribute1}" if you do not want to modify the field name
          #Other Configuration
     	}
     	stdout { 
		          codec => rubydebug 
     	}
}

     This feature can also be used when using Custom Filters to emit events manually.

Related Links :

Web Application for Elasticsearch :
  1. ElasticTab – Elasticsearch to Excel Report (Web Application)
Elasticsearch Plugin:
  1. Elasticsearch Plugin To Generate (Save and E-Mail) Excel Reports
Elasticsearch:
  1. Execute Multiple Search Query in Elasticsearch
  2. Monitor Elasticsearch Servers with Shell Script - E-Mail Notification
  3. Execute Raw Elasticsearch Query using Transport Client – Java API
  4. Elasticsearch – Apply Nested Filter on Nested (Inner) Aggregation
  5. Execute Multiple Search Query in Elasticsearch
  6. Enable CORS to Send Cross Domain Request to Elasticsearch using AJAX
  7. Elasticsearch Java API – Get Index List
  8. Elasticsearch Java API – Get Alias List
  9. Elasticsearch Java API - Get Type List from given Index
  10. Elasticsearch Java API – Get Field List for a given Index and Type
  11. Elasticsearch Java API – Get Index Type List Mapping
  12. Elasticsearch – Use Script Filter/Conditon in Aggregation/Sub-Aggreagtion
  13. Elasticsearch – Compare/ScriptFilter/Condition on Two Fields using Script Filter – REST Query + Java API
  14. Elasticsearch - Date/Time(String)  Add/Subtract Duration - Days,Months,Years,Hours,Minutes,Seconds
Logstash:
  1. Logstash – Process Log File Once and Exit/Stop Logstash After Reading Log File Once
  2. Measure Logstash Performance using Metrics Filter – Issue/Error in Syntax (Unknown setting ‘message’ for stdout)
  3. Logstash – Process Same Log File (File Input) from Beginning/Start
  4. Create Custom Filter/Plugin to Emit New Events Manually in Logstash
Logstash and Elasticsearch:
  1. Query Elasticsearch Cluster in Filter Section when using Logstash
  2. Custom Elasticsearch Index Name/Type Name based on Events in Logstash
MongoDB and Elasticsearch:
  1. Import Data from Mongo DB to Elasticsearch using Elasticsearch River
 Read More...

[ YOU MAY ALSO LIKE ]

Leave a Reply