Custom Elasticsearch Index Name/Type Name based on Events in Logstash

     The article details on how to use custom Index/Type name when using Logstash for proessing logs and Elasticsearch as Output.

input {   
     	file {

filter {
     	grok {
          		pattern => "%{NUMBER:attribute1} %{GREEDYDATA:attribute2}" 		

     ruby {
          		code => '				
               event["custom_index_name"] = event["attribute1"]				
               event["custom_type_name"]  = event["attribute2"]
          #You can also set an field or modify an existing field to suit your needs

output {
     elasticsearch { 
          		host => localhost
          		index => "index_name_%{custom_index_name}"
          index_type => "type_name_%{custom_type_name}"
          #You can also use index => "index_name_%{attribute1}" if you do not want to modify the field name
          #Other Configuration
     	stdout { 
		          codec => rubydebug 

     This feature can also be used when using Custom Filters to emit events manually.

Related Links :

Web Application for Elasticsearch :
  1. ElasticTab – Elasticsearch to Excel Report (Web Application)
Elasticsearch Plugin:
  1. Elasticsearch Plugin To Generate (Save and E-Mail) Excel Reports
  1. Execute Multiple Search Query in Elasticsearch
  2. Monitor Elasticsearch Servers with Shell Script - E-Mail Notification
  3. Execute Multiple Search Query in Elasticsearch
  4. Enable CORS to Send Cross Domain Request to Elasticsearch using AJAX
  5. Elasticsearch – Use Script Filter/Conditon in Aggregation/Sub-Aggreagtion
  6. Elasticsearch - Date/Time(String)  Add/Subtract Duration - Days,Months,Years,Hours,Minutes,Seconds
  7. Elasticsearch Geo-Shape Slow Indexing Performance - Solved
  8. Chrome Elasticsearch Sense Not Working – Solved
  1. Logstash – Process Log File Once and Exit/Stop Logstash After Reading Log File Once
  2. Measure Logstash Performance using Metrics Filter – Issue/Error in Syntax (Unknown setting ‘message’ for stdout)
  3. Logstash – Process Same Log File (File Input) from Beginning/Start
  4. Create Custom Filter/Plugin to Emit New Events Manually in Logstash
Logstash and Elasticsearch:
  1. Query Elasticsearch Cluster in Filter Section when using Logstash
  2. Custom Elasticsearch Index Name/Type Name based on Events in Logstash
MongoDB and Elasticsearch:
  1. Import Data from Mongo DB to Elasticsearch using Elasticsearch River

Leave a Reply