Query Elasticsearch Cluster in Filter Section when using Logstash

Query Elasticsearch Cluster in Filter Section when using Logstash

Query Elasticsearch Cluster in Filter Section when using Logstash

     This article details about on how to query Elasticsearch from Logstash when processing logs or when using any other inputs.

input {   
     	file {
     	}
     #Any other inputs
}

filter {
     	grok {
          		pattern => "%{NUMBER:attribute1} %{GREEDYDATA:attribute2}" 		
     	}

     elasticsearch {
          		hosts => ["HOSTNAME:9200/index_name/type_name"]
          		query => 'field_name.sub_field:%{attribute1}'
          		fields => ["attribute3_elasticsearch", "attribute3_new","attribute4_elasticsearch", "attribute4_new"] 
          #We can any number of fields
          		sort => "attribute3_new:desc"
     }
}

output {
     	stdout { 
		          codec => rubydebug 
     	}
}

     The elasticsearch filters can even take just the hostname and port. If you are using just hostname and port, make sure that only one index and one type exist in elasticsearch cluster of that host. It is better to give the index and type name along with the hostname and port. We can also give multiple hosts to query multiple clusters.

Related Links :

Web Application for Elasticsearch :
  1. ElasticTab – Elasticsearch to Excel Report (Web Application)
Elasticsearch Plugin:
  1. Elasticsearch Plugin To Generate (Save and E-Mail) Excel Reports
Elasticsearch:
  1. Execute Multiple Search Query in Elasticsearch
  2. Monitor Elasticsearch Servers with Shell Script - E-Mail Notification
  3. Execute Raw Elasticsearch Query using Transport Client – Java API
  4. Elasticsearch – Apply Nested Filter on Nested (Inner) Aggregation
  5. Execute Multiple Search Query in Elasticsearch
  6. Enable CORS to Send Cross Domain Request to Elasticsearch using AJAX
  7. Elasticsearch Java API – Get Index List
  8. Elasticsearch Java API – Get Alias List
  9. Elasticsearch Java API - Get Type List from given Index
  10. Elasticsearch Java API – Get Field List for a given Index and Type
  11. Elasticsearch Java API – Get Index Type List Mapping
  12. Elasticsearch – Use Script Filter/Conditon in Aggregation/Sub-Aggreagtion
  13. Elasticsearch – Compare/ScriptFilter/Condition on Two Fields using Script Filter – REST Query + Java API
  14. Elasticsearch - Date/Time(String)  Add/Subtract Duration - Days,Months,Years,Hours,Minutes,Seconds
Logstash:
  1. Logstash – Process Log File Once and Exit/Stop Logstash After Reading Log File Once
  2. Measure Logstash Performance using Metrics Filter – Issue/Error in Syntax (Unknown setting ‘message’ for stdout)
  3. Logstash – Process Same Log File (File Input) from Beginning/Start
  4. Create Custom Filter/Plugin to Emit New Events Manually in Logstash
Logstash and Elasticsearch:
  1. Query Elasticsearch Cluster in Filter Section when using Logstash
  2. Custom Elasticsearch Index Name/Type Name based on Events in Logstash
MongoDB and Elasticsearch:
  1. Import Data from Mongo DB to Elasticsearch using Elasticsearch River
 Read More...

[ YOU MAY ALSO LIKE ]

Leave a Reply