Author Archives: Raghavendar T S

Elasticsearch Plugin To Generate (Save and E-Mail) Excel Reports

ELASTICSEARCH PLUGIN TO GENERATE (SAVE AND E-MAIL) EXCEL REPORTS

Elasticsearch Plugin To Generate (Save and E-Mail) Excel Reports


ElasticTab – Elasticsearch to Excel Report Web Application

  • Simple Web UI
  • Download Excel Reports
  • Schedule/Edit/Delete Reports
  • E-Mail Reports

Go to https://github.com/raghavendar-ts/ElasticTab-Elasticsearch-to-Excel-Report


Elasticsearch Plugin to Generate Excel Reports

           There are many use cases where in one needs to import data from an external system or database to Elasticsearch and many plugins are build to solve the use case. But there are cases where the users need to view the data in Elasticsearch in a structured or tabular form. There are quite a few plugins that help us export the data to CSV and Excel format, but with no manipulations. So here is a plugin which can be used to export data in Elasticsearch to Excel format and we can perform standard operations on the fields of a document.

Plugin Page: Elasticsearch Report Plugin (https://github.com/raghavendar-ts/Elasticsearch-Report-Plugin)

List of Supported Operation:

  1. getValue – Get the Elasticsearch field value
  2. getDValue – Get custom value based on the Elasticsearch field value
  3. Length – Get the length of Elasticsearch field value
  4. Format Number Length – Format integer to given length
  5. Sub String – Get substring of Elasticsearch field value
  6. Character at index – Get the character at given index of Elasticsearch field value
  7. Calculate – Perform arithmetic operation on Elasticsarch field values
  8. Range – Get custom value based on range condition of Elasticsearch field value
  9. Array indexOf(int value) – Get index of given integer value from an Elasticsearch array
  10. Array indexOf(String value) – Get index of given string value from an Elasticsearch array
  11. Array valueAt(index) – Get the value of given index from an Elasticsearch array

           A simple JSON input to the plugin can generate (save to server or E-Mail) the required Excel file. The plugin page details information on the following.

  • Installing and Removing the Plugin
  • List of Operation Supported
  • Operation Syntax
  • Syntax Description
  • Detailed Example with Sample Data
  • Sample Input Data
  • Sample HTTP Request to the Plugin
  • Sample Output

Related Links: 

Web Application for Elasticsearch :
  1. ElasticTab – Elasticsearch to Excel Report (Web Application)
Elasticsearch Plugin:
  1. Elasticsearch Plugin To Generate (Save and E-Mail) Excel Reports
Elasticsearch:
  1. Execute Multiple Search Query in Elasticsearch
  2. Monitor Elasticsearch Servers with Shell Script - E-Mail Notification
  3. Execute Raw Elasticsearch Query using Transport Client – Java API
  4. Elasticsearch – Apply Nested Filter on Nested (Inner) Aggregation
  5. Execute Multiple Search Query in Elasticsearch
  6. Enable CORS to Send Cross Domain Request to Elasticsearch using AJAX
  7. Elasticsearch Java API – Get Index List
  8. Elasticsearch Java API – Get Alias List
  9. Elasticsearch Java API - Get Type List from given Index
  10. Elasticsearch Java API – Get Field List for a given Index and Type
  11. Elasticsearch Java API – Get Index Type List Mapping
  12. Elasticsearch – Use Script Filter/Conditon in Aggregation/Sub-Aggreagtion
  13. Elasticsearch – Compare/ScriptFilter/Condition on Two Fields using Script Filter – REST Query + Java API
  14. Elasticsearch - Date/Time(String)  Add/Subtract Duration - Days,Months,Years,Hours,Minutes,Seconds
  15. Elasticsearch Geo-Shape Slow Indexing Performance - Solved
  16. Chrome Elasticsearch Sense Not Working – Solved
Logstash:
  1. Logstash – Process Log File Once and Exit/Stop Logstash After Reading Log File Once
  2. Measure Logstash Performance using Metrics Filter – Issue/Error in Syntax (Unknown setting ‘message’ for stdout)
  3. Logstash – Process Same Log File (File Input) from Beginning/Start
  4. Create Custom Filter/Plugin to Emit New Events Manually in Logstash
Logstash and Elasticsearch:
  1. Query Elasticsearch Cluster in Filter Section when using Logstash
  2. Custom Elasticsearch Index Name/Type Name based on Events in Logstash
MongoDB and Elasticsearch:
  1. Import Data from Mongo DB to Elasticsearch using Elasticsearch River
 Read More...

Execute Multiple Search Query in Elasticsearch

Execute Multiple Search Query in Elasticsearch

Execute Multiple Search Query in Elasticsearch

           The following query can be used to execute multiple search queries on the required index and type. Elasticsearch will process the request and the response is sent as array field named responses which contains the response for each query. 

Note :

  • { “index”: “INDEX_NAME_1”, “type”: “TYPE_NAME_1” } in Example 1 should not span multiple lines
  • { “query”: { “match_all”: {}}} in the Example 1 is the query and make sure this line also doesn’t span multiple lines
  • The first line (POST /_msearch) represents type of query
  • The following two lines will represent a search query.
  • Add any number of query based on your requirement

Example 1:

POST /_msearch
{ "index": "INDEX_NAME_1", "type": "TYPE_NAME_1" }
{ "query": { "match_all": {}}}
{ "index": "INDEX_NAME_2", "type": "TYPE_NAME_2" }
{ "query": { "match_all": {}}}

Sample Response for Example 1 :

{
   "responses": [
      {
         "took": 4700,
         "timed_out": false,
         "_shards": {
            "total": 1060,
            "successful": 1060,
            "failed": 0
         },
         "hits": {
            "total": 24855713,
            "max_score": 1,
            "hits": [......]
         }
      },
      {
         "took": 4629,
         "timed_out": false,
         "_shards": {
            "total": 1060,
            "successful": 1060,
            "failed": 0
         },
         "hits": {
            "total": 520958748,
            "max_score": 1,
            "hits": [......]
         }
      }
   ]
}

Example 2:

POST /_msearch
{ "index": "INDEX_NAME", "type": "TYPE_NAME" }
{ "query": { "match_all": {}}}

Sample Response for Example 2 :

{
   "responses": [
      {
         "took": 4700,
         "timed_out": false,
         "_shards": {
            "total": 1060,
            "successful": 1060,
            "failed": 0
         },
         "hits": {
            "total": 24855713,
            "max_score": 1,
            "hits": [......]
         }
      }
   ]
}

Related Links: 

Web Application for Elasticsearch :
  1. ElasticTab – Elasticsearch to Excel Report (Web Application)
Elasticsearch Plugin:
  1. Elasticsearch Plugin To Generate (Save and E-Mail) Excel Reports
Elasticsearch:
  1. Execute Multiple Search Query in Elasticsearch
  2. Monitor Elasticsearch Servers with Shell Script - E-Mail Notification
  3. Execute Raw Elasticsearch Query using Transport Client – Java API
  4. Elasticsearch – Apply Nested Filter on Nested (Inner) Aggregation
  5. Execute Multiple Search Query in Elasticsearch
  6. Enable CORS to Send Cross Domain Request to Elasticsearch using AJAX
  7. Elasticsearch Java API – Get Index List
  8. Elasticsearch Java API – Get Alias List
  9. Elasticsearch Java API - Get Type List from given Index
  10. Elasticsearch Java API – Get Field List for a given Index and Type
  11. Elasticsearch Java API – Get Index Type List Mapping
  12. Elasticsearch – Use Script Filter/Conditon in Aggregation/Sub-Aggreagtion
  13. Elasticsearch – Compare/ScriptFilter/Condition on Two Fields using Script Filter – REST Query + Java API
  14. Elasticsearch - Date/Time(String)  Add/Subtract Duration - Days,Months,Years,Hours,Minutes,Seconds
  15. Elasticsearch Geo-Shape Slow Indexing Performance - Solved
  16. Chrome Elasticsearch Sense Not Working – Solved
Logstash:
  1. Logstash – Process Log File Once and Exit/Stop Logstash After Reading Log File Once
  2. Measure Logstash Performance using Metrics Filter – Issue/Error in Syntax (Unknown setting ‘message’ for stdout)
  3. Logstash – Process Same Log File (File Input) from Beginning/Start
  4. Create Custom Filter/Plugin to Emit New Events Manually in Logstash
Logstash and Elasticsearch:
  1. Query Elasticsearch Cluster in Filter Section when using Logstash
  2. Custom Elasticsearch Index Name/Type Name based on Events in Logstash
MongoDB and Elasticsearch:
  1. Import Data from Mongo DB to Elasticsearch using Elasticsearch River
 Read More...

Elasticsearch – Apply Nested Filter on Nested (Inner) Aggregation

Elasticsearch - Apply Nested Filter on Nested (Inner) Aggregation

Elasticsearch – Apply Nested Filter on Nested (Inner) Aggregation

           The following query can be used to apply filter on nested (inner) aggregation. In our example aggregationName2 is the nested aggregation. The filter (bold content) in the input JSON will be applied only for the aggregation aggregationName2. aggregationName2 takes in the output of aggregationName1, then applies the filter which is then processed by aggregationName2 for the final results.

 

GET /index/type/_search
{
    "aggs": {
        "aggregationName1": {
            "terms": {
                "field": "fieldName1"
             },
             "aggs": {
                 "filtered": {
                     "filter": {
                         "bool": {
                             "must": [{
                                 "term": {
                                     "filterFieldName": filterFieldValue
                                  }
                             }]
                         }
                     },
                     "aggs": {
                         "aggregationName2": {
                             "terms": {
                                 "field": "fieldName2"
                             }
                         }
                     }
                 }
             }
         }
    },   
    "size": 0
}

Related Links: 

Web Application for Elasticsearch :
  1. ElasticTab – Elasticsearch to Excel Report (Web Application)
Elasticsearch Plugin:
  1. Elasticsearch Plugin To Generate (Save and E-Mail) Excel Reports
Elasticsearch:
  1. Execute Multiple Search Query in Elasticsearch
  2. Monitor Elasticsearch Servers with Shell Script - E-Mail Notification
  3. Execute Raw Elasticsearch Query using Transport Client – Java API
  4. Elasticsearch – Apply Nested Filter on Nested (Inner) Aggregation
  5. Execute Multiple Search Query in Elasticsearch
  6. Enable CORS to Send Cross Domain Request to Elasticsearch using AJAX
  7. Elasticsearch Java API – Get Index List
  8. Elasticsearch Java API – Get Alias List
  9. Elasticsearch Java API - Get Type List from given Index
  10. Elasticsearch Java API – Get Field List for a given Index and Type
  11. Elasticsearch Java API – Get Index Type List Mapping
  12. Elasticsearch – Use Script Filter/Conditon in Aggregation/Sub-Aggreagtion
  13. Elasticsearch – Compare/ScriptFilter/Condition on Two Fields using Script Filter – REST Query + Java API
  14. Elasticsearch - Date/Time(String)  Add/Subtract Duration - Days,Months,Years,Hours,Minutes,Seconds
  15. Elasticsearch Geo-Shape Slow Indexing Performance - Solved
  16. Chrome Elasticsearch Sense Not Working – Solved
Logstash:
  1. Logstash – Process Log File Once and Exit/Stop Logstash After Reading Log File Once
  2. Measure Logstash Performance using Metrics Filter – Issue/Error in Syntax (Unknown setting ‘message’ for stdout)
  3. Logstash – Process Same Log File (File Input) from Beginning/Start
  4. Create Custom Filter/Plugin to Emit New Events Manually in Logstash
Logstash and Elasticsearch:
  1. Query Elasticsearch Cluster in Filter Section when using Logstash
  2. Custom Elasticsearch Index Name/Type Name based on Events in Logstash
MongoDB and Elasticsearch:
  1. Import Data from Mongo DB to Elasticsearch using Elasticsearch River
 Read More...

Append/Write CRON Output to Log File in Linux (Unix)

Append or Write CRON Output to Log File

Append/Write CRON Output to Log File in Linux (Unix)

Append CRON Output to Log :

           The following syntax can be used to append the output of the shell scripts to the given file.

Syntax :

CRON_EXPRESSION COMMAND >> /OUTPUT_LOG_FILE_PATH/LOF_FILE 2>&1

Example :

           The following example will append the output of the shell script test.sh to cron_output.log in the path /usr/local.

45 * * * * sh /usr/local/test.sh >> /usr/local/cron_output.log 2>&1

Overwrite CRON Output to Log :

           The following syntax can be used to write the output of the shell scripts to the given file. If the output file already exist while writing, the old content will be replaced by new content.

Syntax :

CRON_EXPRESSION COMMAND > /OUTPUT_LOG_FILE_PATH/LOF_FILE

 Example :

The following example will write the output of the shell script test.sh to cron_output.log in the path /usr/local.

45 * * * * sh /usr/local/test.sh > /usr/local/cron_output.log