Import Data from Mongo DB to Elasticsearch using Elasticsearch River

Import Data from Mongo DB to Elasticsearch using Elasticsearch River

Import Data from Mongo DB to Elasticsearch using Elasticsearch River

           Importing data from Mongo DB to Elasticsearch requires Mongo DB to have replica set enabled. Once the replica set is enabled, Mongo DB populates the oplog (operation log) as and when we add documents to the collection in the database (except database named local, which is used by Mongo DB to store system level data). This oplog is used by Elasticsearch River plugin to sync up data with Elasticsearch.

           In most cases, we will be running Mongo DB without any replica set (standalone setup). In those cases, we can convert the standalone setup to replica set for development and testing purpose.

Steps to Setup Replicaset

  1. Install Elasticsearch
  2. Install Elasticsearch River plugin for Mongo DB from the website https://github.com/richardwilly98/elasticsearch-river-mongodb or use the command given below.
  3. 1. Go to ES_HOME/bin
    2. plugin --install com.github.richardwilly98.elasticsearch/elasticsearch-river-mongodb/2.0.6
                                            or
     ./plugin --install com.github.richardwilly98.elasticsearch/elasticsearch-river-mongodb/2.0.6
  4.  Install MongoDB 2.4.3. Higher versions are also compatible.
  5. Start Mongo DB using the command ./mongod –dbpath “/usr/local/mongodata/” –replSet “rs0”
  6. Execute the command rs.initiate() in Mongo DB console. This command may take a few seconds or a few seconds to complete based on the system specification.
  7. Use the following REST query to import data from Mongo DB to Elasticsearch 
PUT /_river/sample_name/_meta
{
    "type": "mongodb",
    "mongodb": {
        "servers": [{
             "host": "IP",
             "port": 27017
         }],
    "db": "db_name",
    "collection": "collection_name"
    },
    "index": {
    "name": "index_name",
    "type": "type_name"
    }
}

Test Setup Used:

  • OS: RHEL (Red Hat Enterprise Linux)
  • Mongo DB: v2.4.3 (64bit)
  • Elasticsearch: v1.4.4
  • Mongo River: v2.0.6

           Mongo River plugin syncup data from Mongo DB to Elasticsearch in real-time. Make sure you don’t try to import collection from the database named local since the system level data are maintained in it

Note : 

  • The syncup process will work with no issues if we have Elasticsearch and Mongo DB in same system when using standalone setup. However we faced the issues when Elasticsearch and Mongo DB were on different nodes.

Related Links :

Web Application for Elasticsearch :
  1. ElasticTab – Elasticsearch to Excel Report (Web Application)
Elasticsearch Plugin:
  1. Elasticsearch Plugin To Generate (Save and E-Mail) Excel Reports
Elasticsearch:
  1. Execute Multiple Search Query in Elasticsearch
  2. Monitor Elasticsearch Servers with Shell Script - E-Mail Notification
  3. Execute Raw Elasticsearch Query using Transport Client – Java API
  4. Elasticsearch – Apply Nested Filter on Nested (Inner) Aggregation
  5. Execute Multiple Search Query in Elasticsearch
  6. Enable CORS to Send Cross Domain Request to Elasticsearch using AJAX
  7. Elasticsearch Java API – Get Index List
  8. Elasticsearch Java API – Get Alias List
  9. Elasticsearch Java API - Get Type List from given Index
  10. Elasticsearch Java API – Get Field List for a given Index and Type
  11. Elasticsearch Java API – Get Index Type List Mapping
  12. Elasticsearch – Use Script Filter/Conditon in Aggregation/Sub-Aggreagtion
  13. Elasticsearch – Compare/ScriptFilter/Condition on Two Fields using Script Filter – REST Query + Java API
  14. Elasticsearch - Date/Time(String)  Add/Subtract Duration - Days,Months,Years,Hours,Minutes,Seconds
Logstash:
  1. Logstash – Process Log File Once and Exit/Stop Logstash After Reading Log File Once
  2. Measure Logstash Performance using Metrics Filter – Issue/Error in Syntax (Unknown setting ‘message’ for stdout)
  3. Logstash – Process Same Log File (File Input) from Beginning/Start
  4. Create Custom Filter/Plugin to Emit New Events Manually in Logstash
Logstash and Elasticsearch:
  1. Query Elasticsearch Cluster in Filter Section when using Logstash
  2. Custom Elasticsearch Index Name/Type Name based on Events in Logstash
MongoDB and Elasticsearch:
  1. Import Data from Mongo DB to Elasticsearch using Elasticsearch River
 Read More...

[ YOU MAY ALSO LIKE ]

Leave a Reply