Elasticsearch is a search engine based on the Lucene library. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents.

Logstash is an open source, data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite “stash.” including Elasticsearch.

In this blog we show you a simplistic configuration for inserting data into Elasticsearch using the Logstash JDBC plugin and Simba MongoDB JDBC driver.

 

MongoDB to ElasticSearch

Start logstash instance using a configuration file as below:

input {
 jdbc {
   jdbc_driver_library => "<>\SimbaMongoDBJDBC\libs\MongoDBJDBC42.jar"
   jdbc_driver_class => "com.simba.mongodb.jdbc.Driver"
   jdbc_connection_string => "jdbc:mongodb://localhost/magnitude"
   jdbc_user => ""
   parameters => { "favorite_artist" => "Beethoven" }
   schedule => "1 * * * *"
   statement => "SELECT name from artists"
 }
}
filter {
 mutate { remove_field => [ "_id" ] }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
  }
}

This would store data in the default index created in Elasticsearch.

You may want to use filters in the SQL query to fetch relevant data from MongoDB.

The example above fetches data every 1st minute of every hour every day; update the schedule as per your requirements. Verify in Elasticsearch if the data from MongoDB was indeed inserted.

curl -X GET "localhost:9200/logstash-2019.08.01/_search?q=name: Beethoven"

Now you are all set to visualize and explore MongoDB data in Kibana dashboard.

Now it's your turn !