BIG DATA GURUKUL

Open the world of Knowledge

MongoDB logs on ELK(Elasticsearch Logstash Kibana)

14. April 2017 bigdata-admin ELK (0)
Prerequisites-

1) ElasticSearch

2) Kibana

3) Logstash

4) Mongodb

5) Set JAVA_HOME


Steps to store MongoDB logs on ElasticSearch and visualize them on Kibana


Step1- Download latest version:


  • Elasticsearch
  • Kibana
  • Logstash

  It can be downloaded from “www.elasticsearch.org”


Step2- Run elasticsearch (bin/elasticsearch)


           



Step3-   Create a configuration file in the bin folder of Logstash and save it as “logstash.conf”


logstash.conf

      

       input {

                      file {

                              path => "C:\Data\log\filter-mongologs-2017-03-18\mongodb-current.log"

                              start_position => "beginning"

                            }

                  }

          filter {

                    grok {

                              match => {message => "%{TIMESTAMP_ISO8601:timestamp} %{MONGO3_SEVERITY:severity} %                                                             {MONGO3_COMPONENT:component}%{SPACE}(?:\[%{DATA:context}\])? %                                                                                 {GREEDYDATA:content}.*%{NUMBER:duration}+ms"}

                            }

               mutate {

                              remove_field => ["message" ,"timestamp" , "tags" , "@version" ]

                           }

                   

                   if  [component] != "COMMAND"  {

                                                                          drop { }

                                                                        }

                 }


Output :


             {

                   elasticsearch{ hosts => ["localhost:9200"] index => "mongolog7" }

                                          tdout{codec => "rubydebug"}

                }

 


(Add the path of mongodb log file in input of  logstash configuration file.)



Step4-  Install Mongodb input plugin


                           >  logstash-plugin install logstash-input-mongodb



                 



Step5-   Install elasticsearch output plugin


                      >  logstash-plugin install logstash-output-elasticsearch



                

 


Step6- In command prompt run the following command under /bin directory


                      > /bin/logstash –f logstash.conf



                

   It will create an index in elasticsearch.

 

Step7- 


            KIBANA (bin/kibana)  (in cmd)

            It is running on port (localhost:5601)

            In console window(Dev Tools) check your index is created or not.

            Run  – GET  (index name)/_search



               



Step8- 


  • Go to the management window in kibana. 
  • It will ask to configure index pattern. 
  • Configure the index by entering the index name (same index name which is in conf file).

                


     Click on Create.

     It will show the following result.


            

              



Step9- Go to the “Discover” window in Kibana

 

       Click on New and add index “Index_name”.(refresh the index as per time given in the corner of window if it is giving error while discovering the log)

 


               



It will give following output:


               


Step10- Now Visualize your index.


          Go to visualize window and select the parameters.


               



  • After selecting the parameters then select the add metrics to visualize your data.
  • Add the aggregation which you want to display in your visualization.
  • After adding the aggregation, It will show the following output:

              


     After that save this visualization. Click on the “Save” button on menu bar.


Step11- Create Dashboard in Kibana.

              

            Go to dashboard window. It will give “ Ready to get started?” Window


           



Then click on “Add” on the upper menu bar.

And then add your saved visualization on Dashboard.



            



       If you want to expand your dashboard to get more information then click on the small arrow at the below corner of the visualization.


             



By doing this we can load the MongoDB logs into the Elasticsearch using Logstash and then visualize them using Kibana.