sync mongo data to elastic using logstash - mongodb

Question

I want to sync my mongodb data(local mongodb) to elastic search(local elastic) using logstash-plugin of mongodb
I have install logstash plugin using
bin/logstash-plugin install logstash-input-mongodb .
Then i created a mongodata.conf file in /usr/share/logstash directory.
When I execute the conf file then it shows
--> Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
My config file is:
input{
mongodb{
uri => "mongodb://localhost:27017/reporterDB"
placeholder_db_dir => "/opt/logstash-mongodb/"
placeholder_db_name => "logstash_sqlite.db"
collection => "iam_ms_test"
batch_size => 5000
}
}
filter{
}
output {
stdout { codec => rubydebug }
elasticsearch {
action => "index"
hosts => "localhost:9200"
user => elastic
password => changeme
index => "mongo_log"
document_type => "document_type"
document_id => "%{id}"
}
}
I am getting below lines in logstash-plain.log file
[2019-11-01T15:41:00,869][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-11-01T15:41:00,871][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>6, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>750, :thread=>"#<Thread:0x351f7fd1#/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:245 run>"}
[2019-11-01T15:41:01,068][INFO ][logstash.inputs.mongodb ] Registering MongoDB input
[2019-11-01T15:41:01,116][ERROR][logstash.pipeline ] Error registering plugin {:pipeline_id=>"main", :plugin=>"<LogStash::Inputs::MongoDB uri=>\"mongodb://localhost:27017/anchorReports\", placeholder_db_dir=>\"/opt/logstash-mongodb/\", placeholder_db_name=>\"logstash_sqlite.db\", collection=>\"hi_p5m\", batch_size=>5000, id=>\"ec7682e8c6c5676deca84d5072c5f7865120a107ffce81ce21caa878c6e4ed09\", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>\"plain_441f95b8-cc8a-4b9e-a45f-657ed2011e2b\", enable_metric=>true, charset=>\"UTF-8\">, since_table=>\"logstash_since\", since_column=>\"_id\", since_type=>\"id\", parse_method=>\"flatten\", isodate=>false, retry_delay=>3, generateId=>false, unpack_mongo_id=>false, message=>\"Default message...\", interval=>1>", :error=>"Java::JavaSql::SQLException: path to '/opt/logstash-mongodb/logstash_sqlite.db': '/opt/logstash-mongodb' does not exist", :thread=>"#<Thread:0x351f7fd1#/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:245 run>"}
[2019-11-01T15:41:01,869][ERROR][logstash.pipeline ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<Sequel::DatabaseConnectionError: Java::JavaSql::SQLException: path to '/opt/logstash-mongodb/logstash_sqlite.db': '/opt/logstash-mongodb' does not exist>, :backtrace=>["org.sqlite.core.CoreConnection.open(org/sqlite/core/CoreConnection.java:190)", "org.sqlite.core.CoreConnection.<init>(org/sqlite/core/CoreConnection.java:74)", "org.sqlite.jdbc3.JDBC3Connection.<init>(org/sqlite/jdbc3/JDBC3Connection.java:24)", "org.sqlite.jdbc4.JDBC4Connection.<init>(org/sqlite/jdbc4/JDBC4Connection.java:23)", "org.sqlite.SQLiteConnection.<init>
"(org/sqlite/SQLiteConnection.java:45)",
"org.sqlite.JDBC.createConnection(org/sqlite/JDBC.java:114)",
"org.sqlite.JDBC.connect(org/sqlite/JDBC.java:88)"
I want the records on my elastic search under `index("mongo_log").
I also want to know the uses of placeholder_db_dir and placeholder_db_name and whats should be these values when we are using mongodb as the input database.

Answer 1

Problem solved! actually the directory opt/logstash was not created . So I manually create the logstash folder under opt. After that i gave Write permission to that directory , so that when we execute the command for logstash then it can create file inside this folder.