so I created my logstash conf file, and spun up logstash, kibana, Postgres, and elasticsearch in one docker compose file, it connected seemlessly with my database however it says the table "products" don't exist.
[2023-01-18T14:06:00,182][WARN ][logstash.inputs.jdbc ][main][6a13cd40fa144828caae9db4ed20b978765149c99cc59d5830fa4ccad80b4017] Exception when executing JDBC query {:exception=>"Java::OrgPostgresqlUtil::PSQLException: ERROR: relation \"products\" does not exist\n Position: 15"}
This is my conf
input {
jdbc {
jdbc_connection_string => "jdbc:postgresql://elastic-postgres-1:5432/shopdb"
jdbc_user => "postgres"
jdbc_password => "****"
jdbc_driver_library => "./postgresql-42.2.27.jre7.jar"
jdbc_driver_class => "org.postgresql.Driver"
statement => "SELECT * FROM products;"
schedule => "* * * * *"
}
}
output {
elasticsearch {
hosts => ["http://elasticsearch:9200"]
index => "PostgreSQL"
}
}
granted I did link my postgres to logstash with the conf BEFORE creating the table but I have tried re-starting the containers again. Again the error persists I tried putting in the wrong table to know if it's even updating the conf which it noticed so why isn't it seeing the table "products" which has been created and populated now?
Try explicitly to use the object schema name in your query for avoid this error not found the table.
like:
SELECT * FROM schema_name.object_name
i am trying to pull records form mongo data base with json format to do that i am running below js :
conatin of get_DRMPhysicalresources_Data.js
cursor = db.physicalresources.find().limit(10)
while ( cursor.hasNext() ){
print( JSON.stringify(cursor.next()) );
}
the command i run to get the records :
mongo host_name:port/data_base_name -u user -p 'password' --eval "load(\"get_DRMPhysicalresources_Data.js\")" > DRMPhysicalresources.json
and i am able to get the result as josn formate inside DRMPhysicalresources.json , now i want to switch to other data base using use command i try add use db as below :
conatin of get_DRMPhysicalresources_Data.js
use db2_test
cursor = db.physicalresources.find().limit(10)
while ( cursor.hasNext() ){
print( JSON.stringify(cursor.next()) );
}
the command i run to get the records :
mongo host_name:port/data_base_name -u user -p 'password' --eval "load(\"get_DRMPhysicalresources_Data.js\")" > DRMPhysicalresources.json
but i am getting below errors :
MongoDB shell version v4.2.3
connecting to: "some data base info"
Implicit session: session { "id" : UUID("8c85c6af-ebed-416d-9ab8-d6739a4230cb") }
MongoDB server version: 4.4.11
WARNING: shell and server versions do not match
2022-04-11T13:39:30.121+0300 E QUERY [js] uncaught exception: SyntaxError: unexpected token: identifier :
#(shell eval):1:1
2022-04-11T13:39:30.122+0300 E QUERY [js] Error: error loading js file: get_DRMPhysicalresources_Data.js :
#(shell eval):1:1
2022-04-11T13:39:30.122+0300 E - [main] exiting with code -4
is there are any way to add use db2_test without break it ?
You can try this:
db = new Mongo().getDB("myOtherDatabase");
or
db = db.getSiblingDB('myOtherDatabase')
( this is the exact js equivalent to the 'use database' helper)
docs
I want to sync my mongodb data(local mongodb) to elastic search(local elastic) using logstash-plugin of mongodb
I have install logstash plugin using
bin/logstash-plugin install logstash-input-mongodb .
Then i created a mongodata.conf file in /usr/share/logstash directory.
When I execute the conf file then it shows
--> Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
My config file is:
input{
mongodb{
uri => "mongodb://localhost:27017/reporterDB"
placeholder_db_dir => "/opt/logstash-mongodb/"
placeholder_db_name => "logstash_sqlite.db"
collection => "iam_ms_test"
batch_size => 5000
}
}
filter{
}
output {
stdout { codec => rubydebug }
elasticsearch {
action => "index"
hosts => "localhost:9200"
user => elastic
password => changeme
index => "mongo_log"
document_type => "document_type"
document_id => "%{id}"
}
}
I am getting below lines in logstash-plain.log file
[2019-11-01T15:41:00,869][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-11-01T15:41:00,871][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>6, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>750, :thread=>"#<Thread:0x351f7fd1#/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:245 run>"}
[2019-11-01T15:41:01,068][INFO ][logstash.inputs.mongodb ] Registering MongoDB input
[2019-11-01T15:41:01,116][ERROR][logstash.pipeline ] Error registering plugin {:pipeline_id=>"main", :plugin=>"<LogStash::Inputs::MongoDB uri=>\"mongodb://localhost:27017/anchorReports\", placeholder_db_dir=>\"/opt/logstash-mongodb/\", placeholder_db_name=>\"logstash_sqlite.db\", collection=>\"hi_p5m\", batch_size=>5000, id=>\"ec7682e8c6c5676deca84d5072c5f7865120a107ffce81ce21caa878c6e4ed09\", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>\"plain_441f95b8-cc8a-4b9e-a45f-657ed2011e2b\", enable_metric=>true, charset=>\"UTF-8\">, since_table=>\"logstash_since\", since_column=>\"_id\", since_type=>\"id\", parse_method=>\"flatten\", isodate=>false, retry_delay=>3, generateId=>false, unpack_mongo_id=>false, message=>\"Default message...\", interval=>1>", :error=>"Java::JavaSql::SQLException: path to '/opt/logstash-mongodb/logstash_sqlite.db': '/opt/logstash-mongodb' does not exist", :thread=>"#<Thread:0x351f7fd1#/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:245 run>"}
[2019-11-01T15:41:01,869][ERROR][logstash.pipeline ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<Sequel::DatabaseConnectionError: Java::JavaSql::SQLException: path to '/opt/logstash-mongodb/logstash_sqlite.db': '/opt/logstash-mongodb' does not exist>, :backtrace=>["org.sqlite.core.CoreConnection.open(org/sqlite/core/CoreConnection.java:190)", "org.sqlite.core.CoreConnection.<init>(org/sqlite/core/CoreConnection.java:74)", "org.sqlite.jdbc3.JDBC3Connection.<init>(org/sqlite/jdbc3/JDBC3Connection.java:24)", "org.sqlite.jdbc4.JDBC4Connection.<init>(org/sqlite/jdbc4/JDBC4Connection.java:23)", "org.sqlite.SQLiteConnection.<init>
"(org/sqlite/SQLiteConnection.java:45)",
"org.sqlite.JDBC.createConnection(org/sqlite/JDBC.java:114)",
"org.sqlite.JDBC.connect(org/sqlite/JDBC.java:88)"
I want the records on my elastic search under `index("mongo_log").
I also want to know the uses of placeholder_db_dir and placeholder_db_name and whats should be these values when we are using mongodb as the input database.
Problem solved! actually the directory opt/logstash was not created . So I manually create the logstash folder under opt. After that i gave Write permission to that directory , so that when we execute the command for logstash then it can create file inside this folder.
I am trying to index data from a PostgreSQL database to elastic search using logstash. I have a column in the database which is of type JSON. When I run logstash, I get an error called "Missing Converter handling". I do understand that is java/logstash doesn't recognize the JSON type column.
I can typecast the column to "col_name::TEXT" which works fine. However, I need that column as JSON in an elastic search index. Is there any workaround?
Note: Keys in the JSON object is not fixed, it varies.
Logstash postgres query example:
input {
jdbc {
# Postgres jdbc connection string to our database, bot
jdbc_connection_string => "***"
# The user we wish to execute our statement as
jdbc_user => "***"
jdbc_password => "***"
# The path to our downloaded jdbc driver
jdbc_driver_library => "<ja_file_path>"
# The name of the driver class for Postgresql
jdbc_driver_class => "org.postgresql.Driver"
# our query
statement => "SELECT col_1, col_2, col_3, json_col_4 from my_db;"
}
}
filter{
}
output {
stdout { codec => json_lines }
}
Need JSON object from PostgreSQL to elastic search.
I try to use the mongodb plugin as input for logstash.
Here is my simple configuration:
input {
mongodb {
uri => 'mongodb://localhost:27017/testDB'
placeholder_db_dir => '/Users/TEST/Documents/WORK/ELK_Stack/LogStash/data/'
collection => 'logCollection_ALL'
batch_size => 50
}
}
filter {}
output { stdout {} }
But I'm facing a "loop issue" probably due to a field "timestamp" but I don't know what to do.
[2018-04-25T12:01:35,998][WARN ][logstash.inputs.mongodb ] MongoDB Input threw an exception, restarting {:exception=>#TypeError: wrong argument type String (expected LogStash::Timestamp)>}
With also a DEBUG log:
[2018-04-25T12:01:34.893000 #2900] DEBUG -- : MONGODB | QUERY | namespace=testDB.logCollection_ALL selector={:_id=>{:$gt=>BSON::ObjectId('5ae04f5917e7979b0a000001')}} flags=[:slave_ok] limit=50 skip=0 project=nil |
runtime: 39.0000ms
How can I parametrize my logstash config to get my output in the stdout console ?
It's because of field #timestamp that has ISODate data type.
You must remove this field from all documents.
db.getCollection('collection1').update({}, {$unset: {"#timestamp": 1}}, {multi: true})