How to insert record to mongoDB6 collection using PutMongo processor in Apache Nifi? - mongodb

When I am trying to insert the record(json) to mongodb collection. I am getting exception
{ "ok" : 0.0, "errmsg" : "Unsupported OP_QUERY command: insert. The client driver may require an upgrade. For more details see https://dochub.mongodb.org/core/legacy-opcode-removal", "code" : 352, "codeName" : "UnsupportedOpQueryCommand" }
I have MongoDb 6.0 version configured
MongoDb removes Opal_insert operation in new version.How to resolve it ?

Related

Spark/MongoDB connector with $or pipeline syntax

I'm trying to request a MongoDB collection using Spark with the mongo-spark-connector. Here is the code :
val document = Document.parse("{$or: [{key1:\"A\", key2:\"500\"}, {key1:\"A\", key2:\"100\"}] }")
val mongoDf = mongoCollectionRdd.withPipeline(Seq(document)).toDF(mongoCollectionSchema)
But I get the following error :
com.mongodb.MongoCommandException: Command failed with error 40324 (Location40324): 'Unrecognized pipeline stage name: '$or'' on server hostname.fr:27017.
I ran the query through the mongo shell and the execution is fine and I get the expected result. Here are the versions of the different components :
Scala : 2.12.11
Spark : 3.2.0
MongoDB : 4.2.17
Any ideas ?

MongoDb - command drop requires authentication

I have the following command
mongo clean_collection.js --host localhost --username root --password example
Clean_collection.js file contains:
conn = new Mongo();
db = conn.getDB("MyDb");
var myColl = db.getCollection('MyCollection');
myColl.drop();
But MongoDb returns:
MongoDB shell version v4.2.8
connecting to: mongodb://localhost:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("22d094e1-fb65-46c8-a9e6-386c1678c680") }
MongoDB server version: 4.2.8
2020-07-15T10:33:00.787-0400 E QUERY [js] uncaught exception: Error: drop failed: {
"ok" : 0,
"errmsg" : "command drop requires authentication",
"code" : 13,
"codeName" : "Unauthorized"
} :
_getErrorWithCode#src/mongo/shell/utils.js:25:13
DBCollection.prototype.drop#src/mongo/shell/collection.js:696:15
I'm also using MongoDb .NET Driver and I can drop the collection withou any problems with using .NET driver.

Mongo.getDatabaseNames() occasionally throws CommandFailureException

We have been having some problems where our mongo connection fails to allow us to get the database names from the DB.
We call the method:
com.mongodb.Mongo.getDatabaseNames()
And 1 out of 100 times, it throws the following exception:
Caused by: com.mongodb.CommandFailureException: { "serverUsed" : "localhost:27017" ,
"errmsg" : "exception: can't open database in a read lock. if db was just closed, consider retrying the query. might otherwise indicate an internal error" , "code" : 15927 , "ok" : 0.0}
at com.mongodb.CommandResult.getException(CommandResult.java:76)
at com.mongodb.CommandResult.throwOnError(CommandResult.java:131)
at com.mongodb.Mongo.getDatabaseNames(Mongo.java:397)
Looking at the mongo code, the database is the internal admin database
public List<String> getDatabaseNames(){
BasicDBObject cmd = new BasicDBObject();
cmd.put("listDatabases", 1);
>>CommandResult res = getDB(ADMIN_DATABASE_NAME).command(cmd, getOptions());
res.throwOnError();
This database is not being deleted, but there are several other databases that could be deleted around this time.
Has anyone else had this problem?

elasticsearch jdbc river polling--- load data from mysql repeatedly

When using https://github.com/jprante/elasticsearch-river-jdbc I notice that the following curl statement successfully indexes data the first time. However, the river fails to repeatedly poll the database for updates.
To restate, when I run the following, the river successfully connects to MySQL, runs the query successfully, indexes the results, but never runs the query again.
curl -XPUT '127.0.0.1:9200/_river/projects_river/_meta' -d '{
"type" : "jdbc",
"index" : {
"index" : "test_projects",
"type" : "project",
"bulk_size" : 100,
"max_bulk_requests" : 1,
"autocommit": true
},
"jdbc" : {
"driver" : "com.mysql.jdbc.Driver",
"poll" : "1m",
"strategy" : "simple",
"url" : "jdbc:mysql://localhost:3306/test",
"user" : "root",
"sql" : "SELECT name, updated_at from projects p where p.updated_at > date_sub(now(),interval 1 minute)"
}
}'
Tailing the log, I see:
[2013-09-27 16:32:24,482][INFO ][org.elasticsearch.river.jdbc.strategy.simple.SimpleRiverFlow] next run, waiting 1m
[2013-09-27 16:33:24,488][INFO ]> [org.elasticsearch.river.jdbc.strategy.simple.SimpleRiverFlow] next run, waiting 1m
[2013-09-27 16:34:24,494][INFO ]> [org.elasticsearch.river.jdbc.strategy.simple.SimpleRiverFlow] next run, waiting 1m
But the index stays empty. Running on a macbook pro with elasticsearch version stable 0.90.2, HEAD and mysql-connector-java-5.1.25-bin.jar in the river pligns directory.
I think if you switch your strategy value from "simple" to "poll" you may get what you are looking for - it has worked for me with jdbc on that version of elasticsearch against MS SQL.
Also - you will need to select a field as _id (select primarykey as _id) as this is used in the elasticsearch river for determining what records are added/deleted/updated.

No updatedExisting from getLastError in MongoLab

I am running updates against a database in MongoLab (Heroku) and cannot get information from getLastError.
As an example, below are statements to update a collection in a MongoDB database running locally in my machine (db version v2.0.3-rc1).
ariels-MacBook:mongodb ariel$ mongo
MongoDB shell version: 2.0.3-rc1
connecting to: test
> db.mycoll.insert({'key': '1','data': 'somevalue'});
> db.mycoll.find();
{ "_id" : ObjectId("505bcc5783cdc9e90ffcddd8"), "key" : "1", "data" : "somevalue" }
> db.mycoll.update({'key': '1'},{$set: {'data': 'anothervalue'}});
> db.runCommand('getlasterror');
{
"updatedExisting" : true,
"n" : 1,
"connectionId" : 4,
"err" : null,
"ok" : 1
}
>
All is well locally.
Now I switch to a database in MongoLab and run the same statements to update a document. getLastError is not returning an updatedExisting field. Hence, I am unable to test if my update was successful or otherwise.
ariels-MacBook:mongodb ariel$ mongo ds0000000.mongolab.com:00000/heroku_app00000 -u someuser -p somepassword
MongoDB shell version: 2.0.3-rc1
connecting to: ds000000.mongolab.com:00000/heroku_app00000
> db.mycoll.insert({'key': '1','data': 'somevalue'});
> db.mycoll.find();
{ "_id" : ObjectId("505bcf9b2421140a6b8490dd"), "key" : "1", "data" : "somevalue" }
> db.mycoll.update({'key': '1'},{$set: {'data': 'anothervalue'}});
> db.runCommand('getlasterror');
{
"n" : 0,
"lastOp" : NumberLong("5790450143685771265"),
"connectionId" : 1097505,
"err" : null,
"ok" : 1
}
> db.mycoll.find();
{ "_id" : ObjectId("505bcf9b2421140a6b8490dd"), "data" : "anothervalue", "key" : "1" }
>
Did anyone run into this?
If it matters, my resource at MongoLab is running mongod v2.0.7 (my shell is 2.0.3).
Not exactly sure what I am missing.
I am waiting to hear from their support (I will post here when I hear back) but wanted to check with you fine folks here as well just in case.
Thank you.
This looks to be a limitation of not having admin privileges to the mongod process. You might file a ticket with 10gen as it doesn't seem like a necessary limitation.
When I run Mongo in auth mode on my laptop I need to authenticate as a user in the admin database in order to see an "n" other than 0 or the "updatedExisting" field. When I authenticate as a user in any other database I get similar results to what you're seeing in MongoLab production.
(Full disclosure: I work for MongoLab. As a side note, I don't see the support ticket you mention in our system. We'd be happy to work with you directly if you'd like. You can reach us at support#mongolab.com or http://support.mongolab.com)