I'm trying to request a MongoDB collection using Spark with the mongo-spark-connector. Here is the code :
val document = Document.parse("{$or: [{key1:\"A\", key2:\"500\"}, {key1:\"A\", key2:\"100\"}] }")
val mongoDf = mongoCollectionRdd.withPipeline(Seq(document)).toDF(mongoCollectionSchema)
But I get the following error :
com.mongodb.MongoCommandException: Command failed with error 40324 (Location40324): 'Unrecognized pipeline stage name: '$or'' on server hostname.fr:27017.
I ran the query through the mongo shell and the execution is fine and I get the expected result. Here are the versions of the different components :
Scala : 2.12.11
Spark : 3.2.0
MongoDB : 4.2.17
Any ideas ?
Related
When I am trying to insert the record(json) to mongodb collection. I am getting exception
{ "ok" : 0.0, "errmsg" : "Unsupported OP_QUERY command: insert. The client driver may require an upgrade. For more details see https://dochub.mongodb.org/core/legacy-opcode-removal", "code" : 352, "codeName" : "UnsupportedOpQueryCommand" }
I have MongoDb 6.0 version configured
MongoDb removes Opal_insert operation in new version.How to resolve it ?
i am trying to pull records form mongo data base with json format to do that i am running below js :
conatin of get_DRMPhysicalresources_Data.js
cursor = db.physicalresources.find().limit(10)
while ( cursor.hasNext() ){
print( JSON.stringify(cursor.next()) );
}
the command i run to get the records :
mongo host_name:port/data_base_name -u user -p 'password' --eval "load(\"get_DRMPhysicalresources_Data.js\")" > DRMPhysicalresources.json
and i am able to get the result as josn formate inside DRMPhysicalresources.json , now i want to switch to other data base using use command i try add use db as below :
conatin of get_DRMPhysicalresources_Data.js
use db2_test
cursor = db.physicalresources.find().limit(10)
while ( cursor.hasNext() ){
print( JSON.stringify(cursor.next()) );
}
the command i run to get the records :
mongo host_name:port/data_base_name -u user -p 'password' --eval "load(\"get_DRMPhysicalresources_Data.js\")" > DRMPhysicalresources.json
but i am getting below errors :
MongoDB shell version v4.2.3
connecting to: "some data base info"
Implicit session: session { "id" : UUID("8c85c6af-ebed-416d-9ab8-d6739a4230cb") }
MongoDB server version: 4.4.11
WARNING: shell and server versions do not match
2022-04-11T13:39:30.121+0300 E QUERY [js] uncaught exception: SyntaxError: unexpected token: identifier :
#(shell eval):1:1
2022-04-11T13:39:30.122+0300 E QUERY [js] Error: error loading js file: get_DRMPhysicalresources_Data.js :
#(shell eval):1:1
2022-04-11T13:39:30.122+0300 E - [main] exiting with code -4
is there are any way to add use db2_test without break it ?
You can try this:
db = new Mongo().getDB("myOtherDatabase");
or
db = db.getSiblingDB('myOtherDatabase')
( this is the exact js equivalent to the 'use database' helper)
docs
I have the following command
mongo clean_collection.js --host localhost --username root --password example
Clean_collection.js file contains:
conn = new Mongo();
db = conn.getDB("MyDb");
var myColl = db.getCollection('MyCollection');
myColl.drop();
But MongoDb returns:
MongoDB shell version v4.2.8
connecting to: mongodb://localhost:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("22d094e1-fb65-46c8-a9e6-386c1678c680") }
MongoDB server version: 4.2.8
2020-07-15T10:33:00.787-0400 E QUERY [js] uncaught exception: Error: drop failed: {
"ok" : 0,
"errmsg" : "command drop requires authentication",
"code" : 13,
"codeName" : "Unauthorized"
} :
_getErrorWithCode#src/mongo/shell/utils.js:25:13
DBCollection.prototype.drop#src/mongo/shell/collection.js:696:15
I'm also using MongoDb .NET Driver and I can drop the collection withou any problems with using .NET driver.
I am trying to write data from Spark (using Databricks) to Azure Cosmos DB(Mongo DB). There are no errors when executing notebook but i am getting below error when querying the collection.
I have used jar from databricks website azure-cosmosdb-spark_2.4.0_2.11-2.1.2-uber.jar. My versions is 6.5 (includes Apache Spark 2.4.5, Scala 2.11)
import org.joda.time.format._
import com.microsoft.azure.cosmosdb.spark.schema._
import com.microsoft.azure.cosmosdb.spark.CosmosDBSpark
import com.microsoft.azure.cosmosdb.spark.config.Config
import org.apache.spark.sql.functions._
val configMap = Map(
"Endpoint" -> "https://******.documents.azure.com:***/",
"Masterkey" -> "****==",
"Database" -> "dbname",
"Collection" -> "collectionname"
)
val config = Config(configMap)
val df = spark.sql("select id,name,address from emp_db.employee")
CosmosDBSpark.save(df, config)
when i query the collection i get below response
Error: error: {
"_t" : "OKMongoResponse",
"ok" : 0,
"code" : 1,
"errmsg" : "Unknown server error occurred when processing this request.",
"$err" : "Unknown server error occurred when processing this request."
}
Any help would be much appreciated. Thank you!!!
That error suggests you are using CosmosDB with the MongoDB api.
The spark connector for CosmosDB only supports it when using the SQL api.
Instead you should use the MongoDB connector.
https://learn.microsoft.com/en-us/azure/cosmos-db/spark-connector
Use this instead https://docs.mongodb.com/spark-connector/master/
I have this issue with my meteor app. When I run this query on my chrome console, It returns the expected data
Questions.find({}, {sort:{commentLength: -1}})
but when I run it in the console as db.questions.find({}, {sort:{commentLength: -1}})
it returns this error
error: {
"$err" : "Can't canonicalize query: BadValue Unsupported projection option: sort: { commentLength: -1.0 }",
"code" : 17287
}
Why does this error happen? Thanks
sort has a different syntax when executed in a mongodb shell. Try this instead:
db.questions.find().sort({commentLength: -1})