so I'm trying to learn mongodb via nodeschool with local servers and I noticed in my exercises, that if I don't prefix the variable I am comparing with a '+' character, I don't get anything returned to me.
Here is the actual function in question, arg is first argument from the command line entered by the user:
parrots.find({
age: {$gt: arg}
},
{name: 1, age: 1, _id: 0}).toArray(function(err, documents) {
if (err) {
throw err
}
console.log(documents);
db.close();
});
however if I simply edit the line where I check if age is greater than the age provided by the user to
age: {$gt: +arg}
suddenly I get an JSON object returned to me that was within the database.
I'm just not sure why age: {$gt: arg} doesn't work since even in the documentation for the find method, there was no example that used +
Thank you
The arguments as they come from the command line are Strings, the +arg syntax is a shorthand to cast that string to a JavaScript Number. It is not specific to Mongo, just a JS idiosyncrasy.
Related
In MongoDB (using mongosh or command-line mongo cli), you can query documents, for example using db.mycollection.find({"something":true}) and get the following result:
{
"someDate": ISODate("2022-10-24T17:21:44.980Z"),
"something": true,
"hello": "world"
}
This result, however, is not valid JSON (Due to ISODate). How can I change the query above to make MongoDB return canonical (valid) JSON?
I'm looking for a recursive and generalized way to do this, even for deeply nested documents.
There are a number of existing answers, I'll clarify a few:
Use aggregate to produce the output in JSON format: Playground
db.collection.aggregate([
{
$match: {
something: true
}
},
{
$project: {
_id: 1,
someDate: {
$dateToString: {
format: "%Y-%m-%dT%H:%M:%S:%LZ",
date: "$someDate"
}
},
something: 1,
hello: 1
}
}
])
Loop through your query in your application: (example, node.js)
db.mycollection.find({"something":true}).forEach(function(doc) {
doc.someDate = doc.someDate.toISOString() // or even .toJSON()
})
// Or with await
const records = await db.mycollection.find({"something":true}).map(doc => {
doc.someDate = doc.someDate.toISOString()
return doc
}).toArray()
The details of where you are running this command are very important, can you please share those?
I am guessing that you are probably running this via the (older) mongo utility (as opposed to the newer mongosh). But confirmation of that, and the database version that you are using, would both be helpful. I will retain this assumption for the purposes of this answer.
This result, however, is not valid JSON (Due to ISODate).
The database itself doesn't return some text that has ISODate. In fact, it doesn't return or "speak" JSON at all. Rather, the database communicates via "Binary JSON" or BSON for short. In fact, the "Does MongoDB use BSON or JSON?" section of this page specifically mentions the following:
Firstly, BSON documents may contain Date or Binary objects that are not natively representable in pure JSON.
So when you see things like ISODate() that is the client application wrapping and representing the rich BSON document in a more limited (and text-based) JSON-like form. Importantly, this is often for readability purposes. You should be able to natively pass around and operate on information (documents) returned by the database directly in the application without doing any sort of transformation and without losing rich type information. Additional reading material about BSON is here.
Getting back to the original question, if you want to have the shell print out a valid JSON document than you can do that via additional helpers. In the older mongo utility, a reproduction of the situation described in the question is:
> db.mycollection.findOne({"something": true})
{
"_id" : 1,
"someDate" : ISODate("2022-11-30T14:38:37.711Z"),
"something" : true,
"hello" : "world"
}
The shell itself can understand and operate on ISODate() (and other functions of that nature). If you did want to remove things like ISODate() for some reason, then you can leverage the JSON.stringify() functionality (reformatted with line indents for readability):
> JSON.stringify( db.mycollection.findOne({"something": true}) )
{
"_id":1,
"someDate":"2022-11-30T14:38:37.711Z",
"something":true,
"hello":"world"
}
The newer mongosh shell offers even more utility here:
> EJSON.serialize( db.mycollection.findOne() )
{
_id: 1,
someDate: { '$date': '2022-11-30T14:38:37.711Z' },
something: true,
hello: 'world'
}
Via these EJSON functions, mongosh is attempting to preserve type information when it prints the data in this format. Notice that in the earlier example the date was just represented as a string, but here the shell is using Extended JSON to capture the fact that the type of someDate is a Date.
I have this query which isn't working and I've tried looking for material to better understand the syntax behind it, but I'm not having much luck. If someone could point out some resources for me to look into, I would appreciate it. Specifically, I'm having trouble with ISODate(:#{#start}) and what roles the :# or {#parameter} play in this query.
Also, if you happen to know how to fix this up. Please share!
Edit The dates in the db should be engulfed by the date parameters supplied. I originally had 'foo': :#{#parameter} which worked for looking up values, but here I've had to wrap the string with ISODate() in order to convert it for comparison.
#Query("
{ 'user.id': :#{#id} },
{ 'date.events.start': { $gte: ISODate(:#{#start}) } },
{ 'date.events.end': { $lte: ISODate(:#{#currentDate}) } }
")
List<Case> getUsersInPeriod(#Param("id") String id,
#Param("start") String start, #Param("currentDate") String currentDate);
Edit Playing around in Mongo Compass query box with the following:
{'user.id': '5df2b19006f31c190cc13288' }, { 'date.events.start': {$gte: ISODate('2020-02-13')} }, {'date.events.end': {$lte: ISODate('2020-02-31')}}
and it does not work as expected. So I am not sure what the issue is.
Sample Mongo document values:
start: "2020-01-20T08:30:00.000-06:00" (String)
end: "2020-01-20T15:30:00.000-06:00" (String)
Sample Comparison values:
start: "2020-01-16" (String)
end: "2020-01-31" (String)
Which didn't work, and so I wrapped the comparison values with ISODate() and that still didn't work, which now has me looking at document string values.
Edit: #3
I've converted some values in the document to Date and changed the query in Mongo Compass to:
{$and: [{'user.id': '5df2b19006f31c190cc13288' }, { 'date.events.start': {$gte: ISODate('2020-01-21')} }, {'date.events.end': {$lte: ISODate('2020-01-31')}}]}
which only picks up the document values formed as Date instead of String... so, I think I narrowed down the problem to two things (My original issue still persists with the syntax).
How do I deal with Mongo documents with string dates? Is it possible to parse the document string dates as ISODate while doing the query?
#prasad_ answer to formatting: can parse String or Date, but both must be of the same type. Ideally Date is used.
Edit #4: What I know so far...
User.id checks out. There's no issue there. I know the $and usage is correct.
Sample Document entries:
start: "2020-01-20T08:30:00.000-06:00" (String)
end: "2020-01-20T15:30:00.000-06:00" (String)
The above values should be engulfed by the following parameters:
start: 2020-01-16T19:57:54.949-06:00
end: 2020-01-31T23:59:59.999-06:00
I've converted both sets of strings to String or to Date and neither has returned results from the query made in my application; however ...
MongoDB Compass Community Query filter:
{'$and':[{'user.id':'5df2b19006f31c190cc13288'},{'date.events.start':{'$gte':'2020-01-22T08:30:00.000-06:00'}},{'date.events.end':{'$lte':'2020-01-31T15:30:00.000-06:00'}}]}
Does filter strings correctly, and...
{'$and':[{'user.id':'5df2b19006f31c190cc13288'},{'date.events.start':{'$gte':'2020-01-22T08:30:00.000-06:00'}},{'date.events.end':{'$lte':'2020-01-31T15:30:00.000-06:00'}}]}
Works when the document fields are of type Date and
start: 2020-01-21T14:30:00.000+00:00 (Date)
end: 2020-01-21T21:30:00.000+00:00 (Date)
Since I can get results in Mongo Compass Community, there must be something wrong with my application's query here:
#Query("{'$and': [{ 'user.id': :#{#id} }, { 'date.events.start': { '$gte': :#{#startPeriod} } }, { 'date.events.end': { '$lte': :#{#currentDate} } } ]}")
List<Case> getUsersInPeriod(#Param("id") String id, #Param("startPeriod") String startPeriod, #Param("currentDate") String currentDate);
The document entry is structured as:
date: (Object)
events: (Array)
[0]: (Object)
start: (String)
end: (String)
[1]: (Object)
(...)
(...)
Solution
I was able to find something that put me on the right path:
Internally, MongoDB can store dates as either Strings or as 64-bit integers. If you intend to do any operations using the MongoDB query or aggregate functions, or if you want to index your data by date, you'll likely want to store your dates as integers. If you're using the built-in "Date" data type, or a date wrapped in the ISODate() function, you're also storing your date as an integer.
https://www.compose.com/articles/understanding-dates-in-compose-mongodb/
So I changed everything to Date and now it's working as expected. Not sure what I had done wrong the first time I checked Date types, but oh well.
I still don't understand the syntax I originally asked, so if someone wants to help by providing something to read, please and thank you.
I have a model band that contains a list of tours
Band:
{
name: String,
email: String,
createdAt: String,
tours: Tour[],
...
}
where a Tour is:
{
name: String,
region: String,
published: Boolean,
...
}
The goal is simply to create an end point that receives a Band Name and Tour Name deletes a tour based on this input.
The following works:
bandService.getBandByName(req.getParam("bandName")).flatMap{ b =>
val tour = b.tours.filter(t => t.name == req.getParam("tourName")).head
mongoDataBaseConnector.bands.findOneAndUpdate(
equal("bandName", req.getParam("bandName")),
pull("tours", tour)
).toFuture().flatMap(u => bandService.getBandByName(req.getParam("bandName")))
However, this requires me to first resolve the band by the name received first, filter, find the tour and pass in the exact object in to the pull I am trying to avoid this by using pullByFilter but can't seem to get this to work. Unfortunately couldn't find any examples of this function in the scala driver.
This is what I am trying:
mongoDataBaseConnector.bands.findOneAndUpdate(
and(
equal("bandName", req.getParam("bandName")),
equal("tours.name", req.getParam("tourName"))),
pullByFilter(and(
equal("tours.$.name", req.getParam("tourName")),
equal("tours.$.region", req.getParam("region"))
))
).toFuture().flatMap(u => bandService.getBandByName(req.getParam("bandName")))
this gives the following error:
com.mongodb.MongoCommandException: Command failed with error 2 (BadValue): 'Cannot apply $pull to a non-array value' on server cluster0-shard-00-01-sqs4t.mongodb.net:27017. The full response is {"operationTime": {"$timestamp": {"t": 1568589476, "i": 8}}, "ok": 0.0, "errmsg": "Cannot apply $pull to a non-array value", "code": 2, "codeName": "BadValue", "$clusterTime": {"clusterTime": {"$timestamp": {"t": 1568589476, "i": 8}}, "signature": {"hash": {"$binary": "Qz/DqAdG11H8KRkW8gtvRAAE61Q=", "$type": "00"}, "keyId": {"$numberLong": "6710077417139994625"}}}}
Any ideas are appreciated. Is this even possible with this function?
Okay, I've been working on a nearly identical problem all day. What I have isn't great, but it's enough that I'm willing to accept it and hopefully someone will be able to make it into a more fully-formed solution. Or at least I can save someone else some time.
pullByFilter only accepts one argument. It seems that argument can be another Bson filter which is applied directly to each of the children, or a BsonValue that will be matched against. Since none of the Bson filters step into the child documents, you have to create a BsonValue, specifically, a document. I believe this should solve your problem:
mongoDataBaseConnector.bands.findOneAndUpdate(
and(
equal("bandName", req.getParam("bandName")),
equal("tours.name", req.getParam("tourName"))),
pullByFilter(BsonDocument().append("tours", BsonDocument()
.append("name", req.getParam("tourName"))
.append("region", req.getParam("region"))
))
).toFuture().flatMap(u => bandService.getBandByName(req.getParam("bandName")))
I would have liked to be able to switch back using Bson filters after getting to the inner document, but it seems that once you're in BsonValues, you can't go back.
Edit: To help whoever comes after me: I think there should be a way to solve this more cleanly with arrayFilters, but I haven't yet found it.
When I tried to update the datatype string to ObjectId (mongodb 3.2.12 version), I got the error Invalid ObjectID length and it converts half of the data as ObjectID. How can I get rid of this error? I have 25k data to update.
db.booking.find({user: {$exists:true}}).forEach( function(x) {
x.user = ObjectId(x.user);
db.booking.update({_id: x._id}, {$set: {user: x.user}});
});
The error suggests that the data is not actually valid for an ObjectId, or at least some of it is. If you think it is okay you can always weed out the invalid length entries by adding $where to the query .find({ user: { $exists: true }, $where: 'this.user.length === 24' }) And then deal with anything that does not meet the length requirement of the hex string required as a separate issue. Bottom line is you have some bad data you need to look at separately
I tried to use positional $ operator to update the value of the object's field in the array, and used the example from the Mongo docs ( docs.mongodb.com/manual/reference/operator/update/positional/ ) - but that example did not work in meteor. Does it my error or some restrictions of mongo in meteor? Here is the code:
if (Meteor.isServer) {
console.info( "Create collection");
students = new Mongo.Collection('students');
console.info( "Clear collection from old data");
students.remove();
//***************************************************
console.info( "Insert data to collection");
if (students.find().count() === 0) {
students.insert({
_id: 4,
grades: [
{ grade: 80, mean: 75, std: 8 },
{ grade: 85, mean: 90, std: 5 },
{ grade: 90, mean: 85, std: 3 }
]
});
}
//***************************************************
console.info( "Update data in collection");
// Example from https://docs.mongodb.com/manual/reference/operator/update/positional/
// Use the positional $ operator to update the value of the std field in the embedded document with the grade of 85:
students.update(
{ _id: 4, "grades.grade": 85 },
{ $set: { "grades.$.std" : 6 } }
);
console.info( "Update data in array with id 4 in collection - OK");
//***************************************************
console.info( "See updated data");
console.info("New data in the field std must be 6 but = " + students.findOne({_id: "4"}).grades[1].std);}
The result is the same old value "5" without any error message. What may be the reason that example from Mongo doc does not work?
This answer is based on the info you gave in the comments above. You state that the code started working perfectly as soon as you put quotation marks in and ask if this is some special Meteor or JavaScript feature. The answer is no, it isn't a special feature
The reason you need quotation marks to make the code work is that you are saving your number as Strings instead of Numbers. You need to refactor your insertion/update code to make sure you are inserting 85 as a number not a string. Then you can remove the quotation marks and the code will work.
You may think there's no need to do this as everything is working as expected but if the numbers are being saved as strings then performing maths on them and querying them using $gt or $lt later will be affected.
To save the numbers as numbers instead of strings just make sure there are no quotation marks when you insert/update them. It seems your current code doesn't have quotation marks but your comment states that it did previously. So reset your DB by running meteor reset, the insertion script will run and then if you remove the quotation marks you just added the code will run as expected.