Pymongo Find & Update problems - mongodb

Background
I'm making an application which takes an input through a Discord bot and then puts the input in a Google Document. During that process, I'm using Flask with Pymongo to find documents whose isNew property is set to true and send it back to Google Docs script. After that's done I want to set the isNew property of the documents to set to false. Alternatively, I can set the isNew to false before I send, since i can't call a function successfully, after I have returned the response in Flask.
Based on what I've experimented with, whenever I get new documents using a <CollectionName>.find() request, and tried to update the isNew property subsequently using a CollectionName.update_many() command, it results in the variable storing the original find command documents to go blank.
Code Attempts:
1. update many while trying to return new documents
import pymongo
# mongodb db connection string
connectionString = ('mongodb://localhost:27017')
client = pymongo.MongoClient(connectionString)
db = client['tempBotDB']
# 3 types of questions: quiz, cross, hunt
# separate collections for each type of strings
quizQs = db['quizQs']
crossQs = db['crossQs']
huntQs = db['huntQs']
# Properties of a question: isnew, type, question, answer
def saveQ(qObj): # Saves a question to the db
qType = qObj.get('type')
# creating own qObj
q = {
'question': qObj.get('question'),
'answer': qObj.get('answer'),
'isNew': bool(True)
}
if qType == 'quiz':
result = quizQs.insert_one(q)
elif qType == 'cross':
result = crossQs.insert_one(q)
elif qType == 'hunt':
result = huntQs.insert_one(q)
else:
result = ('Error: "type" of question can only be'
'one of : cross, hunt, quiz')
return result
def getQs(qType): # Gets all the questions by given type
if qType == 'quiz':
result = quizQs.update_many(
{'isNew': bool(True)},
{'$set': {'isNew': bool(False)}},
{returnOriginal: True}
)
print(result)
else:
result = ('Error: "type" of question can only be'
'one of : cross, hunt, quiz')
getQs('quiz')
Error: NameError: name 'returnOriginal' is not defined
Also tried with returnNewDocument instead of returnOriginal. But got: NameError: name 'returnNewDocument' is not defined
Using a find followed by an update within the same function
def getQs(qType): # Gets all the questions by given type
if qType == 'quiz':
selector = {'isNew': False}
modifier = {'$set': {'isNew': True}}
result = quizQs.find(selector)
#fakeRes = quizQs.update_many(selector, modifier)
print('res', result)
for i in result:
print('result', i)
#print('fakeres', fakeRes)
# for i in fakeRes:
# print('FakeRes', i)
else:
result = ('Error: "type" of question can only be'
'one of : cross, hunt, quiz')
If I uncomment the update command, the result of the find command becomes uniteratable (I get an error message), which to the best of my knowledge means that it's empty. Here's what happens:
def getQs(qType): # Gets all the questions by given type
if qType == 'quiz':
selector = {'isNew': False}
modifier = {'$set': {'isNew': True}}
result = quizQs.find(selector)
fakeRes = quizQs.update_many(selector, modifier)
print('res', result)
for i in result:
print('result', i)
print('fakeres', fakeRes)
for i in fakeRes:
print('FakeRes', i)
else:
result = ('Error: "type" of question can only be'
'one of : cross, hunt, quiz')
I get an the following output:
res <pymongo.cursor.Cursor object at 0x10e34ca50>
fakeres <pymongo.results.UpdateResult object at 0x10e35bfa0>
Traceback (most recent call last):
File "db.py", line 63, in <module>
getQs('quiz')
File "db.py", line 55, in getQs
for i in fakeRes:
TypeError: 'UpdateResult' object is not iterable
Notice how iterating through the result object did not print anything, pointing to the fact that it goes empty.
So, to conclude, my main question is:
How do I successfully find and update multiple documents in mongodb, while retrieving the update documents, OR just the found documents (not updated) [EITHER of them will work as i won't need to access the property I update]
Lastly, would it be logical to first do a find and then update_one on each document while generating a list of dictionaries of the updated documents and then finally returning that list to the user?
I'd really appreciate any help.

There's a few ways to do this, but you can't get the list of updated documents from an update_many - Mongo doesn't support that.
So your best approach is:
for record in db.mycollection.find({'isNew': False}):
result = db.mycollection.update_one({'_id': record['_id']}, {'$set': {'isNew': True}})
print(record)
# Add in logic here to assemble records for return to the user

Related

$sum inside aggregate in mongomock seems not to be working

I have an aggregate Mongo query that projects some fields and calculates two other ones using $sum. The query works as expected, so I created an unit test for it, and to my surprise the test was failing.
I created a minimal, complete, and verifiable example to test my hypothesis that this was a problem with MongoMock, and it seems to be!
Here is the code:
import mongoengine as mongo
from mongoengine import connect
from mongoengine.queryset import QuerySet
class ValuesList(mongo.EmbeddedDocument):
updated_value = mongo.DecimalField()
class ValuesHistory(mongo.Document):
name = mongo.StringField()
base_value = mongo.DecimalField()
values_list = mongo.EmbeddedDocumentListField(ValuesList, required=False)
meta = {
'collection' : 'values_history'
}
def __str__(self):
return 'name: {}\nbase_value: {}\n'.format(self.name, self.base_value)
def migrate_data(new_collection):
ValuesHistory.objects.aggregate(
{'$project': {'name': 1,
'base_value': {'$sum': ['$base_value', {'$arrayElemAt': ['$values_list.updated_value', -1]}]}
}
},
{'$out': "{}".format(new_collection)}
)
def clear_tables_and_insert_test_data(db):
db.test.values_history.drop()
db.test.updated_values.drop()
ValuesHistory(name='first',
base_value=100,
values_list=[ValuesList(updated_value=5),
ValuesList(updated_value=15)]).save()
def run_aggregate_query_with_db(db):
new_collection_name = 'updated_values'
migrate_data(new_collection_name)
new_group = ValuesHistory.switch_collection(ValuesHistory(), new_collection_name)
aggregated_values = QuerySet(ValuesHistory, new_group._get_collection()).all()
for value in aggregated_values:
print(value)
db.close()
A quick explanation about the code above.
ValuesHistory is a class that contains a String name, a numeric base_value and a list of values (ValuesList class).
The method clear_tables_and_insert_test_data clears the two tables used in this test and inserts some test data.
The query in migrate_data method creates a new collection (through the $out operator) and the base_value of the newly created collection should be the sum of the current value and the last value in the values_list list. In my case it should be 115 (being 100 the current value and 15 the last value on the list).
If I run the code using a connection to my local MongoDB, like this:
if __name__ == '__main__':
db = connect('test') # connect to real instance of Mongo
clear_tables_and_insert_test_data(db)
run_aggregate_query_with_db(db)
I get 115 as a result, which is exactly what is expected.
If I, instead, use a connection to MongoMock:
if __name__ == '__main__':
db = connect('test', host='mongomock://localhost') # Connect to MongoMock instance
clear_tables_and_insert_test_data(db)
run_aggregate_query_with_db(db)
I get 100 as result, which is odd! Looks like the $sum operator did not do it's job properly, since the sum of 100 and 15 resulted in 100!
EDIT: I also tried using the $add operator, but the problem remains the same, yielding 100 when it should be 115.
TL;DR;
Question: How should I use $sum (or $add) inside an aggregate pipeline on MongoMock so that it yields the correct value?
The problem described was, indeed, a bug on Mongomock that existed up to version 3.14.0.
After the original question was posted, I opened an issue on Mongomock's github describing the problem. Shortly after it was fixed and version 3.15.0 has been released a few days ago. I ran the code on the question and the issue is now solved for both $add and $sum operators!
TL;DR
Updating to Mongomock 3.15.0 is enough to solve the problem.

MongoDb script : Cannot compare two types correctly [duplicate]

I have a straightforward tool for building collections of documents and then automatically formatting them for EPUB or LaTeX rendering, written on top of ExpressJS. I'm using Coffeescript, if that matters (I doubt it).
Using Mongoose, I have the following:
DocumentSchema = new Schema
title: String
Offrefs = new Schema
ref: { type: ObjectId }
isa: String
BinderSchema = new Schema
title: String
contains: [Offrefs]
Offrefs doesn't specify what it refers to because because I want to be able to contain some binders in other binders, to create logical collections: "These are for the printer," "These are for epub," "These are web only," etc. (I've stripped out all the miscellaneous stuff out.)
Unfortunately, I have run into queries where, for retrieved objects
(story._id == offref.ref) -> True
And the two do indeed look the same. But:
(binder._id == offref.ref) -> False
(String(binder._id) == String(offref.ref)) -> True
And a visual comparison of the two references in the last two, they are the same ID number, but the ObjectId objects don't compare correctly.
I don't want to have to do string conversions constantly, which is a strong possiblity when I'm converting these complex objects into trees of data. Tree relationships are a bear in any DB; they shouldn't be difficult in MongoDB.
How do you do ObjectId comparisons in MongoDB?
A straight == (or ===) comparison will compare the two objects by reference, not value. So that will only evaluate to true if they both reference the very same instance.
Instead, you should be using the equals method of ObjectID to compare their values:
story._id.equals(offref.ref)
As #bendytree notes in the comments, if either value could be null (and you want nulls to compare as equal), then you can use the following instead:
String(story._id) === String(offref.ref)
This goes somewhat beyond the original asked question, but I have found that the .equals method of ObjectID's will return false in some cases where a string comparison will return true even when the values are not null. Example:
var compare1 = invitationOwningUser.toString() === linkedOwningUser.toString();
var compare2 = invitationOwningUser.equals(linkedOwningUser);
var compare3 = String(invitationOwningUser) === String(linkedOwningUser);
logger.debug("compare1: " + compare1 + "; " + "compare2: " + compare2 + "; " + "compare3: " + compare3);
Output:
compare1: true; compare2: false; compare3: true
This occurred when invitationOwningUser (an ObjectID) came from a Collection created using a Mongoose schema, and linkedOwningUser (also an ObjectID) came from a Collection not created using Mongoose (just regular MongoDB methods).
Here is the document containing invitationOwningUser (the owningUser field):
{
"_id" : ObjectId("5782faec1f3b568d58d09518"),
"owningUser" : ObjectId("5781a5685a06e69b158763ea"),
"capabilities" : [
"Read",
"Update"
],
"redeemed" : true,
"expiry" : ISODate("2016-07-12T01:45:18.017Z"),
"__v" : 0
}
Here is the document containing linkedOwningUser (the owningUser field):
{
"_id" : ObjectId("05fb8257c95d538d58be7a89"),
"linked" : [
{
"owningUser" : ObjectId("5781a5685a06e69b158763ea"),
"capabilities" : [
"Read",
"Update"
]
}
]
}
So, as a bottom line for me, I'll be using the string comparison technique to compare ObjectID's, not the .equals method.

Mongo db conditional query

being a newbie to mongo, stuck in a conditional query:
I want to perform a search on the basis of 3 criteria, first name, last name and email id:
below query works perfect when all the fields exist:
db.students.find( { $and: [ { first_name: /^Abc$/i }, { last_name: 'Xyz'},{email_id:'gd#he'} ]})
the problem is when I don't give an email id , the query dosen't returns any result as it considers the email id to be null and searches for the combination 'Abc Firtis null',
where as I want the below scenario to be fulfilled:
I have a collection of students:
- FirstName: 1. ABC 2. ABC 3.ABC
- LastName: 1.XYZ 2. XY 3. XZ
- EmailID: 1.abc#xyz 2.Ab#xy 3.Ab#xz
if one enters only the first name in the search it should return all the 3 results
if user enters first name and last name it should return first two results and if the user enters all three details it should return only 1 result.
Any leads would be highly appreciated.
You seem to be talking about "input" data being different for the queries you want to issue and how to contruct the query to ignore fields as criteria for which you have no input.
This is all really about how the input is being collected as to how you handle it, but it all boils down to that you "conditionally build" the query ( which is just a data structure anyway ) rather than statically define a query and somehow ignore null or empty data.
So if you have seperate variables, then you test each value and build the query:
var firstName = "abc",
lastName = "xy,
email = null,
query = {};
if (firstName) {
query.firstName = new RegExp("^"+firstName,"i")
}
if (lastName) {
query.lastName = new RegExp("^"+lastName,"i")
}
if (email) {
query.email = new RegExp("^"+email,"i")
}
db.students.find(query)
This would build a query object that would end up like this based on the inputs:
{ "firstName": /^abc/i, "lastName": /^xy/i }
So since there is no value in email then the condition is not included. The end result is the condition not provided is not even queried for and then you get the relevant matches.
The same approach is basically simplified if you have some structured input to begin with:
var params = {
firstName = "abc",
lastName = "xy"
};
var query = {};
Object.keys(params).forEach(function(key) {
if (params[key])
query[key] = new RegExp("^"+key,"i");
});
db.students.find(query);
And it's the same thing, but since you have all parameter keys in one place then you can iterate them to build the query rather than test individually.
This is generally the case where you have input from something like a web request with parameters that come into req.params or even req.body depending on your method of input. So if you structure your code to accept input into a similar object ( or already have it ) then you use it to build your query.
Also note that all MongoDB query arguments are implicitly an "AND" condition by definition, so there is rarely any need to use $and unless you explicitly have multiple conditions to meet for the same document property. Even then there are generally better syntax alternates.
No Need to give and you can simply try this
find( { first_name: { $regex: '/^Abc$/i' }, last_name:'Xyz',email_id:'gd#he'}

Document field names can't start with '$' (Bad Key: '$set')

BasicDBObject u = new BasicDBObject();
DBObject q = new BasicDBObject();
q.put("orgId", orgId);
u.append("orgId", orgId);
if(accounts.keySet().size() > 0) {
BasicDBObject setObj = new BasicDBObject();
for (String key : accounts.keySet()) {
String fieldToUpdate = "accounts." + key;
setObj.append(fieldToUpdate, accounts.get(key));
}
u.append("$set", setObj);
}
DBCollection collection = db.getCollection(collectionName);
WriteResult result = collection.update(q, u, true, false);
I get following error, what is wrong:
java.lang.RuntimeException: java.lang.IllegalArgumentException: Document field names can't start with '$' (Bad Key: '$set')
u.toString output is :
{ "orgId" : 2 , "$set" : { "accounts.001" : "EXAMPLE"}}
You should not have the { "orgId" : 2 } in the update document.
Remove this line from the code and it should work fine.
u.append("orgId", orgId);
The reason you triggered the error was that there are two ways to specify the update for a document and you created a cross bread of both. The options are:
Provide the complete document for the update. For this model the existing document is overwritten by the provided document.
Uses update operators to modify the existing document in the collection.
If you use the second version then all of the "top level keys" in the update document will start with a $. If you use the first option then none of the top level keys will start with a $. The code looked at the first field, thought it was a replacement document and then failed when it tried to validate the rest of the document was valid since keys in documents cannot start with a $ (lest the be confused with update or query documents).
Edit:
In the case of an upsert (e.g., the document does not already exist and you flag the update to allow the upsert) query's exact match operators are used to seed the document. For the above example we get a seed document of { "orgId" : 2 }. The server will then apply the update operators and save the result.

get number of updated documents mongo

Is there a function in mongo to return the number of documents that were updated in an update statement?
I have tried to use count() but apparently the update statement only returns true or false so i think I'm getting the count of a string.
Thanks
Use the getLastError command to get information about the result of your operation.
I don't know the ruby driver, but most drivers automatically do this in 'safe mode'. In safe mode, each write will examine the result of getLastError to make sure the write was successful. The update operation should return an object that looks like the JSON object further down, and it includes the number of updated documents (n). You can fine-tune the safe mode settings, but be warned that the default mode is "fire and forget", so safe mode is a good idea for many use cases.
In the shell,
> db.customers.update({}, {$set : {"Test" : "13232"}}, true, true);
> db.runCommand( "getlasterror" )
{
"updatedExisting" : true,
"n" : 3,
"connectionId" : 18,
"err" : null,
"ok" : 1
}
Here, I updated n = 3 documents. Note that by default, update operations in mongodb only apply to the first matched document. In the shell, the fourth parameter is used to indicate we want to update multiple documents.
For future reference as of the latest version of mongoDB, you can save the result of the update command and look for the property modifiedCount. Note that if the update operation results in no change to the document, such as setting the value of the field to its current value, it may be less than the number of documents that were matched in the query.
For example (in JS):
var updateResult = await collection.updateOne( .... )
if(updateResult.modifiedCount < 1) throw new Error("No docs updated");
There are also two fields result.n and result.nModified in the response object that tell you the number of documents matched, and the number updated, respectively. Here's a portion of the response object that mongoDB returns
result: {
n: 1,
nModified: 1,
...
},
...
modifiedCount: 1
More info here: https://docs.mongodb.com/manual/reference/command/update/