let say i have this huge documents.
2 of them got this array of object;
{
status: "A",
group: "public",
"created.dt": ....
}
{
status: "A",
group: "private",
"created.dt": ....
}
i indexed and ensure like this :
db.collection.ensureIndex({"created.dt":-1});
db.collection.ensureIndex({"created.dt":-1, "status":1});
db.collection.ensureIndex({"created.dt":-1, "group":1});
db.collection.ensureIndex({"created.dt":-1, "status":1, "group":1});
Query:
db.collection.find(
{
"status": {
$in: ["A", "I"]
},
"asset_group": "public"
},
{
sort: {
'created.dt':1
}
}
).count();
is it wrong ?
after i make this index still slow.
please help me proper index.thank you
for the following query:
db.collection.find(
{
"status": {
$in: ["A", "I"]
},
"asset_group": "public"
},
{
sort: {
'created.dt':1
}
}
).count();
The best index will be this:
db.collection.ensureIndex({"status":1, "asset_group":1, "created.dt":1});
or
db.collection.ensureIndex({"asset_group":1, "status":1, "created.dt":-1});
Since you are querying on
status, asset_group - these values can be switched in the index prefix
and sort on created.dt field - therefore created.at shuold be the last value in the index prefix. Note: On sort the index can traverse the reverse order.
For other queries, other indexes might be more suitable.
Read more about compound indexes.
Related
I would like to update each elements (object) in an array of a company.
Here my actual data :
{
_id: ObjectId("60d31024860ce0400b586111")
contracts:
[
{
name: 1.pdf
url: "https://someurl"
createdAt: 2021-06-23T10:42:44.594+00:00
}
{
name: 2.pdf
url: "https://someurl"
}
{
name: 3.pdf
url: "https://someurl"
}
]
}
I would like to add a defined date on each object (in contracts) that has no "updatedAt" key.
Here what I tried :
db.companies.update({ _id: ObjectId("60d31024860ce0400b586111"),"contracts.createdAt": { $exists: false } },{ $set: { "contracts.$.createdAt": "test" } })
but I got this error :
"The positional operator did not find the match needed from the query."
I have also tried this and it works, but I don't wanna query by file name. I just wanna add "createdAt" on each elements found that has no "createdAt"
db.companies.update({ "contracts.name": "2.pdf" },{ $set: { "contracts.$.createdAt": "atest" } })
I think you need to use the filtered position operator:
$ - updates the first matched array element
$[] - updates all the matched elements with a specific condition
The specific condition is mentioned in the arrayFilters key.
db.students.update(
{ },
{ $set: { "contracts.$[element].createdAt" : "atest"} },
{ multi: true,
arrayFilters: [ { "element.createdAt": { $exists: false } } ]
}
)
multi - true is to apply the operation on all the matching documents.
Also notice, how the first query parameter is empty, which means the query runs for all documents. I used it based on the second query you wrote but you can also add in an ObjectID query there.
I have the following collection:
{
_id: 12345,
quizzes: [
{
_id: 111111,
questions: []
}
]
},
{
_id: 78910,
quizzes: [
{
_id: 22222
}
]
}
I want to select the documents of a certain quiz from the quizzes that do not have the questions array and want to make sure that it uses the appropriate questions index. So I use the following query:
Answer.find({ 'quizzes.0.questions': { $exists: false } }).explain('queryPlanner');
Which returns:
{
queryPlanner: {
plannerVersion: 1,
namespace: 'iquiz.answers',
indexFilterSet: false,
parsedQuery: { 'quizzes.0.questions': [Object] },
winningPlan: { stage: 'COLLSCAN', filter: [Object], direction: 'forward' },
rejectedPlans: []
}
}
The query is not using any index as seen from the output. I have tried the following indexes and none get used:
{ quizzes.$**: 1 }
{ quizzes.questions: 1 }
{ quizzes.[$**].questions: 1 }
{ quizzes: 1 }
The only 1 that actually gets used:
{ quizzes.0.questions: 1 }
However this is not really practical as I may target any quiz from the quizzes array not just the first one. Is there a certain syntax for the index in my case or this is a current limitation of mongodb? Thanks!
Indexes generally do not help to answer queries in the form of "X does not exist". See also mongodb indexes covering missing values.
To verify whether the index is used, look up some data using a positive condition.
I am trying to construct a query that will accept multiple fields that can be searched over using regex for partial field matching that also has a hard constraint on other fields.
Example:
Collection: "Projects"
Required Information: { propertyId: "abc", clientId: "xyz" }
Fields to be Searched: name, serviceType.name, manager.name
Currently, I have a query like this, but if there are no results it returns all the results, which isn't helpful.
{
'$and': [
{ propertyId: '7sHGCHT4ns6z9j6BC' },
{ clientId: 'xyz' },
{ '$or':
[
{ name: /HVAC/gi },
{ 'serviceType.name': /HVAC/gi },
{ 'manager.name': /HVAC/gi }
]
}
]
}
If anyone has any insight into this it would be much appreciated.
Example Document:
{
_id: "abc",
propertyId: "7sHGCHT4ns6z9j6BC",
clientId: "xyz"
name: "16.000.001",
serviceType: {
_id: "asdf",
name: "HVAC"
},
manager: {
_id: "dfgh",
name: "Patrick Lewis",
}
}
The expected result is to only find documents where propertyId = 7sHGCHT4ns6z9j6BC AND one at least one of the following keys: name, serviceType.name, or manager.name match an inputted string, in this case, it's HVAC and if none of the regex fields match, then return nothing.
UPDATE
The issue was with MongoDB, after restarting it, everything worked.
Try following script:
db.collection.find({
$and:[
{propertyId:"7sHGCHT4ns6z9j6BC"},
{
$or:[
{name: /HVAC/i},
{"serviceType.name": /HVAC/i},
{"manager.name": /HVAC/i}
]
}]
})
Query above will return a document or documents if and only if propertyId matches and either of name, serviceType.name or manager.name matches desired regex.
Is there a way to conditionally $addToSet based on a specific key field in a subdocument on an array?
Here's an example of what I mean - given the collection produced by the following sample bootstrap;
cls
db.so.remove();
db.so.insert({
"Name": "fruitBowl",
"pfms" : [
{
"n" : "apples"
}
]
});
n defines a unique document key. I only want one entry with the same n value in the array at any one time. So I want to be able to update the pfms array using n so that I end up with just this;
{
"Name": "fruitBowl",
"pfms" : [
{
"n" : "apples",
"mState": 1111234
}
]
}
Here's where I am at the moment;
db.so.update({
"Name": "fruitBowl",
},{
// not allowed to do this of course
// "$pull": {
// "pfms": { n: "apples" },
// },
"$addToSet": {
"pfms": {
"$each": [
{
"n": "apples",
"mState": 1111234
}
]
}
}
}
)
Unfortunately, this adds another array element;
db.so.find().toArray();
[
{
"Name" : "fruitBowl",
"_id" : ObjectId("53ecfef5baca2b1079b0f97c"),
"pfms" : [
{
"n" : "apples"
},
{
"n" : "apples",
"mState" : 1111234
}
]
}
]
I need to effectively upsert the apples document matching on n as the unique identifier and just set mState whether or not an entry already exists. It's a shame I can't do a $pull and $addToSet in the same document (I tried).
What I really need here is dictionary semantics, but that's not an option right now, nor is breaking out the document - can anyone come up with another way?
FWIW - the existing format is a result of language/driver serialization, I didn't choose it exactly.
further
I've gotten a little further in the case where I know the array element already exists I can do this;
db.so.update({
"Name": "fruitBowl",
"pfms.n": "apples",
},{
$set: {
"pfms.$.mState": 1111234,
},
}
)
But of course that only works;
for a single array element
as long as I know it exists
The first limitation isn't a disaster, but if I can't effectively upsert or combine $addToSet with the previous $set (which of course I can't) then it the only workarounds I can think of for now mean two DB round-trips.
The $addToSet operator of course requires that the "whole" document being "added to the set" is in fact unique, so you cannot change "part" of the document or otherwise consider it to be a "partial match".
You stumbled on to your best approach using $pull to remove any element with the "key" field that would result in "duplicates", but of course you cannot modify the same path in different update operators like that.
So the closest thing you will get is issuing separate operations but also doing that with the "Bulk Operations API" which is introduced with MongoDB 2.6. This allows both to be sent to the server at the same time for the closest thing to a "contiguous" operations list you will get:
var bulk = db.so.initializeOrderedBulkOp();
bulk.find({ "Name": "fruitBowl", "pfms.n": "apples": }).updateOne({
"$pull": { "pfms": { "n": "apples" } }
});
bulk.find({ "Name": "fruitBowl" }).updateOne({
"$push": { "pfms": { "n": "apples", "state": 1111234 } }
})
bulk.execute();
That pretty much is your best approach if it is not possible or practical to move the elements to another collection and rely on "upserts" and $set in order to have the same functionality but on a collection rather than array.
I have faced the exact same scenario. I was inserting and removing likes from a post.
What I did is, using mongoose findOneAndUpdate function (which is similar to update or findAndModify function in mongodb).
The key concept is
Insert when the field is not present
Delete when the field is present
The insert is
findOneAndUpdate({ _id: theId, 'likes.userId': { $ne: theUserId }},
{ $push: { likes: { userId: theUserId, createdAt: new Date() }}},
{ 'new': true }, function(err, post) { // do the needful });
The delete is
findOneAndUpdate({ _id: theId, 'likes.userId': theUserId},
{ $pull: { likes: { userId: theUserId }}},
{ 'new': true }, function(err, post) { // do the needful });
This makes the whole operation atomic and there are no duplicates with respect to the userId field.
I hope this helpes. If you have any query, feel free to ask.
As far as I know MongoDB now (from v 4.2) allows to use aggregation pipelines for updates.
More or less elegant way to make it work (according to the question) looks like the following:
db.runCommand({
update: "your-collection-name",
updates: [
{
q: {},
u: {
$set: {
"pfms.$[elem]": {
"n":"apples",
"mState": NumberInt(1111234)
}
}
},
arrayFilters: [
{
"elem.n": {
$eq: "apples"
}
}
],
multi: true
}
]
})
In my scenario, The data need to be init when not existed, and update the field If existed, and the data will not be deleted. If the datas have these states, you might want to try the following method.
// Mongoose, but mostly same as mongodb
// Update the tag to user, If there existed one.
const user = await UserModel.findOneAndUpdate(
{
user: userId,
'tags.name': tag_name,
},
{
$set: {
'tags.$.description': tag_description,
},
}
)
.lean()
.exec();
// Add a default tag to user
if (user == null) {
await UserModel.findOneAndUpdate(
{
user: userId,
},
{
$push: {
tags: new Tag({
name: tag_name,
description: tag_description,
}),
},
}
);
}
This is the most clean and fast method in the scenario.
As a business analyst , I had the same problem and hopefully I have a solution to this after hours of investigation.
// The customer document:
{
"id" : "1212",
"customerCodes" : [
{
"code" : "I"
},
{
"code" : "YK"
}
]
}
// The problem : I want to insert dateField "01.01.2016" to customer documents where customerCodes subdocument has a document with code "YK" but does not have dateField. The final document must be as follows :
{
"id" : "1212",
"customerCodes" : [
{
"code" : "I"
},
{
"code" : "YK" ,
"dateField" : "01.01.2016"
}
]
}
// The solution : the solution code is in three steps :
// PART 1 - Find the customers with customerCodes "YK" but without dateField
// PART 2 - Find the index of the subdocument with "YK" in customerCodes list.
// PART 3 - Insert the value into the document
// Here is the code
// PART 1
var myCursor = db.customers.find({ customerCodes:{$elemMatch:{code:"YK", dateField:{ $exists:false} }}});
// PART 2
myCursor.forEach(function(customer){
if(customer.customerCodes != null )
{
var size = customer.customerCodes.length;
if( size > 0 )
{
var iFoundTheIndexOfSubDocument= -1;
var index = 0;
customer.customerCodes.forEach( function(clazz)
{
if( clazz.code == "YK" && clazz.changeDate == null )
{
iFoundTheIndexOfSubDocument = index;
}
index++;
})
// PART 3
// What happens here is : If i found the indice of the
// "YK" subdocument, I create "updates" document which
// corresponds to the new data to be inserted`
//
if( iFoundTheIndexOfSubDocument != -1 )
{
var toSet = "customerCodes."+ iFoundTheIndexOfSubDocument +".dateField";
var updates = {};
updates[toSet] = "01.01.2016";
db.customers.update({ "id" : customer.id } , { $set: updates });
// This statement is actually interpreted like this :
// db.customers.update({ "id" : "1212" } ,{ $set: customerCodes.0.dateField : "01.01.2016" });
}
}
}
});
Have a nice day !
It looks like you(/I ) cannot have both upsert and an array element update operation.
If you do (python):
findDct = {
"_id": ObjectId("535e3ab9c36b4417d031402f"),
'events.ids': '176976332'
}
print col.update(findDct, {"$set" : {"events.$.foo": "bar"} }, upsert=True)
It will throw:
pymongo.errors.DuplicateKeyError: insertDocument :: caused by :: 11000 E11000
duplicate key error index: test.col.$_id_ dup key: { : ObjectId('535e3ab9c36b4417d031402f') }
This happens because "_id" is of course an index and mongo tries to insert the document as a new since the find query fails on its 'events.ids': '176976332' part (cheat).
Is it possible to update an unknown element in array with upsert True/how?
Yes it is, but you are going about it in the wrong way. Rather than make "finding" the element that you are not sure whether it exists or not, then try to apply the $addToSet operator instead:
db.collection.update(
{ "_id": ObjectId("535e3ab9c36b4417d031402f" },
{
"$addToSet": { "events": { "foo": "bar" } }
},
{ "upsert": true }
)
Please also note from the positional $ operator documentation that you should not use the $ operator with "upserts" as this will result in the field name being interpreted as a "literal" ( which includes the value as in "events.$.foo" ) and that will be the actual field inserted into the document.
Try to make sure that your array "insert/upsert" operations specify the whole array content in order to make this work.
Another adaptation is with the "bulk" methods, the pymongo driver already has a nice API for this, but this is a general form:
db.runCommand({
"update": "collection",
"updates": [
{
"q": { "_id": ObjectId("535e3ab9c36b4417d031402f" } },
"u": {
"$addToSet": {
"events": {
"foo": "bar", "bar": "baz"
}
}
},
"upsert": true
},
{
"q": { "_id": ObjectId("535e3ab9c36b4417d031402f" } },
"u": {
"$set": { "events.foo": "bar" }
}
}
]
})
But still being very careful that you are not producing duplicates in your sub-document array if you can clearly see the case there. But it is a method, as each update will cascade down even if the first form failed to add anything. Not the best case example, but I hope you see the point.