Query - Not to display data where restrict_id = 13546456566
Suppose we have a collection and in it a field name restrict_id with value 13546456566 i want not to display data any data before restrict_id have value from next it will display how can i make this query mongodb
{
"_id": ObjectID("5e9eb49efc81c83087396ff2"),
"group_id": ObjectID("5e4e755b380054797d9db627"),
"date": 1587459230192,
"text": "First"
}
{
"_id": ObjectID("5e9eb49efc81c83087396ff2"),
"group_id": ObjectID("5e4e755b380054797d9db627"),
"date": 1587459230192,
"text": "Second"
}
{
"_id": ObjectID("5e9eb49efc81c83087396ff2"),
"group_id": ObjectID("5e4e755b380054797d9db627"),
"restrict_id": "13546456566"
"date": 1587459230192,
"text": "Third"
}
{
"_id": ObjectID("5e9eb49efc81c83087396ff2"),
"group_id": ObjectID("5e4e755b380054797d9db627"),
"date": 1587459230192,
"text": "Fourth"
}
So here my Texture collection say you can skip thinking same Object _id its copied those are different but question is I would like to skip all first three because it has flag restrict_id = 13546456566 only to display fourth text value how this possible in mongodb
The distinction of "has any previously seen document contained this value" is meaningless unless the documents are sorted.
MongoDB does not have any kind of query state variable that could be set to true and persist between documents to indicate that this has happened.
Using aggregation you could:
- sort the documents
- push them all into an array
- reduce the array to build a new array with the condition applied
- unwind the array
- restore the documents to the state they had before
db.collection.aggregate([
{$sort:{sortField:1}},
{$group:{
_id:null,
documents:{$push:"$$ROOT"}
}},
{$project:{
keepers:{
$reduce:{
input:"$documents",
initialValue:{keep:false,documents:[]},
in:{
keep:{$or:[
"$$value.keep",
{$eq:["$$this.restrict_id","13546456566"]}
]},
documents:{
$cond:[
"$$value.keep",
{$concatArrays:["$$value.documents",["$$this"]]},
[]
]}
}
}
}
}},
{$unwind:"$keepers.documents"},
{$replaceRoot:{newRoot:"$keepers.documents"}}
])
Playground
Related
Hi I am trying to increment the count of the matching requirement in an array. My sample collection looks like the following:
{
"_id": ObjectId("60760ba2e870fa518f2ae48b"),
"userId": "6075f7289822d94dca8066b4",
"requirements": [
{
"searchText": "zee5",
"planType": "basic",
"mode": "PRIVATE",
"count": 32.0
},
{
"searchText": "sony",
"planType": "standard",
"mode": "PUBLIC",
"count": 12.0
},
{
"searchText": "prime",
"planType": "premium",
"mode": "PRIVATE",
"count": 2
}
]
}
If a user searches for prime, with filter premium and PRIVATE, then the count of the last requirement should be updated. If he searches for prime, with filter standard and PRIVATE, then the new requirement will be inserted with count 1.
I am doing in two steps. First I fire an update with the following query and then if no update, I fire a push query with count 1:
db.getCollection('userProfile').update({ "$and" : [{ "requirements.searchText" : {$eq:"prime"}}, {"requirements.mode" : {$eq: "PUBLIC"}}, {"requirements.planType": {$eq: "standard"}}, { "userId" : "6075f7289822d94dca8066b4"}]}, {$inc: {"requirements.$.count" : 1}})
I was expecting that the above query will not update any requirement, since there is no exact match. Interestingly, it increments the count of the second requirement with (sony, standard, public). What is wrong with the query? How can I get it right?
Demo - with Update - https://mongoplayground.net/p/-ISXaAayxxv
Demo No update - https://mongoplayground.net/p/88bTj3lz7U_
Use $elemMatch to make sure all properties are present in the same object inside the array
The $elemMatch operator matches documents that contain an array field with at least one element that matches all the specified query criteria.
db.collection.update(
{
"requirements": {
$elemMatch: { "searchText": "prime","mode": "PUBLIC", "planType": "standard" }
},
"userId": "6075f7289822d94dca8066b4"
},
{ $inc: { "requirements.$.count": 1 } }
)
Problem -
Your current query will match any document with all these fields in
requirements array in any object, if they match 1 property in 1 index of the array and another match in the next index query will find the document valid.
"searchText": "prime",
"mode": "PUBLIC",
"planType": "standard"
I am beginner in MongoDB and struck at a place I am trying to fetch data from nested array but is it taking so long time as data is around 50K data, also it is not much accurate data, below is schema structure please see once -
{
"_id": {
"$oid": "6001df3312ac8b33c9d26b86"
},
"City": "Los Angeles",
"State":"California",
"Details": [
{
"Name": "Shawn",
"age": "55",
"Gender": "Male",
"profession": " A science teacher with STEM",
"inDate": "2021-01-15 23:12:17",
"Cars": [
"BMW","Ford","Opel"
],
"language": "English"
},
{
"Name": "Nicole",
"age": "21",
"Gender": "Female",
"profession": "Law student",
"inDate": "2021-01-16 13:45:00",
"Cars": [
"Opel"
],
"language": "English"
}
],
"date": "2021-01-16"
}
Here I am trying to filter date with date and Details.Cars like
db.getCollection('news').find({"Details.Cars":"BMW","date":"2021-01-16"}
it is returning details of other persons too which do not have cars- BMW , Only trying to display details of person like - Shawn which have BMW or special array value and date too not - Nicole, rest should not appear but is it not happening.
Any help is appreciated. :)
A combination of $match on the top-level fields and $filter on the array elements will do what you seek.
db.foo.aggregate([
{$match: {"date":"2021-01-16"}}
,{$addFields: {"Details": {$filter: {
input: "$Details",
as: "zz",
cond: { $in: ['BMW','$$zz.Cars'] }
}}
}}
,{$match: {$expr: { $gt:[{$size:"$Details"},0] } }}
]);
Notes:
$unwind is overly expensive for what is needed here and it likely means "reassembling" the data shape later.
We use $addFields where the new field to add (Details) already exists. This effectively means "overwrite in place" and is a common idiom when filtering an array.
The second $match will eliminate docs where the date matches but not a single entry in Details.Cars is a BMW i.e. the array has been filtered down to zero length. Sometimes you want to know this info so if this is the case, do not add the final $match.
I recommend you look into using real dates i.e. ISODate instead of strings so that you can easily take advantage of MongoDB date math and date formatting functions.
Is a common mistake think that find({nested.array:value}) will return only the nested object but actually, this query return the whole object which has a nested object with desired value.
The query is returning the whole document where value BMW exists in the array Details.Cars. So, Nicole is returned too.
To solve this problem:
To get multiple elements that match the criteria you can do an aggregation stage using $unwind to separate the different objects into array and match by the criteria you want.
db.collection.aggregate([
{
"$match": { "Details.Cars": "BMW", "date": "2021-01-26" }
},
{
"$unwind": "$Details"
},
{
"$match": { "Details.Cars": "BMW" }
}
])
This query first match by the criteria to avoid $unwind over all collection.
Then $unwind to get every document and $match again to get only the documents you want.
Example here
To get only one element (for example, if you match by _id and its unique) you can use $elemMatch in this way:
db.collection.find({
"Details.Cars": "BMW",
"date": "2021-01-16"
},
{
"Details": {
"$elemMatch": {
"Cars": "BMW"
}
}
})
Example here
You can use $elemenMatch into query or projection stage. Docs here and here
Using $elemMatch into query the way is this:
db.collection.find({
"Details": {
"$elemMatch": {
"Cars": "BMW"
}
},
"date": "2021-01-16"
},
{
"Details.$": 1
})
Example here
The result is the same. In the second case you are using positional operator to return, as docs says:
The first element that matches the query condition on the array.
That is, the first element where "Cars": "BMW".
You can choose the way you want.
I have a collection containing objects with the following structure
{
"dep_id": "some_id",
"departament": "dep name",
"employees": [{
"name": "emp1",
"age": 31
},{
"name": "emp2",
"age": 35
}]
}
I would like to sort and save the array of employees for the object with id "some_id", by employees.age, descending. The best outcome would be to do this atomically using mongodb's query language. Is this possible?
If not, how can I rearrange the subdocuments without affecting the parent's other data or the data of the subdocuments? In case I have to download the data from the database and save back the sorted array of children, what would happen if something else performs an update to one of the children or children are added or removed in the meantime?
In the end, the data should be persisted to the database like this:
{
"dep_id": "some_id",
"departament": "dep name",
"employees": [{
"name": "emp2",
"age": 35
},{
"name": "emp1",
"age": 31
}]
}
The best way to do this is to actually apply the $sort modifier as you add items to the array. As you say in your comment "My actual objects have a "rank" and 'created_at'", which means that you really should have asked that in your question instead of writing a "contrived" case ( don't know why people do that ).
So for "sorting" by multiple properties, the following reference would adjust like this:
db.collection.update(
{ },
{ "$push": { "employees": { "$each": [], "$sort": { "rank": -1, "created_at": -1 } } } },
{ "multi": true }
)
But to update all the data you presently have "as is shown in the question", then you would sort on "age" with:
db.collection.update(
{ },
{ "$push": { "employees": { "$each": [], "$sort": { "age": -1 } } } },
{ "multi": true }
)
Which oddly uses $push to actually "modify" an array? Yes it's true, since the $each modifier says we are not actually adding anything new yet the $sort modifier is actually going to apply to the array in place and "re-order" it.
Of course this would then explain how "new" updates to the array should be written in order to apply that $sort and ensure that the "largest age" is always "first" in the array:
db.collection.update(
{ "dep_id": "some_id" },
{ "$push": {
"employees": {
"$each": [{ "name": "emp": 3, "age": 32 }],
"$sort": { "age": -1 }
}
}}
)
So what happens here is as you add the new entry to the array on update, the $sort modifier is applied and re-positions the new element between the two existing ones since that is where it would sort to.
This is a common pattern with MongoDB and is typically used in combination with the $slice modifier in order to keep arrays at a "maximum" length as new items are added, yet retain "ordered" results. And quite often "ranking" is the exact usage.
So overall, you can actually "update" your existing data and re-order it with "one simple atomic statement". No looping or collection renaming required. Furthermore, you now have a simple atomic method to "update" the data and maintain that order as you add new array items, or remove them.
In order to get what you want you can use the following query:
db.collection.aggregate({
$unwind: "$employees" // flatten employees array
}, {
$sort: {
"employees.name": -1 // sort all documents by employee name (descending)
}
}, {
$group: { // restore the previous structure
_id: "$_id",
"dep_id": {
$first: "$dep_id"
},
"departament": {
$first: "$departament"
},
"employees": {
$push: "$employees"
},
}
}, {
$out: "output" // write everything out to a separate collection
})
After this step you would want to drop your source table and rename the "output" collection to match your source table name.
This solution will, however, not deal with the concurrency issue. So you should remove write access from the collection first so nobody modifies it during the process and then restore it once you're done with the migration.
You could alternatively query all data first, then sort the employees array on the client side and then use either single update queries or - faster but more complicated - a bulk write operation with all the individual update calls in order to update the existing documents. Here, you could use the entire document that you've initially read as a filter for the update operation. So if an individual update does not modify any document you'd know straight away, that some other change must have modified the document you read before. Those cases you'd need to retry later (or straight away until the update does actually modify a document).
Currently I'm hitting at a problem to process the mongodb documents and get the field wise values. For example, say mongo contains these documents:
[
{ "name": "test1", "age": 20, "gender": "male" },
{ "name": "test2", "age": 21, "gender": "female" },
{ "name": "test3", "age": 30, "gender": "male"}
]
Expected Output:
{
"name": ["test1","test2","test3"],
"age": [20,21,30],
"gender": ["male","female", "male"]
}
Is it possible to retrieve data from mongo in the above format? I dont want to write some javascript functions to process this. Looking at retrieving the data by using mongo functions along with the find query.
You'd need to use the aggregation framework to get the desired result. Run the following aggregation pipeline which filters the documents in the collection getting into the pipeline for grouping using the $match operator. This is similar to the find() query filter.
db.collection.aggregate([
{ "$match": { "age": { "$gte": 20 } } }, // filter on users with age >= 20
{
"$group": {
"_id": null,
"name": { "$push": "$name" },
"age": { "$push": "$age" },
"gender": { "$push": "$gender" }
}
},
{
"$project": {
"_id": 0,
"name": 1,
"age": 1,
"gender": 1
}
}
])
Sample Output
{
"name": ["test1", "test2", "test3"],
"age": [20, 21, 30],
"gender": ["male", "female", "male"]
}
In the above pipeline, the first pipeline step is the $match operator which is similar to SQL's WHERE clause. The above example filters incoming documents on the age field (age greater than or equal to 20).
One thing to note here is when executing a pipeline, MongoDB pipes operators into each other. "Pipe" here takes the Linux meaning: the output of an operator becomes the input of the following operator. The result of each operator is a new collection of documents. So Mongo executes the previous pipeline as follows:
collection | $match | $group | $project => result
The next pipeline stage is the $group operator. Inside the $group pipeline, you are now grouping all the filtered documents where you can specify an _id value of null to calculate accumulated values for all the input documents as a whole. Use the available accumulators to return the desired aggregation on the grouped documents. The accumulator operator $push is used in this grouping operation because it returns an array of expression values for each group.
Accumulators used in the $group stage maintain their state (e.g. totals, maximums, minimums, and related data) as documents progress through the pipeline.
To get the documents with the desired field, the $project operator which is similar to SELECT in SQL is used to rename the field names and select/deselect the fields to be returned, out of the grouped fields. If you specify 0 for a field, it will NOT be sent in the pipeline to the next operator.
You cannot do this with the find command.
Try using mongodb's aggregation pipeline.
Specifically use $group in combination with $push
See here: https://docs.mongodb.com/manual/reference/operator/aggregation/group/#pipe._S_group
I have a Mongo query which I want to effectively use the $group in the same way as GROUP BY in SQL.
This isn't working for me unless I set the _id of the new document one of the group categories which doesn't work for me and also, I am not able to get the values I want which come from potentially THREE documents which I am merging together in Mongo.
In SQL, I would write something like to illustrate the grouping and select that I am using as the basis of my aggregation in Mongo:
SELECT entity_id, connection_id, cycle_id, objectOriginAPI,accountBalance
FROM raw_originBusinessData
WHERE objectStatus = 'UPROCESSED'
AND (objectOriginAPI = 'Profit & Loss'
OR objectOriginAPI = 'Balance Sheet'
OR objectOriginAPI = 'Bank Summary')
GROUP BY entity_id, connection_id, cycle_id;
I have paraphrased to simplify what my Mongo script is doing with embedded arrays.
db.getCollection('raw_originBusinessData').aggregate([
{ "$match": {
objectStatus : "UNPROCESSED"
, $or: [
{ objectOriginAPI : "Profit & Loss"}
,{objectOriginAPI : "Balance Sheet"}
,{objectOriginAPI : "Bank Summary"}
]}
},
// don't worry about this, this is all good
{ "$unwind": "$objectRawOriginData.Reports" }
,{ "$unwind": "$objectRawOriginData.Reports.Rows" }
,{ "$unwind": "$objectRawOriginData.Reports.Rows.Rows" },
// this is where I believe I'm having my problem
{ "$group": {"_id": "$entity_id"
// , "$connection_id"
// , "objectCycleID"
, "accountBalances": { "$push": "$objectRawOriginData.Reports.Rows.Rows.Cells.Value" }
}},
{$project: {objectClass: {$literal: "Source Data"}
, objectCategory: {$literal: "Application"}
, objectType: {$literal: "Account Balances"}
, objectOrigin: {$literal: "Xero"}
, entity_ID: "$_id"
, connection_ID: "$connection_ID"
, accountBalances: "$accountBalances"}
}
]
// ,{$out: "std_sourceBusinessData"}
)
So each of the documents I am combining into a single document have the same entity_id, connection_id and cycle_id which I want to put into the new document. I also want to ensure that the new document has it's own unique object_id.
Your help is very much appreciated - Mongo documentation doesn't cover anything about $group other than _id is mandatory but if I don't set the _id to something that I want to group by (in the above script it is set to entity_id) it doesn't group properly.
Put simply, the _id needs to be a "composite" value, and therefore comprised of three "sub-keys":
{ "$group":{
"_id": {
"entity_id": "$entity_id"
"connection_id": "$connection_id",
"objectCycleID": "$objectCycleID"
},
"accountBalances": {
"$push": "$objectRawOriginData.Reports.Rows.Rows.Cells.Value"
}
}},
{ "$project": {
"_id": 0,
"objectClass": { "$literal": "Source Data" },
"objectCategory": { "$literal": "Application"},
"objectType": { "$literal": "Account Balances"},
"objectOrigin": { "$literal": "Xero"},
"entity_ID": "$_id.entity_id",
"connection_ID": "$_id.connection_id",
"accountBalances": "$accountBalances"
}}
And then of course, referncing any of those values in the later $project requires you now prefix with $_id as that is now the parent key.
Just as with any MongoDB document, the _id can be anything that is a valid BSON Object in representation. So in this case, the combination means "group on all these field values".