i have a collection with more then 1000 documents and there are some documents with same value in some fields, i need to get those
the collection is:
[{_id,fields1,fields2,fields3,etc...}]
what query can i use to get all the elements that have the same 3 fields for example:
[
{_id:1,fields1:'a',fields2:1,fields3:'z'},
{_id:2,fields1:'a',fields2:1,fields3:'z'},
{_id:3,fields1:'f',fields2:2,fields3:'g'},
{_id:4,fields1:'f',fields2:2,fields3:'g'},
{_id:5,fields1:'j',fields2:3,fields3:'g'},
]
i need to get
[
{_id:2,fields1:'a',fields2:1,fields3:'z'},
{_id:4,fields1:'f',fields2:2,fields3:'g'},
]
in this way i can easly get a list of "duplicate" that i can delete if needed, it's not really important get id 2 and 4 or 1 and 3
but 5 would never be included as it's not 'duplicated'
EDIT:
sorry but i forgot to mention that there are some document with null value i need to exclude those
This is the perfect use case of window field. You can use $setWindowFields to compute $rank in the grouping/partition you want. Then, get those rank not equal to 1 to get the duplicates.
db.collection.aggregate([
{
$match: {
fields1: {
$ne: null
},
fields2: {
$ne: null
},
fields3: {
$ne: null
}
}
},
{
"$setWindowFields": {
"partitionBy": {
fields1: "$fields1",
fields2: "$fields2",
fields3: "$fields3"
},
"sortBy": {
"_id": 1
},
"output": {
"duplicateRank": {
"$rank": {}
}
}
}
},
{
$match: {
duplicateRank: {
$ne: 1
}
}
},
{
$unset: "duplicateRank"
}
])
Mongo Playground
I think you can try this aggregation query:
First group by the feilds you want to know if there are multiple values.
It creates an array with the _ids that are repeated.
Then get only where there is more than one ($match).
And last project to get the desired output. I've used the first _id found.
db.collection.aggregate([
{
"$group": {
"_id": {
"fields1": "$fields1",
"fields2": "$fields2",
"fields3": "$fields3"
},
"duplicatesIds": {
"$push": "$_id"
}
}
},
{
"$match": {
"$expr": {
"$gt": [
{
"$size": "$duplicatesIds"
},
1
]
}
}
},
{
"$project": {
"_id": {
"$arrayElemAt": [
"$duplicatesIds",
0
]
},
"fields1": "$_id.fields1",
"fields2": "$_id.fields3",
"fields3": "$_id.fields2"
}
}
])
Example here
Related
Here, I have multiple fields from multiple tables those values needs to compared and need to display desired result.
SQL QUERY:
select pd.service_id,ps.service_id from player pd, service ps where pd.subject_id=ps.subject_id and pd.service_id = ps.service_id
Mongo query:
db.player.aggregate([
{
"$lookup":{
"from":"service",
"localField":"player.subject_id",
"foreignField":"subject_id",
"as":"ps"
}
},
{
"$unwind":"$ps"
},
{
"$match":{
"service_id":{
"$eq": "ps.service_id"
}
}
}
];
sample input records:
player:
[{subject_id:23,service_id:1},{subject_id:76,service_id:9}]
service:
[{subject_id:76,service_id:9},{subject_id:99,service_id:10}]
The match is not working. I have to match service_id's of both collections. Need to get matched records. But not able to see any result. Can anyone please help me to find out the mistake...
In your query, if you want to compare 2 values from the document itself, you need to use $expr operator
{
"$match":{
"$expr":{
"$eq": ["$service_id", "$ps.service_id"]
}
}
}
MongoPlayground
Alternative solution: You need to use Uncorrelated sub-query to "* join" with 2 o more conditions
db.player.aggregate([
{
"$lookup": {
"from": "service",
"let": {
subject_id: "$subject_id",
service_id: "$service_id"
},
"pipeline": [
{
$match: {
$expr: {
$and: [
{
$eq: [
"$$subject_id",
"$subject_id"
]
},
{
$eq: [
"$$service_id",
"$service_id"
]
}
]
}
}
}
],
"as": "ps"
}
},
// Remove non matched results
{
$match: {
"ps.0": {
$exists: true
}
}
},
// Remove temporal "ps" field
{
$addFields: {
"ps": "$$REMOVE"
}
}
])
MongoPlayground
My doucments:
[{
"_id":"621c6e805961def3332bcf97",
"title":"monk plus",
"brand":"venture electronics",
"category":"earphones",
"variant":[
{
"price":1100,
"impedance":"16ohm"
},
{
"price":1600,
"impedance":"64ohm"
}],
"salesCount":185,
"buysCount":182,
"viewsCount":250
},
{
"_id":"621c6dab5961def3332bcf92",
"title":"nokia1",
"brand":"nokia",
"category":"mobile phones",
"variant":[
{
"price":10000,
"RAM":"4GB",
"ROM":"32GB"
},
{
"price":15000,
"RAM":"6GB",
"ROM":"64GB"
},
{
"price":20000,
"RAM":"8GB",
"ROM":"128GB"
}],
"salesCount":34,
"buysCount":21,
"viewsCount":80
}]
expected output
[{
_id:621c6e805961def3332bcf97
title:"monk plus"
brand:"venture electronics"
category:"earphones"
salesCount:185
viewsCount:250
variant:[
{
price:1100
impedance:"16ohm"
}]
}]
I have tried this aggregation method
[{
$match: {
'variant.price': {
$gte: 0,$lte: 1100
}
}},
{
$project: {
title: 1,
brand: 1,
category: 1,
salesCount: 1,
viewsCount: 1,
variant: {
$filter: {
input: '$variant',
as: 'variant',
cond: {
$and: [
{
$gte: ['$$variant.price',0]
},
{
$lte: ['$$variant.price',1100]
}
]
}
}
}
}}]
This method returns the expected output, now my question is there any other better approach that return the expected output.Moreover thank you in advance, and as I am new to nosql database so I am curious to learn from the community.Take a note on expected output all properties of particular document must return only the variant array of object I want to filter based on the price.
There's nothing wrong with your aggregation pipeline, and there are other ways to do it. If you just want to return matching documents, with only the first matching array element, here's another way to do it. (The .$ syntax only returns the first match unfortunately.)
db.collection.find({
// matching conditions
"variant.price": {
"$gte": 0,
"$lte": 1100
}
},
{
title: 1,
brand: 1,
category: 1,
salesCount: 1,
viewsCount: 1,
// only return first array element that matched
"variant.$": 1
})
Try it on mongoplayground.net.
Or, if you want to use an aggregation pipeline and return all matching documents in entirety except for the filtered array, you could just "overwrite" the array with the elements you want using "$set" (or its alias "$addFields"). Doing this means you won't need to "$project" anything.
db.collection.aggregate([
{
"$match": {
"variant.price": {
"$gte": 0,
"$lte": 1100
}
}
},
{
"$set": {
"variant": {
"$filter": {
"input": "$variant",
"as": "variant",
"cond": {
"$and": [
{ "$gte": [ "$$variant.price", 0 ] },
{ "$lte": [ "$$variant.price", 1100 ] }
]
}
}
}
}
}
])
Try it on mongoplayground.net.
your solution is good, just make sure to apply your $match and pagination before applying this step for faster queries
I've nested JSON like this. I want to retrieve the value of "_value" in second level. i,e. "Living Organisms" This is my JSON document.
{
"name": "Biology Book",
"data": {
"toc": {
"_version": "1",
"ge": [
{
"_name": "The Fundamental Unit of Life",
"_id": "5a",
"ge": [
{
"_value": "Living Organisms",
"_id": "5b"
}
]
}
]
}
}
}
This is what I've tried, using the "_id", I want to retrieve it's "_value"
db.products.aggregate([{"$match":{ "data.toc.ge.ge._id": "5b"}}])
This is the closest I could get to the output you mentioned in the comment above. Hope it helps.
db.collection.aggregate([
{
$match: {
"data.toc.ge.ge._id": "5b"
}
},
{
$unwind: "$data.toc.ge"
},
{
$unwind: "$data.toc.ge.ge"
},
{
$group: {
_id: null,
book: {
$push: "$data.toc.ge.ge._value"
}
}
},
{
$project: {
_id: 0,
first: {
$arrayElemAt: [
"$book",
0
]
},
}
}
])
Output:
[
{
"first": "Living Organisms"
}
]
You can check what I tried here
If you are using Mongoid:
(1..6).inject(Model.where('data.toc.ge.ge._id' => '5b').pluck('data.toc.ge.ge._value').first) { |v| v.values.first rescue v.first rescue v }
# => "Living Organisms"
6 is the number of containers to trim from the output (4 hashes and 2 arrays).
If I understand your question correctly, you only care about _value, so it sounds like you might want to use a projection:
db.products.aggregate([{"$match":{ "data.toc.ge.ge._id": "5b"}}, { "$project": {"data.toc.ge.ge._value": 1}}])
var otherLanguages=[ "English","Arabic","French"];
var first, second;
db.collection.find({ $and: [ { "Language" : { $nin : otherLanguages} },{"Language":{ $ne:null}} ]}).forEach(function(obj){
shell out 341 docs one by one. In these docs,I want to find out documents that satisfy two if statements. Later, I want to collect the count it.
if (obj.find({ $and: [{'POS': { $eq: "Past" } },{'Desp': { $ne: null } }] })) { first= first+1;}
if (obj.find({ $and: [{'POS': { $eq: "Past" } },{'Desp': { $eq: null } }] })) {second= second+1;}
});
print (first,second)
I know that I cannot use find() function on the obj, but Is there a way to search on this "bson obj" to find the count.
If this is not feasible, then please suggest a way to get the desired result.
If I understand your question correctly you can achieve that by using the aggregation framework like so:
db.collection.aggregate({
// filter out all documents that you don't care about
$match: {
"Language": { $nin: otherLanguages, $ne: null },
"POS": "Past"
},
}, {
// then split into groups...
$group: {
_id: { $eq: [ "$Desp", null ] }, // ...one for the "eq: null" and one for the "ne: null"
"count": { $sum: 1 } // ...and count the number of documents in each group
}
})
I have a mongodb collection with structure like that:
[
{
name: "name1",
instances: [{value:1, score:2}, {value:2, score:5}, {value:2.5, score:9}]
},
{
name: "name2",
instances: [{value:6, score:3}, {value:1, score:6}, {value:3.7, score:5.2}]
}
]
When I want to get all the data from a document, I use aggregate because I want each instance returned as a separate document:
db.myCollection.aggregate([{$match:{name:"name1"}}, {$unwind:"$instances"}, {$project:{name:1, value:"$instances.value", score:"$instances.score"}}])
And everything works like I want it to.
Now for my question: I want to filter the returned data by score or by value. For example, I want an array of all the subdocuments of name1 which have a value greater or equal to 2.
I tried to add to the $match object 'instances.value':{$gte:2}, but it didn't filter anything, and I still get all 3 documents for this query.
Any ideas?
After unwinding instances then again used $match as below
db.collectionName.aggregate({
"$match": {
"name": "name1"
}
}, {
"$unwind": "$instances"
}, {
"$match": {
"instances.value": {
"$gte": 2
}
}
}, {
$project: {
name: 1,
value: "$instances.value",
score: "$instances.score"
}
})
Or if you tried $match after project then used as below
db.collectionName.aggregate([{
$match: {
name: "name1"
}
}, {
$unwind: "$instances"
}, {
$project: {
name: 1,
value: "$instances.value",
score: "$instances.score"
}
}, {
"$match": {
"value": {
"$gte": 2
}
}
}])