I have large documents and I want to project them with only field and value.
It goes like this;
{"p":{"s":{"status":"b"},"m":{"pd":{"tt":{"bi":"2","psi":"4","ircsi":true}},"mi":"TT","et":"2020-09-07T14:34:00+03:00"}}}
{"p":{"s":{"status":"b"},"m":{"pd":{"tt":{"bi":"20","psi":"1","ircsi":true}},"mi":"TT","et":"2020-12-29T08:28:06+03:00"}}}
.........
Is there any way for its to look like this;
{"status":"b","bi":"2","psi":"4","ircsi":true,"mi":"TT","et":"2020-09-07T14:34:00+03:00"}}}
{"status":"b","bi":"20","psi":"1","ircsi":true,"mi":"TT","et":"2020-09-07T14:34:00+03:00"}}}
.........
And my query is like this;
db.s.aggregate([{
"$match": {
"p.m.etm": {
"$gt": 1585688401000,
"$lt": 1610565947499
},
"p.m.mi":"TT"
}
},
{
"$project": {
"p.m.pd.tt.bi": 1,
"p.m.pd.tt.psi": 1,
"p.m.pd.tt.ircsi": 1,
"p.m.mi":1,
"p.s.status":1,
"p.m.et":1,
"_id": 0
}
}],
{
"allowDiskUse": true
});
What should I change to reach what I need ?
You can use $project to create new fields. (I just remove $match for easiness)
db.collection.aggregate([
{
"$project": {
"bi": "$p.m.pd.tt.bi",
"psi": "$p.m.pd.tt.psi",
"ircsi": "$p.m.pd.tt.ircsi",
"mi": "$p.m.mi",
"status": "$p.s.status",
"et": "$p.m.et",
"_id": 0
}
}
])
Working Mongo playground
Related
i have a collection with more then 1000 documents and there are some documents with same value in some fields, i need to get those
the collection is:
[{_id,fields1,fields2,fields3,etc...}]
what query can i use to get all the elements that have the same 3 fields for example:
[
{_id:1,fields1:'a',fields2:1,fields3:'z'},
{_id:2,fields1:'a',fields2:1,fields3:'z'},
{_id:3,fields1:'f',fields2:2,fields3:'g'},
{_id:4,fields1:'f',fields2:2,fields3:'g'},
{_id:5,fields1:'j',fields2:3,fields3:'g'},
]
i need to get
[
{_id:2,fields1:'a',fields2:1,fields3:'z'},
{_id:4,fields1:'f',fields2:2,fields3:'g'},
]
in this way i can easly get a list of "duplicate" that i can delete if needed, it's not really important get id 2 and 4 or 1 and 3
but 5 would never be included as it's not 'duplicated'
EDIT:
sorry but i forgot to mention that there are some document with null value i need to exclude those
This is the perfect use case of window field. You can use $setWindowFields to compute $rank in the grouping/partition you want. Then, get those rank not equal to 1 to get the duplicates.
db.collection.aggregate([
{
$match: {
fields1: {
$ne: null
},
fields2: {
$ne: null
},
fields3: {
$ne: null
}
}
},
{
"$setWindowFields": {
"partitionBy": {
fields1: "$fields1",
fields2: "$fields2",
fields3: "$fields3"
},
"sortBy": {
"_id": 1
},
"output": {
"duplicateRank": {
"$rank": {}
}
}
}
},
{
$match: {
duplicateRank: {
$ne: 1
}
}
},
{
$unset: "duplicateRank"
}
])
Mongo Playground
I think you can try this aggregation query:
First group by the feilds you want to know if there are multiple values.
It creates an array with the _ids that are repeated.
Then get only where there is more than one ($match).
And last project to get the desired output. I've used the first _id found.
db.collection.aggregate([
{
"$group": {
"_id": {
"fields1": "$fields1",
"fields2": "$fields2",
"fields3": "$fields3"
},
"duplicatesIds": {
"$push": "$_id"
}
}
},
{
"$match": {
"$expr": {
"$gt": [
{
"$size": "$duplicatesIds"
},
1
]
}
}
},
{
"$project": {
"_id": {
"$arrayElemAt": [
"$duplicatesIds",
0
]
},
"fields1": "$_id.fields1",
"fields2": "$_id.fields3",
"fields3": "$_id.fields2"
}
}
])
Example here
My doucments:
[{
"_id":"621c6e805961def3332bcf97",
"title":"monk plus",
"brand":"venture electronics",
"category":"earphones",
"variant":[
{
"price":1100,
"impedance":"16ohm"
},
{
"price":1600,
"impedance":"64ohm"
}],
"salesCount":185,
"buysCount":182,
"viewsCount":250
},
{
"_id":"621c6dab5961def3332bcf92",
"title":"nokia1",
"brand":"nokia",
"category":"mobile phones",
"variant":[
{
"price":10000,
"RAM":"4GB",
"ROM":"32GB"
},
{
"price":15000,
"RAM":"6GB",
"ROM":"64GB"
},
{
"price":20000,
"RAM":"8GB",
"ROM":"128GB"
}],
"salesCount":34,
"buysCount":21,
"viewsCount":80
}]
expected output
[{
_id:621c6e805961def3332bcf97
title:"monk plus"
brand:"venture electronics"
category:"earphones"
salesCount:185
viewsCount:250
variant:[
{
price:1100
impedance:"16ohm"
}]
}]
I have tried this aggregation method
[{
$match: {
'variant.price': {
$gte: 0,$lte: 1100
}
}},
{
$project: {
title: 1,
brand: 1,
category: 1,
salesCount: 1,
viewsCount: 1,
variant: {
$filter: {
input: '$variant',
as: 'variant',
cond: {
$and: [
{
$gte: ['$$variant.price',0]
},
{
$lte: ['$$variant.price',1100]
}
]
}
}
}
}}]
This method returns the expected output, now my question is there any other better approach that return the expected output.Moreover thank you in advance, and as I am new to nosql database so I am curious to learn from the community.Take a note on expected output all properties of particular document must return only the variant array of object I want to filter based on the price.
There's nothing wrong with your aggregation pipeline, and there are other ways to do it. If you just want to return matching documents, with only the first matching array element, here's another way to do it. (The .$ syntax only returns the first match unfortunately.)
db.collection.find({
// matching conditions
"variant.price": {
"$gte": 0,
"$lte": 1100
}
},
{
title: 1,
brand: 1,
category: 1,
salesCount: 1,
viewsCount: 1,
// only return first array element that matched
"variant.$": 1
})
Try it on mongoplayground.net.
Or, if you want to use an aggregation pipeline and return all matching documents in entirety except for the filtered array, you could just "overwrite" the array with the elements you want using "$set" (or its alias "$addFields"). Doing this means you won't need to "$project" anything.
db.collection.aggregate([
{
"$match": {
"variant.price": {
"$gte": 0,
"$lte": 1100
}
}
},
{
"$set": {
"variant": {
"$filter": {
"input": "$variant",
"as": "variant",
"cond": {
"$and": [
{ "$gte": [ "$$variant.price", 0 ] },
{ "$lte": [ "$$variant.price", 1100 ] }
]
}
}
}
}
}
])
Try it on mongoplayground.net.
your solution is good, just make sure to apply your $match and pagination before applying this step for faster queries
this is my document in the sensor collection
sensorId:1
sensorType:1
BuildingId:1
CalcItems:
0: Object
Ts:2019-08-30T18:30:00.000+00:00
Value:1
1:Object
Ts:2019-08-31T19:00:00.000+00:00
Value:2
i need to get sum of all value attributes with respect to same Ts date value
output like this
sensorType:1
BuildingId:1
CalcItems:
0: Object
Ts:2019-08-31T18:30:00.000+00:00
Value:23
1:Object
Ts:2019-08-31T19:00:00.000+00:00
Value:43
give me any suggestions
Try this..
Sample live demo
[
{
"$unwind": "$CalcItems"
},
{
$group: {
_id: {
"sensorType": "$sensorType",
"BuildingId": "$BuildingId",
"Ts": "$CalcItems.Ts"
},
"total": {
"$sum": "$CalcItems.Value"
}
}
},
{
"$project": {
"_id": 0,
"sensorType": "$_id.sensorType",
"BuildingIdVal": "$_id.BuildingId",
"CalcItems.Ts": "$_id.Ts",
"CalcItems.Value": "$total"
}
}
]
Reference
Mongodb $sum
Mongodb $group
Mongodb $project
I have these documents.
db.test.find({"house.floor":1})
db.test.insertMany([{
"name":"homer",
"house": {
"floor": 1,
"room":
{
"bed": "bed_pink",
"chair":"chair_pink"
}
}
},
{
"name":"marge",
"house": {
"floor": 1,
"room":
{
"bed": "bed_blue",
"chair":"chair_red"
}
}
}]
)
db.test.find({"house.room.bed":"bed_blue"})
I want to return only the value of the last level of my search. in this case:
{
"bed": "bed_blue",
"chair":"chair_red"
}
the answer I get, is the whole document, I want to advance to a certain level of the query. how can I do it?
You can use aggregation for it
db.test.aggregate([
{ $match: { "house.room.bed":"bed_blue" } },
{ $replaceRoot: { newRoot: "$house.room" } }
])
You can use aggregate together with $match and $project for this as follows:
db.test.aggregate([
{
$match: {
"house.room.bed": "bed_blue"
}
},
{
$project: {
"_id": 0,
"bed": "$house.room.bed",
"chair": "$house.room.chair"
}
}
])
Further, you can modify the $project part as needed.
Here is a demo: https://mongoplayground.net/p/Fvll0LFIvQy
I have a document in the following format:
"summary":{
"HUL":{
"hr_0":{
"ts":None,
"Insights":{
"sentiments":{
"pos":37,
"neg":3,
"neu":27
},
"topics":[
"Basketball",
"Football"
],
"geo":{
"locations":{
"Delhi":34,
"Kolkata":56,
"Pune":79,
"Bangalore":92,
"Mumbai":54
},
"mst_act":{
"loc":Bangalore,
"lat_long":None
}
}
}
},
"hr_1":{....},
"hr_2":{....},
.
.
"hr_23":{....}
I want to run an aggregation in pymongo that sums up the pos, neg and neu sentiments for all hours of the day "hr_0" to "hr_23".
I am having trouble in constructing the pipeline command in order to do this as the fields I am interested in are in nested dictionaries. Would really appreciate your suggestions.
Thanks!
It's going to be pretty difficult to come up with an aggregation pipeline that will give you the desired aggregates because your document schema has some dynamic keys which you can't use as an identifiey expression in the group operator pipeline.
However, a workaround using the current schema would be to iterate over the find cursor and extract the values you want to add up within the loop. Something like the following:
pos_total = 0
neg_total = 0
neu_total = 0
cursor = db.collection.find()
for doc in cursor:
for i in range(0, 24):
pos_total += doc["summary"]["HUL"]["hr_"+str(i)]["Insights"]["sentiments"]["pos"]
neg_total += doc["summary"]["HUL"]["hr_"+str(i)]["Insights"]["sentiments"]["neg"]
neu_total += ddoc["summary"]["HUL"]["hr_"+str(i)]["Insights"]["sentiments"]["neu"]
print(pos_total)
print(neg_total)
print(neu_total)
If you could do with changing the schema, then the following schema would be ideal for using the aggregation framework:
{
"summary": {
"HUL": [
{
"_id": "hr_0",
"ts": None,
"Insights":{
"sentiments":{
"pos":37,
"neg":3,
"neu":27
},
"topics":[
"Basketball",
"Football"
],
"geo":{
"locations":{
"Delhi":34,
"Kolkata":56,
"Pune":79,
"Bangalore":92,
"Mumbai":54
},
"mst_act":{
"loc":Bangalore,
"lat_long":None
}
}
}
},
{
"_id": "hr_2",
"ts": None,
"Insights":{
"sentiments":{
"pos":37,
"neg":3,
"neu":27
},
...
}
},
...
{
"_id": "hr_23",
"ts": None,
"Insights":{
"sentiments":{
"pos":37,
"neg":3,
"neu":27
},
...
}
}
]
}
}
The aggregation pipeline that would give you the required totals is:
var pipeline = [
{
"$unwind": "$summary.HUL"
},
{
"$group": {
"_id": "$summary.HUL._id",
"pos_total": { "$sum": "$summary.HUL.Insights.sentiments.pos" },
"neg_total": { "$sum": "$summary.HUL.Insights.sentiments.neg" },
"neu_total": { "$sum": "$summary.HUL.Insights.sentiments.neu" },
}
}
]
result = db.collection.aggregate(pipeline)