Display field document in mongoDB after execute Query aggregate - mongodb

This is an example of a data document
{
"_id" : ObjectId("5f437e7846103b2ad0fc5d7d"),
"order_no" : "O-200824-AGFJDQW",
"shipment_no" : "S-200824-AGWCRRM",
"member_id" : 2200140,
"ponta_id" : "9990010100280214",
"plu" : 14723,
"product_name" : "AQUA Air Mineral Botol Air Pet 600ml",
"qty" : 2,
"store_id" : "TD46",
"stock_on_hand" : 0,
"transaction_date" : ISODate("2020-08-24T08:28:29.931Z"),
"created_date" : ISODate("2020-08-24T08:46:48.441Z")
}
this is the data query that I run
var bulan = 12 //month is written with number. example: August = 8
db.log_stock_oos.aggregate([
{
$project: {
month: {
$month: '$transaction_date'
}
}
},
{
$match: {month: bulan}
}
]);
but the result is like this after I run the query
{
"_id" : ObjectId("5f44689607fe453fbfba433e"),
"month" : 12
}
how to make the output exactly like the document display that I attached above??
this is my reference

When you use the projection, its kind of if your value 1 then include the field, if your value 0 then exclude the field from the whole documents. Projection
You can do two things
Use the projection
db.collection.aggregate([
{
$project: {
month: {
$month: "$transaction_date"
},
order_no: 1,
shipment_no: 1,
member_id: 1,
//other fields like above with the value 1
}
},
// match stages
])
Use $addFields
use $addFields incited of $project in your code. If will create a filed if not exists in your document, else it will overwrite the field

Related

Mongodb array of object data conversion

This is my single document of a collection. I want to convert this format data to
{
"_id" : ObjectId("5e311e8bb94999f1be0d5ead"),
"dealer_code" : "123",
"mappinginfo" : [
{
"territory" : "MORADABAD",
"area" : "UPH",
"zone" : "N"
}
],
"active" : NumberInt(1),
}
like this data
{
"_id" : ObjectId("5e311e8bb94999f1be0d5ead"),
"dealer_code" : "123",
"territory" : "MORADABAD",
"area" : "UPH",
"zone" : "N"
"active" : NumberInt(1),
}
Here is the Query.
db.collection.aggregate([
{
$unwind: "$mappinginfo"
},
{
$project: {
_id: 1,
active: 1,
dealer_code: 1,
territory: "$mappinginfo.territory",
area: "$mappinginfo.area",
zone: "$mappinginfo.zone",
}
},
])
db.collection.aggregate([
{ $unwind: "$mappinginfo" },
{
$project:
{
_id: 1,
dealer_code: 1,
active: 1,
territory: "$mappinginfo.territory",
area: "$mappinginfo.area",
zone: "$mappinginfo.zone"
}
},
{ $out: "existing_collection_name" }
])
Make use of aggregation pipeline.
Solution -
db.test.aggregate([
{ $unwind: "$mappinginfo" },
{ $project: {
dealer_code: 1,
territory: "$mappinginfo.territory",
area: "$mappinginfo.area",
zone: "$mappinginfo.zone",
active: 1
}}
]);
Explanation
Aggregation pipeline as the name is, works on data in pipeline. A pipeline has an input, an operator and an output. For the example above -
First stage is an unwind stage which will be fed with an entire collection. The unwind operator will iterate on individual documents and for each document it will create as many copies as are elements in mappinginfo field. For your case it will only create just one copy. After the unwind stage the resulting document would look like this -
[{
"_id" : ObjectId("5e311e8bb94999f1be0d5ead"),
"dealer_code" : "123",
"mappinginfo": {
"territory" : "MORADABAD",
"area" : "UPH",
"zone" : "N"
}
"active" : NumberInt(1),
}]
Notice that the mappinginfo is no more a list.
Next stage is a projection stage. $project operator will simply take the above list of documents as input and for each document either it will simply project the field as it is or will change the value of the field based on what is available in the current document. { "active": 1 } implies projecting the value as it is. { "zone": "$mappinginfo.zone" } implies projecting the value of zone inside mappinginfo field under the name zone at the root level.
More info on both operators -
- $unwind
- $project

Querying date range in aggregate query returns nothing or ignores dates

In my aggregate query I'm trying to add conditions in the $match statement to return only records within given date range. Without converting to ISOString, I get a set of records that ignores the date range completely. When I convert to ISOString, I get nothing (returns empty set). I've tried using the $and operator, still nothing.
I've tried all the solutions on stack to no avail. Here's my code:
$match: {
$and: [
{'author.id': { $ne: req.user._id }},
{'blurtDate': { $gte: test1.toISOString() }},
{'blurtDate': { $lte: test2.toISOString() }}
]
}
test1 and test2 are correct, I checked them on console log they reflect as follows:
2019-06-02T12:44:39.000Z -- 2019-07-02T12:44:39.928Z
I also tried without the $and operator like so:
$match: {
'author.id': { $ne: req.user._id },
'blurtDate': { $gte: test1.toISOString() },
'blurtDate': { $lte: test2.toISOString() }
}
Which again returns nothing. Any help much appreciated!
EDIT: Wanted to emphasize that test1 and test2 are new date objects:
test1 = new Date(qryDateFrom); //Tried .toISOString() as well
test2 = new Date(qryDateTo);
Without .toISOString(), I get a return of values that ignores the dates. With .toISOString I get an empty return.
Here's an example document that should be returned:
{
"_id" : ObjectId("5d0a807345c85d00ac4b7217"),
"text" : "<p>Seriously RV style.</p>",
"blurtDate" : ISODate("2019-06-19T18:35:31.156Z"),
"blurtImg" : "04643410-92c1-11e9-80b6-a3262311afff.png",
"vote" : 0,
"author" : {
"id" : ObjectId("5cb5df0ef7a3570bb4ac6e05"),
"name" : "Benjamin Paine"
},
"__v" : 0
}
When I remove .toISOString(), I get documents outside of the expected date range, such as this one in May (query should only return between june 2 and july 2).
{
"_id" : ObjectId("5d07ebaf9a035117e4546349"),
"text" : "<p>A start to something more...</p>",
"blurtDate" : ISODate("2019-05-15T19:36:15.737Z"),
"blurtImg" : "2be7a160-9137-11e9-933f-6966b2e503c7.png",
"vote" : 0,
"author" : {
"id" : ObjectId("5cb5df0ef7a3570bb4ac6e05"),
"name" : "Benjamin Paine"
},
"__v" : 0
}
Your docs contain actual Date objects, so remove the .toISOString()s from your query. But you'll also need to combine your $gte and $lte terms into a single object:
$match: {
'author.id': { $ne: req.user._id },
'blurtDate': { $gte: test1, $lte: test2 }
}

Query to count number of occurrence in array grouped by day

I have the following document structure:
(trackerEventsCollection) =
{
"_id" : ObjectId("5b26c4fb7c696201040c8ed1"),
"trackerId" : ObjectId("598fc51324h51901043d76de"),
"trackingEvents" : [
{
"type" : "checkin",
"eventSource" : "app",
"timestamp" : ISODate("2017-08-25T06:34:58.964Z")
},
{
"type" : "power",
"eventSource" : "app",
"timestamp" : ISODate("2017-08-25T06:51:23.795Z")
},
{
"type" : "position",
"eventSource" : "app",
"timestamp" : ISODate("2017-08-25T06:51:23.985Z")
}
]
}
I would like to write a query that would count number of trackingEvents with type "type" : "power" grouped by day. This seems to be quite tricky to me because parent document does not have date and I should rely on timestamp field that belongs to the trackingEvents array members.
I'm not really experienced mongodb user and couldn't understand how can this be achieved so far.
Would really appreciate any help, thanks
To process your nested array as a separate documents you need to use $unwind. In the next stage you can use $match to filter out by type. Then you can group by single days counting occurences. The point is that you have to build grouping key containing year, month and day like in following code:
db.trackerEvents.aggregate([
{ $unwind: "$trackingEvents" },
{ $match: { "trackingEvents.type": "power" } },
{
$group: {
_id: {
year: { $year:"$trackingEvents.timestamp" },
month:{ $month:"$trackingEvents.timestamp" },
day: { $dayOfMonth:"$trackingEvents.timestamp" }
},
count: { $sum: 1 }
}
}
])

MongoDb aggregation Group by Date

I'm trying to group by timestamp for the collection named "foo" { _id, TimeStamp }
db.foos.aggregate(
[
{$group : { _id : new Date (Date.UTC({ $year : '$TimeStamp' },{ $month : '$TimeStamp' },{$dayOfMonth : '$TimeStamp'})) }}
])
Expecting many dates but the result is just one date. The data i'm using is correct (has many foo and different dates except 1970). There's some problem in the date parsing but i can not solve yet.
{
"result" : [
{
"_id" : ISODate("1970-01-01T00:00:00.000Z")
}
],
"ok" : 1
}
Tried this One:
db.foos.aggregate(
[
{$group : { _id : { year : { $year : '$TimeStamp' }, month : { $month : '$TimeStamp' }, day : {$dayOfMonth : '$TimeStamp'} }, count : { $sum : 1 } }},
{$project : { parsedDate : new Date('$_id.year', '$_id.month', '$_id.day') , count : 1, _id : 0} }
])
Result :
uncaught exception: aggregate failed: {
"errmsg" : "exception: disallowed field type Date in object expression (at 'parsedDate')",
"code" : 15992,
"ok" : 0
}
And that one:
db.foos.aggregate(
[
{$group : { _id : { year : { $year : '$TimeStamp' }, month : { $month : '$TimeStamp' }, day : {$dayOfMonth : '$TimeStamp'} }, count : { $sum : 1 } }},
{$project : { parsedDate : Date.UTC('$_id.year', '$_id.month', '$_id.day') , count : 1, _id : 0} }
])
Can not see dates in the result
{
"result" : [
{
"count" : 412
},
{
"count" : 1702
},
{
"count" : 422
}
],
"ok" : 1
}
db.foos.aggregate(
[
{ $project : { day : {$substr: ["$TimeStamp", 0, 10] }}},
{ $group : { _id : "$day", number : { $sum : 1 }}},
{ $sort : { _id : 1 }}
]
)
Group by date can be done in two steps in the aggregation framework, an additional third step is needed for sorting the result, if sorting is desired:
$project in combination with $substr takes the first 10 characters (YYYY:MM:DD) of the ISODate object from each document (the result is a collection of documents with the fields "_id" and "day");
$group groups by day, adding (summing) the number 1 for each matching document;
$sort ascending by "_id", which is the day from the previous aggregation step - this is optional if sorted result is desired.
This solution can not take advantage of indexes like db.twitter.ensureIndex( { TimeStamp: 1 } ), because it transforms the ISODate object to a string object on the fly. For large collections (millions of documents) this could be a performance bottleneck and more sophisticated approaches should be used.
It depends on whether you want to have the date as ISODate type in the final output. If so, then you can do one of two things:
Extract $year, $month, $dayOfMonth from your timestamp and then reconstruct a new date out of them (you are already trying to do that, but you're using syntax that doesn't work in aggregation framework).
If the original Timestamp is of type ISODate() then you can do date arithmetic to subtract the hours, minutes, seconds and milliseconds from your timestamp to get a new date that's "rounded" to the day.
There is an example of 2 here.
Here is how you would do 1. I'm making an assumption that all your dates are this year, but you can easily adjust the math to accommodate your oldest date.
project1={$project:{_id:0,
y:{$subtract:[{$year:"$TimeStamp"}, 2013]},
d:{$subtract:[{$dayOfYear:"$TimeStamp"},1]},
TimeStamp:1,
jan1:{$literal:new ISODate("2013-01-01T00:00:00")}
} };
project2={$project:{tsDate:{$add:[
"$jan1",
{$multiply:["$y", 365*24*60*60*1000]},
{$multiply:["$d", 24*60*60*1000]}
] } } };
Sample data:
db.foos.find({},{_id:0,TimeStamp:1})
{ "TimeStamp" : ISODate("2013-11-13T19:15:05.600Z") }
{ "TimeStamp" : ISODate("2014-02-01T10:00:00Z") }
Aggregation result:
> db.foos.aggregate(project1, project2)
{ "tsDate" : ISODate("2013-11-13T00:00:00Z") }
{ "tsDate" : ISODate("2014-02-01T00:00:00Z") }
This is what I use in one of my projects :
collection.aggregate(
// group results by date
{$group : {
_id : { date : "$date" }
// do whatever you want here, like $push, $sum...
}},
// _id is the date
{$sort : { _id : -1}},
{$orderby: { _id : -1 }})
.toArray()
Where $date is a Date object in mongo. I get results indexed by date.

Upsert with pymongo and a custom _id field

I'm attempting to store pre-aggregated performance metrics in a sharded mongodb according to this document.
I'm trying to update the minute sub-documents in a record that may or may not exist with an upsert like so (self.collection is a pymongo collection instance):
self.collection.update(query, data, upsert=True)
query:
{ '_id': u'12345CHA-2RU020130304',
'metadata': { 'adaptor_id': 'CHA-2RU',
'array_serial': 12345,
'date': datetime.datetime(2013, 3, 4, 0, 0, tzinfo=<UTC>),
'processor_id': 0}
}
data:
{ 'minute': { '16': { '45': 1.6693091}}}
The problem is that in this case the 'minute' subdocument always only has the last hour: { minute: metric} entry, the minute subdocument does not create new entries for other hours, it's always overwriting the one entry.
I've also tried this with a $set style data entry:
{ '$set': { 'minute': { '16': { '45': 1.6693091}}}}
but it ends up being the same.
What am I doing wrong?
In both of the examples listed you are simply setting a field ('minute')to a particular value, the only reason it is an addition the first time you update is because the field itself does not exist and so must be created.
It's hard to determine exactly what you are shooting for here, but I think what you could do is alter your schema a little so that 'minute' is an array. Then you could use $push to add values regardless of whether they are already present or $addToSet if you don't want duplicates.
I had to alter your document a little to make it valid in the shell, so my _id (and some other fields) are slightly different to yours, but it should still be close enough to be illustrative:
db.foo.find({'_id': 'u12345CHA-2RU020130304'}).pretty()
{
"_id" : "u12345CHA-2RU020130304",
"metadata" : {
"adaptor_id" : "CHA-2RU",
"array_serial" : 12345,
"date" : ISODate("2013-03-18T23:28:50.660Z"),
"processor_id" : 0
}
}
Now let's add a minute field with an array of documents instead of a single document:
db.foo.update({'_id': 'u12345CHA-2RU020130304'}, { $addToSet : {'minute': { '16': {'45': 1.6693091}}}})
db.foo.find({'_id': 'u12345CHA-2RU020130304'}).pretty()
{
"_id" : "u12345CHA-2RU020130304",
"metadata" : {
"adaptor_id" : "CHA-2RU",
"array_serial" : 12345,
"date" : ISODate("2013-03-18T23:28:50.660Z"),
"processor_id" : 0
},
"minute" : [
{
"16" : {
"45" : 1.6693091
}
}
]
}
Then, to illustrate the addition, add a slightly different entry (since I am using $addToSet this is required for a new field to be added:
db.foo.update({'_id': 'u12345CHA-2RU020130304'}, { $addToSet : {'minute': { '17': {'48': 1.6693391}}}})
db.foo.find({'_id': 'u12345CHA-2RU020130304'}).pretty()
{
"_id" : "u12345CHA-2RU020130304",
"metadata" : {
"adaptor_id" : "CHA-2RU",
"array_serial" : 12345,
"date" : ISODate("2013-03-18T23:28:50.660Z"),
"processor_id" : 0
},
"minute" : [
{
"16" : {
"45" : 1.6693091
}
},
{
"17" : {
"48" : 1.6693391
}
}
]
}
I ended up setting the fields like this:
query:
{ '_id': u'12345CHA-2RU020130304',
'metadata': { 'adaptor_id': 'CHA-2RU',
'array_serial': 12345,
'date': datetime.datetime(2013, 3, 4, 0, 0, tzinfo=<UTC>),
'processor_id': 0}
}
I'm setting the metrics like this:
data = {"$set": {}}
for metric in csv:
date_utc = metric['date'].astimezone(pytz.utc)
data["$set"]["minute.%d.%d" % (date_utc.hour,
date_utc.minute)] = float(metric['metric'])
which creates data like this:
{"$set": {'minute.16.45': 1.6693091,
'minute.16.46': 1.566343,
'minute.16.47': 1.22322}}
So that when self.collection.update(query, data, upsert=True) is run it updates those fields.