As I am new to mongo, I have a slight problem getting something done the way I need it.
I am trying to group the collection by "token_address", count the occurencies of that as totalTransfers and also for each token_address sum the values of the "decimal" property.
The desired outlook would look like this
{
"token_address" : "0x2a746fb4d7338e4677c102f0ce46dae3971de1cc",
"totalTransfers" : 4.0, //occurencies per token_address in collection
"decimal" :132.423 //the $sum of each decimal per token_address
}
This is how the documents look
{
"_id" : "BrBr1vuhNRmmaZliYopQocD2",
"from_address" : "0x7ed77e237fa0a87fc12afb92d2999e6f90e1d43b",
"log_index" : 442,
"to_address" : "0x31d5e41636c2331d8be9ea9c4393a0ff4e597b6c",
"transaction_hash" : "0x1a80b66839b021ef9c1a902f19d28b77d8e688b2e3ebb9bfc185443ae1830403",
"_created_at" : ISODate("2022-03-21T14:09:49.894Z"),
"_updated_at" : ISODate("2022-03-21T14:09:49.894Z"),
"block_hash" : "0x80dfe8642f998ce7fb3e692ab574d9786efdd81ba6aeace060ae9cc919a8acbf",
"block_number" : 14209975,
"block_timestamp" : ISODate("2022-02-15T09:47:30.000Z"),
"confirmed" : true,
"decimal" : NumberDecimal("0.1206"),
"historical" : true,
"token_address" : "0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2",
"transaction_index" : 278,
"value" : "120600000000000000"
}
This is my aggregation which gives which results in the last output
db.EthTokenTransfers.aggregate([
{ $project: {
token_address : 1 // Inclusion mode
}},
{ $group : { _id : '$token_address', totalTransfers : { $sum : 1 }, decimal: {$sum:"$decimal"}
} }
])
{
"_id" : "0x2a746fb4d7338e4677c102f0ce46dae3971de1cc",
"totalTransfers" : 4.0,
"decimal" : 0
}
Can some one point me towards the correct way of doing this? I been trying for over an hour for such a simple task.
The problem is your are removing fields in the $project stage.
Check this example only with the first stage where the output is only the field token_address.
So to the next aggregation stage this is the income data, only the token_address. And that's why totalTransfersis correct (you group by the existing token_address field and $sum one for each one) but decimal is always 0, because the field does not exists.
One solution can be add decimal into $project like this example
Related
I'm trying to fetch a set of records in the most efficient way from MongoDB, but it goes wrong when I add a sorting stage to the pipeline. The server does not use my intended index. According to the documentation it should however match the prefix:
https://docs.mongodb.com/manual/tutorial/sort-results-with-indexes/#sort-and-non-prefix-subset-of-an-index
I have an index which looks like this:
{
"v" : 2,
"key" : {
"account_id" : 1,
"cdr_block" : 1,
"billing.total_billed" : 1,
"times.created" : -1
},
"name" : "IDX_by_account_and_block_sorted"
}
So I would suppose that when I filter on account_id, cdr_block and billing.total_billed, followed by a sort on times.created, the index would be used.
However that is not the case; when I check the query explanations in MongoDB shell;
this one does NOT use the index, but uses an index that is composed of times.created only, so it takes a few minutes:
db.getCollection("cdr").aggregate(
[
{
"$match" : {
"account_id" : 160.0,
"cdr_block" : ObjectId("5d11e0364f853f15824aff47"),
"billing.total_billed" : {
"$gt" : 0.0
}
}
},
{
"$sort" : {
"times.created" : -1.0
}
}
],
{
"allowDiskUse" : true
}
);
If I leave out the $sort stage, it does use my above mentioned index.
I was thinking that it was perhaps due to the fact that it's an aggregation, but this 'regular' query also doesn't use the index:
db.getCollection("cdr").find({
"account_id" : 160.0,
"cdr_block" : ObjectId("5d11e0364f853f15824aff47"),
"billing.total_billed" : {
"$gt" : 0.0
}
}).sort({"times.created" : -1 });
$sort Operator and Performance
$sort operator can take advantage of an index when placed at the beginning of the pipeline or placed before the $project, $unwind, and $group aggregation operators. If $project, $unwind, or $group occur prior to the $sort operation,
$sort cannot use any indexes.
For a given record id, how do I get the average of a sub document field if I have the following in MongoDB:
/* 0 */
{
"item" : "1",
"samples" : [
{
"key" : "test-key",
"value" : "1"
},
{
"key" : "test-key2",
"value" : "2"
}
]
}
/* 1 */
{
"item" : "1",
"samples" : [
{
"key" : "test-key",
"value" : "3"
},
{
"key" : "test-key2",
"value" : "4"
}
]
}
I want to get the average of the values where key = "test-key" for a given item id (in this case 1). So the average should be $avg (1 + 3) = 2
Thanks
You'll need to use the aggregation framework. The aggregation will end up looking something like this:
db.stack.aggregate([
{ $match: { "samples.key" : "test-key" } },
{ $unwind : "$samples" },
{ $match : { "samples.key" : "test-key" } },
{ $project : { "new_key" : "$samples.key", "new_value" : "$samples.value" } },
{ $group : { `_id` : "$new_key", answer : { $avg : "$new_value" } } }
])
The best way to think of the aggregation framework is like an assembly line. The query itself is an array of JSON documents, where each sub-document represents a different step in the assembly.
Step 1: $match
The first step is a basic filter, like a WHERE clause in SQL. We place this step first to filter out all documents that do not contain an array element containing test-key. Placing this at the beginning of the pipeline allows the aggregation to use indexes.
Step 2: $unwind
The second step, $unwind, is used for separating each of the elements in the "samples" array so we can perform operations across all of them. If you run the query with just that step, you'll see what I mean.
Long story short :
{ name : "bob",
children : [ {"name" : mary}, { "name" : "sue" } ]
}
becomes two documents :
{ name : "bob", children : [ { "name" : mary } ] }
{ name : "bob", children : [ { "name" : sue } ] }
Step 3: $match
The third step, $match, is an exact duplicate of the first $match stage, but has a different purpose. Since it follows $unwind, this stage filters out previous array elements, now documents, that don't match the filter criteria. In this case, we keep only documents where samples.key = "test-key"
Step 4: $project (Optional)
The fourth step, $project, restructures the document. In this case, I pulled the items out of the array so I could reference them directly. Using the example above..
{ name : "bob", children : [ { "name" : mary } ] }
becomes
{ new_name : "bob", new_child_name : mary }
Note that this step is entirely optional; later stages could be completed even without this $project after a few minor changes. In most cases $project is entirely cosmetic; aggregations have numerous optimizations under the hood such that manually including or excluding fields in a $project should not be necessary.
Step 5: $group
Finally, $group is where the magic happens. The _id value what you will be "grouping by" in the SQL world. The second field is saying to average over the value that I defined in the $project step. You can easily substitute $sum to perform a sum, but a count operation is typically done the following way: my_count : { $sum : 1 }.
The most important thing to note here is that the majority of the work being done is to format the data to a point where performing the operation is simple.
Final Note
Lastly, I wanted to note that this would not work on the example data provided since samples.value is defined as text, which can't be used in arithmetic operations. If you're interested, changing the type of a field is described here: MongoDB How to change the type of a field
I have a number of Mongodb documents of the following form:
{
"auditedId" : "53d0f648e4b064e8d746b31c",
"modifications" : [
{
"auditRecordId" : ObjectId("53d0f648e4b064e8d746b31d"),
"modified" : "2014-07-22 18:33:05"
},
{
"auditRecordId" : ObjectId("53d0f648e4b064e8d746b31e"),
"modified" : "2014-07-24 14:15:27"
},
{
"auditRecordId" : ObjectId("53d0f648e4b064e8d746b31f"),
"modified" : "2014-07-24 12:04:24"
}
]
}
For each of these documents I want to find "auditRecordId" value which corresponds to the latest modification. In the given example I want to retrieve
"auditRecordId" : ObjectId("53d0f648e4b064e8d746b31e")
Or, even better:
{
"auditRecordId" : ObjectId("53d0f648e4b064e8d746b31e"),
"modified" : "2014-07-24 14:15:27"
}
Is there any way how I can do this without writing map-reduce functions?
Whenever you have an array in your document, the aggregate method is your friend :)
db.foo.aggregate([
// De-normalize the 'modifications' array
{"$unwind":"$modifications"},
// Sort by 'modifications.modified' descending
{"$sort":{"modifications.modified":-1}},
// Pick the first one i.e., the max
{"$limit":1}
])
Output:
{
"result" : [
{
"_id" : ObjectId("53d12be57a462c7459b6f1c7"),
"auditedId" : "53d0f648e4b064e8d746b31c",
"modifications" : {
"auditRecordId" : ObjectId("53d0f648e4b064e8d746b31e"),
"modified" : "2014-07-24 14:15:27"
}
}
],
"ok" : 1
}
Just to illustrate the $unwind operator, I used the above query with $limit. If you have multiple documents of the above format, and you want to retrieve the latest modification in each, you'll have to add another $group phase in your aggregation pipeline and use the $first operator:
db.foo.aggregate([
{"$unwind":"$modifications"},
{"$sort":{"modifications.modified":-1}},
{"$group":{
"_id" : "$auditedId",
"modifications" : {$first:"$modifications"}}}
])
For a given record id, how do I get the average of a sub document field if I have the following in MongoDB:
/* 0 */
{
"item" : "1",
"samples" : [
{
"key" : "test-key",
"value" : "1"
},
{
"key" : "test-key2",
"value" : "2"
}
]
}
/* 1 */
{
"item" : "1",
"samples" : [
{
"key" : "test-key",
"value" : "3"
},
{
"key" : "test-key2",
"value" : "4"
}
]
}
I want to get the average of the values where key = "test-key" for a given item id (in this case 1). So the average should be $avg (1 + 3) = 2
Thanks
You'll need to use the aggregation framework. The aggregation will end up looking something like this:
db.stack.aggregate([
{ $match: { "samples.key" : "test-key" } },
{ $unwind : "$samples" },
{ $match : { "samples.key" : "test-key" } },
{ $project : { "new_key" : "$samples.key", "new_value" : "$samples.value" } },
{ $group : { `_id` : "$new_key", answer : { $avg : "$new_value" } } }
])
The best way to think of the aggregation framework is like an assembly line. The query itself is an array of JSON documents, where each sub-document represents a different step in the assembly.
Step 1: $match
The first step is a basic filter, like a WHERE clause in SQL. We place this step first to filter out all documents that do not contain an array element containing test-key. Placing this at the beginning of the pipeline allows the aggregation to use indexes.
Step 2: $unwind
The second step, $unwind, is used for separating each of the elements in the "samples" array so we can perform operations across all of them. If you run the query with just that step, you'll see what I mean.
Long story short :
{ name : "bob",
children : [ {"name" : mary}, { "name" : "sue" } ]
}
becomes two documents :
{ name : "bob", children : [ { "name" : mary } ] }
{ name : "bob", children : [ { "name" : sue } ] }
Step 3: $match
The third step, $match, is an exact duplicate of the first $match stage, but has a different purpose. Since it follows $unwind, this stage filters out previous array elements, now documents, that don't match the filter criteria. In this case, we keep only documents where samples.key = "test-key"
Step 4: $project (Optional)
The fourth step, $project, restructures the document. In this case, I pulled the items out of the array so I could reference them directly. Using the example above..
{ name : "bob", children : [ { "name" : mary } ] }
becomes
{ new_name : "bob", new_child_name : mary }
Note that this step is entirely optional; later stages could be completed even without this $project after a few minor changes. In most cases $project is entirely cosmetic; aggregations have numerous optimizations under the hood such that manually including or excluding fields in a $project should not be necessary.
Step 5: $group
Finally, $group is where the magic happens. The _id value what you will be "grouping by" in the SQL world. The second field is saying to average over the value that I defined in the $project step. You can easily substitute $sum to perform a sum, but a count operation is typically done the following way: my_count : { $sum : 1 }.
The most important thing to note here is that the majority of the work being done is to format the data to a point where performing the operation is simple.
Final Note
Lastly, I wanted to note that this would not work on the example data provided since samples.value is defined as text, which can't be used in arithmetic operations. If you're interested, changing the type of a field is described here: MongoDB How to change the type of a field
I have a document with sub-document which looks something like:
{
"name" : "some name1"
"like" : [
{ "date" : ISODate("2012-11-30T19:00:00Z") },
{ "date" : ISODate("2012-12-02T19:00:00Z") },
{ "date" : ISODate("2012-12-01T19:00:00Z") },
{ "date" : ISODate("2012-12-03T19:00:00Z") }
]
}
Is it possible to fetch documents "most liked" (average value for the last 7 days) and sort by the count?
There are a few different ways to solve this problem. The solution I will focus on uses mongodb's aggregation framework. First, here is an aggregation pipeline that will solve your problem, following it will be an explanation/breakdown of what is happening in the command.
db.testagg.aggregate(
{ $unwind : '$likes' },
{ $group : { _id : '$_id', numlikes : { $sum : 1 }}},
{ $sort : { 'numlikes' : 1}})
This pipeline has 3 main commands:
1) Unwind: this splits up the 'likes' field so that there is 1 'like' element per document
2) Group: this regroups the document using the _id field, incrementing the numLikes field for every document it finds. This will cause numLikes to be filled with a number equal to the number of elements that were in "likes" before
3) Sort: Finally, we sort the return values in ascending order based on numLikes. In a test I ran the output of this command is:
{"result" : [
{
"_id" : 1,
"numlikes" : 1
},
{
"_id" : 2,
"numlikes" : 2
},
{
"_id" : 3,
"numlikes" : 3
},
{
"_id" : 4,
"numlikes" : 4
}....
This is for data inserted via:
for (var i=0; i < 100; i++) {
db.testagg.insert({_id : i})
for (var j=0; j < i; j++) {
db.testagg.update({_id : i}, {'$push' : {'likes' : j}})
}
}
Note that this does not completely answer your question as it avoids the issue of picking the date range, but it should hopefully get you started and moving in the right direction.
Of course, there are other ways to solve this problem. One solution might be to just do all of the sorting and manipulations client-side. This is just one method for getting the information you desire.
EDIT: If you find this somewhat tedious, there is a ticket to add a $size operator to the aggregation framework, I invite you to watch and potentially upvote it to try and speed to addition of this new operator if you are interested.
https://jira.mongodb.org/browse/SERVER-4899
A better solution would be to keep a count field that will record how many likes for this document. While you can use aggregation to do this, the performance will likely be not very good. Having a index on the count field will make read operation fast, and you can use atomic operation to increment the counter when inserting new likes.
You can use this simplify the above aggregation query by the following from mongodb v3.4 onwards:
> db.test.aggregate([
{ $unwind: "$like" },
{ $sortByCount: "$_id" }
]).pretty()
{ "_id" : ObjectId("5864edbfa4d3847e80147698"), "count" : 4 }
Also as #ACE said you can now use $size within a projection instead:
db.test.aggregate([
{ $project: { count: { $size : "$like" } } }
]);
{ "_id" : ObjectId("5864edbfa4d3847e80147698"), "count" : 4 }