How is the aggregation achieved with MongoDB ?
using mongoDB for some near real time aggregation and how it scale if the aggregation pipeline results are large ?
Is there any performance tuning methods during aggregation
Is there any performance tuning methods during aggregation
There are many. You should refer the documentation about query optimization
using mongoDB for some near real time aggregation and how it scale if the aggregation pipeline results are large ?
Refer this
Limit the number of documents - to handle network demand
use projection to return only the necessary fields.
Related
Is there any documentation that talks about MongoDb aggregation pipeline operators' algorithm performance. For example: If I do a $merge involving 5 records and the same done on 500 records, what is the order of the algorithm? O(1) or O(n). Similarly, I would like know know what each operator is depdendent on. Please help
I have a doubt like,i am using mongodb aggregation framework but i have multiple stages of $lookup in the pipeline does it going to affect the performance.Is there any limitation on number of stages in the aggregation pipeline?
There is no limitation on the number of stages in a pipeline. However, there are result size and memory limitations, refer to the online doc. $lookup doesn't, at least for now, take advantage of indexes. The more data and stages you have, the more time mongo engines needs to process.
The MongoDB documentation says about map-reduce:
For most aggregation operations, the Aggregation Pipeline provides better performance and more coherent interface. However, map-reduce operations provide some flexibility that is not presently available in the aggregation pipeline.
Does it mean that there are some aggregation operations that cannot be performed in the usual MongoDB aggregation framework but are possible using map-reduce?
In particular I'm looking for an example of map-reduce that cannot be implemented in the MongoDB aggregation framework
Thanks!
An example of "flexibility". Basically, if you have any logic, that does not fit into standard aggregation operators, map-reduce is the only option to do it serverside.
I wanted to know about some examples or scenarios related to Mongo DB which can be done by map-reduce but not aggregation framework ?
Map-reduce is considered to be very powerful tool/mechanism of aggregating data. Then can some of you please share few scenarios where it is not possible for map-reduce to do it ?
Thanks & Best Regards.
In MongoDB currently aggregation framework is limited to 16MB of returned results.
MapReduce can write its output to a collection and has no size limitations.
MapReduce can group entire documents, aggregation framework works on field level. MapReduce can map keys to values and values to keys which can't be done any other way. MapReduce can also call/use various JavaScript built-in functions where aggregation is limited to functions and expressions which are built-in to its framework.
I know that aggregation framework is suitable if there is an initial $match pipeline to limit the collection to be aggregated. However, there may be times that the filtered collection may still be large, say around 2 million and the aggregation will involve $group. Is the aggregation framework fit to work on such a collection given a requirement to output results in at most 5 seconds. Currently I work on a single node. By performing the aggregation on a shard set, will there be a significant improvement in the performance.
As far as I know the only limitations are that the result of the aggregation can't surpass the limit of 16MB, since what it returns is a document and that's the limit size for a document in MongoDB. Also you can't use more than 10% of the total memory of the machine, for that usually $match phases are used to reduce the set you work with, or a $project phase to reduce the data per document.
Be aware that in a sharded environment after $group or $sort phases the aggregation is brought back to the MongoS before sending it to the next phase of the pipeline. Potentially the MongoS could be running in the same machine as your application and could hurt your application performance if not handled correctly.