Spring data mongodb - aggregation framework integration - mongodb

I started to use MongoDB database in my application and for data access I have chosen Spring Data for MongoDB.
I skimmed API reference and documentation and I can see that there is map-reduce integration but what about aggregation framework? I can see that it supports group by operation, which would indicate that it supports $group operator judging from this: http://docs.mongodb.org/manual/reference/sql-aggregation-comparison/, but what about other operators, are that not supported for now?
I am asking this question because I wanted to know what kind of integration with MongoDB Sping Data provides so I know what to expect, so to speak.

Spring Data 1.3.0.RC1 is available and it does support the aggregation framework.
For example:
The shell aggregation comand:
db.eft_transactions.aggregate(
{$match:
{
service:"EFT",
source:"MARKUP",
}
},
{$group:
{
_id:"$card_acceptor_id",
tran_count:{$sum:1},
amount_sum:{$sum:"$amount"}
}
}
)
is run like this from java:
AggregationOperation match = Aggregation.match(Criteria.where("service").is("EFT").and("source").is("MARKUP"));
AggregationOperation group = Aggregation.group("card_acceptor").and("amount_sum").sum("amount").and("tran_count").count();
Aggregation aggregation = newAggregation(match, group);
AggregationResults<StoreSummary> result = this.mongoTemplate.aggregate(aggregation, "eft_transactions", StoreSummary.class);
The documentation is here
NOTE: We recently had to switch to using the BUILD-SNAPSHOT build of version 1.3.0. This change necessitated the change to 2 of the above lines which have changed to:
AggregationOperation group = Aggregation.group("card_acceptor").sum("amount").as("amount_sum").count().as("tran_count");
Aggregation aggregation = Aggregation.newAggregation(match, group);

The Spring Data MongoOperations.group() method is mapped to db.collection.group() MongoDB command and not the $group aggregation function. Currently there is no support in Spring Data MongoDB for aggregation framework. Map reduce, as you have mentioned, is supported though

Aggregation aggregation = newAggregation(
match(Criteria.where("salesyear").is(year)),
group("brand","salesyear").sum("numberOfCars").as("total"),
sort(Sort.Direction.ASC, previousOperation(), "brand")
);

Here is how to get the sum of a particular field.
private Map<String, Long> getTotalMap(){
/*
db.pDSSummaryModel.aggregate([{
$group: {
_id: null,
total: {
$sum: '$totalUniqueCustomerCount'
}
}
}])
*/
Aggregation aggregations = newAggregation(
group("null").sum("totalUniqueUserCount").as("userTotal")
.sum("totalUniqueCustomerCount").as("customerTotal"),
project("customerTotal", "userTotal")
);
AggregationResults<DBObject> results = mongoTemplate.aggregate(aggregations, "pDSSummaryModel", DBObject.class);
List<DBObject> fieldList = results.getMappedResults();
Map<String, Long> map = new HashMap<>();
if(fieldList != null && !fieldList.isEmpty()) {
for(DBObject db: fieldList){
map.put("userTotal", parseLong(db.get("userTotal").toString()));
map.put("customerTotal", parseLong(db.get("customerTotal").toString()));
}
}
return map;
}

Related

How to use DateOperators in Spring Mongo Data aggregation

We have an aggregation pipeline setup in Spring Data Mongo like so:
MatchOperation matchStage = ...
Fields groupingFields = Fields.fields();
groupingFields.and(name1, target1);
groupingFields.and(name2, target2);
...
GroupOperation groupStage = Aggregation.group(groupingFields);
List<AggregationOperation> aggStages = new ArrayList<>();
aggStages.add(matchStage);
aggStages.add(groupStage);
Aggregation aggregation = Aggregation.newAggregation(aggStages);
Now, we want to be able to use aggregation over dates using Date operators in mongodb. This is fairly straightforward in mongodb, example below:
db.getCollection('Collection').aggregate([
{"$match": {"state": "XY"}},
{"$group": {
"_id": {
"city": "$city",
"dayOfYear": {"$dayOfYear": "$date"}
},
"totalProfit": {"$sum": "$profit"}
}}
])
My question is, how can I use the $dayOfYear operator in the Spring pipeline we have. Spring has support for DateOperators like DateOperators.DayOfWeek etc. but I am unable to incorporate it into the pipeline as we have it. How do I modify the groupStage so I can group by various date related parts as required?
For some operation I used to follow Bson document styles
#Autowired
private MongoTemplate mongoTemplate;
public List<Object> test() {
Aggregation aggregation = Aggregation.newAggregation(
match(Criteria.where("state").is("XY"))
p-> new Document("$group",
new Document("_id",
new Document("city","$city")
.append("dayOfYear",
new Document("$dayOfYear", "$date")
)
).append("totalProfit",
new Document("$sum","$$profit")
)
)
).withOptions(AggregationOptions.builder().allowDiskUse(Boolean.TRUE).build());
return mongoTemplate.aggregate(aggregation, mongoTemplate.getCollectionName(YOUR_COLLECTION.class), Object.class).getMappedResults();
}
This should work, if the above aggregation you posted is working. You can refer Trick to convert.
I haven't tried DateOperator yet. But I'm curious how does it work. I will update if I get to know about it. Until then you have no blockers

Spring WebFlux + MongoDB: Tailable Cursor and Aggregation

I´m new with WebFlux and MongoDB. I´m trying to use aggregation in a capped collection with tailable cursor, but I´m nothing getting sucessful.
I´d like to execute this mongoDB query:
db.structures.aggregate(
[
{
$match: {
id: { $in: [8244, 8052]}
}
},
{ $sort: { id: 1, lastUpdate: 1} },
{
$group:
{
_id: {id: "$id"},
lastUpdate: { $last: "$lastUpdate" }
}
}
]
)
ReactiveMongoOperations gives me option to "tail" or "aggregation".
I´m able to execute aggregation:
MatchOperation match = new MatchOperation(Criteria.where("id").in(8244, 8052));
GroupOperation group = Aggregation.group("id", "$id").last("$lastUpdate").as("lastUpdate");
Aggregation aggregate = Aggregation.newAggregation(match, group);
Flux<Structure> result = mongoOperation.aggregate(aggregate,
"structures", Structure.class);
Or tail cursor
Query query = new Query();
query.addCriteria(Criteria.where("id").in(8244, 8052));
Flux<Structure> result = mongoOperation.tail(query, Structure.class);
Is it possible? Tail and Aggregation together?
Using aggregation was the way that I found to get only the last inserted document for each id.
Without aggregation I get:
query without aggregation
With aggregation:
query with aggregation
Tks in advance
The tailable cursor query creates a Flux that never completes (never emits onComplete event) and that Flux emits records as they are inserted in the database. Because of that fact I would think aggregations are not allowed by the database engine with the tailable cursor.
So the aggregation doesn't make sense in a way because on every newly inserted record the aggregation would need to be recomputed. Technically you can do a running aggregation where for every returned record you compute the wanted aggregate record and send it downstream.
One possible solution would be to do the aggregations programmatically on the returned "infinite" Flux:
mongoOperation.tail(query, Structure.class)
.groupBy(Structure::id) // create independent Fluxes based on id
.flatMap(groupedFlux ->
groupedFlux.scan((result, nextStructure) -> { // scan is like reduce but emits intermediate results
log.info("intermediate result is: {}", result);
if (result.getLastUpdate() > nextStructure.getLastUpdate()) {
return result;
} else {
result.setLastUpdate(nextStructure.getLastUpdate());
return result;
}
}));
On the other hand you should probably revisit your use case and what you need to accomplish here and see if something other than capped collection should be used or maybe the aggregation part is redundant (i.e. if newly inserted records always have the lastUpdate property larger then the previous record).

spring mongodb - How to provide match condition to check for empty array using spring data mongodb api?

how to perform below operation (it is the actual mongo query) using MatchOperation in Spring data mongodb ?
$match: { "docs": { $ne: [] } }
here docs is an array field and want to check that it is not empty.
I also had a similar problem, but I solved is as below.
MatchOperation mathOpertaion = match(Criteria.where("docs")
.elemMatch(new Criteria().exists(true)));

How to get Count of aggregation query in spring mongo template

I'm using spring mongo template to run an agreegation query on mongodb. I'm wondering is there any way to find out the count of aggregation result in spring mongo template?
Here is my Aggregation sample :
Aggregation agg = newAggregation(Class1.class,
match(criteria),
group("prop1", "prop2")
.avg("x").as("averageX")
);
I just need to know how to get count of this aggregation result in spring mongo template.
My response comes very late but it might help others. To get the count for an aggregation you need to add a new group at the end:
Add at the end of the aggregation -> Aggregation.group().count().as("count") to get the count
Aggregation aggregation = newAggregation(
Aggregation.match(Criteria.where("x").is(x).and("y").exists(true)),
Aggregation.group("x", "y"),
Aggregation.group().count().as("count")
);
To get the count:
Long.parseLong(results.getMappedResults().get(0).get("count").toString());
for spring data mongo 2.1.4
if you assume your return class is like this:
class Output{
Object x;
Object y;
/**getter and setter**/
}
you can use this code:
1.Hold grouping data:
AggregationResults<Output> result =
mongoTemplate.aggregate(
agg ,
"YOUR_DOCUMENT",
Output.class
);
int count = result.getMappedResults().size();
2. only get count: (grouping not effect until you use first(...) or last(...) usage after that)
Aggregation agg = Aggregation.newAggregation(
match(criteria),
count().as("x")/*using x only for mapping count*/
);
AggregationResults<Output> result =
mongoTemplate.aggregate(
agg ,
"YOUR_DOCUMENT",
Output.class
);
int count = result.getMappedResults().get(0).getX().get(0);/**o_O**/

Export mongodb aggregation framework result to a new collection

I want to save the aggregation framework result to a new collection.
I know that's impossible with the framework at the moment with the command itself.
Is there a workaround in the shell?
Starting with Mongo 2.6.0 you can do this natively without any additional manipulation.
db.<collection>.aggregate( [
{ <operation> },
{ <operation> },
...,
{ $out : "<output-collection>" }
] )
Check the new aggregation operator $out for more detailed example.
P.S. using this way you are not limited to 16Mb size.
Update: See Salvador's answer for a more efficient way to do this in MongoDB 2.6+.
Save the aggregation result in a variable and then insert its result property into a new collection:
var result = db.foo.aggregate(...);
db.bar.insert(result.result);
result.result is no longer working, try toArray().
var result = db.coll.aggregate(...);
db.bar.insert(result.toArray());