How to use DateOperators in Spring Mongo Data aggregation - mongodb

We have an aggregation pipeline setup in Spring Data Mongo like so:
MatchOperation matchStage = ...
Fields groupingFields = Fields.fields();
groupingFields.and(name1, target1);
groupingFields.and(name2, target2);
...
GroupOperation groupStage = Aggregation.group(groupingFields);
List<AggregationOperation> aggStages = new ArrayList<>();
aggStages.add(matchStage);
aggStages.add(groupStage);
Aggregation aggregation = Aggregation.newAggregation(aggStages);
Now, we want to be able to use aggregation over dates using Date operators in mongodb. This is fairly straightforward in mongodb, example below:
db.getCollection('Collection').aggregate([
{"$match": {"state": "XY"}},
{"$group": {
"_id": {
"city": "$city",
"dayOfYear": {"$dayOfYear": "$date"}
},
"totalProfit": {"$sum": "$profit"}
}}
])
My question is, how can I use the $dayOfYear operator in the Spring pipeline we have. Spring has support for DateOperators like DateOperators.DayOfWeek etc. but I am unable to incorporate it into the pipeline as we have it. How do I modify the groupStage so I can group by various date related parts as required?

For some operation I used to follow Bson document styles
#Autowired
private MongoTemplate mongoTemplate;
public List<Object> test() {
Aggregation aggregation = Aggregation.newAggregation(
match(Criteria.where("state").is("XY"))
p-> new Document("$group",
new Document("_id",
new Document("city","$city")
.append("dayOfYear",
new Document("$dayOfYear", "$date")
)
).append("totalProfit",
new Document("$sum","$$profit")
)
)
).withOptions(AggregationOptions.builder().allowDiskUse(Boolean.TRUE).build());
return mongoTemplate.aggregate(aggregation, mongoTemplate.getCollectionName(YOUR_COLLECTION.class), Object.class).getMappedResults();
}
This should work, if the above aggregation you posted is working. You can refer Trick to convert.
I haven't tried DateOperator yet. But I'm curious how does it work. I will update if I get to know about it. Until then you have no blockers

Related

$lookup is applied again on all subfields of a looked up object

I'm using Spring Data MongoDB and applying $lookup on a field with ObjectId value,
what I'm getting is the referenced document like I want but also the fields of that document are also populated, $lookup is applied again on the fields of the referenced document which causes the query to take a very long time to execute (around 5mins).
The query works fine using a raw query from a mongo client without spring data MongoDB, the lookup is applied only once and I get a document that has ObjectId value in the fields instead of those fields being populated too.
This is the raw query
{$match : {_id : new ObjectId("63282356bb1b311dd0e63f67")}},
{$lookup : {
"from":"paper",
"let":{
"userId":"$_id"
},
"pipeline":[
{$facet:{
sent:[
{$match:{
"$expr":{"$eq":["$to", "$$userId"]}
}}
],
}}
],
"as":"papers"
}}
])
And I'm translating this to spring data mongodb like this
Aggregation aggregation = Aggregation.newAggregation(
matchStage,
aggregationOperationContext -> {
Document lookupPipeline = new Document("$lookup",
new Document("from", "paper")
.append("let", new Document("userId", "$_id"))
.append("pipeline", Arrays.<Object> asList(
new Document("$facet",
new Document("sent",Arrays.<Object>asList(
new Document("$match", new Document("$expr",
new Document("$eq", Arrays.<Object>asList("$to", "$$userId"))
))
)
)
)
))
.append("as", "papers")
);
return aggregationOperationContext.getMappedObject(lookupPipeline);
}
);
I'm using the custom aggregation because using $lookup with the pipeline field is not supported in spring data MongoDB as of now
Also I'm using #DocumentReference
public class User{
#ReadOnlyProperty
#DocumentReference
private List<Paper> papers;
}

Aggregation from MongoDB to Spring Boot Aggregation Framework

I am new to Spring And Mongo. I am using Spring Batch for getting some reports. My Query needs some aggregation which is not supported by MongoItemReader so I extended that class as per below stackoverflow link.
How to use Aggregation Query with MongoItemReader in spring batch
But there is something wrong with my Aggregation. I made aggregation which works fine in mongoDB but not able to convert it into Spring mongo aggregation.
MongoDb Aggregation which works as expected.
db.getCollection('orders').aggregate([
{$match: {orderDate: {$gt:"2021-03-15",$lt: "2021-03-17"}, "status" :{"$in": ["GREEN", "YELLOW"]}}},
{$group: {_id: {orderDate: "$orderDate", node: "$node", code1:"$code1", code2:"$code2"}, orderUnts: {$sum: 1}}},
{"$project": {orderDate:"$_id.orderDate", node:"$_id.node", code1:"$_id.code1", code2:"$_id.code2", orderUnts:"$orderUnts"}}
])
Spring Mongo Aggregation which gives error.
String[] fields = {"orderDate", "node", "code1", "code2"};
String[] projectionFields = {"orderDate", "orderDate", "code1", "code2"};
MatchOperation matchOp = Aggregation.match(Criteria.where("orderDate").gt(startDate).and("orderDate").lt(endDate).and("status").in("GREEN", "YELLOW"));
GroupOperation groupOp = Aggregation.group(fields).sum("orderUnts").as("_id");
ProjectionOperation projectOp = Aggregation.project(projectionFields);
SortOperation sortOp = Aggregation.sort(Sort.by(Sort.Direction.ASC, "orderDate"));
Aggregation aggregation = Aggregation.newAggregation(matchOp, groupOp, projectOp, sortOp);
I am getting below unique field error.
Caused by: java.lang.IllegalStateException: An implementation of MongoOperations is required.
at org.springframework.util.Assert.state(Assert.java:76)
at org.springframework.batch.item.data.MongoItemReader.afterPropertiesSet(MongoItemReader.java:238)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1847)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1784)
... 81 common frames omitted
MongoItemReader extended class.
#Data
public class CustomMongoItemReader<T> extends MongoItemReader<T> {
private MongoTemplate template;
private Class<? extends T> type;
private String collection;
private MatchOperation matchOperation;
private GroupOperation groupOperation;
private ProjectionOperation projectionOperation;
private SortOperation sortOperation;
private Aggregation aggregation;
#Override
protected Iterator<T> doPageRead() {
Pageable page = PageRequest.of(this.page, this.pageSize);
if(matchOperation != null && groupOperation != null) {
Aggregation agg = Aggregation.newAggregation(matchOperation,
groupOperation,
projectionOperation,
sortOperation,
Aggregation.skip(Long.valueOf(page.getPageNumber() * page.getPageSize())),
Aggregation.limit(page.getPageSize())
);
return (Iterator<T>) template.aggregate(agg, collection, this.type).iterator();
}
else {
return Collections.emptyIterator();
}
}
}
Let me know if question needs more info. Thanks in Advance.
java.lang.IllegalStateException: An implementation of MongoOperations is required.
Your item reader extends MongoItemReader which requires a MongoOperations (typically a MongoTemplate). This error means that your item reader was not configured with a MongoOperations.
You need to set one on the reader before using it.
Also override with afterPropertiesSet() throws Exception{} with properties which are required for aggregation, for Reference check MongoItemReader<T>.

mongodb collation doesn't work insinde aggregation with facet stage

I want to sort my products after using aggregation, in my aggregation pipeline i use $sort inside $facet stage. Actually sorting was working just fine for all fields until i try to sort a string field(Name) ascending/descending.
Problem is sorting doesn't work properly in my language which is Turkish. So i am using Collation options.
I am using C#.
Normally i use Collation locale setting like below
PipelineDefinition<BsonDocument, Model> pipeline = new BsonDocument[]
{
//some stages,
new BsonDocument().Add("$sort", new BsonDocument().Add("sortingParameter", -1)),
//some stages
};
var options = new AggregateOptions() { Collation = new Collation("tr") };
var cursor = _collection.Aggregate(pipeline, options);
and it is working, no problem here.
But some of my queries more complex than above example, such as
(below i am showing shortened BsonDocument version of my query)
db.getCollection('collName').aggregate([
{"$match": { // match operations }
},
{"$facet": {
"product_list": [
{"$group": {"_id": { /some grouping parameters }}},
{ "$sort": { "_id.Name": -1.0 } },
{ "$project": { // some projects }}
]}}])
for above query i use the same aggregate options
var options = new AggregateOptions() { Collation = new Collation("tr") };
but sorted results are incorrect. When i try to remove $match stage i don't know how but it works.
So, is there any way to solve this or is it impossible with this kind of query?
Instead of using collation-locale setting inside the AggregateOptions, if i create the collection with collation-locale setting, the query works properly.

Spring Mongodata Aggregation query

I tried to implement an aggregation with the simple sum operation using spring-data-mongodb -> 1.10.11.RELEASE
I tried the following query
db.transaction.aggregate([{
$group: {
_id:"null",
netBalance: {
$sum: "$netBalance"
},
referalBalance: {
$sum: "$referalBalance"
}
}
}])
And the output on my terminal is
{ "_id" : "null", "netBalance" : 587432, "referalBalance" : 2940 }
When I tried same query, using spring mongodata
Aggregation aggregation =
Aggregation.newAggregation(
Aggregation.group("netBalance")
.sum("netBalance")
.as("netBalance"));
The result is not same as the terminal output, also I am not able to add a second field on the query. How can I modify the spring mongodata call to have the same query?
with spring you're grouping by netBalance instead of null in shell. To reproduce, just leave group param empty (null will throw an error)
Then you can apply sum on second field .
Try this code (not tested, but must work):
Aggregation aggregation =
Aggregation.newAggregation(
Aggregation.group()
.sum("netBalance").as("netBalance")
.sum("referalBalance").as("referalBalance"));

Spring data mongodb - aggregation framework integration

I started to use MongoDB database in my application and for data access I have chosen Spring Data for MongoDB.
I skimmed API reference and documentation and I can see that there is map-reduce integration but what about aggregation framework? I can see that it supports group by operation, which would indicate that it supports $group operator judging from this: http://docs.mongodb.org/manual/reference/sql-aggregation-comparison/, but what about other operators, are that not supported for now?
I am asking this question because I wanted to know what kind of integration with MongoDB Sping Data provides so I know what to expect, so to speak.
Spring Data 1.3.0.RC1 is available and it does support the aggregation framework.
For example:
The shell aggregation comand:
db.eft_transactions.aggregate(
{$match:
{
service:"EFT",
source:"MARKUP",
}
},
{$group:
{
_id:"$card_acceptor_id",
tran_count:{$sum:1},
amount_sum:{$sum:"$amount"}
}
}
)
is run like this from java:
AggregationOperation match = Aggregation.match(Criteria.where("service").is("EFT").and("source").is("MARKUP"));
AggregationOperation group = Aggregation.group("card_acceptor").and("amount_sum").sum("amount").and("tran_count").count();
Aggregation aggregation = newAggregation(match, group);
AggregationResults<StoreSummary> result = this.mongoTemplate.aggregate(aggregation, "eft_transactions", StoreSummary.class);
The documentation is here
NOTE: We recently had to switch to using the BUILD-SNAPSHOT build of version 1.3.0. This change necessitated the change to 2 of the above lines which have changed to:
AggregationOperation group = Aggregation.group("card_acceptor").sum("amount").as("amount_sum").count().as("tran_count");
Aggregation aggregation = Aggregation.newAggregation(match, group);
The Spring Data MongoOperations.group() method is mapped to db.collection.group() MongoDB command and not the $group aggregation function. Currently there is no support in Spring Data MongoDB for aggregation framework. Map reduce, as you have mentioned, is supported though
Aggregation aggregation = newAggregation(
match(Criteria.where("salesyear").is(year)),
group("brand","salesyear").sum("numberOfCars").as("total"),
sort(Sort.Direction.ASC, previousOperation(), "brand")
);
Here is how to get the sum of a particular field.
private Map<String, Long> getTotalMap(){
/*
db.pDSSummaryModel.aggregate([{
$group: {
_id: null,
total: {
$sum: '$totalUniqueCustomerCount'
}
}
}])
*/
Aggregation aggregations = newAggregation(
group("null").sum("totalUniqueUserCount").as("userTotal")
.sum("totalUniqueCustomerCount").as("customerTotal"),
project("customerTotal", "userTotal")
);
AggregationResults<DBObject> results = mongoTemplate.aggregate(aggregations, "pDSSummaryModel", DBObject.class);
List<DBObject> fieldList = results.getMappedResults();
Map<String, Long> map = new HashMap<>();
if(fieldList != null && !fieldList.isEmpty()) {
for(DBObject db: fieldList){
map.put("userTotal", parseLong(db.get("userTotal").toString()));
map.put("customerTotal", parseLong(db.get("customerTotal").toString()));
}
}
return map;
}