I am new to Spring And Mongo. I am using Spring Batch for getting some reports. My Query needs some aggregation which is not supported by MongoItemReader so I extended that class as per below stackoverflow link.
How to use Aggregation Query with MongoItemReader in spring batch
But there is something wrong with my Aggregation. I made aggregation which works fine in mongoDB but not able to convert it into Spring mongo aggregation.
MongoDb Aggregation which works as expected.
db.getCollection('orders').aggregate([
{$match: {orderDate: {$gt:"2021-03-15",$lt: "2021-03-17"}, "status" :{"$in": ["GREEN", "YELLOW"]}}},
{$group: {_id: {orderDate: "$orderDate", node: "$node", code1:"$code1", code2:"$code2"}, orderUnts: {$sum: 1}}},
{"$project": {orderDate:"$_id.orderDate", node:"$_id.node", code1:"$_id.code1", code2:"$_id.code2", orderUnts:"$orderUnts"}}
])
Spring Mongo Aggregation which gives error.
String[] fields = {"orderDate", "node", "code1", "code2"};
String[] projectionFields = {"orderDate", "orderDate", "code1", "code2"};
MatchOperation matchOp = Aggregation.match(Criteria.where("orderDate").gt(startDate).and("orderDate").lt(endDate).and("status").in("GREEN", "YELLOW"));
GroupOperation groupOp = Aggregation.group(fields).sum("orderUnts").as("_id");
ProjectionOperation projectOp = Aggregation.project(projectionFields);
SortOperation sortOp = Aggregation.sort(Sort.by(Sort.Direction.ASC, "orderDate"));
Aggregation aggregation = Aggregation.newAggregation(matchOp, groupOp, projectOp, sortOp);
I am getting below unique field error.
Caused by: java.lang.IllegalStateException: An implementation of MongoOperations is required.
at org.springframework.util.Assert.state(Assert.java:76)
at org.springframework.batch.item.data.MongoItemReader.afterPropertiesSet(MongoItemReader.java:238)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1847)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1784)
... 81 common frames omitted
MongoItemReader extended class.
#Data
public class CustomMongoItemReader<T> extends MongoItemReader<T> {
private MongoTemplate template;
private Class<? extends T> type;
private String collection;
private MatchOperation matchOperation;
private GroupOperation groupOperation;
private ProjectionOperation projectionOperation;
private SortOperation sortOperation;
private Aggregation aggregation;
#Override
protected Iterator<T> doPageRead() {
Pageable page = PageRequest.of(this.page, this.pageSize);
if(matchOperation != null && groupOperation != null) {
Aggregation agg = Aggregation.newAggregation(matchOperation,
groupOperation,
projectionOperation,
sortOperation,
Aggregation.skip(Long.valueOf(page.getPageNumber() * page.getPageSize())),
Aggregation.limit(page.getPageSize())
);
return (Iterator<T>) template.aggregate(agg, collection, this.type).iterator();
}
else {
return Collections.emptyIterator();
}
}
}
Let me know if question needs more info. Thanks in Advance.
java.lang.IllegalStateException: An implementation of MongoOperations is required.
Your item reader extends MongoItemReader which requires a MongoOperations (typically a MongoTemplate). This error means that your item reader was not configured with a MongoOperations.
You need to set one on the reader before using it.
Also override with afterPropertiesSet() throws Exception{} with properties which are required for aggregation, for Reference check MongoItemReader<T>.
Related
I have a collection named Location which has couple of attributes of which I want to fetch only distinct country code and country name.
Below query works fine in Mongo DB.
db.location.aggregate([{"$group": {"_id": { countryCode: "$countryCode", countryName: "$countryName" }}}]);
I want to convert same query into SpringMongoReactiveAggretate Query. Below code has some issues. Please help me figure frame the correct code.
#Repository
public class AggregateQueryRepository {
#Autowired
ReactiveMongoTemplate reactiveMongoTemplate;
public Flux<Location> getAllCountryCodeAndCountry(String countryCode, String countryName) {
Aggregation aggregation = newAggregation(
match(Criteria.where("_id").is(countryCode).and(countryName)),
group("_id").push("location").as("location")
);
Flux<Location> output
= reactiveMongoTemplate.aggregate(aggregation, "location", Location.class);
return output;
}
}
You don't need match stage and you should pass the field names to group as parameter:
public Flux<Location> getAllCountryCodeAndCountry() {
Aggregation aggregation = newAggregation(
group("countryCode", "countryName")
);
return reactiveMongoTemplate.aggregate(aggregation, "location", Location.class);
}
We have an aggregation pipeline setup in Spring Data Mongo like so:
MatchOperation matchStage = ...
Fields groupingFields = Fields.fields();
groupingFields.and(name1, target1);
groupingFields.and(name2, target2);
...
GroupOperation groupStage = Aggregation.group(groupingFields);
List<AggregationOperation> aggStages = new ArrayList<>();
aggStages.add(matchStage);
aggStages.add(groupStage);
Aggregation aggregation = Aggregation.newAggregation(aggStages);
Now, we want to be able to use aggregation over dates using Date operators in mongodb. This is fairly straightforward in mongodb, example below:
db.getCollection('Collection').aggregate([
{"$match": {"state": "XY"}},
{"$group": {
"_id": {
"city": "$city",
"dayOfYear": {"$dayOfYear": "$date"}
},
"totalProfit": {"$sum": "$profit"}
}}
])
My question is, how can I use the $dayOfYear operator in the Spring pipeline we have. Spring has support for DateOperators like DateOperators.DayOfWeek etc. but I am unable to incorporate it into the pipeline as we have it. How do I modify the groupStage so I can group by various date related parts as required?
For some operation I used to follow Bson document styles
#Autowired
private MongoTemplate mongoTemplate;
public List<Object> test() {
Aggregation aggregation = Aggregation.newAggregation(
match(Criteria.where("state").is("XY"))
p-> new Document("$group",
new Document("_id",
new Document("city","$city")
.append("dayOfYear",
new Document("$dayOfYear", "$date")
)
).append("totalProfit",
new Document("$sum","$$profit")
)
)
).withOptions(AggregationOptions.builder().allowDiskUse(Boolean.TRUE).build());
return mongoTemplate.aggregate(aggregation, mongoTemplate.getCollectionName(YOUR_COLLECTION.class), Object.class).getMappedResults();
}
This should work, if the above aggregation you posted is working. You can refer Trick to convert.
I haven't tried DateOperator yet. But I'm curious how does it work. I will update if I get to know about it. Until then you have no blockers
I tried creating a Custom AggregationOperation based on https://github.com/krishnaiitd/learningJava/tree/master/spring-boot-sample-data-mongodb
When I used a custom aggregation in my aggregation for a lookup, it threw an exception saying the "as" field is not found on the entity.
If anybody has tried using custom AggregationOperation please share your code or let me know where I am going wrong.
Below is my code,
String lookup = "{ $lookup : { from: 'ITEM', localField : 'item_id', foreignField : '_id', as : 'item' } }";
TypedAggregation<Order> aggregation = Aggregation.newAggregation(Order.class,
new CustomAggregationOperation(lookup),
unwind("item", false));
The exception:
org.springframework.data.mapping.PropertyReferenceException: No property item found for type Order!
A TypedAggregation is a special Aggregation that holds information of the input aggregation type.
https://docs.spring.io/spring-data/mongodb/docs/current/api/org/springframework/data/mongodb/core/aggregation/TypedAggregation.html
That means that Spring will verify after each stage that your documents have not changed the structure.
Since you are trying transform original document, you need to use a standard Aggregation.
Aggregation aggregation = Aggregation.newAggregation(
new CustomAggregationOperation(lookup),
unwind("item", false)
);
List<Order> result = mongoTemplate.aggregate(aggregation, mongoTemplate.getCollectionName(Order.class), Order.class).getMappedResults();
I started to use MongoDB database in my application and for data access I have chosen Spring Data for MongoDB.
I skimmed API reference and documentation and I can see that there is map-reduce integration but what about aggregation framework? I can see that it supports group by operation, which would indicate that it supports $group operator judging from this: http://docs.mongodb.org/manual/reference/sql-aggregation-comparison/, but what about other operators, are that not supported for now?
I am asking this question because I wanted to know what kind of integration with MongoDB Sping Data provides so I know what to expect, so to speak.
Spring Data 1.3.0.RC1 is available and it does support the aggregation framework.
For example:
The shell aggregation comand:
db.eft_transactions.aggregate(
{$match:
{
service:"EFT",
source:"MARKUP",
}
},
{$group:
{
_id:"$card_acceptor_id",
tran_count:{$sum:1},
amount_sum:{$sum:"$amount"}
}
}
)
is run like this from java:
AggregationOperation match = Aggregation.match(Criteria.where("service").is("EFT").and("source").is("MARKUP"));
AggregationOperation group = Aggregation.group("card_acceptor").and("amount_sum").sum("amount").and("tran_count").count();
Aggregation aggregation = newAggregation(match, group);
AggregationResults<StoreSummary> result = this.mongoTemplate.aggregate(aggregation, "eft_transactions", StoreSummary.class);
The documentation is here
NOTE: We recently had to switch to using the BUILD-SNAPSHOT build of version 1.3.0. This change necessitated the change to 2 of the above lines which have changed to:
AggregationOperation group = Aggregation.group("card_acceptor").sum("amount").as("amount_sum").count().as("tran_count");
Aggregation aggregation = Aggregation.newAggregation(match, group);
The Spring Data MongoOperations.group() method is mapped to db.collection.group() MongoDB command and not the $group aggregation function. Currently there is no support in Spring Data MongoDB for aggregation framework. Map reduce, as you have mentioned, is supported though
Aggregation aggregation = newAggregation(
match(Criteria.where("salesyear").is(year)),
group("brand","salesyear").sum("numberOfCars").as("total"),
sort(Sort.Direction.ASC, previousOperation(), "brand")
);
Here is how to get the sum of a particular field.
private Map<String, Long> getTotalMap(){
/*
db.pDSSummaryModel.aggregate([{
$group: {
_id: null,
total: {
$sum: '$totalUniqueCustomerCount'
}
}
}])
*/
Aggregation aggregations = newAggregation(
group("null").sum("totalUniqueUserCount").as("userTotal")
.sum("totalUniqueCustomerCount").as("customerTotal"),
project("customerTotal", "userTotal")
);
AggregationResults<DBObject> results = mongoTemplate.aggregate(aggregations, "pDSSummaryModel", DBObject.class);
List<DBObject> fieldList = results.getMappedResults();
Map<String, Long> map = new HashMap<>();
if(fieldList != null && !fieldList.isEmpty()) {
for(DBObject db: fieldList){
map.put("userTotal", parseLong(db.get("userTotal").toString()));
map.put("customerTotal", parseLong(db.get("customerTotal").toString()));
}
}
return map;
}
I recently spent some time trying to use the $pull operator through Spring's Data MongoOperations interface, so I thought it would be nice to share my findings in case anyone bumps into a similar problem.
So here it goes...
I have 2 java POJOs like so :
#Document
public class OutterObject{
private String id;
private String name;
private List<InnerDocument> innerDocs;
//SETTERS - GETTERS Ommited
public class InnerDocument{
private String id;
private String name;
//SETTERS - GETTERS Ommited
This is stored in a Mongo collection like so :
"_id" : "doc2",
"_class" : "OutterObject",
"name" : "doc2",
"innerDocs" : [{
"_id" : "innerDoc21",
"name" : "innerDoc21"
}, {
"_id" : "innerDoc22",
"name" : "innerDoc22"
}]
I'm trying to use the $pull operator in order to remove all objects inside the innerDoc collection having a name value = "innerDoc22".
I know how to accomplish this using the mongo driver like so :
List<String> ids =
Arrays.asList("innerDoc22");
BasicDBObject find = new BasicDBObject();
match.put("innerDocs.name",
BasicDBObjectBuilder.start("$in", ids).get());
BasicDBObject update = new BasicDBObject();
update.put(
"$pull",
BasicDBObjectBuilder.start("innerDocs",
BasicDBObjectBuilder.start("name", "innerDoc22").get()).get());
DBCollection col= mongoOperations.getDb().getCollection("outterObject");
col.update(find , update);
I'm trying to accomplish the same thing using Spring's MongoOperations Interface.
Here is my code using the MongoOperations interface :
List<String> ids = Arrays.asList("innerDoc22");
Query removeQuery = Query.query(Criteria.where("innerDocs.name").in(ids));
WriteResult wc = mongoOperations.upsert(
removeQuery,
new Update().pull("innerDocs.name", "innerDoc22"),
OutterObject.class);
System.out.println(wc.getLastError());
I'm not getting any errors when calling getLastError() the update is simply not done in the database.
I know a similar question has already been asked here but the answer that was given does not use the MongoOperations interface.
After searching a bit and looking at the source code I realized that I needed to pass an InnerDocument object as a second parameter to the pull method so that the spring classes would be able to do the mapping correctly.
As it turns out I can navigate objects while selecting objects (I'm using "innerDocs.name" in the removeQuery) but I cannot (or havent found a way) do the same when updating a document.
Below is how I implemented the query using MongoOperations :
List<String> ids = Arrays.asList("innerDoc22", "innerDoc21");
Query removeQuery = Query.query(Criteria.where("innerDocs.name").in(ids));
WriteResult wc =
mongoOperations.upsert(removeQuery,
new Update().pull("innerDocs",
new InnerDocument("innerDoc22", null)),
OutterObject.class);
System.out.println(wc.getLastError());
You can also use the BasicDBObject instead of the InnerDocument I found this out by accident. By printing out the update object, you can see the actual mongo shell query json, which is super helpful for debugging.
:
Update updateObj = new Update()
.pull("innerDocs", new BasicDBObject("innerDocs.name","innerDoc22"));
System.out.println("UPDATE OBJ: " + updateObj.toString());
results in:
UPDATE OBJ: { "$pull" : { "innerDocs" : { "innerDocs.name" : "innerDoc22"}}}
I tried the solution given by med116 and I had to modify it to work :
Update updateObj = new Update().pull("innerDocs", new BasicDBObject("name","innerDoc22"));
Because otherwise there was no matching object in my array.
in spring data mongo,just like this:
//remove array's one elem
UpdateResult wc = mongoTemplate.upsert(removeQuery,new Update().pull("tags",Query.query(Criteria.where("tag").is("java"))),TestPull.class);
//remove array's multi-elem
UpdateResult wc1 = mongoTemplate.upsert(removeQuery,new Update().pull("tags",Query.query(Criteria.where("tag").in(Arrays.asList("mongo", "netty")))),TestPull.class);
If you simply want to remove an element from array which does not have any other property like name then write the query you wish and
Update update = new Update();
update.pull("Yourarrayname","valueyouwishtodelete");
mongoTemplate.upsert(query,update,"collectionname");
That's my solution - thanks to #ufasoli:
public Mono<ProjectChild> DeleteCritTemplChild(String id, String idch) {
Query query = new Query();
query.addCriteria(Criteria
.where("_id")
.is(id)
.and("tasks._id")
.is(idch)
);
Update update = new Update();
update.set("tasks.$", null);
return template
// findAndModify:
// Find/modify/get the "new object" from a single operation.
.findAndModify(
query, update,
new FindAndModifyOptions().returnNew(true), ProjectChild.class
);
}
This works for me!
UpdateResult updateResult = mongoTemplate.updateMulti(new Query(where("id").is(activity.getId())), new Update().pull("comments", new Query(where("id").is(commentId))), Activity.class);
It will generate:
Calling update using query: { "_id" : { "$oid" : "61f900e7c7b79442eb3855ea"}} and update: { "$pull" : { "comments" : { "_id" : "61fac32e3f9ab5646e016bc8"}}} in collection: activity