Is there a way to return a cursor with spring data's mongodb aggregation?
Aggregation agg = newAggregation(
match(Criteria.where("_id").is(objId)),
unwind("taskResultContent"),
project("taskResultContent.executionUUID","taskResultContent.returnContent","taskResultContent.sequency").and("resultID").previousOperation(),
match(Criteria.where("executionUUID").is(executionUUID)),
sort(DESC,"sequency")
).withOptions(Aggregation.newOptions().cursor(cursor).build());
Solution
Quoting here :
From spring-data-mongo version 2.0.0.M4 onwards (AFAIK) MongoTemplate got an aggregateStream method.
So you can do the following:
AggregationOptions aggregationOptions = Aggregation.newAggregationOptions()
// this is very important: if you do not set the batch size,
// you'll get all the objects at once and you might run out of memory
// if the returning data set is too large
.cursorBatchSize(mongoCursorBatchSize)
.build();
data = mongoTemplate.aggregateStream(Aggregation.newAggregation(
Aggregation.group("person_id")
.count()
.as("count"))
.withOptions(aggregationOptions), collectionName, YourClazz.class);
Spring data for mongodb does not support the use of a cursor when aggregating. The MongoDB java driver must be used instead.
Related
After upgrading from Spring boot 2.2.6.RELEASE to 2.4, the following mapping does not work & returns null for all the mapped values. I'm unable to find any mongo documentation describing any update to aggregation or mapping. However the rawResults contains the required data. I've tried changing the mongo versions, spring boot version upgrades.
The same code works without a issue in spring boot 2.2.6.RELEASE version.
final Aggregation agg = newAggregation(match(Criteria.where("id.cid").is(cId)),
group("sId.assignmentId", "status").count().as("total"),
sort(Sort.Direction.DESC, "sId.assignmentId", "status"));
final AggregationResults<StatusSummeryDTO> groupResults = mongoTemplate.aggregate(agg,
Submission.class, StatusSummeryDTO.class);
return groupResults.getMappedResults();
Turns out after MongoDB driver version being changed from 3.x to 4.x in the result DTO class we will need to specify the exact field value using #Field annotation. Unlike in 3.x it does not map automatically which resulted in the above issue.
Since there are breaking changes in the aggregation when moving from 3.x to 4.x I thought that might be the issue. to be on the better side I've changed my code to be compatible with 4.x using new classes as follows.
MatchOperation match = match(Criteria.where("id.cid").is(cId));
GroupOperation group = group("sId.assignmentId", "status").count().as("total");
SortOperation sort = sort(Sort.Direction.DESC, "sId.assignmentId", "status");
Aggregation agg = newAggregation(match, group, sort);
AggregationResults<StatusSummeryDTO> groupResults = mongoTemplate.aggregate(agg, Submission.class, StatusSummeryDTO.class);
return groupResults.getMappedResults();
Consider a aggregation, which is formed like:
MatchOperation matchOperation = match(Criteria.where("group").is("A")
.andOperator(Criteria.where("score").lt(50))) ;
SortOperation sortOperation = sort(Sort.by(Direction.DESC, "department"));
Aggregation aggregation = Aggregation.newAggregation(matchOperation, sortOperation) ;
I want to issue a bulk remove (or basically remove each document found as a result of above) using a Spring Boot application.
Problem:
Unable to find a suitable method to do so, and I want to achieve this using Aggregation only ie. don't want to switch to Query.
What I've tried already:
I have already tried using mongoTemplate.findAllAndRemove(), but it accepts a Query and I am using Aggregation.
I have also tried using
BulkOperations bulkOps = mongoTemplate.bulkOps(BulkMode.ORDERED, EntityName.class) ;
And then, bulkOps.remove() but this method also accepts a List<Query>
You can't remove using aggregation so I think there are two options:
Using your aggregate query to get all _ids and do a new Query to remove these _ids.
Something like this (not tested):
List<Document> result = mongoTemplate.aggregate(aggregation, "my_collection", Document.class).getMappedResults();
List<ObjectId> resultList = result.stream().map(item -> item.get("_id")).collect(Collectors.toList());
mongoTemplate.remove(new Query(Criteria.where("_id").in(resultList)), YourClass.class);
Create a new delete query. As you need to use delete maybe you can avoid aggregation simply using this:
mongoTemplate.remove(new Query(Criteria.where("group").is("A").andOperator(Criteria.where("score").lt(50))), YourClass.class);
Note that the criteria is the same as you have in the match. This is because is the same use it in a find/remove/update query and into an aggregation query
By the way, I don't understand why sort if you want to delete.
I want to implement a delete method which deletes all the documents in a collection. I am using mongo db with Spring Data.This could be done using db.myCollection.remove({}) in Mongo shell. But I want to write a method in my data access layer to do this.I am not using MongodbTemplate in my Dao class. I want to know how can I do this with Query.
Query query = new Query();
Could anybody please tell me how can I do it.
You can use MongoTemplate directly
mongoTemplate.remove(new Query(), collectionName);
Use MongoRepository's deleteAll(). Utilizes mongoTemplate behind the scene to call the remove method.
From calling method someRepository.deleteAll()
Drop collection may be efficient as other answer has noted. For that you will need to use MongoTemplate directly and call dropCollection with entity class or collection name.
You can drop the collection and create it again:
mongoTemplate.dropCollection("collectionName");
mongoTemplate.createCollection("collectionName");
You'd better drop the entire collection (if possible) than deleting all documents. For performance and allocation sake.
You can try something such as:
MongoClient mongoClient = new MongoClient(new ServerAddress("localhost", 27017));
MongoDatabase db = mongoClient.getDatabase("test");
MongoCollection collection = db.getCollection("test_collection");
collection.drop();
If you do want to delete all (I think), instead of:
collection.drop();
use:
Bson filter = new Document();
collection.deleteMany(filter);
Query is part of spring-data-mongodb, if you can't use a MongoTemplate, probably Query is irrelevant as well.
If you have considerable changes in schema, better is to drop the collection.
MongoTemplate.dropCollection(collectionName).
But the requirement is to delete all documents you have use
MongoTemplate.remove(new Query(), collectionName)or
CrudRepository.deleteAll()
If all collections are to be deleted, this could also be done:
mongoTemplate.collectionNames
.forEach { mongoTemplate.dropCollection(it) }
I was looking to use the $natural operator in Spring Data MongoDB, as documented here:
https://docs.mongodb.org/v3.0/reference/operator/meta/natural/
Is this possible to do using the MongoTemplate class? Thanks.
It is possible to use $natural in at least three styles where 1. and 2. are probably what you're looking for:
1. Using Sort with Query
Query query = new Query().with(new Sort(Direction.ASC, "$natural"));
use the query afterwards with MongoTemplate. The query carries a sort document like:
{ "$natural" : 1}
2. Using BasicQuery
BasicQuery allows using own DBObjects for the query document, the fields ("projection") and sorting.
BasicQuery basicQuery = new BasicQuery(new BasicDBObject());
basicQuery.setSortObject(new BasicDBObject("$natural", 1));
3. Using execute and CollectionCallback
This is the most extensive way in which you're getting access to the DBCollection and you can use the native MongoDB driver API. See Spring Data Mongo docs for more details.
I have a query that works in MongoDB Shell but it is throwing an "aggregation result exceeds maximum document size (16MB)" exception using Spring Data MongoDB when I increase the number of documents to return.
Here is the shell query that returns without error:
db.myCollection.aggregate([{"$skip":0},{"$limit":10000},{"$match":{"_id"="550b2552e4b03562d6329a39"}}], {"allowDiskUse":true})
Here is the Spring Data snippet:
List<AggregrationOperation> ops = new ArrayList<AggregationOperation>();
ops.add(new SkipOperation(0));
ops.add(new LimitOperation(10000));
ops.add(new MatchOperation(Criteria.where("_id").is("550b2552e4b03562d6329a39")));
TypeAggregation<MyCollection> aggregation = Aggregation.newAggregation(MyCollection.class, ops).withOptions(Aggregation.newAggregationOptions().allowDiskUse(true).build());
AggregationResults<MyCollection> result = mongoTemplate.aggregate(aggregation, MyCollection.class);
List<MyCollection> myResults = result.getMappedResults()
When I limit to say 100 or 1000 records everything works fine. When I increase to 10,000 I get the "aggregation result exceeds maximum document size (16MB)" exception.
For reference here are the versions I am using:
MongoDB = 3.0.0
Mongo Java Driver = 2.13.0
Spring Data MongoDB = 1.6.0
Spring Data Commons = 1.9.0
UPDATE:
Background: I came to the aggregate solution because I was exceeding the 32MB sort limit when using a find(). I understand that adding an index will solve this problem. Unfortunately this solution doesn't scale. I want to have sorting on all of our columns in our List Grids which would mean 10+ columns to index. Of course from the UI I could limit sorting to specific columns but once again I am trying to avoid that solution and hence the reason I tried an aggregation.
It appears that using a cursor is my only solution. Can anyone confirm that Spring Data MongoDB does not provide direct cursor support meaning I would have to use MongoDB's API?
I had the same issue using RoboMongo (mongo client).
Try using db.runCommand() instead of .aggregate().
Is seems that using the aggregate method on some clients will add some data to the query and allowDiskUse will be ignored.
When I used ranCommand, it worked fine:
db.runCommand(
{ aggregate: "collection_name",
pipeline: [
{$group: ...},
{$match: ...},
{$group: ...}
],
allowDiskUse: true
}
You can use outputMode(AggregationOptions.OutputMode.CURSOR) as ouput option instead of the limited 16MB document by default. See Aggregation Cursor Usage.
Aggregation.newAggregationOptions().outputMode(AggregationOptions.OutputMode.CURSOR).allowDiskUse(true).build()