The slice method in aggreation for array within an embedded document is not working for me using spring mongo template.
Example:
Invoice collection:
{
"reference_number": "aaa111",
"historical_data": {
"year": 2010,
"data": [
{
"item_name": "Apple",
"price": 50
},
{
"item_name": "Book",
"price": 200
}
]
}
}
Using mongoTemplate I would like to get only the historical data in slices.
For the arrays which needs to be sliced that appear directly under the root I had found a solution using aggregation
Refer : Spring mongo repository slice
Applying a similar query for array in an embedded document returns an empty list even if there is data.
The query that I was trying was :
TypedAggregation<Invoice> agg = newAggregation(Invoice.class,
match(where("reference_number").is(referenceNumber)),
project.andExpression("historicalData.data").slice(limit, offset));
AggregationResults<Invoice> result = mongoTemplate.aggregate(agg, Invoice.class, Invoice.class);
But this returns an empty list of data.
Is there any other alternative way to get the sliced result for arrays within an embedded document?
Invoice.java
#Data
#Document(collection = "invoice")
public class Invoice {
#Id
private String id;
#NotEmpty
#Indexed(unique = true)
#Field("reference_number")
private String referenceNumber = UUID.randomUUID().toString();
#Valid
#Field("historical_data")
private HistoricalData historicalData = new HistoricalData();
}
HistoricalData:
#Data
public class HistoricalData {
#NotEmpty
#Field("year")
private Intger year;
#Valid
#NotNull
#Field("data")
private List<InvoiceData> data = new LinkedList<>();
}
Note : I have also tried :
TypedAggregation<Invoice> agg = newAggregation(Invoice.class,
match(where("reference_number").is(referenceNumber)),
project.andExpression("historical_data.data").slice(limit, offset));
AggregationResults<Invoice> result = mongoTemplate.aggregate(agg, Invoice.class, Invoice.class);
But this gave me a PropertyPathException.
Thanks in advance!!
After a weeks struggle I have figured out a solution for this:
ProjectionOperation project = project().and("historicalRevisionData.data").slice(limit, offset).as("historical_revision_data.data")
.andInclude("id")
.and("referenceNumber").as("reference_number");
TypedAggregation<Invoice> agg = newAggregation(Invoice.class,
match(where("reference_number").is(referenceNumber)),
project);
AggregationResults<TaxInvoice> aggregate = mongoTemplate.aggregate(agg, Invoice.class, Invoice.class);
Hoping that this would help someone else too.
Related
I have an issue getting the sum of an array of objects, Document is mentioned below.
{
"orderNumber":123,
"items":[
{
"price":2
},
{
"price":10
}
]
}
I need the below document
{
"orderNumber":123,
"totalItemsValue":12,
"items":[
{
"price":2
},
{
"price":10
}
]
}
Sum of prices in totalItemsValue field.I need a solution in spring mongo data.I will be thankful.
There ae lot of ways, easiest way is given below. Spring data doesn't provide any operation for $addFields. So we do a Trick.
Autowire the mongoTemplate
#Autowired
private MongoTemplate mongoTemplate;
And the method is like
public List<Object> test() {
Aggregation aggregation = Aggregation.newAggregation(
a-> new Document("$addFields",
new Document("totalItemsValue",
new Document("$reduce"
new Document()
.append("input","$items")
.append("initialValue",0)
.append("in",
new Document("$add",Arrays.asList("$$value",1))
)
)
)
)
).withOptions(AggregationOptions.builder().allowDiskUse(Boolean.TRUE).build());
return mongoTemplate.aggregate(aggregation, mongoTemplate.getCollectionName(YOUR_COLLECTION.class), Object.class).getMappedResults();
}
Working Mongo playground
I have a search criteria in which I have to provide pagination on searched result along with total number of records in collection. Suppose in a collection of 10 records, I want to get only 5 records along with count of total records. The resulted data, I want to push them into separate object having count and searchResult properties. Total count of records will have to map to count and paginated records to searchResult. I have applied aggregation and it is working well except inclusion of CountOperation and ProjectOperation. When I add countOperation and ProjectOperation in aggregation it gives invalid reference "_id!" exception.
the expected query would be like this.
db.customer.aggregate([
{
$facet:{
searchResult:[{$match:{"name" : { "$regex" : "xyz", "$options" : "i" }}}],
count: [{ $count: 'count' }]
}
}
])
and the output would be like this.
[
{
"searchResult":[{...},{...},{...}, ...],
"count":[{"count":100}]
}
]
Search logic:
public List<SampleSearchResult> findListByRequest(ListRequest queryParams, Class<T> clazz) {
String collectionName = mongoTemplate.getCollectionName(clazz);
MatchOperation matchOperation = getMatchOperation(queryParams);
SortOperation sortOperation = getSortOperation(queryParams);
SkipOperation skipOperation = Aggregation.skip((long) queryParams.getPageNumber() * queryParams.getSize());
LimitOperation limitOperation = Aggregation.limit(queryParams.getSize());
CountOperation countOperation = Aggregation.count().as("count");
ProjectionOperation projectionOperation = getProjectionOperation();
AggregationResults<SampleSearchResult> results = mongoTemplate
.aggregate(Aggregation.newAggregation(matchOperation, sortOperation, skipOperation, limitOperation, countOperation, projectionOperation ), collectionName, SampleSearchResult.class);
return (List<SampleSearchResult>) results.getMappedResults();
}
Projection operation logic
private ProjectionOperation getProjectionOperation() {
return Aggregation.project("count").and("_id").previousOperation();
}
SortOperation logic:
private SortOperation getSortOperation(ListRequest listRequest) {
// setting defaults
if (StringUtils.isEmpty(listRequest.getSortBy())) {
listRequest.setSortBy("_id");
listRequest.setAsc(false);
}
Sort sort = listRequest.isAsc() ? new Sort(Direction.ASC, listRequest.getSortBy())
: new Sort(Direction.DESC, listRequest.getSortBy());
return Aggregation.sort(sort);
}
MatchOperation logic:
private MatchOperation getMatchOperation(ListRequest listRequest) {
Criteria criteria = new Criteria();
// build match operation logic with listRequest parameters
return Aggregation.match(criteria);
}
The resultant object which will hold aggregation result
public class SampleSearchResult {
private List<Object> searchResult;
private int count;
public List<Object> getSearchResult() {
return searchResult;
}
public void setSearchResult(List<Object> searchResult) {
this.searchResult = searchResult;
}
public int getCount() {
return count;
}
public void setCount(int count) {
this.count = count;
}
}
I need to write CountOperation and ProjectionOperation properly to map the data to SampleSearchResult but I'm not that efficient to do since I'm new to MongoDB opreations.
Well this might be coming very late, but let me drop my answer here for who ever might be having this same trouble in future. The right way to go is truly using facet as described here.
I could use the facet in Java, but mapping the result of the query to a pojo class was also a problem for me. But here is how I was able to get past that challenge.
First looking at the expected output as seen in the question:
[
{
"searchResult":[{...},{...},{...}, ...],
"count":[{"count":100}]
}
]
A pojo to fit this output must be modelled like this:
#Getter //lombok stuff to create getters and setters
#Setter
public class CustomerSearchResult {
private List<CustomerData> searchResult;
private List<CountDto> count;
}
So here is CountDto.java
#Setter
#Getter
public class CountDto {
private Long count;
}
And here is CustomerData.java
#Getter
#Setter
public class CustomerData {
private Long dateCreated;
private String Id;
private String firstName;
private String lastName;
}
Next, MongoClient must be instantiated with a custom or default code registry like this:
import com.mongodb.ConnectionString;
import com.mongodb.MongoClientSettings;
import com.mongodb.client.MongoClient;
import com.mongodb.client.MongoClients;
import org.bson.codecs.configuration.CodecRegistry;
import org.bson.codecs.pojo.PojoCodecProvider;
import static org.bson.codecs.configuration.CodecRegistries.fromProviders;
import static org.bson.codecs.configuration.CodecRegistries.fromRegistries;
public class MongoSource {
private MongoClient mongoClient;
public MongoClient createClient(){
String connectionString = "your.database.connection.string";
ConnectionString connection = new ConnectionString(connectionString);
CodecRegistry defaultCodec = MongoClientSettings.getDefaultCodecRegistry();
CodecRegistry fromProvider = fromProviders(PojoCodecProvider.builder().automatic(true).build());
CodecRegistry pojoCodecRegistry = fromRegistries(defaultCodec, fromProvider);
MongoClientSettings.Builder builder = MongoClientSettings.builder();
builder.applyConnectionString(connection);
MongoClientSettings settings = builder.codecRegistry(pojoCodecRegistry).build();
mongoClient = MongoClients.create(settings);
return mongoClient;
}
public MongoClient getMongoClient(){
return mongoClient;
}
}
Using Mongo Java Driver 4.0.X
Not sure the version of the mongo driver or third-party API that was used in the question but I think the use of criteria has been deprecated or was never part of mongo API natively. But this is how I was able to achieve this using the Mongo Java Driver 4.0.x
To achieve a search with paginated result and the total count, using facet, it would be required to build two separate pipelines
One to do the search with sort, limit and skip
The other to do the actual count of the total result without limit
and skip
These two pipelines would finally be used to construct the facets to be used with the aggregate API.
Consider the pojo below used to define the respective query fields
import lombok.Getter;
import lombok.Setter;
#Getter
#Setter
public class SearchQuery {
String firstName;
String lastName;
int pageNumber;
int pageSize;
}
And here is the final snippet that performs the search and returns the CustomerSearchResult intsance that contains the count and the paginated result defined by the pageSize and pageNumber.
import java.util.*;
import static com.mongodb.client.model.Aggregates.*;
import static com.mongodb.client.model.Filters.*;
import static com.mongodb.client.model.Projections.fields;
import static com.mongodb.client.model.Projections.include;
import org.apache.commons.lang3.StringUtils;
public CustomerSearchResult findListByRequest(SearchQuery queryParam){
MongoSource mongoSource = new MongoSource();
MongoDatabase database = mongoSource.createClient().getDatabase("customers_db");
MongoCollection<CustomerSearchResult> collection = database.getCollection("customer", CustomerSearchResult.class);
List<Bson> queryPipeline = new ArrayList<>();
List<Bson> countPipeline = new ArrayList<>();
List<Bson> andMatchFilter = new ArrayList<>();
//you might want to check for null before using the fields in the query param object
andMatchFilter.add(regex("firstName", Pattern.compile(queryParam.getFirstName(), Pattern.CASE_INSENSITIVE)));
andMatchFilter.add(regex("lastName", Pattern.compile(queryParam.getLastName(), Pattern.CASE_INSENSITIVE)));
if(queryParam.getPageNumber() == 0){
queryParam.setPageNumber(1);
}
if(queryParam.getPageSize() == 0){
queryParam.setPageSize(30);
}
queryPipeline.add(match(and(andMatchFilter)));
queryPipeline.add(sort(eq("dateCreated", -1)));
queryPipeline.add(skip(queryParam.getPageSize() * (queryParam.getPageNumber() - 1)));
queryPipeline.add(limit(queryParam.getPageSize()));
queryPipeline.add(project(fields(include("Id","firstName","lastName"))));
countPipeline.add(match(and(andMatchFilter)));
countPipeline.add(count());
Facet resultFacet = new Facet("searchResult", queryPipeline);
Facet totalDocFacet = new Facet("count", countPipeline);
return collection.aggregate(Collections.singletonList(facet(resultFacet, totalDocFacet))).first();
}
Note that the CustomerSearchResult fields are the same as the names given to the facets defined for each pipeline respectively. This will allow the defined coderegistry to map correctly the output document to your pojo class (CustomerSearchResult)
then with that you can do like so to access the count:
SearchQuery searchQuery = new SearchQuery();
searchQuery.setPageNumber(1);
searchQuery.setPageSize(15);
searchQuery.setFirstName("John");
searchQuery.setLastName("Doe");
CustomerSearchResult result = findListByRequest(query);
long count = result.getCount()!= null? result.getCount().get(0).getCount() : 0;
List<CustomerData> data = result.getSearchResult();
IMongoDatabase.ListCollections return a cursor over BsonDocument.
Why doesn't it return a cursor over IMongoCollection<T> instead?
I was trying to write a generic GetCollection method to retrieve the collection given just the document type, something like this:
private IMongoCollection<T> GetCollection<T>()
{
var client = new MongoClient("connectionString");
var db = client.GetDatabase("dbName");
var coll = db.ListCollectionsAsync().Result.ToListAsync().Result
// Find collection of document of type T
// Collection is a BsonDocument instead
.Find(collection => typeof(collection) == typeof(T));
return coll;
}
The driver doesn't know what kind of document is in a collection, which is why it takes a type parameter T. MongoDB itself isn't aware of how the documents in the database map to the types in your application.
It is not possible to take a connection to a "generic" MongoDB deployment and simply discover the collections and types in them. This is code you would need to write, and probably won't work out well as it'll be something akin to trial and error.
If you are simply trying to create a factory type, you will need to build the backing list of collections before calling GetCollection<T>.
You could try using the type name as the collection name. This would make the collection name repeatable (unless the type name is changed). But I've never tested it and it might have some idiosyncrasies in the real world.
public class MyDatabase
{
private readonly IMongoClient _client;
public MyDatabase(string connectionString)
{
_client = new MongoClient(connectionString);
}
public IMongoCollection<T> GetCollection<T>()
{
var db = _client.GetDatabase("dbName");
return db.GetCollection<T>(typeof(T).Name);
}
}
If you prefer collection names to be pluralized, something like Humanizer can help there.
I generally prefer to create a type that has the collections as fields on the class. For example:
public class MyDatabase
{
private readonly IMongoClient _client;
public IMongoCollection<Foo> Foos { get; }
public IMongoCollection<Bar> Bars { get; }
public MyDatabase(string connectionString)
{
_client = new MongoClient(connectionString);
var db = _client.GetDatabase("myDb");
Foos = db.GetCollection<Foo>("foos");
Bars = db.GetCollection<Bar>("bars");
}
}
MongoDB collections have a flexible scheme that allows you to insert any document with any structure into the collection. For example, We can insert the following 3 objects into the test collection and it's valid:
> db.test.insertMany([{one: 1}, {two:2}, {three: 3}])
{
"acknowledged" : true,
"insertedIds" : [
ObjectId("5c87c954ed372bf469367e57"),
ObjectId("5c87c954ed372bf469367e58"),
ObjectId("5c87c954ed372bf469367e59")
]
}
> db.test.find().pretty()
{ "_id" : ObjectId("5c87c954ed372bf469367e57"), "one" : 1 }
{ "_id" : ObjectId("5c87c954ed372bf469367e58"), "two" : 2 }
{ "_id" : ObjectId("5c87c954ed372bf469367e59"), "three" : 3 }
Thus you can't map a MongoDB collection to a .NET Type as the collection has no knowledge of the type.
In a spring-boot data mongodb application I whould like to return the last element of an embeded collection.
My document is :
#Document
public class ConnectedObject {
#Id
private String uuid;
private List<Measure> measures = new ArrayList<>();
}
public class Measure {
private LocalDateTime timestamp;
private long stepsCount;
}
Exemple of data in mongoDb:
{
"_id":"aaaa",
"measures":[
{"timestamp":"2018-04-05T08:20:33.561Z","stepsCount":"0"},
{"timestamp":"2018-04-05T08:21:35.561Z","stepsCount":"10"},
{"timestamp":"2018-04-05T08:20:35.561Z","stepsCount":"0"}
]
}
I whould like to get the last measure (filter by timestamp field) of the connectedObject (filter onthe uuid).
I don't know how to write the query using MongoTemplate.
I already have custom repository in the project.
Something like the query below should give what you need
db.collection.aggregate([
{$match: {'$uuid':'12345'}},
{$sort:{'$measure.timestamp':1}},
{$project:{
uuid: 1,
last: { $arrayElemAt: [ "$measures", -1 ] }
}
}
])
After a lot of try, the one running is :
public Measure getLastMeasureOf(String uuid) {
final Aggregation agg = newAggregation(
match(Criteria.where("uuid").is(uuid)),
unwind("measures"),
sort(Sort.Direction.DESC, "measures.timestamp"),
group("uuid").first("$$ROOT").as("result"),
project("result.uuid").and("result.measures").as("last")
);
final AggregationResults<ObjectWithLastMeasure> results
= template.aggregate(agg, ConnectedObject.class, ObjectWithLastMeasure.class);
final ObjectWithLastMeasure owlm = results.getUniqueMappedResult();
return owlm == null ? null : owlm.getLast();
}
I am a dummy in using spring aggregations.
I do have this entity document:
#Document(collection = "DocumentFile")
public class DocumentFile {
private String projectId;
private String originalFileName;
and I will get the amount of documentFiles which have the same projectId grouped by originalFileName (so DocumentFile's with same name should only be counted once)
This is my approach but I don't know how to get now the result/amount.
final Aggregation agg = newAggregation(match(Criteria.where("projectId").in(projectId)),
group("originalFileName").count().as("amountOfDocumentFiles"));
Assuming that aggregate present in the post is correct. Here is the sample code to execute the aggregate using MongoOperations and get the result.
In my project, I get the MongoOperations object like this.
public MongoOperations getMongoConnection() {
return (MongoOperations) new AnnotationConfigApplicationContext(SpringMongoConfig.class)
.getBean("mongoTemplate");
}
Execute aggregate and get results:-
Aggregation aggregate = newAggregation(match(Criteria.where("projectId").in(projectId)),
group("originalFileName").count().as("amountOfDocumentFiles"));
AggregationResults<DocumentFile> documentFileAggregate = mongoOperations.aggregate(aggregate,
"DocumentFile", DocumentFile.class);
if (documentFileAggregate != null) {
System.out.println("Output ====>" + documentFileAggregate.getRawResults().get("result"));
System.out.println("Output ====>" + documentFileAggregate.getRawResults().toMap());
}