I am a dummy in using spring aggregations.
I do have this entity document:
#Document(collection = "DocumentFile")
public class DocumentFile {
private String projectId;
private String originalFileName;
and I will get the amount of documentFiles which have the same projectId grouped by originalFileName (so DocumentFile's with same name should only be counted once)
This is my approach but I don't know how to get now the result/amount.
final Aggregation agg = newAggregation(match(Criteria.where("projectId").in(projectId)),
group("originalFileName").count().as("amountOfDocumentFiles"));
Assuming that aggregate present in the post is correct. Here is the sample code to execute the aggregate using MongoOperations and get the result.
In my project, I get the MongoOperations object like this.
public MongoOperations getMongoConnection() {
return (MongoOperations) new AnnotationConfigApplicationContext(SpringMongoConfig.class)
.getBean("mongoTemplate");
}
Execute aggregate and get results:-
Aggregation aggregate = newAggregation(match(Criteria.where("projectId").in(projectId)),
group("originalFileName").count().as("amountOfDocumentFiles"));
AggregationResults<DocumentFile> documentFileAggregate = mongoOperations.aggregate(aggregate,
"DocumentFile", DocumentFile.class);
if (documentFileAggregate != null) {
System.out.println("Output ====>" + documentFileAggregate.getRawResults().get("result"));
System.out.println("Output ====>" + documentFileAggregate.getRawResults().toMap());
}
Related
I have a search criteria in which I have to provide pagination on searched result along with total number of records in collection. Suppose in a collection of 10 records, I want to get only 5 records along with count of total records. The resulted data, I want to push them into separate object having count and searchResult properties. Total count of records will have to map to count and paginated records to searchResult. I have applied aggregation and it is working well except inclusion of CountOperation and ProjectOperation. When I add countOperation and ProjectOperation in aggregation it gives invalid reference "_id!" exception.
the expected query would be like this.
db.customer.aggregate([
{
$facet:{
searchResult:[{$match:{"name" : { "$regex" : "xyz", "$options" : "i" }}}],
count: [{ $count: 'count' }]
}
}
])
and the output would be like this.
[
{
"searchResult":[{...},{...},{...}, ...],
"count":[{"count":100}]
}
]
Search logic:
public List<SampleSearchResult> findListByRequest(ListRequest queryParams, Class<T> clazz) {
String collectionName = mongoTemplate.getCollectionName(clazz);
MatchOperation matchOperation = getMatchOperation(queryParams);
SortOperation sortOperation = getSortOperation(queryParams);
SkipOperation skipOperation = Aggregation.skip((long) queryParams.getPageNumber() * queryParams.getSize());
LimitOperation limitOperation = Aggregation.limit(queryParams.getSize());
CountOperation countOperation = Aggregation.count().as("count");
ProjectionOperation projectionOperation = getProjectionOperation();
AggregationResults<SampleSearchResult> results = mongoTemplate
.aggregate(Aggregation.newAggregation(matchOperation, sortOperation, skipOperation, limitOperation, countOperation, projectionOperation ), collectionName, SampleSearchResult.class);
return (List<SampleSearchResult>) results.getMappedResults();
}
Projection operation logic
private ProjectionOperation getProjectionOperation() {
return Aggregation.project("count").and("_id").previousOperation();
}
SortOperation logic:
private SortOperation getSortOperation(ListRequest listRequest) {
// setting defaults
if (StringUtils.isEmpty(listRequest.getSortBy())) {
listRequest.setSortBy("_id");
listRequest.setAsc(false);
}
Sort sort = listRequest.isAsc() ? new Sort(Direction.ASC, listRequest.getSortBy())
: new Sort(Direction.DESC, listRequest.getSortBy());
return Aggregation.sort(sort);
}
MatchOperation logic:
private MatchOperation getMatchOperation(ListRequest listRequest) {
Criteria criteria = new Criteria();
// build match operation logic with listRequest parameters
return Aggregation.match(criteria);
}
The resultant object which will hold aggregation result
public class SampleSearchResult {
private List<Object> searchResult;
private int count;
public List<Object> getSearchResult() {
return searchResult;
}
public void setSearchResult(List<Object> searchResult) {
this.searchResult = searchResult;
}
public int getCount() {
return count;
}
public void setCount(int count) {
this.count = count;
}
}
I need to write CountOperation and ProjectionOperation properly to map the data to SampleSearchResult but I'm not that efficient to do since I'm new to MongoDB opreations.
Well this might be coming very late, but let me drop my answer here for who ever might be having this same trouble in future. The right way to go is truly using facet as described here.
I could use the facet in Java, but mapping the result of the query to a pojo class was also a problem for me. But here is how I was able to get past that challenge.
First looking at the expected output as seen in the question:
[
{
"searchResult":[{...},{...},{...}, ...],
"count":[{"count":100}]
}
]
A pojo to fit this output must be modelled like this:
#Getter //lombok stuff to create getters and setters
#Setter
public class CustomerSearchResult {
private List<CustomerData> searchResult;
private List<CountDto> count;
}
So here is CountDto.java
#Setter
#Getter
public class CountDto {
private Long count;
}
And here is CustomerData.java
#Getter
#Setter
public class CustomerData {
private Long dateCreated;
private String Id;
private String firstName;
private String lastName;
}
Next, MongoClient must be instantiated with a custom or default code registry like this:
import com.mongodb.ConnectionString;
import com.mongodb.MongoClientSettings;
import com.mongodb.client.MongoClient;
import com.mongodb.client.MongoClients;
import org.bson.codecs.configuration.CodecRegistry;
import org.bson.codecs.pojo.PojoCodecProvider;
import static org.bson.codecs.configuration.CodecRegistries.fromProviders;
import static org.bson.codecs.configuration.CodecRegistries.fromRegistries;
public class MongoSource {
private MongoClient mongoClient;
public MongoClient createClient(){
String connectionString = "your.database.connection.string";
ConnectionString connection = new ConnectionString(connectionString);
CodecRegistry defaultCodec = MongoClientSettings.getDefaultCodecRegistry();
CodecRegistry fromProvider = fromProviders(PojoCodecProvider.builder().automatic(true).build());
CodecRegistry pojoCodecRegistry = fromRegistries(defaultCodec, fromProvider);
MongoClientSettings.Builder builder = MongoClientSettings.builder();
builder.applyConnectionString(connection);
MongoClientSettings settings = builder.codecRegistry(pojoCodecRegistry).build();
mongoClient = MongoClients.create(settings);
return mongoClient;
}
public MongoClient getMongoClient(){
return mongoClient;
}
}
Using Mongo Java Driver 4.0.X
Not sure the version of the mongo driver or third-party API that was used in the question but I think the use of criteria has been deprecated or was never part of mongo API natively. But this is how I was able to achieve this using the Mongo Java Driver 4.0.x
To achieve a search with paginated result and the total count, using facet, it would be required to build two separate pipelines
One to do the search with sort, limit and skip
The other to do the actual count of the total result without limit
and skip
These two pipelines would finally be used to construct the facets to be used with the aggregate API.
Consider the pojo below used to define the respective query fields
import lombok.Getter;
import lombok.Setter;
#Getter
#Setter
public class SearchQuery {
String firstName;
String lastName;
int pageNumber;
int pageSize;
}
And here is the final snippet that performs the search and returns the CustomerSearchResult intsance that contains the count and the paginated result defined by the pageSize and pageNumber.
import java.util.*;
import static com.mongodb.client.model.Aggregates.*;
import static com.mongodb.client.model.Filters.*;
import static com.mongodb.client.model.Projections.fields;
import static com.mongodb.client.model.Projections.include;
import org.apache.commons.lang3.StringUtils;
public CustomerSearchResult findListByRequest(SearchQuery queryParam){
MongoSource mongoSource = new MongoSource();
MongoDatabase database = mongoSource.createClient().getDatabase("customers_db");
MongoCollection<CustomerSearchResult> collection = database.getCollection("customer", CustomerSearchResult.class);
List<Bson> queryPipeline = new ArrayList<>();
List<Bson> countPipeline = new ArrayList<>();
List<Bson> andMatchFilter = new ArrayList<>();
//you might want to check for null before using the fields in the query param object
andMatchFilter.add(regex("firstName", Pattern.compile(queryParam.getFirstName(), Pattern.CASE_INSENSITIVE)));
andMatchFilter.add(regex("lastName", Pattern.compile(queryParam.getLastName(), Pattern.CASE_INSENSITIVE)));
if(queryParam.getPageNumber() == 0){
queryParam.setPageNumber(1);
}
if(queryParam.getPageSize() == 0){
queryParam.setPageSize(30);
}
queryPipeline.add(match(and(andMatchFilter)));
queryPipeline.add(sort(eq("dateCreated", -1)));
queryPipeline.add(skip(queryParam.getPageSize() * (queryParam.getPageNumber() - 1)));
queryPipeline.add(limit(queryParam.getPageSize()));
queryPipeline.add(project(fields(include("Id","firstName","lastName"))));
countPipeline.add(match(and(andMatchFilter)));
countPipeline.add(count());
Facet resultFacet = new Facet("searchResult", queryPipeline);
Facet totalDocFacet = new Facet("count", countPipeline);
return collection.aggregate(Collections.singletonList(facet(resultFacet, totalDocFacet))).first();
}
Note that the CustomerSearchResult fields are the same as the names given to the facets defined for each pipeline respectively. This will allow the defined coderegistry to map correctly the output document to your pojo class (CustomerSearchResult)
then with that you can do like so to access the count:
SearchQuery searchQuery = new SearchQuery();
searchQuery.setPageNumber(1);
searchQuery.setPageSize(15);
searchQuery.setFirstName("John");
searchQuery.setLastName("Doe");
CustomerSearchResult result = findListByRequest(query);
long count = result.getCount()!= null? result.getCount().get(0).getCount() : 0;
List<CustomerData> data = result.getSearchResult();
To reduce the DB hits to read the data from DB using the query, I am planning to keep resultant in the cache. To do this I am using guava caching.
studentController.java
public Map<String, Object> getSomeMethodName(Number departmentId, String departmentType){
ArrayList<Student> studentList = studentManager.getStudentListByDepartmentType(departmentId, departmentType);
----------
----------
}
StudentHibernateDao.java(criteria query )
#Override
public ArrayList<Student> getStudentListByDepartmentType(Number departmentId, String departmentType) {
Criteria criteria =sessionFactory.getCurrentSession().createCriteria(Student.class);
criteria.add(Restrictions.eq("departmentId", departmentId));
criteria.add(Restrictions.eq("departmentType", departmentType));
ArrayList<Student> studentList = (ArrayList)criteria.list();
return studentList;
}
To cache the criteria query resultant i started off with building CacheBuilder, like below.
private static LoadingCache<Number departmentId, String departmentType, ArrayList<Student>> studentListCache = CacheBuilder
.newBuilder().expireAfterAccess(1, TimeUnit.MINUTES)
.maximumSize(1000)
.build(new CacheLoader<Number departmentId, String departmentType, ArrayList<Student>>() {
public ArrayList<Student> load(String key) throws Exception {
return getStudentListByDepartmentType(departmentId, departmentType);
}
});
Here I dont know where to put CacheBuilder function and how to pass multiple key parameters(i.e departmentId and departmentType) to CacheLoader and call it.
Is this the correct way of caching using guava. Am I missing anything?
Guava's cache only accepts two type parameters, a key and a value type. If you want your key to be a compound key then you need to build a new compound type to encapsulate it. Effectively it would need to look like this (I apologize for my syntax, I don't use Java that often):
// Compound Key type
class CompoundDepartmentId {
public CompoundDepartmentId(Long departmentId, String departmentType) {
this.departmentId = departmentId;
this.departmentType = departmentType;
}
}
private static LoadingCache<CompoundDepartmentId, ArrayList<Student>> studentListCache =
CacheBuilder
.newBuilder().expireAfterAccess(1, TimeUnit.MINUTES)
.maximumSize(1000)
.build(new CacheLoader<CompoundDepartmentId, ArrayList<Student>>() {
public ArrayList<Student> load(CompoundDepartmentId key) throws Exception {
return getStudentListByDepartmentType(key.departmentId, key.departmentType);
}
});
The slice method in aggreation for array within an embedded document is not working for me using spring mongo template.
Example:
Invoice collection:
{
"reference_number": "aaa111",
"historical_data": {
"year": 2010,
"data": [
{
"item_name": "Apple",
"price": 50
},
{
"item_name": "Book",
"price": 200
}
]
}
}
Using mongoTemplate I would like to get only the historical data in slices.
For the arrays which needs to be sliced that appear directly under the root I had found a solution using aggregation
Refer : Spring mongo repository slice
Applying a similar query for array in an embedded document returns an empty list even if there is data.
The query that I was trying was :
TypedAggregation<Invoice> agg = newAggregation(Invoice.class,
match(where("reference_number").is(referenceNumber)),
project.andExpression("historicalData.data").slice(limit, offset));
AggregationResults<Invoice> result = mongoTemplate.aggregate(agg, Invoice.class, Invoice.class);
But this returns an empty list of data.
Is there any other alternative way to get the sliced result for arrays within an embedded document?
Invoice.java
#Data
#Document(collection = "invoice")
public class Invoice {
#Id
private String id;
#NotEmpty
#Indexed(unique = true)
#Field("reference_number")
private String referenceNumber = UUID.randomUUID().toString();
#Valid
#Field("historical_data")
private HistoricalData historicalData = new HistoricalData();
}
HistoricalData:
#Data
public class HistoricalData {
#NotEmpty
#Field("year")
private Intger year;
#Valid
#NotNull
#Field("data")
private List<InvoiceData> data = new LinkedList<>();
}
Note : I have also tried :
TypedAggregation<Invoice> agg = newAggregation(Invoice.class,
match(where("reference_number").is(referenceNumber)),
project.andExpression("historical_data.data").slice(limit, offset));
AggregationResults<Invoice> result = mongoTemplate.aggregate(agg, Invoice.class, Invoice.class);
But this gave me a PropertyPathException.
Thanks in advance!!
After a weeks struggle I have figured out a solution for this:
ProjectionOperation project = project().and("historicalRevisionData.data").slice(limit, offset).as("historical_revision_data.data")
.andInclude("id")
.and("referenceNumber").as("reference_number");
TypedAggregation<Invoice> agg = newAggregation(Invoice.class,
match(where("reference_number").is(referenceNumber)),
project);
AggregationResults<TaxInvoice> aggregate = mongoTemplate.aggregate(agg, Invoice.class, Invoice.class);
Hoping that this would help someone else too.
In a spring-boot data mongodb application I whould like to return the last element of an embeded collection.
My document is :
#Document
public class ConnectedObject {
#Id
private String uuid;
private List<Measure> measures = new ArrayList<>();
}
public class Measure {
private LocalDateTime timestamp;
private long stepsCount;
}
Exemple of data in mongoDb:
{
"_id":"aaaa",
"measures":[
{"timestamp":"2018-04-05T08:20:33.561Z","stepsCount":"0"},
{"timestamp":"2018-04-05T08:21:35.561Z","stepsCount":"10"},
{"timestamp":"2018-04-05T08:20:35.561Z","stepsCount":"0"}
]
}
I whould like to get the last measure (filter by timestamp field) of the connectedObject (filter onthe uuid).
I don't know how to write the query using MongoTemplate.
I already have custom repository in the project.
Something like the query below should give what you need
db.collection.aggregate([
{$match: {'$uuid':'12345'}},
{$sort:{'$measure.timestamp':1}},
{$project:{
uuid: 1,
last: { $arrayElemAt: [ "$measures", -1 ] }
}
}
])
After a lot of try, the one running is :
public Measure getLastMeasureOf(String uuid) {
final Aggregation agg = newAggregation(
match(Criteria.where("uuid").is(uuid)),
unwind("measures"),
sort(Sort.Direction.DESC, "measures.timestamp"),
group("uuid").first("$$ROOT").as("result"),
project("result.uuid").and("result.measures").as("last")
);
final AggregationResults<ObjectWithLastMeasure> results
= template.aggregate(agg, ConnectedObject.class, ObjectWithLastMeasure.class);
final ObjectWithLastMeasure owlm = results.getUniqueMappedResult();
return owlm == null ? null : owlm.getLast();
}
How do I execute "select distinct ename from emp" using GreenDao
I am trying to get distinct values of a column of sqlite DB using GreenDao. How do I do it? Any help appreciated.
You have to use a raw query for example like this:
private static final String SQL_DISTINCT_ENAME = "SELECT DISTINCT "+EmpDao.Properties.EName.columnName+" FROM "+EmpDao.TABLENAME;
public static List<String> listEName(DaoSession session) {
ArrayList<String> result = new ArrayList<String>();
Cursor c = session.getDatabase().rawQuery(SQL_DISTINCT_ENAME, null);
try{
if (c.moveToFirst()) {
do {
result.add(c.getString(0));
} while (c.moveToNext());
}
} finally {
c.close();
}
return result;
}
Of course you can add some filter-criteria to the query as well.
The static String SQL_DISTINCT_ENAME is used for performance, so that the query string doesn't have to be built every time.
EmpDao.Properties and EmpDao.TABLENAME is used to always have the exact column-names and table-names as they are generated by greendao.