GroupBy data based on date range in spring data mongo db - mongodb

How can i GroupBy data, based on date range in spring data Mongo db using
ArithmeticOperators.Subtract
I have wide range of timed data, and i should summarize data, based on average in interval time:
I partitioned data based on interval time and run this aggregation for each interval. this method is too time consuming operation when i use for loop and change start and until throw loop
MatchOperation match = match(Criteria.where("type").is(id).andOperator(Criteria.where("time").gt(start),
Criteria.where("time").lt(until)));
matchOperationL.add(match);
GroupOperation as = group("type").avg("value").as("mean");
ProjectionOperation previousOperation = project("mean").and("type").previousOperation();
Aggregation avg = Aggregation.newAggregation(match, as, previousOperation);
AggregationResults<Data2> aggregate = mongotemplate.aggregate(avg, Data1.class, Data2.class);
how can i use
ArithmeticOperators.Subtract
to solve my problem?
my EntityClass
public class Data1 {
#Id
private String id;
//value of data
private Number So;
//recorded time
private long time;
}
Can any one help me to convert this query to spring data form?
db.collection.aggregate([ {$group: { _id:{'$add': {'$trunc': { '$subtract' :[ {'$divide' : ['$time', 15000 ]}, { '$mod' : [{'$divide' :['$time',15000]},1] } ]},'$time'}}, avg : {$avg : "$So"} }}]

#Veeram second option is like this:
AggregationOperation matchTime = new AggregationOperation() {
#Override
public DBObject toDBObject(AggregationOperationContext aggregationOperationContext) {
return new BasicDBObject("$match", BasicDBObject
.parse("{ 'time': {$gt: " + s + ", $lt: " + e + "} } }"));
}
};
AggregationOperation matchSId = new AggregationOperation() {
#Override
public DBObject toDBObject(AggregationOperationContext aggregationOperationContext) {
return new BasicDBObject("$match", BasicDBObject
.parse("{'sd' : '" + sd + "'}"));
}
};
AggregationOperation groupOperation = new AggregationOperation() {
#Override
public DBObject toDBObject(AggregationOperationContext aggregationOperationContext) {
return new BasicDBObject("$group",
BasicDBObject.parse("{ _id:{ '$trunc': { '$subtract' :[ {'$divide' : ['$time', " + dist
+ " ]},{ '$mod' : [{'$divide' :['$time'," + dist
+ " ]},1] } ]}},avg : {$avg : '$So'}}}"));
}
};
AggregationOperation sortOperation = new AggregationOperation() {
#Override
public DBObject toDBObject(AggregationOperationContext aggregationOperationContext) {
return new BasicDBObject("$sort", BasicDBObject.parse("{'_id':1}"));
}
};
Aggregation agg = Aggregation.newAggregation(matchSd,matchTime, groupOperation, sortOperation);
AggregationResults<Result> aggregate = mongoOps.aggregate(agg, Data.class, Result.class);

Related

how should migrate my DBRef fields with mongock?

I have a problem with migrating an entity which has DBRef fields. If I change any field in this entity, my DBRefs are gone. What should I do?
Query query = new Query();
List<Criteria> criteriaList = new ArrayList<Criteria>();
criteriaList.add(Criteria.where("created").exists(false));
criteriaList.add(Criteria.where("updated").exists(false));
query.addCriteria(new Criteria().orOperator(criteriaList.toArray(new Criteria[criteriaList.size()])));
List<Asset> results = mongoTemplate.find(query, Asset.class);
results.forEach(asset -> {
asset.setCreated(asset.getCreatedBy().getCreated());
mongoTemplate.save(asset);
});
in summary, its problem is the MongoTemplate. mongoTemplate.save(...) creates a new entity and saves it to DB. As a consequence, if you want to change/update a field in your entity, you should use mongoTemplate.updateFirst(...). My solution is as the following.
public void createAssetCreateOrUpdateDateIfNot(MongockTemplate mongockTemplate) {
MongoTemplate mongoTemplate = mongockTemplate.getImpl();
Query queryGetAssetsCreatedOrUpdatedIsNot = new Query();
List<Criteria> criteriaList = new ArrayList<Criteria>();
criteriaList.add(Criteria.where("created").exists(false));
criteriaList.add(Criteria.where("updated").exists(false));
queryGetAssetsCreatedOrUpdatedIsNot
.addCriteria(new Criteria().orOperator(criteriaList.toArray(new Criteria[criteriaList.size()])));
List<Asset> results = mongoTemplate.find(queryGetAssetsCreatedOrUpdatedIsNot, Asset.class);
if (results.size() > 0) {
for (Asset asset : results) {
Date createdDate = (asset.getCreatedBy().getCreated() != null)
? asset.getCreatedBy().getCreated()
: new Date();
Date updateDate = (asset.getModifiedBy().getCreated() != null)
? asset.getModifiedBy().getCreated()
: new Date();
Query queryGetAssetViaId = new Query();
queryGetAssetViaId.addCriteria(Criteria.where("id").is(asset.getId()));
Update update = new Update()
.set("created", createdDate)
.set("updated", updateDate);
mongoTemplate.updateFirst(queryGetAssetViaId, update, Asset.class);
}
}
}

MongoDB aggregate framework very slow when have lookup

this is my method, when the server runs this mothod, it will take 18s.
but when i run the same query, use NoSQLBoost, it just takes 5s.
public Integer queryUserEduTotal(UserAdminV1StepListRequest body) throws ParseException {
Criteria criteria = Criteria.where(Column.SYSTEM_STATUS).is(true);
criteria.and(UserColumn.APP_ID).is(body.getAppId());
criteria.and(UserColumn.STATUS).is(2);
Criteria criteria1 = Criteria.where(UserColumn.USER_TYPE).exists(false);
Criteria criteria2 = Criteria.where(UserColumn.USER_TYPE).is(UserTypeEnum.USER_REGISTER.getKey());
Criteria criteria3 = new Criteria();
criteria3.orOperator(criteria1, criteria2);
criteria.andOperator(criteria3, Criteria.where(UserColumn.SYSTEM_CREATE_TIME).gte(sdf.parse(body.getStart())), Criteria.where(UserColumn.SYSTEM_CREATE_TIME).lte(sdf.parse(body.getEnd())));
LookupOperation lookup = lookup("user_edu_item_info", "_id", UserEduItemColumn.USER_ID, "edu_info");
Criteria criteria4 = Criteria.where(Column.SYSTEM_STATUS).is(true);
criteria4.and(UserColumn.APP_ID).is(body.getAppId());
criteria4.and("edu_info._id").exists(true);
Aggregation typedAggregation1 = Aggregation.newAggregation(
match(criteria),
lookup,
match(criteria4),
Aggregation.count().as(Constant.COUNT)
);
AggregationResults<BasicDBObject> aggregationResults = mongoTemplate.aggregate(typedAggregation1, User.class, BasicDBObject.class);
int count = 0;
if (aggregationResults.getMappedResults().size() == 0) {
count = 0;
} else {
String document = JSON.toJSONString(aggregationResults.getMappedResults().get(0));
JSONObject result = JSON.parseObject(document);
count = result.getInteger(Constant.COUNT);
}
return count;
}
and, i run the same code on the local by using the java program.it takes 5s too.
i dont know why the server takes so much time.
MongoDB configuration:
MongoClientOptions.Builder builder = new xxx()
builder.connectionsPerHost(100);
builder.minConnectionsPerHost(10);
builder.connectTimeout(30000);
builder.threadsAllowedToBlockForConnectionMultiplier(10);
builder.serverSelectionTimeout(30000);
builder.socketTimeout(0);
builder.maxWaitTime(120000);
builder.heartbeatConnectTimeout(30000);
builder.heartbeatSocketTimeout(30000);
builder.heartbeatFrequency(10000);
builder.minHeartbeatFrequency(500);
builder.localThreshold(15);
this is my local java code
public class Test {
public static void main(String[] args) {
// Link code omitted
MongoTemplate mongoTemplate = new MongoTemplate(mongoDbFactory);
search(mongoTemplate);
try {
mongoDbFactory.destroy();
} catch (Exception e) {
e.printStackTrace();
}
}
public static void search(MongoTemplate mongoTemplate){
Criteria criteria = Criteria.where("systemStatus").is(true);
//criteria code omitted, sane whit the server code
Aggregation typedAggregation1 = Aggregation.newAggregation(
match(criteria),
lookup,
match(criteria4),
Aggregation.count().as("count")
);
AggregationResults<BasicDBObject> aggregationResults = mongoTemplate.aggregate(typedAggregation1, User.class, BasicDBObject.class);
}
}

Dynamic JPQL query qith JOIN

Had to write a jpql query, based on the input need to add and condition and for some input had to need JOIN queries.
#Override
public List<IncidentHdr> fetchIncidents(IncidentHdrDto incidentHdrDto) {
StringBuilder query = new StringBuilder();
query.append(ReposJPQL.GET_INCIDENT_DETAILS);
Map<String, Object> parameters = new HashMap<String, Object>();
List<String> criteria = new ArrayList<String>();
if(incidentHdrDto.getIncidentId() > 0) {
criteria.add("inc.incidentId = :incidentId");
parameters.put("incidentId", incidentHdrDto.getIncidentId());
}
if(incidentHdrDto.getCatCode() > 0) {
criteria.add("inc.catCode = :catCode");
parameters.put("catCode", incidentHdrDto.getCatCode());
}
if(incidentHdrDto.getType != null) {
//here i need to generate a join query
//SELECT * FROM INCIDENT JOIN CATEGORY_MAST ON(INCIDENT.CAT_CODE = CATEGORY_MAST.CAT_CODE) WHERE CATEGORY_MAST.TYPE_CODE = 16
}
Query q = em.createQuery(query.toString());
logger.info("Get Incidents Query : "+query.toString());
for (Entry<String, Object> entry : parameters.entrySet()) {
q.setParameter(entry.getKey(), entry.getValue());
}
List<IncidentHdr> incidentHdrs = q.getResultList();
return incidentHdrs;
}
where as ReposJPQL is the base query which had a where condition.
public interface ReposJPQL {
public String GET_INCIDENT_DETAILS = "SELECT inc FROM IncidentHdr inc WHERE 1 = 1" ;
}

Spring data mongodb aggregation scenario

How can I translate
$group : {
_id : 1 ,
likes : { $avg: { $size: ["$likes"] }},
count: { $sum: 1 },
}
into a spring-data-mongodb aggregation query?
You can use Following Code :
ArrayList<String> list = new ArrayList<>();
list.add("$likes");
BasicDBObject idObj = new BasicDBObject("_id", 1);
BasicDBObject likesObj = new BasicDBObject("$avg", new BasicDBObject("$size", list));
BasicDBObject countObj = new BasicDBObject("$sum", 1);
idObj.append("likes", likesObj).append("count", countObj);
BasicDBObject group = new BasicDBObject("$group", idObj);
List<DBObject> pipeline = new ArrayList<>();
pipeline.add(group);
MongoTemplate mongoTemplate = new MongoTemplate(new MongoClient(), "dbname");
AggregationOutput aggregationOutput = mongoTemplate.getCollection("collectionName")
.aggregate(pipeline);
List<DBObject> dbObjects = (List<DBObject>) aggregationOutput.results();
//convert dbObjects into appropriate output
You can use also following way for spring data mongodb
List<AggregationOperation> stages = new ArrayList<>();
GroupOperation groupOperation = group()
.count().as("count")
.avg(ArrayOperators.Size.lengthOfArray("likes")).as("likes");
stages.add(groupOperation);
AggregationResults<ResultDTO> result = mongoOperation.aggregate(newAggregation(stages),
"collectionName", ResultDTO.class);
public class ResultDTO {
private Integer count;
private Double likes;
//getter setter
}

How to retrieve an embedded list of object of Entity?

I have a simple problem storing and retrieving an embedded collection of entity to mongo. I have checked theses question :
how to serialize class? and Mongodb saves list of object
what I understand is to save a list objects the class of that objects must extends ReflactionDBObject. This worked for saving the object, by retrieving it with the embedded collection does not work.
here a simple test show that retrieving embedded entities does not work !
#Test
public void whatWasStoredAsEmbeddedCollectionIsRetrieved2() {
BasicDBObject country = new BasicDBObject();
country.put("name", "Bulgaria");
List<City> cities = Lists.newArrayList(new City("Tarnovo"));
country.put("listOfCities", cities);
DBCollection collection = db().get().getCollection("test_Collection");
collection.save(country);
DBCursor object = collection.find(new BasicDBObject().append("name", "Bulgaria"));
DBObject returnedCity = object.next();
DBObject embeddedCities = (DBObject) returnedCity.get("listOfCities");
System.out.println(embeddedCities);
}
Here is the City Class
class City extends ReflectionDBObject {
String name;
City() {
}
City(String name) {
this.name = name;
}
public String getName() {
return name;
}
#Override
public boolean equals(Object o) {
if (this == o) return true;
if (!(o instanceof City)) return false;
City city = (City) o;
if (name != null ? !name.equals(city.name) : city.name != null) return false;
return true;
}
#Override
public int hashCode() {
return name != null ? name.hashCode() : 0;
}
#Override
public String toString() {
return "City{" +
"name='" + name + '\'' +
'}';
}
}
The out put of the System.out.println statement is [ { "_id" : null }]
Now how can get back the embedded object and the embedded list in it ?
If you do not have a requirement to define your own class City, you can define subdocuments using the BasicDBObjects. I only added the 'name' field to the citySubDoc1 and citySubDoc2, but of course, you can add more fields to these subdocuments.
// Define subdocuments
BasicDBObject citySubDoc1 = new BasicDBObject();
citySubDoc1.put("name", "Tarnovo");
BasicDBObject citySubDoc2 = new BasicDBObject();
citySubDoc2.put("name", "Sofia");
// add to list
List<DBObject> cities = new ArrayList <DBObject>();
cities.add(citySubDoc1);
cities.add(citySubDoc2);
country.put("listOfCities", cities);
collection.save(country);
// Specify query condition
BasicDBObject criteriaQuery = new BasicDBObject();
criteriaQuery.put("name", "Bulgaria");
// Perform the read
DBCursor cursor = collection.find(criteriaQuery);
// Loop through the results
try {
while (cursor.hasNext()) {
List myReturnedListOfCities = (List) cursor.next().get("listOfCities");
System.out.println(myReturnedListOfCities);
}
} finally {
cursor.close();
}