MongoDb lte/ lt query behaving unexpectedly - mongodb

I have connected to mongoDb using Spring. While creating query using Criteria, lt/ lte are not behaving the way they should i.e they are giving random outputs.
I want to Find a store within "x" miles of a zipcode
Query Creation/ execution code:
System.out.println("Please Enter your zipCode");
String zipCode = br.readLine();
System.out.println("Enter the distance (miles) to find store");
Integer distance = br.read();
Query query = new Query();
query.addCriteria(Criteria.where("storeDistance").lt(distance).and("storezipCode").is(zipCode));
List<Store>storeList = mongoOperation.find(query, Store.class);
if(storeList.isEmpty()){
System.out.println("Oops...no store found nearby...!!");
}else{
for(int idx=0; idx<storeList.size(); idx++){
System.out.println(storeList.get(idx).storeIdentificationumber+"\t"+storeList.get(idx).storeDistance);
}
}
Store model class
package com.storeApp.model;
import org.springframework.data.annotation.Id;
import org.springframework.data.mongodb.core.mapping.Document;
#Document(collection= "stores")
public class Store {
#Id
public String storeIdentificationumber;
public String storeName;
public String storezipCode;
public String storeCity;
public Integer storeDistance;
public Store(){}
public Store(String id, String name, String city, String zipCode, Integer distance){
this.storeIdentificationumber = id;
this.storeName = name;
this.storezipCode = zipCode;
this.storeCity = city;
this.storeDistance = distance;
}
}
Entries in Database :
{
"_id" : "store1",
"storeName" : "store One",
"storezipCode" : "123",
"storeCity" : "city",
"storeDistance" : 51
}
{
"_id" : "store03",
"storeName" : "Store Three",
"storezipCode" : "123",
"storeCity" : "city",
"storeDistance" : 54
}
Input :
Welcome to Store Application....!!
Please select choice from menu below
1. Add a Store
2. Find a Store
3. Remove a Store
2
Please Enter your zipCode
123
Enter the distance (miles) to find store
50
Expected Output :
Oops...no store found nearby...!!
Actual Output :
store1 51
According to the criteria and database, there should be no store whose distance is less than 50 miles, but still one record is being returned.

Issue is with Integer distance = br.read(); Assuming br is instance of BufferedReader. read method just reads a single character and gives the integer value of that character. You can verify by printing the distance variable.
You need to use readLine and convert the String number using Integer.parseInt

Yeah It's little weird, thing is if you want to add multiple options in your query, you should keep adding Criteria, One Criteria cannot take multiple fields.
In your case storezipCode is removing storeDistance from the Criteria chief field.
Try this
Query query = new Query();
query.addCriteria(Criteria.where("storeDistance").lt(distance));
query.addCriteria(Criteria.where("storezipCode").is(zipCode));
So question is in what scenario you would use 'and', You need to lookout for examples but it doesn't work the way we want it to work.

Related

How to create/guarantee a unique timestamp in MongoDb collection using Spring Boot, to limit queries sorted by timestamp without paging

The application is syncing data records between devices via an online Mongo DB collection. Multiple devices can send batches of new or modified records to the server Mongo collection at any time. Devices get all record updates for them that they don't already have, by requesting records added or modified since their last get request.
Approach 1 - was to add a Date object field (called stored1) to the records before saving to MongoDb. When a device requests records , mongoDb paging is used to skip entries up to the current page, and then limit to 1000. Now that the data set is large, each page request is taking a long time, and mongo hit a memory error.
https://docs.mongodb.com/manual/reference/limits/#operations
Setting allowDiskUse(true) as shown in the posted code in my current configuration isn't fixing the memory error for some reason. If that can be fixed, it still wouldn't be a long term solution as the query times with the paging are already too long.
Approach 2:
What is the best way for pagination on mongodb using java
https://arpitbhayani.me/blogs/benchmark-and-compare-pagination-approach-in-mongodb
The 2nd approach considered is to change from Mongo paging skipping returned records, to just asking for stored time > largest stored time last received, until the number of records in a return is less than the limit. This requires the stored timestamp to be unique between all records matching the query, or it could miss records or get duplicate records etc.
In the example code, using the stored2 field, there's still a chance of duplicate timestamps, even if the probability is low.
Mongo has a BSON timestamp that guarantees unique values per collection, but I don't see a way to use it with document save(), or query on it in Spring Boot. It would need to be set on each record newly inserted, or replaced, or updated.
https://docs.mongodb.com/manual/reference/bson-types/#timestamps
Any suggestions on how to do this?
#Getter
#Setter
public abstract class DataModel {
private Map<String, Object> data;
#Id // maps this field name to the database _id field, automatically indexed
private String uid;
/** Time this entry is written to the db (new or modified), to support querying for changes since last query */
private Date stored1; //APPROCAH 1
private long stored2; //APPROACH 2
}
/** SpringBoot+MongoDb database interface implementation */
#Component
#Scope("prototype")
public class SpringDb implements DbInterface {
#Autowired
public MongoTemplate db; // the database
#Override
public boolean set(Collection<?> newRecords, Collection<?> updatedRecords) {
// get current time for this set
Date date = new Date();
int randomOffset = ThreadLocalRandom.current().nextInt(0, 500000);
long startingNanoSeconds = Instant.now().getEpochSecond() * 1000000000L + instant.getNano() + randomOffset;
int ns = 0;
if (updatedRecords != null && updatedRecords.size() > 0) {
for (Object entry : updatedRecords) {
entry.setStored1(date); //APPROACH 1
entry.setStored2(startingNs + ns++); //APPROCH 2
db.save(entry, repoName);
}
}
// for new documents only
if (newRecords != null && newRecords.size() > 0) {
for (DataModel entry : newRecords) {
entry.setStored1(date); //APPROACH 1
entry.setStored2(startingNs + ns++); // APPROACH 2
}
//multi record insert
db.insert(newRecords, repoName);
}
return true;
}
#Override
public List<DataModel> get(Map<String, String> params, int maxResults, int page, String sortParameter) {
// generate query
Query query = buildQuery(params);
//APPROACH 1
// do a paged query
Pageable pageable = PageRequest.of(page, maxResults, Direction.ASC, sortParameter);
List<T> queryResults = db.find(query.allowDiskUse(true).with(pageable), DataModel.class, repoName); //allowDiskUse(true) not working, still get memory error
// count total results
Page<T> pageQuery = PageableExecutionUtils.getPage(queryResults, pageable,
() -> db.count(Query.of(query).limit(-1).skip(-1), clazz, getRepoName(clazz)));
// return the query results
queryResults = pageQuery.getContent();
//APPROACH 2
List<T> queryResults = db.find(query.allowDiskUse(true), DataModel.class, repoName);
return queryResults;
}
#Override
public boolean update(Map<String, String> params, Map<String, Object> data) {
// generate query
Query query = buildQuery(params);
//This applies the same changes to every entry
Update update = new Update();
for (Map.Entry<String, Object> entry : data.entrySet()) {
update.set(entry.getKey(), entry.getValue());
}
db.updateMulti(query, update, DataModel.class, repoName);
return true;
}
private Query buildQuery(Map<String, String> params) {
//...
}
}
The solution I ended up using was to define, and index on, another field called storedId, which is a string concatenation of the modified record storedTime, and the _id. This guarantees all these storedId record fields are unique, because _id is unique.
Here's an example to show how indexing and querying on the concatenated storedTime+_id field works, while indexing and querying on the separate storedTime and _id fields fails:
public abstract class DataModel {
private Map<String, Object> data;
#Indexed
private String _id; // Unique id
#Indexed
private String storedTime; // Time this entry is written to the db (new or modified)
#Indexed
String storedId; // String concatenation of storedTime and _id field
}
//Querying on separate fields and indexes:
{
//storedTime, _id
"time1", "id2"
"time1", "id3"
"time1", "id4"
"time2", "id1"
"time2", "id5"
}
get (storedTime>"time0", _id>"id0", limit=2) // returns _id's 2,3 (next query needs to check for more at storedTime="time1" but skip _id’s <="id3")
get (storedTime>="time1", _id>"id3", limit=2) // returns _id's 4,5
//FAILS because this second query MISSES _id 1 (Note any existing _id record can be modified at any time, so the _id fields are not in storedTime order)
//Querying on the combined field and index:
{
//storedId
"time1-id2"
"time1-id3"
"time1-id4"
"time2-id1"
"time2-id5"
}
get (storedId>"time0", limit=2) // returns _id's 2,3 (next query for values greater than the greatest last value returned)
get (storedId>"time1-id3", limit=2) // returns _id's 4,1 (next query for values greater than the greatest last value returned)
get (storedId>"time2-id1", limit=2) //: returns _id 5
//WORKS, this doesn't miss or duplicate any records

Spring Data save date and time in specific format

I don't understand how can I save date and time for example "eventDateFrom" in MongoDB in specific format.
I try do it like that:
Part of Event class:
public class Event {
#Id
private String id;
private String eventTitle;
#DateTimeFormat(pattern = "MM/dd/yyyy HH:mm:ss")
private Date eventDateFrom;
private String eventDateTo;
private Integer minEventPartcipants;
private String eventDateConfirmTo;
private String location;
private String minEventTime;
private List<Participant> participantList = new ArrayList<>();
Part of RunAtStart class:
Event event = new Event();
event.setEventTitle("Wyjazd na rower Turbacz");
event.setEventDateFrom(new Date("05/27/2017 10:00:00"));
event.setEventDateTo("27.05.2017 22:00:00");
event.setMinEventPartcipants(2);
event.setEventDateConfirmTo("26.05.2017 10:00:00");
event.setLocation("Lindego 13a");
event.setMinEventTime("06:00:00");
But the time "eventDateFrom" is wrong.
MongoDB document:
{
"_id" : ObjectId("592d34dcc5f2fd2d204d7db4"),
"_class" : "info.wzielezicki.app.MeetHelp.model.Event",
"eventTitle" : "Wyjazd na rower Turbacz",
"eventDateFrom" : ISODate("2017-05-27T08:00:00.000Z"),
"eventDateTo" : "27.05.2017 22:00:00",
"minEventPartcipants" : 2,
"eventDateConfirmTo" : "26.05.2017 10:00:00",
"location" : "Lindego 13a",
"minEventTime" : "06:00:00",
"participantList" : []
}
What I'm doing wrong?
Is the better way to define date and time format?
If you want to store a Date in a specific format and not the one, the store uses, you don't really want to store a Date, but a String, so just make the property of type String. You can still have additional accessors that use Date to use inside your application.

Compare pair of attributes using "IN"

The domain that I'm describing is not the same that I use (with this in mind, there is no need to worry with the fact the the sample class doesn't use Enum for the type, and other things), but describe well my problem. I want to make a query that filter transactions by username and type.
public class Transaction {
private String userName;
private String type;
private BigDecimal value;
...
}
I would like to search all the transactions that match a list of [{'username', 'type'}], where the couple {"username", "type"} is chosen by the user, and the user can choose as many as he wants
For instance: Find all :
"Debit" transactions made by "Rafael"
"Credit" transactions made by "Daniel"
"Debit" transactions made by "Monica"
"Credit" transactions made by "Monica"
I have two possible solutions for my problem, but I don't like any
POSSIBLE SOLUTION 1
I could dynamically add n "OR" clauses to my query, like:
"WHERE (t.userName = 'Rafael' AND t.type = 'Debit')
OR (t.userName = 'Daniel' AND t.type = 'Credit')
... "
POSSIBLE SOLUTION 2
I could concatenate the "username" to the "type" and check if the result is contained by a list of concatenated values of "username" with "type" which was generated dynamically.
CONCAT(t.userName, '-', t.type) IN (list)
for the example, the variable "list" would have the following values:
['Rafael-Debit', 'Daniel-Credit', 'Monica-Debit', 'Monica-Credit']
I'm using JPA, does anyone have a suggestion?

How can I query to find mongo entities whose list of sub-entities contain a field matching a string?

I have a collection of entities that look like this:
public class ClientEntity {
#Id
private String id;
#Indexed(unique = true)
private String clientId;
private String name;
#DBRef
private List<ClientMachineEntity> machines;
...
}
...where ClientMachineEntity looks like:
public class ClientMachineEntity {
#Id
private String id;
#Indexed(unique = true)
private String clientMachineId;
private String hostName;
...
}
I have a working search that finds ClientEntities by matching against "clientId" and "name":
public List<ClientEntity> searchByIdAndName(String id, String name) {
Criteria idCriteria = Criteria.where("clientId").regex(id, "i");
Criteria nameCriteria = Criteria.where("name").regex(name, "i");
Query query = new Query(new Criteria().orOperator(idCriteria, nameCriteria));
...
}
So my question is, how can I expand this search so that it also matches against "clientMachineId" in the list of sub-entities? I tried adding the following criteria:
Criteria machineCriteria = Criteria.where("machines.clientMachineId").regex(id, "i");
...but that doesn't work, presumably because machines is a LIST of entities, not just a single sub-entity.
UPDATE: It seems like what I'm looking for is the .elemMatch() functionality, but when I try that:
Criteria machineCriteria = Criteria.where("machines").elemMatch(Criteria.where("clientMachineId").regex(id, "i"));
...I get the following error:
org.springframework.data.mapping.model.MappingException: No mapping metadata found for class com.mongodb.BasicDBObject
You can't query by fields in subentities linked with DBRef. If ClientMachineEntity would be embedded in ClientMachine - then you could use dot notation or $elemMatch depending on needs.
In your particular example - couldn't field ClientMachineEntity.clientMachineId be saved as _id and used as a primary key? Then you could get the results you need - take a look at: How to query mongodb with DBRef
My suggestion for development with Spring Data MongoDB is - first learn how to (and if it's possible) do it in plain Javascript with MongoDB console, then learn how to do the same with Spring Data MongoDB.

spring data - Mongodb - findBy Method for nested objects

I have two domain objects,
#Document
public class PracticeQuestion {
private int userId;
private List<Question> questions;
// Getters and setters
}
#Document
public class Question {
private int questionID;
private String type;
// Getters and setters
}
My JSON doc is like this,
{
"_id" : ObjectId("506d9c0ce4b005cb478c2e97"),
"userId" : 1,
"questions" : [
{
"questionID" : 1,
"type" : "optional"
},
{
"questionID" : 3,
"type" : "mandatory"
}
]
}
I have to update the "type" based on userId and questionId, so I have written a findBy query method inside the custom Repository interface,
public interface CustomRepository extends MongoRepository<PracticeQuestion, String> {
List<PracticeQuestion> findByUserIdAndQuestionsQuestionID(int userId,int questionID);
}
My problem is when I execute this method with userId as 1 and questionID as 3, it returns the entire questions list irrespective of the questionID. Is the query method name valid or how should I write the query for nested objects.
Thanks for any suggestion.
Just use the #Query annotation on that method.
public interface CustomRepository extends MongoRepository<PracticeQuestion, String> {
#Query(value = "{ 'userId' : ?0, 'questions.questionID' : ?1 }", fields = "{ 'questions.questionID' : 1 }")
List<PracticeQuestion> findByUserIdAndQuestionsQuestionID(int userId, int questionID);
}
By adding the fields part of the #Query annotation, you are telling Mongo to only return that part of the document. Beware though, it still returns the entire document in the same format - just missing everything you did not specify. So your code will still have to return List<PracticeQuestion> and you will have to do:
foreach (PracticeQuestion pq : practiceQuestions) {
Question q = pq.getQuestions().get(0); // This should be your question.
}
Property expressions
Property expressions can refer only to a direct property of the managed entity, as shown in the preceding example. At query creation time you already make sure that the parsed property is a property of the managed domain class. However, you can also define constraints by traversing nested properties. Assume Persons have Addresses with ZipCodes. In that case a method name of List<Person> findByAddressZipCode(ZipCode zipCode);
creates the property traversal x.address.zipCode. The resolution algorithm starts with interpreting the entire part (AddressZipCode) as the property and checks the domain class for a property with that name (uncapitalized). If the algorithm succeeds it uses that property. If not, the algorithm splits up the source at the camel case parts from the right side into a head and a tail and tries to find the corresponding property, in our example, AddressZip and Code. If the algorithm finds a property with that head it takes the tail and continue building the tree down from there, splitting the tail up in the way just described. If the first split does not match, the algorithm move the split point to the left (Address, ZipCode) and continues.
Although this should work for most cases, it is possible for the algorithm to select the wrong property. Suppose the Person class has an addressZip property as well. The algorithm would match in the first split round already and essentially choose the wrong property and finally fail (as the type of addressZip probably has no code property). To resolve this ambiguity you can use _ inside your method name to manually define traversal points. So our method name would end up like so:
UserDataRepository:
List<UserData> findByAddress_ZipCode(ZipCode zipCode);
UserData findByUserId(String userId);
ProfileRepository:
Profile findByProfileId(String profileId);
UserDataRepositoryImpl:
UserData userData = userDateRepository.findByUserId(userId);
Profile profile = profileRepository.findByProfileId(userData.getProfileId());
userData.setProfile(profile);
Sample Pojo :
public class UserData {
private String userId;
private String status;
private Address address;
private String profileId;
//New Property
private Profile profile;
//TODO:setter & getter
}
public class Profile {
private String email;
private String profileId;
}
For the above Document/POJO in your Repository Class:
UserData findByProfile_Email(String email);
For ref : http://docs.spring.io/spring-data/data-commons/docs/1.6.1.RELEASE/reference/html/repositories.html
You need to use Mongo Aggregation framework :
1) Create custom method for mongo repository : Add custom method to Repository
UnwindOperation unwind = Aggregation.unwind("questions");
MatchOperation match = Aggregation.match(Criteria.where("userId").is(userId).and("questions.questionId").is(questionID));
Aggregation aggregation = Aggregation.newAggregation(unwind,match);
AggregationResults<PracticeQuestionUnwind> results = mongoOperations.aggregate(aggregation, "PracticeQuestion",
PracticeQuestionUnwind.class);
return results.getMappedResults();
2) You need to cretae a class(Because unwind operation has changed the class structure) like below :
public class PracticeQuestionUnwind {
private String userId;
private Question questions;
This will give you only those result which matches the provide userId and questionId
Result for userId: 1 and questionId : 111 :
{
"userId": "1",
"questions": {
"questionId": "111",
"type": "optional"
}
}
i too had similar issue. for that i added $ before the nested class attributes.
try below query
#Query(value = "{ 'userId' : ?0, 'questions.$questionID' : ?1 }") List<PracticeQuestion> findPracticeQuestionByUserIdAndQuestionsQuestionID(int userId, int questionID);