Spring data mongo pagination - mongodb

I want to implement pagination with Spring Data Mongo. There are many tutorials and docs suggest to use PagingAndSortingRepository, like this:
StoryRepo extends PagingAndSortingRepository<Story, String>{}
And so because PagingAndSortingRepository provides api for query with paging, I can use it like:
Page<Story> story = storyRepo.findAll(pageable);
My question is where actually is this findAll method here implemented? Do I need to write its implementation by myself?
The StoryRepoImpl which implements StoryRepo needs to implement this method?

You do not need to implement the method as when you autowired the Spring object PagingAndSortingRepository, it automatically implements the method for you.
Please note that since you are using Mongodb, you can extend MongoRepository instead.
Then in Spring, enable pagination using this:
#RequestMapping(value="INSERT YOUR LINK", method=RequestMethod.GET)
public List<Profile> getAll(int page) {
Pageable pageable = new PageRequest(page, 5); //get 5 profiles on a page
Page<Profile> page = repo.findAll(pageable);
return Lists.newArrayList(page);

I got it working by writing my own implementations, something like this:
List<Story> stories = null;
Query query = new Query();
query.with(pageable);
stories = getTemplate().find(query, Story.class);
long total = getTemplate().count(query, Story.class);
Page<Story> storyPage = new PageImpl<Story>(stories, pageable, total);
return storyPage;
I'm working with spring data & mongodb, using mongo template to query data.

In Spring Data, you create an interface and add a method using the naming conventions used by Spring data and the framework will generate the implementation of that method.
To implement pagination, i create this method declaration in my repository:
public interface PersonRepository extends MongoRepository<Person, ObjectId> {
Page<Person> findByName(String name, Pageable pageable);
}
Then, in my service i call this method like this:
Page<Person> persons = personRepository.findByName("Alex", PageRequest.of(0, 100));
Here, the page will contain 100 element.

To paginate a query, you can use something like below:
public interface PersonRepository extends MongoRepository<Person, String> {
Page<Person> findByFirstname(String firstname, Pageable pageable);
}
For more details, please refer to the second query in Example 6.6 in https://docs.spring.io/spring-data/mongodb/docs/1.2.0.RELEASE/reference/html/mongo.repositories.html

The method is implemented by a store-specific class. For the Spring Data JPA module, it's SimpleJpaRepository. You usually let a DI container create instances for these repository interfaces. With Spring you'd activate Spring Data repositories by either using #EnableJpaRepository on a JavaConfig class or
<jpa:repositories base-package="com.acme.repositories" />
This will create a proxy instance for the repo, so that you can get it injected into your clients:
class MyClient {
#Inject
public MyClient(PersonRepository repository) {
…
}
}

Query query1 = new Query();
Integer startIndex = page * size;
Integer endIndex = (page * size) + size;
List<dto> totalRecord = mongoOperation.find(query1, dto.class);
query1.limit((endIndex > totalRecord.size() ? totalRecord.size() : endIndex));
List<dto> responseList = mongoOperation.find(query1, dto.class);
int end = (endIndex > (totalRecord.size() - 1) ? totalRecord.size() - 1 : endIndex);
if (totalRecord.size() > 0 && end == 0)
end = 1;
if (totalRecord.size() > 0)
responseList = responseList.subList(startIndex, end);
int totalPages = totalRecord.size() / size + (totalRecord.size() % size == 0 ? 0 : 1);

As other answer says: you don't need to implement the repository interface classes. The generating magic will handle it.
I wrote this answer because of say implementation way of PageRequest deprecated
-the Pageable pageable = new PageRequest(fromIndex, toIndex); one.
If you want to implement a pageable request nowadays, you need to use like following:
PageRequest page = PageRequest.of(pageNum, pageSize);
List<MyEntity> result =myEntityRepository.findAll(page).toList();

Related

Is possible to return a projection when query-by-example?

I need to write a query with many optional parameters. I don't want to write my query with many 'is-null' checking.Is that possible to return a projection when we use query-by-example in spring data jpa?
User user = new User();
user.setName("John");
Example<User> ex = Example.of(name);
repository interface
public interface EnglishNameRepos extends JpaRepository<User,Long>{
//how to custom method?
Page<UserProjection> findXXX(Example<User> ex, Pageable pageable);
}

How to find top N elements in Spring Data Jpa?

In Spring Data Jpa to get first 10 rows I can do this findTop10By...(). In my case the number or rows is not defined and comes as a parameter.
Is there something like findTopNBy...(int countOfRowsToGet)?
Here is another way without native query. I added Pageable as a parameter to the method in the interface.
findAllBySomeField(..., Pageable pageable)
I call it like this:
findAllBySomeField(..., PageRequest.of(0, limit)) // get first N rows
findAllBySomeField(..., Pageable.unpaged()) // get all rows
I don't know of a way to do exactly what you want, but if you are open to using #Query in your JPA repository class, then a prepared statement is one alternative:
#Query("SELECT * FROM Entity e ORDER BY e.id LIMIT :limit", nativeQuery=true)
Entity getEntitiesByLimit(#Param("limit") int limit);
Did it by using pagination, as described in the first answer. Just adding a more explicit example.
This example will give you the first 50 records ordered by id.
Repository:
#Repository
public interface MyRepository extends JpaRepository<MyEntity, String> {
Page<MyEntity> findAll(Pageable pageable);
}
Service:
#Service
public class MyDataService {
#Autowired
MyRepository myRepository;
private static final int LIMIT = 50;
public Optional<List<MyEntity>> getAllLimited() {
Page<MyEntity> page = myRepository.findAll(PageRequest.of(0, LIMIT, Sort.by(Sort.Order.asc("id"))));
return Optional.of(page.getContent());
}
}
Found the original idea here:
https://itqna.net/questions/16074/spring-data-jpa-does-not-recognize-sql-limit-command
(which will also link to another SO question btw)

How to inject spring aop advice for MongoDb call?

I am new to Spring Aop, but I have case to implement AOP advice for a mongo db call(monog db update). I am trying in different way but getting 'Point cut not well formed' error or 'warning no match for this type name: arg string [Xlint:invalidAbsoluteTypeName]'(even if I give absolute name of the argument). Anyone can help on this as how to inject advice for mongo db update call?
#Aspect
#Component
public class DBStatsLoggerAspect {
private static final Logger log = LoggerFactory
.getLogger(DBStatsLoggerAspect.class);
private static final Document reqStatsCmdBson = new Document(
"getLastRequestStatistics", 1);
private DbCallback<Document> requestStatsDbCallback = new DbCallback<Document>() {
#Override
public Document doInDB(MongoDatabase db) throws MongoException,
DataAccessException {
return db.runCommand(reqStatsCmdBson);
}
};
#After("execution( public * com.mongodb.client.MongoCollection.*(..)) && args(org.bson.conversions.Bson.filter,..)")
public void requestStatsLoggerAdvice(JoinPoint joinPoint) {
MongoTemplate mongoTemplate = (MongoTemplate) joinPoint.getTarget();
log.info(mongoTemplate.execute(requestStatsDbCallback).toJson());
}
}
Actual db call method where I need to inject advice:(filter, updatePart all are org.bson.conversions.Bson data type) and here 'collection' is com.mongodb.client.MongoCollection.collection
Document result = collection.findOneAndUpdate(filter, updatePart, new FindOneAndUpdateOptions().upsert(false));
I am not a Spring or MongoDB user, just an AOP expert. But from what I see I am wondering:
You are intercepting execution(public * com.mongodb.client.MongoCollection.*(..)), so joinPoint.getTarget() is a MongoCollection type. Why do you think you can cast it to MongoTemplate? That would only work if your MongoCollection happened to be a MongoTemplate subclass. To me this looks like a bug.
Class MongoCollection is not a Spring component but a third-party class. Spring AOP can only intercept Spring component calls by means of creating dynamic proxies for those components and adding aspect interceptors to said proxies. so no matter how correct or incorrect your pointcut, it should never trigger.
What you can do instead is switch from Spring AOP to full-blown AspectJ. The standard way to do this is to activate AspectJ load-time weaving (LTW).

MongoDB and Large Datasets when using a Repository pattern

Okay so at work we are developing a system using MVC C# & MongoDB. When first developing we decided it would probably be a good idea to follow the Repository pattern (what a pain in the ass!), here is the code to give an idea of what is currently implemented.
The MongoRepository class:
public class MongoRepository { }
public class MongoRepository<T> : MongoRepository, IRepository<T>
where T : IEntity
{
private MongoClient _client;
private IMongoDatabase _database;
private IMongoCollection<T> _collection;
public string StoreName {
get {
return typeof(T).Name;
}
}
}
public MongoRepository() {
_client = new MongoClient(ConfigurationManager.AppSettings["MongoDatabaseURL"]);
_database = _client.GetDatabase(ConfigurationManager.AppSettings["MongoDatabaseName"]);
/* misc code here */
Init();
}
public void Init() {
_collection = _database.GetCollection<T>(StoreName);
}
public IQueryable<T> SearchFor() {
return _collection.AsQueryable<T>();
}
}
The IRepository interface class:
public interface IRepository { }
public interface IRepository<T> : IRepository
where T : IEntity
{
string StoreNamePrepend { get; set; }
string StoreNameAppend { get; set; }
IQueryable<T> SearchFor();
/* misc code */
}
The repository is then instantiated using Ninject but without that it would look something like this (just to make this a simpler example):
MongoRepository<Client> clientCol = new MongoRepository<Client>();
Here is the code used for the search pages which is used to feed into a controller action which outputs JSON for a table with DataTables to read. Please note that the following uses DynamicLinq so that the linq can be built from string input:
tmpFinalList = clientCol
.SearchFor()
.OrderBy(tmpOrder) // tmpOrder = "ClientDescription DESC"
.Skip(Start) // Start = 99900
.Take(PageLength) // PageLength = 10
.ToList();
Now the problem is that if the collection has a lot of records (99,905 to be exact) everything works fine if the data in a field isn't very large for example our Key field is a 5 character fixed length string and I can Skip and Take fine using this query. However if it is something like ClientDescription can be much longer I can 'Sort' fine and 'Take' fine from the front of the query (i.e. Page 1) however when I page to the end with Skip = 99900 & Take = 10 it gives the following memory error:
An exception of type 'MongoDB.Driver.MongoCommandException' occurred
in MongoDB.Driver.dll but was not handled in user code
Additional information: Command aggregate failed: exception: Sort
exceeded memory limit of 104857600 bytes, but did not opt in to
external sorting. Aborting operation. Pass allowDiskUse:true to opt
in..
Okay so that is easy to understand I guess. I have had a look online and mostly everything that is suggested is to use Aggregation and "allowDiskUse:true" however since I use IQueryable in IRepository I cannot start using IAggregateFluent<> because you would then need to expose MongoDB related classes to IRepository which would go against IoC principals.
Is there any way to force IQueryable to use this or does anyone know of a way for me to access IAggregateFluent without going against IoC principals?
One thing of interest to me is why the sort works for page 1 (Start = 0, Take = 10) but then fails when I search to the end ... surely everything must be sorted for me to be able to get the items in order for Page 1 but shouldn't (Start = 99900, Take = 10) just need the same amount of 'sorting' and MongoDB should just send me the last 5 or so records. Why doesn't this error happen when both sorts are done?
ANSWER
Okay so with the help of #craig-wilson upgrading to the newest version of MongoDB C# drivers and changing the following in MongoRepository will fix the problem:
public IQueryable<T> SearchFor() {
return _collection.AsQueryable<T>(new AggregateOptions { AllowDiskUse = true });
}
I was getting a System.MissingMethodException but this was caused by other copies of the MongoDB drivers needing updated as well.
When creating the IQueryable from an IMongoCollection, you can pass in the AggregateOptions which allow you to set AllowDiskUse.
https://github.com/mongodb/mongo-csharp-driver/blob/master/src/MongoDB.Driver/IMongoCollectionExtensions.cs#L53

How to query with Pageable with Spring Data and Couchbase

I just try to query with a PageRequest:
Page<MyEntity> entityPage = myRepository.findAll(new PageRequest(1, 20));
and this is my Repository
#Repository
public interface MyRepository extends CouchbasePagingAndSortingRepository<MyEntity, String> {
}
But always get an empty Page object back.
I'm using 2.1.5.RELEASE for spring-data-couchbase. I read in this older question that it wasn't implemented yet - but this was 2015 and in the Spring Data for Couchbase documentation it's described in detail. So I guess it should work by now ...
You are requesting for second page. Pages are zero indexed, thus providing 0 for page will return the first page.
Try this:
Page<MyEntity> entityPage = myRepository.findAll(new PageRequest(0, 20));