Jetpack Paging3 RemoteMediator returns same PagingSatate on #load append - android-paging

I'm following this codelab to build an paging3 app with github API and a local DB. While the first 2 pages load fine, the mediator hits a loop when trying to load the 3rd page when scrolling to bottom - the same PagingState is passed to load() function over and over.
Just wondering if anyone knows what could be the possible root cause here?
Some implementation details:
RemoteMediator: (the prevPage and currentPage is from github API's pagination response header and saved to a local DB.)
// RepositoryMediator
override suspend fun load(
loadType: LoadType,
state: PagingState<Int, Repository>
): MediatorResult {
return when (loadType) {
LoadType.REFRESH -> {
fireRequestForPage(1, true /*clear DB*/)
return Success(endOfPaginationReached = false)
}
LoadType.APPEND -> {
// !!!!!!! kept getting the same state when APPEND is triggered, resulting in same currentPage and nextPage
// get currentPage, nextPage from state.lastItemOrNull
if(currentPage < nextPage) {
fireRequestForPage(nextPage)
Success(endOfPaginationReached = false)
} else {
return Success(endOfPaginationReached = true)
}
LoadType.PREPEND -> {
// get currentPage, prevPage from state.firstItemOrNull
if(currentPage > prevPage) {
fireRequestForPage(prevPage)
Success(endOfPaginationReached = false)
} else {
return Success(endOfPaginationReached = true)
}
}
}
}
Observable: I'm using liveData instead of flow to from the Pager:
fun searchRepositoryWithUserId(userLoginName: String): LiveData<PagingData<Repository>> {
// need to create a new Pager each time because the search query is different
return Pager(
config = PagingConfig(pageSize = PAGE_SIZE, enablePlaceholders = false),
remoteMediator = RepositoryMediator()
) {
repoDao().getRepositoriesOfUser(userLoginName)
}.liveData
}
Dao: just a plain query
#Query("SELECT * FROM repository_table WHERE login = :ownerLoginName")
fun getRepositoriesOfUser(ownerLoginName: String): PagingSource<Int, Repository>

For anyone interested, the fix is from Dao, need to update the query to sort on reponame, otherwise the query will return the same last Page for PagingSource even if there're new items inserted into DB, confusing the Mediator.
#Query("SELECT * FROM repository_table WHERE login = :ownerLoginName ORDER BY repository_name ASC")
fun getRepositoriesOfUser(ownerLoginName: String): PagingSource<Int, Repository>

Had a similar issue just now. Trying to sort by different fields had led to RemoteMediator getting stuck in a loop on different page numbers.
Turns out I couldn't rely on item ID's assigned by backend to be primary keys for my Room DB Entity. Assigning primary key ID's locally (starting from zero) seems to have fixed the issue.

Related

How to use gatling feed / foreach methods correctly?

I'm new in gatling/scala
The main idea of my scenario: I need to pass each item from some prepared List of String into the method, and check that response correct
so it's my scenario
object ForeachScenario extends Scenario {
override def profile(): OpenInjectionStep = {
atOnceUsers(3)
}
val rules_new = List("one", "two", "three")
val custom_feed = rules_new.map(el=> Map("expressions_rules" -> el)).iterator
override def createScenario(): ChainBuilder = {
group(name()) {
tryMax(SCENARIO_MAX_RETRIES) {
tryMax(NUMBER_OF_ATTEMPTS) {
// Creating a user and his context
exec(
UserCreator.createUser(
"userName", "userUid"
)
)
}.exitHereIfFailed
.tryMax(NUMBER_OF_ATTEMPTS) {
feed(custom_feed)
.exec(
someClassWithMethod.method("${userName}", "${userUid}", "request-bodies/rulesDS/rules-expressions.json")
.requestBuilder
.check(status.is(200))
.check(jsonPath("$.evaluations[0].result").find.is("100"))
)
}
}
}
}
}
in general, it works as expected, but... I would like to change this code for using only one virtual user, I have some separate scenario for checking functionality related to UserCreator.createUser and in this case, the main task create a user and under his context check another API, so, for this task user creation for each element in my list is redundantly
override def profile(): OpenInjectionStep = {
atOnceUsers(1)
}
I can't understand how can I realize this, because feeder uses only the first item from the list in the case with one virtual user (atOnceUsers(1)), probably I can save somehow userName and UserUid
exec(
UserCreator.createUser(
"userName", "userUid"
)
)
and use them further OR probably have to use another approach for iteration?
can somebody help me with that?

Rx.Net - Publish method missing first few items when subscribing to Cold Observable

Inspired by Akavache I am trying to create a solution that provides me with an IObservable<IArticle>. The method essentially first try to get all the articles that are present in the database, then it tries to fetch updated articles from the webservice and as it is getting the latest articles from webservice it tries to save them back to the database.
Since the webservice is essentially a cold observable and I don't want to subscribe twice, I used Publish to connect to it. My understanding is that I am using the correct version of the Publish method, however, many times the method tend to miss first couple of Articles from the GetNewsArticles. This was observed through the UI and also the Trace calls added in the call below.
Apart from solving the problem, it would be great to also understand how to debug/test this code (apart from introducing DI to inject NewsService).
public IObservable<IArticle> GetContents(string newsUrl, IScheduler scheduler)
{
var newsService = new NewsService(new HttpClient());
scheduler = scheduler ?? TaskPoolScheduler.Default;
var fetchObject = newsService
.GetNewsArticles(newsUrl,scheduler)
.Do(x => Trace.WriteLine($"Parsing Articles {x.Title}"));
return fetchObject.Publish(fetchSubject =>
{
var updateObs = fetchSubject
.Do( x =>
{
// Save to database, all sync calls
})
.Where(x => false)
.Catch(Observable.Empty<Article>());
var dbArticleObs = Observable.Create<IArticle>(o =>
{
return scheduler.ScheduleAsync(async (ctrl, ct) =>
{
using (var session = dataBase.GetSession())
{
var articles = await session.GetArticlesAsync(newsUrl, ct);
foreach (var article in articles)
{
o.OnNext(article);
}
}
o.OnCompleted();
});
});
return
dbArticleObs // First get all the articles from dataBase cache
.Concat(fetchSubject // Get the latest articles from web service
.Catch(Observable.Empty<Article>())
.Merge(updateObs)) // Update the database with latest articles
.Do(x => Trace.WriteLine($"Displaying {x.Title}"));
});
}
UPDATE - Added GetArticles
public IObservable<IContent> GetArticles(string feedUrl, IScheduler scheduler)
{
return Observable.Create<IContent>(o =>
{
scheduler = scheduler ?? DefaultScheduler.Instance;
scheduler.ScheduleAsync(async (ctrl, ct) =>
{
try
{
using (var inputStream = await Client.GetStreamAsync(feedUrl))
{
var settings = new XmlReaderSettings
{
IgnoreComments = true,
IgnoreProcessingInstructions = true,
IgnoreWhitespace = true,
Async = true
};
//var parsingState = ParsingState.Channel;
Article article = null;
Feed feed = null;
using (var reader = XmlReader.Create(inputStream, settings))
{
while (await reader.ReadAsync())
{
ct.ThrowIfCancellationRequested();
if (reader.IsStartElement())
{
switch (reader.LocalName)
{
...
// parsing logic goes here
...
}
}
else if (reader.LocalName == "item" &&
reader.NodeType == XmlNodeType.EndElement)
{
o.OnNext(article);
}
}
}
o.OnCompleted();
}
}
catch (Exception e)
{
o.OnError(e);
}
});
return Disposable.Empty;
});
}
UPDATE 2
Sharing the link to source code here.
There's a few things I don't like about your code. I assume NewsService is an IDisposable as it takes an HttpClient (which is disposable). You're not doing a proper clean up.
Also, you haven't provided a complete method - because you've tried cutting it down for the question - but that makes it hard to reason about how to rewrite the code.
That said, the one thing that sticks out to me as quite horrid looking is the Observable.Create. Can you please try this code instead and see if it helps things work for you?
var dbArticleObs =
Observable
.Using(
() => dataBase.GetSession(),
session =>
from articles in Observable.FromAsync(ct => session.GetArticlesAsync(newsUrl, ct))
from article in articles
select article);
Now, if that does, try rewriting fetchObject to use the same Observable.Using when newing up the `NewService.
In any case, it would be good if you could provide a complete implementation of GetContents, NewsService and your dataBase code in your question.

EBeans update does not save changed field items

I upgrade from Play 2.5 to 2.7, and am having a problem with saving my forms.
When fields are changed and I call the Model.update() the changes are not persisted in the database (even though they show changed when debugging before the update is done)
When however I set them specifically, then they do persists. So it must have to do something with the fact that it does not detect the change and does not see the object as changed. I use getter and setters in the model, and all the properties are private.
This is the controller function (with the two lines to persist those two fields)
#Check(UserTask.MANAGER)
public Result updateSceneSet(Http.Request request) {
Messages messages = messagesApi.preferred(request);
Form<StreamingSceneSet> form = formFactory.form(StreamingSceneSet.class).bindFromRequest(request);
if (form.hasErrors()) {
if (form.rawData().get("id") != null && form.rawData().get("id").length() > 0) {
long itemId = Long.parseLong(form.rawData().get("id"));
StreamingSceneSet item = StreamingSceneSet.findById(itemId);
return badRequest(views.html.streaming.editSceneSetView.render(form, item, messages, request));
} else {
return badRequest(views.html.streaming.createSceneSetView.render(form,messages, request));
}
}
// Form is OK, has no errors we can proceed
StreamingSceneSet item = form.get();
item.setName(item.getName());
item.setDescription(item.getDescription());
// Insert or update?
if (item.getId() == null) {
item.insert();
flash("success", messages().at("addedSceneSet", item.getName()));
} else {
item.update();
flash("success", messages().at("updatedSceneSet", item.getName()));
}
return redirect(routes.Streaming.sceneSets());
}
It seems because when I started the upgrade I had some legacy classes I didn't have getters and setters, and as I had some issue I put in:
play.forms.binding.directFieldAccess = true
Removing this made everything work again.

Grails updates the model before saving

I am having trouble in validating and reseting some fields based on the role of a user.
I am trying to develop a rest api with grails and my problem appears when i try to reset some fields based on the role of an user. I send a json with the desired "not allowed" changes via PUT to the controller. I modify the not allowed fields to ones that are correct for me and then call .save() and the "not alowed" fields are updated with their sent value, not with the modified by me values. Here is the code.
THE MODEL
package phonebook
class User {
String firstName
String lastName
String phoneNo
String address
String email
String password
boolean active = false
String hash
String authToken = ""
String role = "user"
static hasMany = [contacts:Contact]
static constraints = {
firstName(blank: false)
lastName(blank: false)
address(blank: true)
phoneNo(unique: true)
email(blank: false, unique: true)
password(blank: false)
role(blank: false, inList: ["user", "admin"])
hash(blank: true)
authToken(blank: true)
active(inList:[true,false])
}
}
THE METHOD FROM CONTROLLER:
#Transactional
def update(User userInstance) {
if (!isAuthenticated()){
notAllowed()
return
}
if (userInstance == null) {
notFound()
return
}
//if(isAdmin()){
def userBackup = User.findById(userInstance.id)
userInstance.role = userBackup.role
userInstance.active = userBackup.active
userInstance.hash = userBackup.hash
userInstance.authToken = userBackup.authToken
//}
if (userInstance.hasErrors()) {
respond userInstance.errors, view:'edit'
return
}
userInstance.save flush:false
request.withFormat {
'*'{ respond userInstance, [status: OK] }
}
}
THE JSON SENT VIA PUT
{
"id":"1",
"firstName": "Modified Name 23",
"role":"admin",
"active":"true",
"hash":"asdasd"
}
The above code should not modify my values for hash or active or role even if they are sent.
Any ideas?
Thanks.
The reason your changes are being saved is because by default any changes made to a domain instance will be flushed at the end of the session. This is known as open session in view with automatic session flushing. I recommend you do some reading on some of the main issues people face with GORM.
Proper use of discard may solve your issue. Discard your instance changes before you exit your controller.
For example:
if (!isAuthenticated()){
notAllowed()
userInstance.discard()
return
}
Edit
Based on conversation in the comments this perhaps may be the way to address your issue. A combination of discard and attach.
userInstance.discard()
def userBackup = User.findById(userInstance.id)
userInstance.role = userBackup.role
userInstance.active = userBackup.active
userInstance.hash = userBackup.hash
userInstance.authToken = userBackup.authToken
userInstance.attach()
I was helped by this method.
getPersistentValue
Example
def update(ShopItem shopItemInstance) {
if (shopItemInstance == null) {
notFound()
return
}
if (!shopItemInstance.itemPhoto){
shopItemInstance.itemPhoto =
shopItemInstance.getPersistentValue("itemPhoto");
}
if (shopItemInstance.hasErrors()) {
respond shopItemInstance.errors, view:'edit'
return
}
shopItemInstance.save flush:true
redirect(action: "show", id: shopItemInstance.id)
}
In your case:
userInstance.role = userInstance.getPersistentValue("role")
userInstance.active = userInstance.getPersistentValue("active")
userInstance.hash = userInstance.getPersistentValue("hash")
userInstance.authToken = userInstance.getPersistentValue("authToken")
It's better if you'll use the command objects feature. You can bind a command object with the request payload, validate it and than find and update the domain object.
You can find more details here:
http://grails.org/doc/2.3.x/guide/theWebLayer.html#commandObjects
And off the record you shoudn't use #Transactional in your controller. You can move that code into a service.
Eq:
def update(Long id, UserCommand cmd){
// Grails will map the json object into the command object and will call the validate() method if the class is annotated with #Validatable
}

Caching a result from EF

I have this method for retrieving a result from my context and caching it using MemoryCache.
public IEnumerable<CustomerRole> GetCustomerRoles()
{
string key = String.Format(CC_CACHE_CUSTOMER_ROLE_ALL, "all");
return _cacheManager.Get(key, () =>
{
return from r in _customerRoleRepository.Table select r;
}
);
}
I then use this in my view like
#foreach (CustomerRole role in Model)
{
}
The problem I have is that because the actual result isn't executed until the data is accessed (in my view), it's not actually caching the result.
How do I force this query to run via my caching function rather than waiting until the data is used?
I've not included what _cacheManager.Get() does as I know it's caching whatever I send to it properly but if you think that is the problem, let me know and I will post the relative code.
Note: I have tried doing it this way hoping it would force the query to run but still no luck
public IEnumerable<CustomerRole> GetCustomerRoles()
{
string key = String.Format(CC_CACHE_CUSTOMER_ROLE_ALL, "all");
return _cacheManager.Get(key, () =>
{
var roles = from r in _customerRoleRepository.Table select r;
return roles.Take(roles.Count());
}
);
}
You need to call a method like ToList() to force linq to get the data. Then just add that list to your cache.