RxJava - Somewhat complicated API requests, together with details of progress - rx-java2

Question

To start of, I've only been playing around with RxJava/RxAndroid for a couple of days.
In psuedocode I'm trying to accomplish something like this when loading data from the Guild Wars 2 API:
refreshDatabase {
getAllIds - single API call (50.000+ ids)
chunk ids into chunks of 100 each
for each chunk
getItems - single API call which fetches 100 items
update ui on the progress (# chunks out of # chunks done)
loop through all items when all chunks are fetched
determine if the item is something we're after
save it in our own database
update ui on progress (# items out of # items done)
return bool telling if successful or not
}
I've read through a lot of tutorials and documentation about it, but I'm not really grasping how to code this.
I've got finished and working logic for all of it if I want the sorting/saving to the database to run directly after the chunk has been fetched, but then I'm unable to update the UI with the details.
Since this is an operation that takes quite a while on the first run I'd like the progress to be pretty detailed rather than saving .5-2 seconds of loading time.
Any tips on how I get make it something like what I put in the psuedocode?

Answer 1

Divide an conquer.
You actually have 2 or 3 different operations, so don't try to do this in one function.
First of all, you should use the WorkManager from AndroidX Jetpack to perform the sync the local DB, and Room for reading and writing to a local SQLite database.
Starting with fetching all relevant items and storing them in your DB (not doing UI udpates here):
class MyRepository(private val api: MyApi, private val dao:MyItemDao) {
fun isItemInteresting(item: Item): Boolean {
return true
}
fun fetchAllInterestingItems(): Single<List<Item>> {
return api.getAllIds()
.flatMapIterable { it }
.buffer(100)
.flatMap { api.getItems(it) }
.flatMapIterable { it }
.filter { isItemInteresting(it) }
.toList()
}
fun updateDatabase(items: List<Item>): Completable {
return dao.storeItems(items)
}
}
class SyncWorker(
context: Context,
params: WorkerParameters,
private val repository: MyRepository
) : RxWorker(context, params) {
override fun createWork(): Single<Result> {
return repository.fetchAllInterestingItems()
.flatMapCompletable { repository.updateDatabase(it) }
.subscribeOn(Schedulers.io())
.toSingle { Result.success()}
}
}
This will fetch all items, check which are relevant, and store those in your database (MyApi would be a Retrofit interface and MyItemDao a Room Dao interface).
Reporting progress to the UI can be done through the progress reporting API in WorkManager (version 2.3.0, currently in beta).