How to ensure result count and caching with varying parameters - rx-java2

Question

I have an API endpoint that can different result count based on request parameters. Parameters are page, per_page, query and others.
fun getItems(params : Map<String, String>) : Single<ItemsResponse>
data class ItemsResponse(
val hasMore : Boolean,
val items : List<Items>
)
API is not trustworthy and could return less than per_page. I want to ensure, that I always get result count I need and cache remainder for next request cycle.
For example something
val page : Int = 1
fun fetchItems(requestedItems : Int = 20) : Single<List<Items>> {
...
.map { buildParams(page, perPage, query) }
.flatMap { api.getItems(it) }
.doOnSuccess { page++ }
.buffer(requestedItems)
}
fun buildParams(page: Int, perPage: Int, query : String) : Map<String, String> {
...
}
Example scenario:
Caller requests 20 items for the first time.
Call to api.getItems() with page: 1, per_page is always 20.
Call returns 16 items
Call to api.getItems() with page: 2
Call return 19 items
20 items were returned to caller and 15 remaining items were cached for next caller request.
Caller requests 20 items for 2nd time.
Call to api.getItems() with page: 3
Call returns 12 items
20 items were returned to caller (15 older ones and 5 from last response) and 7 remaining items were cached for next caller requests.
And so on and so forth.
This looks like Producer-Consumer pattern, but is doable in RxJava2?

Answer 1

Edit: based on the additional info
Requires: RxJava 2 Extensions library: compile "com.github.akarnokd:rxjava2-extensions:0.17.0"
import hu.akarnokd.rxjava2.expr.StatementObservable
import io.reactivex.Observable
import io.reactivex.functions.BooleanSupplier
import io.reactivex.subjects.PublishSubject
import java.util.concurrent.Callable
import java.util.concurrent.ConcurrentLinkedQueue
import java.util.concurrent.ThreadLocalRandom
var counter = 0;
fun service() : Observable<String> {
return Observable.defer(Callable {
val n = ThreadLocalRandom.current().nextInt(21)
val c = counter++;
Observable.range(1, n).map({ v -> "" + c + " | " + v })
})
}
fun getPage(pageSignal : Observable<Int>, pageSize: Int) : Observable<List<String>> {
return Observable.defer(Callable {
val queue = ConcurrentLinkedQueue<String>()
pageSignal.concatMap({ _ ->
StatementObservable.whileDo(
service()
.toList()
.doOnSuccess({ v -> v.forEach { queue.offer(it) }})
.toObservable()
, BooleanSupplier { queue.size < pageSize })
.ignoreElements()
.andThen(
Observable.range(1, pageSize)
.concatMap({ _ ->
val o = queue.poll();
if (o == null) {
Observable.empty()
} else {
Observable.just(o)
}
})
.toList()
.toObservable()
)
})
})
}
fun main(args: Array<String>) {
val pages = PublishSubject.create<Int>();
getPage(pages, 20)
.subscribe({ println(it) }, { it.printStackTrace() })
pages.onNext(1)
pages.onNext(2)
}