RxJs: request list from server, consume values, re-request when we're almost out of values - rest

Question

I'm fetching a list of items from a REST api. The user interacts with each one via a click, and when there are only, say, a couple left unused, I'd like to repeat the request to get more items. I'm trying to do this using a proper RxJs (5) stream-oriented approach.
So, something like:
var userClick$ = Observable.fromEvent(button.nativeElement, 'click');
var needToExtend$ = new BehaviorSubject(1);
var list$ = needToExtend$
.flatMap( () => this.http.get("http://myserver/get-list") )
.flatMap( x => x['list'] );
var itemsUsed$ = userClick$.zip(list$, (click, item) => item);
itemsUsed$.subscribe( item => use(item) );
and then, to trigger a re-load when necessary:
list$.subscribe(
if (list$.isEmpty()) {
needToExtend$.next(1);
}
)
This last bit is wrong, and manually re-triggering doesn't seem very "stream-oriented" even if it did work as intended. Any ideas?
This is similar to Rxjs - Consume API output and re-query when cache is empty but I can't make assumptions about the length of the list returned by the API, and I'd like to re-request before the list is completely consumed. And the solution there feels a bit too clever. There must be a more readable way, right?

Answer 1

How about something like this:
const LIST_LIMIT = 3;
userClick$ = Observable.fromEvent(button.nativeElement, 'click');
list$ = this.http.get("http://myserver/get-list").map(r => r.list);
clickCounter$ = this.userClick$.scan((acc: number, val) => acc + 1, 0);
getList$ = new BehaviorSubject([]);
this.getList$
.switchMap(previousList => this.list$)
.switchMap(list => this.clickCounter$, (list, clickCount) => { return {list, clickCount}; })
.filter(({list, clickCount}) => clickCount >= list.length - LIST_LIMIT)
.map(({list, clickCount}) => list)
.subscribe(this.getList$);
The logic here if you define a list getter stream, and a signal to trigger it.
First, the signal causes switchMap to fetch a new list, which is then fed into another switchmap that resubscribes to a click counter. You combine the result of both streams and feed that to filter, which only emits when the click count is greater than or equal to the list length minus 3 (or whatever you want). Then the signal is subscribed to this whole stream so that it retriggers itself.
Edit: the biggest weakness of this is that you need to set the list value (for display) in a side effect rather than in subscription or with the async pipe. You can rearrange it and multicast though:
const LIST_LIMIT = 3;
userClick$ = Observable.fromEvent(button.nativeElement, 'click');
list$ = this.http.get("http://myserver/get-list").map(r => r.list);
clickCounter$: Observable<number> = this.userClick$.scan((acc: number, val) => acc + 1, 0).startWith(0);
getList$ = new BehaviorSubject([]);
refresh$ = this.getList$
.switchMap(list => this.clickCounter$
.filter(clickCount => list.length <= clickCount + LIST_LIMIT)
.first(),
(list, clickCount) => list)
.switchMap(previousList => this.list$)
.multicast(() => this.getList$);
this.refresh$.connect();
this.refresh$.subscribe(e => console.log(e));
This way has a few advantages, but may be a little less "readable". The pieces are mostly the same, but instead you go to the counter first and let that lead into the switch to the list fetch. and you multicast it to restart the counter.

Answer 2

I'm not clear on how you are tracking getting the next set of items so I will assume it is some form of paging for my answer. I also assume that you don't know the total number of items.
console.clear();
const pageSize = 5;
const pageBuffer = 2;
const data = [...Array(17).keys()]
function getData(page) {
const begin = pageSize * page
const end = begin + pageSize;
return Rx.Observable.of(data.slice(begin, end));
}
const clicks = Rx.Observable.interval(400);
clicks
.scan(count => ++count, 0)
.do(() => console.log('click'))
.map(count => {
const page = Math.floor(count / pageSize) + 1;
const total = page * pageSize;
return { total, page, count }
})
.filter(x => x.total - pageBuffer === x.count)
.startWith({ page: 0 })
.switchMap(x => getData(x.page))
.takeWhile(x => x.length > 0)
.subscribe(
x => { console.log('next: ', x); },
x => { console.log('error: ', x); },
() => { console.log('completed'); }
);
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/5.5.3/Rx.min.js"></script>
Here is an explaination:
Rx.Observable.interval(#): simulates the client click events
.scan(...): accumulates the click events
.map(...): calculates the page index and potential total item count (actual count could be less but it doesn't matter for our purposes
.filter(...): only allow to pass through to get a new page of data if it has just hit the page buffer.
.startWith(...): get the first page without waiting for clicks. The +1 on the page calculation in the .scan accounts for this.
.switchMap(...): get the next page of data.
.takeWhile(...): keep the stream open till we get an empty list.
So it will get an initial page and then go get a new page whenever the number of clicks comes within the designated buffer. Once all items have been retrieved (known by empty list) it will complete.
One thing I didn't figure out how to do is to complete the list when the page length is less than the page size. Not sure if it matters to you.