Can we use state variables in Flink's trigger? - triggers

I am using Apache Flink's sliding window with trigger. I would like to use a state variable in the trigger. However, I did not find any rich function corresponding to trigger function.
inputStream.keyBy(new KeySelector<Integer, String>() {
#Override
public String getKey(Integer i) throws Exception {
return i;
}
}).window(SlidingEventTimeWindows.of(Time.seconds(windowSize), Time.seconds(slideStep)))
.trigger(new Trigger<String, TimeWindow> {
#Override
public TriggerResult onElement(String str, long l, TimeWindow timeWindow, TriggerContext triggerContext) throws Exception {
return TriggerResult.CONTINUE;
}
#Override
public TriggerResult onProcessingTime(long l, TimeWindow timeWindow, TriggerContext triggerContext) throws Exception {
return TriggerResult.FIRE;
}
#Override
public TriggerResult onEventTime(long l, TimeWindow timeWindow, TriggerContext triggerContext) throws Exception {
return TriggerResult.FIRE;
}
#Override
public void clear(TimeWindow timeWindow, TriggerContext triggerContext) throws Exception {
})
.apply(new RichWindowFunction<String, String, String, TimeWindow>() {...}
So my question is, can we use state variables inside trigger block?

Yes, Triggers can be stateful. Look at CountTrigger for an example. You'll see that it keeps a count of how many elements have been added to the window in
private final ReducingStateDescriptor<Long> stateDesc =
new ReducingStateDescriptor<>("count", new Sum(), LongSerializer.INSTANCE);
It then uses that state in its onElement method
public TriggerResult onElement(Object element, long timestamp, W window, TriggerContext ctx)
throws Exception {
ReducingState<Long> count = ctx.getPartitionedState(stateDesc);
count.add(1L);
if (count.get() >= maxCount) {
count.clear();
return TriggerResult.FIRE;
}
return TriggerResult.CONTINUE;
}
and clears it in its clear method
public void clear(W window, TriggerContext ctx) throws Exception {
ctx.getPartitionedState(stateDesc).clear();
}
Keep in mind that if you are implementing a mergeable window type, then your Trigger will have to use state that can be merged, and then merge it when windows are merged. (Since you are using some flavor of sliding window, you can probably ignore this.)

Related

SingleObserver.onError called on background thread instead of main thread

Well as I understood from the documentation using singel.observeOn(Scheduler) will guarantee that any downstream event will be executed on that scheduler.
Apparently the onError called on the same scheduler that thrown the error as I receiving this error -
> Caused by:java.lang.IllegalStateException: Cannot invoke setValue on a background thread
at androidx.lifecycle.LiveData.assertMainThread(LiveData.java:443)
at androidx.lifecycle.LiveData.setValue(LiveData.java:286)
at androidx.lifecycle.MutableLiveData.setValue(MutableLiveData.java:33)
at com.bonimoo.womlauncher.presentation.wizard.registration.RegistrationViewModel$1.onError(RegistrationViewModel.java:89)
at io.reactivex.internal.operators.single.SingleFlatMap$SingleFlatMapCallback$FlatMapSingleObserver.onError(SingleFlatMap.java:116)
at io.reactivex.internal.observers.ResumeSingleObserver.onError(ResumeSingleObserver.java:51)
at io.reactivex.internal.disposables.EmptyDisposable.error(EmptyDisposable.java:78)
at io.reactivex.internal.operators.single.SingleError.subscribeActual(SingleError.java:42)
at io.reactivex.Single.subscribe(Single.java:3603)
out of this code -
public static<T> SingleTransformer<T,T> getSingleTransformer(){
return upstream -> upstream
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread());
}
GetHotelsList getHotelsList = new GetHotelsList(
AsyncTransformers.getSingleTransformer(), networkRepo);
getHotelsList.getHotels()
.map(hotels->
CollectionsUtil.mapList(hotels,
RegistrationMappers::mapHotelToPresentationHotel)
)
.flatMap((Function<List<HotelPresentation>, SingleSource<List<HotelPresentation>>>) hotelPresentations ->
Completable.timer(5,TimeUnit.SECONDS, Schedulers.io())
.andThen(Single.error(new Throwable()))
)
.subscribe(new SingleObserver<List<HotelPresentation>>() {
#Override
public void onSubscribe(Disposable d) {
addDisposable(d);
RegistrationState currentState = stateLiveData.getValue();
stateLiveData.setValue(currentState.newBuilder().setLoadingHotels(true).build());
}
#Override
public void onSuccess(List<HotelPresentation> hotelPresentations) {
RegistrationState currentState = stateLiveData.getValue();
stateLiveData.setValue(currentState.newBuilder().setHotelsList(hotelPresentations).setLoadingHotels(false).build());
}
#Override
public void onError(Throwable e) {
RegistrationState currentState = stateLiveData.getValue();
stateLiveData.setValue(currentState.newBuilder().setLoadingHotels(false).build());
}
});
and the debugger shown that onError called on RxSchedulerIoThread.

unable to add counter in Flink 1.3.2

I am trying to add a counter in Flink as mentioned here, but the issue is that counter.inc() is returning void instead of Integer. Code for my Metric is given as below
private static class myMetric extends RichMapFunction<String,Integer> {
private Counter counter ;
#Override
public void open(Configuration parameters) throws Exception {
super.open(parameters);
this.getRuntimeContext().
getMetricGroup().
counter("countit");
}
#Override
public Integer map(String s) throws Exception {
return this.counter.inc();
}
}
It should work better if you assign a value to your counter:
this.counter = getRuntimeContext()
.getMetricGroup()
.counter("countit");
You may find the documentation helpful.

Wrapping event listeners in Observables

I've seen a lot of examples of how to turn finite things like arrays or Iterables into Observables, but I'm not sure I understand how to make an Observable out of something live and effectively unbounded like an event receiver. I studied the RxJava2 docs and came up with this, using an Android LocationListener as an example.
Is there a simpler and/or more correct way to do this? I'm aware of the "RxBus" concept, but it seems like a way of clinging to the old event bus paradigm.
final Observable<Location> locationObservable = Observable.create(new ObservableOnSubscribe<Location>() {
final LocationManager mLocationManager = (LocationManager) getSystemService(LOCATION_SERVICE);
#Override
public void subscribe(final ObservableEmitter<Location> emitter) throws Exception {
final LocationListener listener = new LocationListener() {
#Override
public void onLocationChanged(final Location location) {
emitter.onNext(location);
}
#Override
public void onStatusChanged(final String s, final int i, final Bundle bundle) {
// TODO ???
}
#Override
public void onProviderEnabled(final String s) {
// TODO ???
}
#Override
public void onProviderDisabled(final String s) {
// TODO ???
}
};
mLocationManager.requestLocationUpdates(LocationManager.GPS_PROVIDER, 0, 0, listener);
emitter.setCancellable(new Cancellable() {
#Override
public void cancel() throws Exception {
mLocationManager.removeUpdates(listener);
}
});
emitter.setDisposable(new Disposable() {
private AtomicBoolean mDisposed;
#Override
public void dispose() {
if(mDisposed.compareAndSet(false, true)) {
mLocationManager.removeUpdates(listener);
}
}
#Override
public boolean isDisposed() {
return mDisposed.get();
}
});
}
});
using Observable.create() is indeed a correct way.
However, with RxJava2 the default way is to extend an Observable, you can see this answer for greater details.
some comments though regarding your implementation:
- there is no point setting both Cancellable and Disposable, as the later one will cancel/dispose the first one, you can see the difference between them here.
- I think it's best practice, to register cancellable/disposable before you start listening to update, in order to prevent weird edge cases races.

Match or Join event and rule from two kafka topics in one datastream

i want to do join between two kafka topics in one Datastream.
In fact the two datastream must have the same id to make the join.
Event are the data coming from the sensors and rule contains rules that will be checked with the CEP(coming from User Interface).
Here is my test but it does not work, can anyone help me please ?
DataStream<Object> evtAndRule=inputEventStream.join(rulesStream)
.where(new KeySelector<TrackEvent, Object>() {
#Override
public Object getKey(Event event) throws Exception {
return event.getId();
}
}).equalTo(new KeySelector<RulesEvent, Object>() {
#Override
public Object getKey(RulesEvent rulesEvent) throws Exception {
return rulesEvent.getId();
}
}).window(TumblingTimeWindows.of(Time.of(10, TimeUnit.SECONDS)))
.apply(new FlatJoinFunction<TrackEvent, RulesEvent, Object>() {
#Override
public void join(TrackEvent trackEvent, RulesEvent rulesEvent, Collector<Object> collector) throws Exception {
....
}
});
I tried this but I do not know how to retrieve the desired rule and if this is the best solution
DataStream<Tuple2<Event , RulesEvent>> evtAndRule= inputEventStream.map(new MapFunction<Event , Tuple2<Event , RulesEvent>>() {
#Override
public Tuple2<Event , RulesEvent> map(final Event event) throws Exception {
return new Tuple2<Event , RulesEvent>(event, new RulesEvent());
}
});

Cannot resolve method 'subscribe(anonymous io.reactivex.functionx.Consumer<java.util.List<...>>)' in rxjava2?

after toList operator performs, original Flowable<<\List>> converts to Single<<\List>>. and it turns out if I create Consumer to subscribe to Single,
the Consumer value type cannot be changed except Object?
#Override
public void loadBannerData(final ADFilterType adFilterType) {
remoteListDataSource.getBannerListData(adFilterType)
.flatMap(new Function<List<BannerBeanList.BannerBean>, Publisher<?>>() {
#Override
public Publisher<?> apply(List<BannerBeanList.BannerBean> bannerBeen) throws Exception {
return Flowable.fromIterable(bannerBeen);
}
})
.toList()
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
/******************************Consume Value Type**************************
.subscribe(new Consumer<List<BannerBeanList.BannerBean>>() {
#Override
public void accept(List<BannerBeanList.BannerBean> bannerBeens) throws Exception {
mainTabView.showMainBanner(bannerBeens);
}
});
*****************************************************************************/
}
From my comment: this happens because you have Publisher<?> instead of Publisher<BannerBeanList.BannerBean> in your code. Often IDEs can't infer types of lambdas or functional interfaces and you'll end up with ? or Object as their type when using some generate/convert refactoring function.