I am getting the data from messageHandler(kafka) and processing it via servicehandler to give me some result
serviceHandler.process(message).getMessage().toString()
Now I need to output this stream of results so that whenever a new record comes through messagehandler, it get processed and output is pushed to front-end(angular)
#Autowired
MessageHandler messageHandler;
#Autowired
ServiceHandler serviceHandler;
public void runHandler() {
Flux<Message> messages = messageHandler.flux();
messages.subscribeOn(Schedulers.parallel())
.doOnNext(message -> serviceHandler.process(message).getMessage().toString())
.subscribe();
}
public Flux pushresult(){ ???? }
Does anyone know of a way to what I need?
Take a look at .map() and .flatMap(). In your case it looks like map() but if the next service you are calling also returns a Mono/Flux you want .flatMap() or you'll end up with something like:
Flux<Flux<String>>
instead of
Flux<String>
.doOnNext is meant for side effects like logging.
Related
I am new to Quarkus. I am trying to write a REST endpoint using quarkus reactive that receives an input, does some validation, transforms the input to a list and then writes a message to kafka. My understanding was converting everything to Uni/Multi, would result in the execution happening on the I/O thread in async manner. In, the intelliJ logs, I could see that the code is getting executed in a sequential manner in the executor thread. The kafka write happens in its own network thread sequentially, which is increasing latency.
#POST
#Consumes(MediaType.APPLICATION_JSON)
#Produces(MediaType.APPLICATION_JSON)
public Multi<OutputSample> send(InputSample inputSample) {
ObjectMapper mapper = new ObjectMapper();
//deflateMessage() converts input to a list of inputSample
Multi<InputSample> keys = Multi.createFrom().item(inputSample)
.onItem().transformToMulti(array -> Multi.createFrom().iterable(deflateMessage.deflateMessage(array)))
.concatenate();
return keys.onItem().transformToUniAndMerge(payload -> {
try {
return producer.writeToKafka(payload, mapper);
} catch (JsonProcessingException e) {
e.printStackTrace();
}
return null;
});
}
#Inject
#Channel("write")
Emitter<String> emitter;
Uni<OutputSample> writeToKafka(InputSample kafkaPayload, ObjectMapper mapper) throws JsonProcessingException {
String inputSampleJson = mapper.writeValueAsString(kafkaPayload);
return Uni.createFrom().completionStage(emitter.send(inputSampleJson))
.onItem().transform(ignored -> new OutputSample("id", 200, "OK"))
.onFailure().recoverWithItem(new OutputSample("id", 500, "INTERNAL_SERVER_ERROR"));
}
I have been on it for a couple of days. Not sure if doing anything wrong. Any help would be appreciated.
Thanks
mutiny as any other reactive library is designed mainly around data flow control.
That being said, at its heart, it will offer a set of capabilities (generally through some operators) to control flow execution and scheduling. This means that unless you instruct munity objects to go asynchronous, they will simply execute in a sequential (old) fashion.
Execution scheduling is controlled using two operators:
runSubscriptionOn: which will cause the code snippet generating the items (which is generally referred to upstream) to execute on a thread from the specified Executor
emitOn: which will cause subscribing code (which is generally referred to downstream) to execute on a thread from the specified Executor
You can then update your code as follows causing the deflation to go asynchronous:
Multi<InputSample> keys = Multi.createFrom()
.item(inputSample)
.onItem()
.transformToMulti(array -> Multi.createFrom()
.iterable(deflateMessage.deflateMessage(array)))
.runSubscriptionOn(Infrastructure.getDefaultExecutor()) // items will be transformed on a separate thread
.concatenate();
EDIT: Downstream on a separate thread
In order to have the full downstream, transformation and writing to Kafka queue done on a separate thread, you can use the emitOn operator as follows:
#POST
#Consumes(MediaType.APPLICATION_JSON)
#Produces(MediaType.APPLICATION_JSON)
public Multi<OutputSample> send(InputSample inputSample) {
ObjectMapper mapper = new ObjectMapper();
return Uni.createFrom()
.item(inputSample)
.onItem()
.transformToMulti(array -> Multi.createFrom().iterable(deflateMessage.deflateMessage(array)))
.emitOn(Executors.newFixedThreadPool(5)) // items will be emitted on a separate thread after transformation
.onItem()
.transformToUniAndConcatenate(payload -> {
try {
return producer.writeToKafka(payload, mapper);
} catch (JsonProcessingException e) {
e.printStackTrace();
}
return Uni.createFrom().<OutputSample>nothing();
});
}
Multi is intended to be used when you have a source that emits items continuously until it emits a completion event, which is not your case.
From Mutiny docs:
A Multi represents a stream of data. A stream can emit 0, 1, n, or an
infinite number of items.
You will rarely create instances of Multi yourself but instead use a
reactive client that exposes a Mutiny API.
What you are looking for is a Uni<List<OutputSample>> because your API you return 1 and only 1 item with the complete result list.
So what you need is to send each message to Kafka without immediately waiting for their return but collecting the generated Unis and then collecting it to a single Uni.
#POST
public Uni<List<OutputSample>> send(InputSample inputSample) {
// This could be injected directly inside your producer
ObjectMapper mapper = new ObjectMapper();
// Send each item to Kafka and collect resulting Unis
List<Uni<OutputSample>> uniList = deflateMessage(inputSample).stream()
.map(input -> producer.writeToKafka(input, mapper))
.collect(Collectors.toList());
// Transform a list of Unis to a single Uni of a list
#SuppressWarnings("unchecked") // Mutiny API fault...
Uni<List<OutputSample>> result = Uni.combine().all().unis(uniList)
.combinedWith(list -> (List<OutputSample>) list);
return result;
}
I have a simple Flink wordcount, which reads from Kafka topic, and outputs its result to another Kafka topic,
DataStream<String> input = env.addSource(new FlinkKafkaConsumer<>(inputTopic, new SimpleStringSchema(), props));
DataStream<Tuple2<String, Long>> counts = ......;
counts.addSink(new FlinkKafkaProducer<>(outputTopic, new WordCountSerializer(), props));
//counts.print();
env.execute("foobar");
The problem is, I see nothing in the output topic via kafka-console-consumer.sh command line.
To address the issue, I try to print the result, and it works fine, I can see the correct workcount result in the log file.
So the guess is that something wrong in WordCountSerializer, which is like,
class WordCountSerializer implements KeyedSerializationSchema<Tuple2<String, Long>>, java.io.Serializable {
public byte[] serializeKey(Tuple2<String, Long> element) {
return new StringSerializer().serialize(null, element.getField(0));
}
public byte[] serializeValue(Tuple2<String, Long> element) {
return new LongSerializer().serialize(null, element.getField(1));
}
public String getTargetTopic(Tuple2<String, Long> element) {
return null;
}
}
After changing the serializeValue to
public byte[] serializeValue(Tuple2<String, Long> element) {
return new StringSerializer().serialize(null, element.getField(1).toString());
}
I can see the count been output to Kafka (the word part of Tuples is still missing), like
1
3
...
My questions are,
I've seen several examples on the internet using the WordCountSerializer mentioned aboved, but it doesn't work for me, am I doing anything wrong here?
After changing the serializeValue method as above, it partially works, but what I actually want is something below, what is the proper way to achieve this?
foo,1
bar,3
...
I'm using RxJava in and Android application with RxAndroid. I'm using mergeDelayError to combine two retro fit network calls into one observable which will process emitted items if either emits one and the error if either has one. This is not working and it is only firing off the onError action when either encounters an error. Now to test this I shifted to a very simple example and still the successAction is never called when I have an onError call. See example below.
Observable.mergeDelayError(
Observable.error(new RuntimeException()),
Observable.just("Hello")
)
.observeOn(AndroidSchedulers.mainThread())
.subscribeOn(Schedulers.io())
.finallyDo(completeAction)
.subscribe(successAction, errorAction);
The success action will only be called if I use two success observables. Am I missing something with how mergeDelayError is supposed to work?
EDIT:
I've found that if I remove the observeOn and subscribeOn everything works as expected. I need to specify threads and thought that was the whole point of using Rx. Any idea why specifying those Schedulers would break the behavior?
Use .observeOn(AndroidSchedulers.mainThread(), true) instead of .observeOn(AndroidSchedulers.mainThread()
public final Observable<T> observeOn(Scheduler scheduler, boolean delayError) {
return observeOn(scheduler, delayError, RxRingBuffer.SIZE);
}
Above is the signature of observeOn function. Following code works.
Observable.mergeDelayError(
Observable.error(new RuntimeException()),
Observable.just("Hello")
)
.observeOn(AndroidSchedulers.mainThread(), true)
.subscribeOn(Schedulers.io())
.subscribe(new Subscriber<String>() {
#Override
public void onCompleted() {
}
#Override
public void onError(Throwable e) {
}
#Override
public void onNext(String s) {
}
});
Got this trick from ConcatDelayError thread: https://github.com/ReactiveX/RxJava/issues/3908#issuecomment-217999009
This still seems like a bug in the mergeDelayError operator but I was able to get it working by duplicating the observerOn and Subscribe on for each observable.
Observable.mergeDelayError(
Observable.error(new RuntimeException())
.observeOn(AndroidSchedulers.mainThread())
.subscribeOn(Schedulers.io()),
Observable.just("Hello")
.observeOn(AndroidSchedulers.mainThread())
.subscribeOn(Schedulers.io())
)
.finallyDo(completeAction)
.subscribe(successAction, errorAction);
I think you don't wait for the terminal event and the main thread quits before the events are delivered to your observer. The following test passes for me with RxJava 1.0.14:
#Test
public void errorDelayed() {
TestSubscriber<Object> ts = TestSubscriber.create();
Observable.mergeDelayError(
Observable.error(new RuntimeException()),
Observable.just("Hello")
)
.subscribeOn(Schedulers.io()).subscribe(ts);
ts.awaitTerminalEvent();
ts.assertError(RuntimeException.class);
ts.assertValue("Hello");
}
i am using the following rxjava dependencies in android:
compile 'io.reactivex.rxjava2:rxjava:2.1.0'
compile 'io.reactivex.rxjava2:rxandroid:2.0.1'
and i am trying to create a method that will take in a observer as a paramter. I am having some issues i think its because this is rxjava2 so things are updated and im a little confused.
Let me show you what i want to accomplish:
private Subscription subscription = Scriptions.empty(); //this isn't working. how to set a empty observer IN RXJAVA2?
protected abstract Observable buildUseCaseObservable(); //RETROFIT WILL BUILD THE OBSERVABLE FOR ME SOMEWHERE ELSE
public void execute(Consumer UseCaseSubscriber){
this.subscription = this.buildUseCaseObservable()
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe(UseCaseSubscriber); //THIS LINE IS NOT WORKING , ERROR IS IN THE PHOTO
}
public void unsubscribe(){
if(!subscription.isUnsubscribed()){
subscription.unsubscribe();
}
}
Basically i am trying to create a method that will accept a observer/consumer as parameter and use that to update the UI after retrofit is done (being the observable).
UPDATE:
ok i changed it to disposables. now i'd like to store the disposable that i get back but its not working.
protected abstract Observable buildUseCaseObservable();
#SuppressWarnings("unchecked")
public void execute(Observer UseCaseSubscriber){
this.subscription = this.buildUseCaseObservable()
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe(UseCaseSubscriber);
}
public void unsubscribe(){
if(!subscription.isUnsubscribed()){
subscription.unsubscribe();
}
}
i am getting the following warning:
The reason i want to store the whole thing in a subscription (or whatever else you recommend) is so i can unsubscribe to it whenever i want.
but from the docs:
Because Reactive-Streams base interface, org.reactivestreams.Publisher defines the subscribe() method as void, Flowable.subscribe(Subscriber) no longer returns any Subscription (or Disposable). The other base reactive types also follow this signature with their respective subscriber types.
so how to save disposable so we can unsubscribe then ?
Subscription has been 'renamed' to Disposable with 2.x version. You can read the rxJava wiki explanation on this change here.
so how to save disposable so we can unsubscribe then ? Flowable.subscribe(Subscriber) doesn't return disposable, but Observable.subscribe(Subscriber) does. If you don't need back-pressure, just cast your Flowable to Observable with .toObservable().
I'm trying to understand RxJava and I'm sure this question is a nonsense... I have this code using RxJava:
public Observable<T> getData(int id) {
if (dataAlreadyLoaded()) {
return Observable.create(new Observable.OnSubscribe<T>(){
T data = getDataFromMemory(id);
subscriber.onNext(data);
});
}
return Observable.create(new Observable.OnSubscribe<T>(){
#Override
public void call(Subscriber<? super String> subscriber) {
T data = getDataFromRemoteService(id);
subscriber.onNext(data);
}
});
}
And, for instance, I could use it this way:
Action1<String> action = new Action<String>() {
#Override
public void call(String s) {
//Do something with s
}
};
getData(3).subscribe(action);
and this another with callback that implements Runnable:
public void getData(int id, MyClassRunnable callback) {
if (dataAlreadyLoaded()) {
T data = getDataFromMemory(id);
callback.setData(data);
callback.run();
} else {
T data = getDataFromRemoteService(id);
callback.setData(data);
callback.run();
}
}
And I would use it this way:
getData(3, new MyClassRunnable()); //Do something in run method
Which are the differences? Why is the first one better?
The question is not about the framework itself but the paradigm. I'm trying to understand the use cases of reactive.
I appreciate any help. Thanks.
First of all, your RxJava version is much more complex than it needs to be. Here's a much simpler version:
public Observable<T> getData(int id) {
return Observable.fromCallable(() ->
dataAlreadyLoaded() ? getDataFromMemory(id) : getDataFromRemoteService(id)
);
}
Regardless, the problem you present is so trivial that there is no discernible difference between the two solutions. It's like asking which one is better for assigning integer values - var = var + 1 or var++. In this particular case they are identical, but when using assignment there are many more possibilities (adding values other than one, subtracting, multiplying, dividing, taking into account other variables, etc).
So what is it you can do with reactive? I like the summary on reactivex's website:
Easily create event streams or data streams. For a single piece of data this isn't so important, but when you have a stream of data the paradigm makes a lot more sense.
Compose and transform streams with query-like operators. In your above example there are no operators and a single stream. Operators let you transform data in handy ways, and combining multiple callbacks is much harder than combining multiple Observables.
Subscribe to any observable stream to perform side effects. You're only listening to a single event. Reactive is well-suited for listening to multiple events. It's also great for things like error handling - you can create a long sequence of events, but any errors are forwarded to the eventual subscriber.
Let's look at a more concrete with an example that has more intrigue: validating an email and password. You've got two text fields and a button. You want the button to become enabled once there is a email (let's say .*#.*) and password (of at least 8 characters) entered.
I've got two Observables that represent whatever the user has currently entered into the text fields:
Observable<String> email = /* you figure this out */;
Observable<String> password = /* and this, too */;
For validating each input, I can map the input String to true or false.
Observable<Boolean> validEmail = email.map(str -> str.matches(".*#.*"));
Observable<Boolean> validPw = password.map(str -> str.length() >= 8);
Then I can combine them to determine if I should enable the button or not:
Observable.combineLatest(validEmail, validPw, (b1, b2) -> b1 && b2)
.subscribe(enableButton -> /* enable button based on bool */);
Now, every time the user types something new into either text field, the button's state gets updated. I've setup the logic so that the button just reacts to the state of the text fields.
This simple example doesn't show it all, but it shows how things get a lot more interesting after you get past a simple subscription. Obviously, you can do this without the reactive paradigm, but it's simpler with reactive operators.