I'm using Spring Webflux & reactor, Java 11, Spring boot 2.4.5, Spring 5.3.6 versions for this reactive application.
Use case:
I need to make a call to API and get data from it. From this data I take uniqueId and then call bunch of API's to get other data, and then finally combine all this data to new Object and return.
Example code:
Mono<Response> 1stAPIResponse = 1stAPI.callMethod(eventId); // response object has ProductId, and other details.
Mono<List<String>> productIds = 1stAPIResponse.map(Response::ProductId).collect(Collectors.toList());
Mono<2ndAPIResponse> 2ndAPIResponse = productIds.flatMap(ids -> 2ndAPI.callMethod(ids));
Mono<3rdAPIResponse> 3rdAPIResponse = productIds.flatMap(ids -> 3rdAPI.callMethod(ids));
...
1stAPIResponse.foreach(response -> {
FinalResponse.builder()
.productId(response.productId)
.val1(2ndAPIResponse.get(response.productId))
.val3(3ndAPIResponse.get(response.productId))
. ...
.build()});
Here the problem is, when ids are passed to 2ndAPI, 3rdAPI,... method, it makes call to 1stAPI and get the data each time. And finally when creating object it makes another call to 1st API. In this example it makes total of 3 calls.
How can I avoid similar multiple calls from occurring?
One way to avoid this is, I can make 1stAPI call blocking but is it correct? doesn't it defeat non-blocking style of coding?
Ex: Response 1stAPIResponse = 1stAPI.callMethod(eventId).toFuture().get();
How can I write a correct reactive program (without blocking) but still make only one call to 1stAPI?
Let me know for any questions.
So, you need to refactor your code in more reactive style and use zip operator for parallel calls:
1stAPI.callMethod(eventId)
.flatmap(response -> // collect your id to list (ids);
return 2ndAPI.callMethod(ids).zipWith(3ndAPI.callMethod(ids))
.flatmap(tuple2 -> FinalResponse.builder() // tuple contains result of 2ndAPI and 3ndAPI
.productId(response.productId)
.val1(2ndAPIResponse.get(response.productId))
.val3(3ndAPIResponse.get(response.productId)))
...
)
Related
I'm new to reactive programming and I try to implement a very basic scenario.
I want to send a message to kafka each time a file is dropped to a specific folder.
I think that I don't understand well the basics things... so please could you help me?
So I have a few questions :
What is the difference between smallrye-reactive-messaging and smallrye-reactive-streams-operators ?
I have this simple code :
#Outgoing( "my-topic" )
public PublisherBuilder<Message<MessageWrapper>> generate() {
if(Objects.isNull(currentMessage)){
//currentMessage is an instance variable which is null when I start the application
return ReactiveStreams.of(new MessageWrapper()).map(Message::of);
}
else {
//currentMessage has been correctly set with the file information
LOGGER.info(currentMessage);
return ReactiveStreams.of(currentMessage).map(Message::of);
}
}
When the code goes in the if statement, everything is ok and I got a JSON serialization of my object will null values. However I don't understand why when my code goes to the else statement, nothing goes to the topic? It seems that the .of instructions of the if statement has broke the streams or something like that...
How to keep a continuous streams that 'react' to the new dropped files ? (or other events like HTTP GET request or something like that) ...
If I don't return an instance of PublisherBuilder but an Integer for example, then my kafka topic will be populated by a very huge stream of Integer value. This is why examples are using some intervals when sending messages...
Should I use some CompletationStage or CompletableFuture ? RxJAva2? It's a bit confusing which lib to use (vertx, smallrye, rxjava2, microprofile, ...)
What are the differences between :
ReactiveStreams.fromCompletionStage
ReactiveStreams.fromProcessor
ReactiveStreams.fromPublisher
ReactiveStreams.fromSubscriber
Which one to use on which scenario ?
Thank you very much !
Let's start with the difference between smallrye-reactive-messaging & smallrye-reactive-streams-operators: smallrye-reactive-streams-operators is the same as smallrye-reactive-messaging but in addition it has a support to MicroProfile-context-propagation. Since most reactive-messaging providers use Vert.x behind the scene, it will process your message in an event-loop style, which means it will run in separate thread. Sometimes you need to propagate some ctx from your base thread into the new thread (ex: populating CDI and Tx context to execute some JPA Entity manager logic). Here where ctx propagation help.
For method signatures. You can take a look at the official documentation of SmallRye-reactive-streams sections 3,4 & 5. Each one has a different use case. It is up to you which flavor do you want to use.
When to use what ? If you are not running within reactive context, you can use the below to send messages.
#Inject
#Channel("my-channel")
Emitter emitter;
For Message consumption you can use method signature like this :
#Incoming("channel-2")
public CompletionStage doSomething(Message anEvent)
Or
#Incoming("channel-2")
public void doSomething(String anEvent)
Hope that helps.
I am going to register interests in ALL_KEYS for my Pivotal GemFire client via Spring Data GemFire, but I find that ClientRegionFactoryBean has one method.
org.springframework.data.gemfire.client.ClientRegionFactoryBean.setInterests(Interest<MyRegionPojo>[] interests)
In this case, I only can set the exact keys, but I want to register interests for all keys. My key is not a simple class like String, or Long, but a complex object MyRegionPojo.
Please help if any method to implement so like GemFire API region.registerInterest("ALL_KEYS");
You problem statement is a bit vague but I assume/suspect you are configuring your Spring (Data GemFire) (SDG) application using Spring JavaConfig?
However, I will quickly add that this is not unlike how you would register interests in all keys using SDG's XML namespace, as shown here.
The JavaConfig approach is similar, but clearly based on "strongly-typed arguments", namely 1 or more sub-type instances of the o.s.d.g.client.Interest class to the o.s.d.g.client.ClientRegionFactoryBean.setInterests(:Interest<K>[]) method.
By way of example, you might do the following...
#Bean("Example")
public ClientRegionFactoryBean<?, ?> exampleRegion(GemFireCache gemfireCache) {
ClientRegionFactoryBean<MyRegionKey, MyRegionValue> exampleRegion =
new ClientRegionFactoryBean<>();
RegexInterest regexInterest = new RegexInterest();
regexInterest.setKey(".*");
exampleRegion.setCache(gemfireCache);
exampleRegion.setShortcut(ClientRegionShortcut.PROXY);
exampleRegion.setInterests(new Interest[] { regexInterest });
exampleRegion.setKeyConstraint(MyRegionKey.class);
exampleRegion.setValueConstraint(MyRegionValue.class);
return exampleRegion;
}
NOTE: updated the example above to reflect the proper way to register (Regex) interests based on SDG 1.9 or earlier. Keep in mind that the `o.s.d.g.client.RegexInterest.getRegex() delegates to getKey() therefore you can set the Regular Expression using setKey(:String) as I have shown above.
Notice the o.s.d.g.client.RegexInterest sub-type registration, which is effectively the same as register interests in "ALL_KEYS", as described here as well.
Hope this helps!
-John
I want to use the deltatrigger in apache flink (flink 1.3) but I have some trouble with this code :
.trigger(DeltaTrigger.of(100, new DeltaFunction[uniqStruct] {
override def getDelta(oldFp: uniqStruct, newFp: uniqStruct): Double = newFp.time - oldFp.time
}, TypeInformation[uniqStruct]))
And I have this error:
error: object org.apache.flink.api.common.typeinfo.TypeInformation is not a value [ERROR] }, TypeInformation[uniqStruct]))
I don't understand why DeltaTrigger need TypeSerializer[T]
and I don't know what to do to remove this error.
Thanks a lot everyone.
I would read into this a bit https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/types_serialization.html sounds like you can create a serializer using typeInfo.createSerializer(config) on your type info. Note what you're passing in currently is a type itself and NOT the type info which is why you're getting the error you are.
You would need to do something more like
val uniqStructTypeInfo: TypeInformation[uniqStruct] = createTypeInformation[uniqStruct]
val uniqStrictTypeSerializer = typeInfo.createSerializer(config)
To quote the page above regarding the config param you need to pass to create serializer
The config parameter is of type ExecutionConfig and holds the
information about the program’s registered custom serializers. Where
ever possibly, try to pass the programs proper ExecutionConfig. You
can usually obtain it from DataStream or DataSet via calling
getExecutionConfig(). Inside functions (like MapFunction), you can get
it by making the function a Rich Function and calling
getRuntimeContext().getExecutionConfig().
DeltaTrigger needs a TypeSerializer because it uses Flink's managed state mechanism to store each element for later comparison with the next one (it just keeps one element, the last one, which is updated as new elements arrive).
You will find an example (in Java) here.
But if all you need is a window that triggers every 100msec, then it'll be easier to just use a TimeWindow, such as
input
.keyBy(<key selector>)
.timeWindow(Time.milliseconds(100)))
.apply(<window function>)
Updated:
To have hour-long windows that trigger every 100msec, you could use sliding windows. However, you would have 10 * 60 * 60 windows, and every event would be placed into each of these 36000 windows. So that's not a great idea.
If you use a GlobalWindow with a DeltaTrigger, then the window will be triggered only when events are more than 100msec apart, which isn't what you've said you want.
I suggest you look at ProcessFunction. It should be straightforward to get what you want that way.
I have a class with a list of users from a server.
Other classes can manipulate the list on the server e.g. call add or delete operation.
My Core-Class has a reference to the other classes which are manipulating the list on the server.
I would like to:
Make a init call in Core-Class to get the list on the beginning
The Core-Class will be notified by plugins each time the list was manipulated on the server, so the Core-Class get the list from the server again.
Notify other classes that the list was reloaded and forward the new list.
My structure
Core {
users: [];
plugin1: Plugin;
plugin2: Plugin;
//Get a new list of users from the server
loadUsers() {
userService.loadUsers.then(function (res) {
this.users = res;
})
}
}
Plugin {
//sends a request to the server to create a special user,
//depending on plugin implementation
createUser();
}
I'm only just starting of using rx. I understand the factory methods, hot vs cold observable and other basic stuff. But i can not imagine how to do it with rx in the right way.
Thanks.
My idea of reactive implementation would be:
In your UserService add the following:
var loadSubject = new Subject();
var usersObservable = loadSubject.flatMap(
Observable.fromPromise(<your http call that returns promise>)).share()
function loadUsers(){
loadSubject.next(true);
}
Then each plugin that made changes will simply call:
userService.loadUsers();
Your Core, plugins, and other classes that would like to get updated will simply do:
loadService.usersObservable.subscribe(function(usersFreshValue){
this.users = usersFreshValue;
})
Note that I used the '.share()' to avoid duplication of the data.
Possible improvements:
Instead of the loadUsers method, have each plugin call userService.loadSubject.next(true);
Use backpressure/buffering operators to avoid too frequent calls. For example debounce (CAUTION! when used incorrectly, might lead to starvation) or bufferWithTime + filter (to ensure you don't trigger on empty buffers)
Use behavioral/replay observables so that new subscribers will quickly get the latest value.
I am looking to open up multiple connections using a netty client bootstrap in order to parse messages coming from multiple sources. The messages all have the same format, however, due to the amount of data that needs to be processed, I must run each connection on separate threads (This is assuming netty creates a thread per client channel, which I couldn't find a reference for - if that's not the case, how would this be achieved?).
This is the code that I use to connect to the data server:
var b = new Bootstrap()
.group(group)
.channel(classOf[NioSocketChannel])
.handler(RawFeedChannelInitializer)
var ch1 = b.clone().connect(host, port).sync().channel();
var ch2 = b.clone().connect(host, port).sync().channel();
The initializer calls RawPacketDecoder, which extends ReplayingDecoder, and is defined here.
The code works well without #Sharable when opening a single connection, but for the purpose of my application I must connect to the same server multiple times.
This results in the runtime error #Sharable annotation is not allowed pointing to my RawPacketDecoder class.
I am not entirely sure on how to get past this issue, short of reimplementing in scala an instantiable class of ReplayingDecoder as my decoder based directly on ByteToMessageDecoder.
Any help would be greatly appreciated.
Note: I am using netty 4.0.32 Final
I found the solution in this StockExchange answer.
My issue was that I was using an object based ChannelInitializer (singleton), and ReplayingDecoder as well as ByteToMessageDecoder are not sharable.
My initializer was created as a scala object, and therefore a single instance allowed. Changing the initializer to a scala class and instantiating for each bootstrap clone solved the problem. I modified the bootstrap code above as follows:
var b = new Bootstrap()
.group(group)
.channel(classOf[NioSocketChannel])
//.handler(RawFeedChannelInitializer)
var ch1 = b.clone().handler(new RawFeedChannelInitializer()).connect(host, port).sync().channel();
var ch2 = b.clone().handler(new RawFeedChannelInitializer()).connect(host, port).sync().channel();
I am not sure whether this ensures multithreading as wanted but it does allow to split the data access into multiple connections to the feed server.
Edit Update: After performing additional research on the subject, I have determined that netty does in fact create a thread per channel; this was verified by printing to console after the creation of each channel:
println("No. of active threads: " + Thread.activeCount());
The output shows an incremental number as channels are created and associated with their respective threads.
By default NioEventLoopGroup uses 2*Num_CPU_cores threads as defined here:
DEFAULT_EVENT_LOOP_THREADS = Math.max(1, SystemPropertyUtil.getInt(
"io.netty.eventLoopThreads",
Runtime.getRuntime().availableProcessors() * 2));
This value can be overriden to something else by setting
val group = new NioEventLoopGroup(16)
and then using the group to create/setup the bootstrap.