Latest overflow strategy with size 1 or any alternatives - reactive-programming

I have a gui events producer. I want to take the latest emission from it and have it processed on a different thread. While processing the emission I need gui producer emissions to be dropped. After emission is processed I want to take latest emission from gui producer.
I was trying to use onBackpressureLatest() overflow strategy but queue size is my problem. With the default 256 queue size I get the expected behaviour but must process 255 emissions which are useless for me. Decreasing queue size to 16 leaves me with 15 useless emissions. I imagine I would achieve expected behaviour with a queue size=1 but 16 is the min value.
I attach code described in the previous paragraph.
import reactor.core.publisher.Flux;
import reactor.core.scheduler.Schedulers;
import java.util.concurrent.TimeUnit;
public class App {
public static void main(String[] args) throws InterruptedException {
System.setProperty("reactor.bufferSize.small", "16");
guiEventsProducer()
.onBackpressureLatest()
.log()
.publishOn(Schedulers.single())
.subscribe(next -> {
System.out.println("Processing " + next);
try {
TimeUnit.MILLISECONDS.sleep(1);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("Processing " + next + " done");
});
TimeUnit.SECONDS.sleep(60);
}
private static Flux<Integer> guiEventsProducer() {
return Flux.range(0, 10000);
}
}

import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
import reactor.core.scheduler.Schedulers;
import java.util.concurrent.TimeUnit;
public class App {
public static void main(String[] args) throws InterruptedException {
Flux.range(1, 10000)
.log()
.onBackpressureLatest()
.log()
.flatMap(next -> Mono.just(next).subscribeOn(Schedulers.single()), 1, 1)
.subscribe(integer -> {
System.out.println("[" + Thread.currentThread().getName() + "] result=" + integer);
try {
TimeUnit.MILLISECONDS.sleep(10);
} catch (InterruptedException e) {
e.printStackTrace();
}
});
TimeUnit.SECONDS.sleep(60);
}
}

Related

Mono subscribe is not executing

I have started learning reactive programming using spring boot and when I tried execute the below piece of code from route function, the mono subscribe part of logic is not executing.
#SpringBootApplication
public class MonotopayloadApplication {
public static void main(String[] args) {
SpringApplication.run(MonotopayloadApplication.class, args);
}
#Bean
RouterFunction<ServerResponse> videoEndPoint() {
return route()
.path("/test", builder -> builder
.POST("", this::handle)
).build();
}
private Mono<ServerResponse> handle(ServerRequest serverRequest) {
serverRequest.bodyToMono(String.class)
.subscribe(result -> {
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("subscribe thread : " + Thread.currentThread().getName());
});
System.out.println("Thread : " + Thread.currentThread().getName() + " is leaving");
return Mono.empty();
}
}
I do not want to call the block/blockFirst/blockLast(). I understand the main thread is leaving the subscribe thread. Kindly help.

RxJava adjust backpressure avoiding observeOn buffer

In the code below I would like the subscriber to control when the Flowable emits an event by holding a reference to the Subscription inside subscribe() and requesting the number of elements I want to be produced.
What I am experiencing is that observeOn()'s buffer with size 2 is hiding my call to subscription.request(3) as the producer is producing 2 elements at a time instead of 3.
public class FlowableExamples {
public static void main(String[] args) throws InterruptedException {
long start = new Date().getTime();
Flowable<Integer> flowable = Flowable
.generate(() -> 0, (Integer state, Emitter<Integer> emitter) -> {
int newValue = state + 1;
log("Producing: " + newValue);
emitter.onNext(newValue);
return newValue;
})
.take(30);
flowable
.subscribeOn(Schedulers.io())
.observeOn(Schedulers.computation(), false, 2)
.subscribe(new Subscriber<Integer>() {
Subscription subscription;
#Override
public void onSubscribe(Subscription subscription) {
this.subscription = subscription;
subscription.request(5);
}
#Override
public void onNext(Integer integer) {
log("\t\treceived: " + integer);
if (integer >= 5) {
sleep(500);
log("Requesting 3 should produce 3, but actually produced 2");
subscription.request(3);
sleep(1000);
}
}
#Override
public void onError(Throwable throwable) {}
#Override
public void onComplete() {
log("Subscription Completed!!!!!!!!");
}
});
sleep(40_000);
System.out.println("Exit main after: " + (new Date().getTime() - start) + " ms");
}
private static void log(String msg) {
System.out.println(Thread.currentThread().getName() + ": " + msg);
}
private static void sleep(long ms) {
try {
Thread.sleep(ms);
} catch (InterruptedException e) {}
}
}
How could I accomplish this?

can Flink receive http requests as datasource?

Flink can read a socket stream, can it read http requests? how?
// socket example
DataStream<XXX> socketStream = env
.socketTextStream("localhost", 9999)
.map(...);
There's an open JIRA ticket for creating an HTTP sink connector for Flink, but I've seen no discussion about creating a source connector.
Moreover, it's not clear this is a good idea. Flink's approach to fault tolerance requires sources that can be rewound and replayed, so it works best with input sources that behave like message queues. I would suggest buffering the incoming http requests in a distributed log.
For an example, look at how DriveTribe uses Flink to power their website on the data Artisans blog and on YouTube.
I write one custom http source. please ref OneHourHttpTextStreamFunction. you need create a fat jar to include apache httpserver classes if you want run my code.
package org.apache.flink.streaming.examples.http;
import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.common.functions.ReduceFunction;
import org.apache.flink.api.java.utils.ParameterTool;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.source.SourceFunction;
import org.apache.flink.streaming.api.windowing.time.Time;
import org.apache.flink.streaming.examples.socket.SocketWindowWordCount.WordWithCount;
import org.apache.flink.util.Collector;
import org.apache.http.HttpException;
import org.apache.http.HttpRequest;
import org.apache.http.HttpResponse;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.bootstrap.HttpServer;
import org.apache.http.impl.bootstrap.ServerBootstrap;
import org.apache.http.protocol.HttpContext;
import org.apache.http.protocol.HttpRequestHandler;
import java.io.IOException;
import java.util.concurrent.TimeUnit;
import static org.apache.flink.util.Preconditions.checkArgument;
import static org.apache.flink.util.Preconditions.checkNotNull;
public class HttpRequestCount {
public static void main(String[] args) throws Exception {
// the host and the port to connect to
final String path;
final int port;
try {
final ParameterTool params = ParameterTool.fromArgs(args);
path = params.has("path") ? params.get("path") : "*";
port = params.getInt("port");
} catch (Exception e) {
System.err.println("No port specified. Please run 'SocketWindowWordCount "
+ "--path <hostname> --port <port>', where path (* by default) "
+ "and port is the address of the text server");
System.err.println("To start a simple text server, run 'netcat -l <port>' and "
+ "type the input text into the command line");
return;
}
// get the execution environment
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
// get input data by connecting to the socket
DataStream<String> text = env.addSource(new OneHourHttpTextStreamFunction(path, port));
// parse the data, group it, window it, and aggregate the counts
DataStream<WordWithCount> windowCounts = text
.flatMap(new FlatMapFunction<String, WordWithCount>() {
#Override
public void flatMap(String value, Collector<WordWithCount> out) {
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
for (String word : value.split("\\s")) {
out.collect(new WordWithCount(word, 1L));
}
}
})
.keyBy("word").timeWindow(Time.seconds(5))
.reduce(new ReduceFunction<WordWithCount>() {
#Override
public WordWithCount reduce(WordWithCount a, WordWithCount b) {
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return new WordWithCount(a.word, a.count + b.count);
}
});
// print the results with a single thread, rather than in parallel
windowCounts.print().setParallelism(1);
env.execute("Http Request Count");
}
}
class OneHourHttpTextStreamFunction implements SourceFunction<String> {
private static final long serialVersionUID = 1L;
private final String path;
private final int port;
private transient HttpServer server;
public OneHourHttpTextStreamFunction(String path, int port) {
checkArgument(port > 0 && port < 65536, "port is out of range");
this.path = checkNotNull(path, "path must not be null");
this.port = port;
}
#Override
public void run(SourceContext<String> ctx) throws Exception {
server = ServerBootstrap.bootstrap().setListenerPort(port).registerHandler(path, new HttpRequestHandler(){
#Override
public void handle(HttpRequest req, HttpResponse rep, HttpContext context) throws HttpException, IOException {
ctx.collect(req.getRequestLine().getUri());
rep.setStatusCode(200);
rep.setEntity(new StringEntity("OK"));
}
}).create();
server.start();
server.awaitTermination(1, TimeUnit.HOURS);
}
#Override
public void cancel() {
server.stop();
}
}
Leave you comment, if you want the demo jar.

How to detect every mouse event in system using java

How to detect every mouse event in system using java?
I have tried using Point class to catch mouse motion but that thing was not so convenient.
import org.jnativehook.GlobalScreen;
import org.jnativehook.NativeHookException;
import org.jnativehook.keyboard.NativeKeyEvent;
import org.jnativehook.keyboard.NativeKeyListener;
class GlobalKeyListenerExample implements NativeKeyListener {
public void nativeKeyPressed(NativeKeyEvent e) {
System.out.println("Key Pressed: " + NativeKeyEvent.getKeyText(e.getKeyCode()));
if (e.getKeyCode() == NativeKeyEvent.VK_ESCAPE) {
}
}
public void nativeKeyReleased(NativeKeyEvent e) {
System.out.println("Key Released: " + NativeKeyEvent.getKeyText(e.getKeyCode()));
}
public void nativeKeyTyped(NativeKeyEvent e) {
System.out.println("Key Typed: " + e.getKeyText(e.getKeyCode()));
}
public GlobalKeyListenerExample()
{
try {
GlobalScreen.registerNativeHook();
}
catch (NativeHookException ex) {
System.err.println("There was a problem registering the native hook.");
System.err.println(ex.getMessage());
System.exit(1);
}
GlobalScreen.getInstance().addNativeKeyListener(this);
}
public static void main(String[] args) {
try {
GlobalScreen.registerNativeHook();
}
catch (NativeHookException ex) {
System.err.println("There was a problem registering the native hook.");
System.err.println(ex.getMessage());
System.exit(1);
}
//Construct the example object and initialze native hook.
GlobalScreen.getInstance().addNativeKeyListener(new GlobalKeyListenerExample());
}
}

How to call getPage from HtmlUnit WebClient and have setTimeout not wait forever?

I have the same problem as described in the question Call getPage from htmlunit WebClient with JavaScript disabled and setTimeout set to 10000 waits forever.
There is only one relevant (complicated) possible answer there (by theytoo). So I was wondering if:
Does someone have a simpler answer?
Can someone verify the solution works?
Code I used:
package main;
import java.io.IOException;
import java.net.MalformedURLException;
import com.gargoylesoftware.htmlunit.FailingHttpStatusCodeException;
import com.gargoylesoftware.htmlunit.WebClient;
public class Test {
public static void main(final String[] args) {
final WebClient webClient = new WebClient();
webClient.setTimeout(1000);
try {
System.out.println("Querying");
webClient.getPage("http://www.google.com");
System.out.println("Success");
} catch (final FailingHttpStatusCodeException e) {
System.out.println("One");
e.printStackTrace();
} catch (final MalformedURLException e) {
System.out.println("Two");
e.printStackTrace();
} catch (final IOException e) {
System.out.println("Three");
e.printStackTrace();
} catch (final Exception e) {
System.out.println("Four");
e.printStackTrace();
}
System.out.println("Finished");
}
}
Output (removed all CSS and JS warnings):
Querying
Success
Finished
After changing timeout from 1000 to 1 (I won't hit google in less than 1 ms):
Querying
Three
org.apache.http.conn.ConnectTimeoutException: Connect to www.google.com:80 timed out
at com.gargoylesoftware.htmlunit.SocksSocketFactory.connectSocket(SocksSocketFactory.java:92)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148)
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:149)
at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:121)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:573)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:425)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:776)
at com.gargoylesoftware.htmlunit.HttpWebConnection.getResponse(HttpWebConnection.java:152)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseFromWebConnection(WebClient.java:1439)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponse(WebClient.java:1358)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:307)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:373)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:358)
at main.Test.main(Test.java:17)
Finished
Conclusion: I can't reproduce it and it works as expected