I'm trying to build an algorithm to explore a tree with RxJava.
My reason to use RxJava is that my node processing function is already using RxJava to send RPCs to other services. Note that the order in which the different branches of the tree are explored is not important.
Ideally, I would like to have something like a QueueFlowable<Node> extends Flowable<Node> which would expose a push function that Observers could use to add new nodes in the queue after processing them.
private static main(String[] args) {
QueueFlowable<Node> nodes = new QueueFlowable<>();
nodes.push(/* the root */);
nodes.concatMap(node -> process(node)).map(nodes::push));
nodes.blockingSubscribe();
}
// Processes the node and returns a Flowable of other nodes to process.
private static Flowable<Node> process(Node node) { ... }
This sounds like something relatively common (as I would expect web crawlers to implement something similar) but I still don't manage to make it work.
My latest attempt was to use a PublishProcessor as follows:
PublishProcessor<Mode> nodes = PublishProcessor.create();
nodes.concatMap(node -> process(node)).subscribe(nodes);
nodes.onNext(/* the root node */);
nodes.blockingSubscribe();
Of course, this never terminates as the processor does not complete.
Any help would be much appreciated!
Related
I'm trying to write a Vert.x Web handler that could be used to hide any processing latencies from an API to prevent figuring out existence of accounts as well as other information from an API. I would like to be able to just write something like:
router
.post("/uri")
.handler(new LatencyNormalizer())
.handler(new UriHandler());
In other words, make it as easy to use as possible and as easy to integrate into existing code bases as possible. Looking at the docs for Router and RoutingContext, I see only the following method as a possible candidate for implementing this:
https://vertx.io/docs/apidocs/io/vertx/ext/web/RoutingContext.html#addHeadersEndHandler-io.vertx.core.Handler-
I could then write code like:
public void handle(RoutingContext ctx) {
long start = System.nanoTime();
ctx.addHeadersEndHandler(v -> {
public void handle(RoutingContext ctx) {
long end = System.nanoTime();
Thread.sleep(...);
});
ctx.next();
}
Of course, this doesn't work, since sleep here blocks the thread. It looks like the handlers in the addHeaderEndHandlers list maintained internally by the RoutingContext are called synchronously, so there is no way to use e.g. vertx.SetTimer() inside the addHeaderEndHandler.
In other words, does Vert.x offer any interface that allows creating a handler which is called asynchronously before writing out to the wire (and with nothing written until the async call finishes)? This is for example how Netty works under the hood, which Vert.x leverages. I know I could implement this LatencyNormalizer as a base class for my other handlers, but it would not be as easy to integrate in existing code in that case.
For example, my tree is:
class TreeNode {
List<TreeNode> children;
}
I'm looking for/hoping to have something in producer like:
#ProducerModule
class RecursiveModule {
#Produces
ListenableFuture<TreeNode> produceNode(/*...?*/) {
// Somehow recursively get a node.
}
}
So that it can dynamically parse some external source, and construct a node, recursively, for me.
A more concrete little example use case may be to build a HN news reader. In their API there is an Item that may have multiple children Items. So to read a news item with all comments, one needs to fetch the root Item and recursively fetch its children.
*I'm new to dagger producers, and I'm trying to learn what it can do. I'm not sure if this recursiveness breaks the "acyclic" in dagger's name, but I'm curious to see whether this is possible.
I (kinda) have a workaround:
Let the component have the production executor passed in as a parameter.
#ProductionComponent(/* ... */)
class TreeComponent {
// ...
#ProductionComponent.Builder
interface Builder {
#BindsInstance
Builder executor(#Production Executor executor);
#BindsInstance
Builder id(#NodeId int id); // Say we need an id to know which node to process, or how to process a node.
// ...
}
}
Then for each node, process the node itself, and create one producer per child.
In this way I'm managing the producer tree myself. But the producer library still takes care of the async coordination. I think there is still benefit to it.
Whats a clean way to make a Observable/Flowable for a global api like say
public void log(String s)
All components use this log method that currently has listeners and the various listeners log to various kinds of logging. I want to make it an observable so that multiple subscribers can can subscribe to this observable and log to various files/db.
I am hoping that I won't have to rework the legacy classes (do have access to code of the library).
Currently I add the string to a queue like data structure which is my observable and use subscribers to achieve what I want.
A more generic question would be , how do you create an observable for an infinite source (for me the log method is the source) and what data structure would be recommended for hold the data of this infinite source that can become a good observable.
Any pointers/recommendations/snippets will be greatly appreciated.
[edit]
As suggested in the response comment, using PublishSubject solved my problem.
When I need to subscribe I access the subject object and call the subscribe on it.
static PublishSubject<LogEvent> subject = PublishSubject.create();
public static void log (LogEvent evt) {
subject.onNext(evt);
}
I implemented in Java what I called a "foldable queue", i.e., a LinkedBlockingQueue used by an ExecutorService. The idea is that each task as a unique id that if is in the queue while another task is submitted via that same id, it is not added to the queue. The Java code looks like this:
public final class FoldablePricingQueue extends LinkedBlockingQueue<Runnable> {
#Override
public boolean offer(final Runnable runnable) {
if (contains(runnable)) {
return true; // rejected, but true not to throw an exception
} else {
return super.offer(runnable);
}
}
}
Threads have to be pre-started but this is a minor detail. I have an Abstract class that implements Runnable that takes a unique id... this is the one passed in
I would like to implement the same logic using Scala and Akka (Actors).
I would need to have access to the mailbox, and I think I would need to override the ! method and check the mailbox for the event.. has anyone done this before?
This is exactly how the Akka mailbox works. The Akka mailbox can only exist once in the task-queue.
Look at:
https://github.com/jboner/akka/blob/master/akka-actor/src/main/scala/akka/dispatch/Dispatcher.scala#L143
https://github.com/jboner/akka/blob/master/akka-actor/src/main/scala/akka/dispatch/Dispatcher.scala#L198
Very cheaply implemented using an atomic boolean, so no need to traverse the queue.
Also, by the way, your Queue in Java is broken since it doesn't override put, add or offer(E, long, TimeUnit).
Maybe you could do that with two actors. A facade one and a worker one. Clients send jobs to facade. Facade forwards then to worker, and remember them in its internal state, a Set queuedJobs. When it receives a job that is queued, it just discard it. Each time the worker starts processing a job (or completes it, whichever suits you), it sends a StartingOn(job) message to facade, which removes it from queuedJobs.
The proposed design doesn't make sense. The closest thing to a Runnable would be an Actor. Sure, you can keep them in a list, and not add them if they are already there. Such lists are kept by routing actors, which can be created from ready parts provided by Akka, or from a basic actor using the forward method.
You can't look into another actor's mailbox, and overriding ! makes no sense. What you do is you send all your messages to a routing actor, and that routing actor forwards them to a proper destination.
Naturally, since it receives these messages, it can do any logic at that point.
if I have the following class, will I run into a problem if 100 people are requesting the page at the same time? If there is only one copy of UpdateUser, will all the requests have to queue up and wait for their turns? Thank you.
public static UserManager
{
public static void UpdateUser(int UserID)
{
// this process takes up 2 seconds
UserDataAccessor DA = new UserDataAccessor();
DA.Update();
}
}
It depends on what all your other code is doing, but that particular code will not cause any problems.
In general, if you are writing a web application using Java servlets, you need to design your classes to allow for multiple threads. The application server you deploy your code into will call this one class many times, possibly simultaneously if there are many users at once. This is quite common.
The code you posted here looks fine. There are field variables being shared between threads. Each thread of execution will invoke your method and create a DA variable that is private to the thread.