I'm trying to create several libraries and to call them. My objective is to call a method of the second one into the first one.
For example, I created two libraries, library1 and library2.
Library1 code
class LibraryScriptTest1 {
def log
def context
def testRunner
def library2
def LibraryScriptTest1(log, context, testRunner) {
this.log = log
this.context = context
this.testRunner = testRunner
library = testRunner.testCase.testSuite.project.testSuites["LibraryTestSuiteTest"]
module = library.testCases["LibraryTestCaseTest2"].testSteps["LibraryScriptTest2"]
module.run(testRunner, context)
this.library2 = context.library2
}
int add(int firstNumber, int secondNumber) {
return firstNumber + secondNumber
}
int multiply(int firstNumber, int secondNumber) {
return firstNumber * secondNumber
}
int divide(firstNumber, secondNumber) {
return this.library2.divide(firstNumber, secondNumber)
}
}
context.setProperty("library1", new LibraryScriptTest1(log, context, testRunner))
Library2 code:
class LibraryScriptTest2 {
def log
def context
def testRunner
def LibraryScriptTest2(log, context, testRunner) {
this.log = log
this.context = context
this.testRunner = testRunner
}
int substract(int firstNumber, int secondNumber) {
return firstNumber - secondNumber
}
int divide(int firstNumber, int secondNumber) {
return firstNumber / secondNumber
}
}
context.setProperty("library2", new LibraryScriptTest2(log, context, testRunner))
Error
However I'm facing to this error in library1:
groovy.lang.MissingPropertyException: No such property: library for
class: LibraryScriptTest1 Possible solutions: library2 error at line:
XX
The corresponding line is the next one:
context.setProperty("library1", new LibraryScriptTest1(log, context, testRunner))
Anyone can help me to understand and fix that please? Thank you.
Related
I want to convert
x.foo(a, b);
into
x.foo(a).bar(b);
I can easily match for acme.X foo(acme.A, acme.B), but how do I build a JavaTemplate that can do that replacement for me?
When I run
#Override
protected TreeVisitor<?, ExecutionContext> getVisitor() {
return new JavaIsoVisitor<>() {
private final JavaTemplate template = JavaTemplate.builder(this::getCursor,
"foo(#{any(java.lang.String)}).bar(#{any(java.lang.String)})")
.build();
#Override
public J.MethodInvocation visitMethodInvocation(J.MethodInvocation method, ExecutionContext executionContext) {
J.MethodInvocation m = super.visitMethodInvocation(method, executionContext);
if (....matches(method)) {
List<Expression> arguments = m.getArguments();
m = m.withTemplate(template, m.getCoordinates().replace(), arguments.get(0), arguments.get(1));
}
return m;
}
};
}
I get
foo(a).bar(b);
instead of
x.foo(a).bar(b);
This worked for me (credit to Patrick Way on slack):
private static final MethodMatcher MATCHER =
new MethodMatcher("org.optaplanner.core.api.score.stream.ConstraintStream " +
"penalize(java.lang.String, org.optaplanner.core.api.score.Score)");
#Override
protected TreeVisitor<?, ExecutionContext> getVisitor() {
return new JavaIsoVisitor<>() {
private final JavaTemplate template = JavaTemplate.builder(() -> getCursor().getParentOrThrow(),
"#{any(org.optaplanner.core.api.score.stream.ConstraintStream)}" +
".penalize(#{any(org.optaplanner.core.api.score.Score)})" +
".asConstraint(#{any(java.lang.String)})"
).build();
#Override
public Expression visitExpression(Expression expression, ExecutionContext executionContext) {
Expression e = super.visitExpression(expression, executionContext);
if (MATCHER.matches(e)){
J.MethodInvocation mi = (J.MethodInvocation) e;
e = e.withTemplate(template,
e.getCoordinates().replace(), mi.getSelect(),
mi.getArguments().get(1), mi.getArguments().get(0));
}
return e;
}
};
}
After a few days researching why my Flink application is not working properly I've came to the conclusion that the problem resides in a MinMaxPriorityQueue I am using.
It seems that this structure is not serializable. I've tried several ways to serialize it:
env.getConfig.registerTypeWithKryoSerializer(classOf[MinMaxPriorityQueue[Double]], classOf[JavaSerializer])
env.getConfig.registerTypeWithKryoSerializer(classOf[MinMaxPriorityQueue[java.lang.Double]], classOf[ProtobufSerializer]);
env.getConfig().addDefaultKryoSerializer(MyCustomType.class, TBaseSerializer.class);
all of them without luck.
However I've found this: Serializing Guava's ImmutableTable
Is there an equivalent to MinMaxPriorityQueue, or a way to serialize it?
Update
I've translated Tomasz into scala:
class MinMaxPriorityQueueSerializer extends Serializer[MinMaxPriorityQueue[Object]] {
private[this] val log = LoggerFactory.getLogger(this.getClass)
setImmutable(false)
setAcceptsNull(false)
val OPTIMIZE_POSITIVE = true
override def read(kryo: Kryo, input: Input, aClass: Class[MinMaxPriorityQueue[Object]]): MinMaxPriorityQueue[Object] = {
log.error("Kryo READ")
val comparator: Ordering[Object] = kryo.readClassAndObject(input).asInstanceOf[Ordering[Object]]
val size = input.readInt(OPTIMIZE_POSITIVE)
val queue: MinMaxPriorityQueue[Object] = MinMaxPriorityQueue.orderedBy(comparator)
.expectedSize(size)
.create()
(0 to size).foreach(_ => queue.offer(kryo.readClassAndObject(input)))
queue
}
override def write(kryo: Kryo, output: Output, queue: MinMaxPriorityQueue[Object]): Unit = {
log.error("Kryo WRITE")
kryo.writeClassAndObject(output, queue.comparator)
val declaredSize = queue.size
output.writeInt(declaredSize, OPTIMIZE_POSITIVE)
val actualSize = queue.toArray.foldLeft(0) {
case (z, q) =>
kryo.writeClassAndObject(output, q)
z + 1
}
Preconditions.checkState(
declaredSize == actualSize,
"Declared size (%s) different than actual size (%s)", declaredSize, actualSize)
}
}
And set kryo in flink to use that Serializer:
env.getConfig.addDefaultKryoSerializer(classOf[MinMaxPriorityQueue[Double]], classOf[MinMaxPriorityQueueSerializer])
env.getConfig.registerTypeWithKryoSerializer(classOf[MinMaxPriorityQueue[Double]], classOf[MinMaxPriorityQueueSerializer])
However it seems it gets never called, since I do not see anywhere in the logs the outputs of log.error("Kryo READ") and log.error("Kryo WRITE")
And the transformation still returns an empty MinMaxPriorityQueue, even I am updating it.
Update 2
I've implemented the SerializerTester, but I am getting a bufferUnderflow:
object Main {
def main(args: Array[String]) {
val tester = new MinMaxPriorityQueueSerializerTester()
val inQueue: MinMaxPriorityQueue[java.lang.Double] = MinMaxPriorityQueue.create()
inQueue.add(1.0)
val outputStream = new ByteArrayOutputStream()
tester.serialize(outputStream, inQueue)
val inputStream = new ByteArrayInputStream(outputStream.toByteArray())
val outQueue: MinMaxPriorityQueue[java.lang.Double] = tester.deserialize(inputStream);
System.out.println(inQueue);
System.out.println(outQueue);
}
class MinMaxPriorityQueueSerializerTester {
val kryo = new Kryo
kryo.setInstantiatorStrategy(new StdInstantiatorStrategy)
registerMinMaxSerializer();
// allowForClassesWithoutNoArgConstructor(); // needed to serialize Ordering
def registerMinMaxSerializer() {
kryo.addDefaultSerializer(classOf[MinMaxPriorityQueue[java.lang.Double]], new MinMaxPriorityQueueSerializer());
}
def serialize(out: OutputStream, queue: MinMaxPriorityQueue[java.lang.Double]) {
// try (Output output = new Output(out)) {
val output = new Output(out)
kryo.writeClassAndObject(output, queue)
// kryo.writeObject(output, queue)
//}
output.flush
}
def deserialize(in: InputStream): MinMaxPriorityQueue[java.lang.Double] = {
//try (Input input = new Input(in)) {
val input = new Input(in)
//kryo.readObject(input, classOf[MinMaxPriorityQueue[java.lang.Double]])
kryo.readClassAndObject(input).asInstanceOf[MinMaxPriorityQueue[java.lang.Double]]
//p}
}
}
You can use a custom Kryo Serializer.
Here is a sample one (in Java):
class MinMaxPriorityQueueSerializer extends Serializer<MinMaxPriorityQueue<Object>> {
private static final boolean OPTIMIZE_POSITIVE = true;
protected MinMaxPriorityQueueSerializer() {
setAcceptsNull(false);
setImmutable(false);
}
#Override
public void write(Kryo kryo, Output output, MinMaxPriorityQueue<Object> queue) {
kryo.writeClassAndObject(output, queue.comparator());
int declaredSize = queue.size();
output.writeInt(declaredSize, OPTIMIZE_POSITIVE);
int actualSize = 0;
for (Object element : queue) {
kryo.writeClassAndObject(output, element);
actualSize++;
}
Preconditions.checkState(
declaredSize == actualSize,
"Declared size (%s) different than actual size (%s)", declaredSize, actualSize
);
}
#Override
public MinMaxPriorityQueue<Object> read(Kryo kryo, Input input, Class<MinMaxPriorityQueue<Object>> type) {
#SuppressWarnings("unchecked")
Comparator<Object> comparator = (Comparator<Object>) kryo.readClassAndObject(input);
int size = input.readInt(OPTIMIZE_POSITIVE);
MinMaxPriorityQueue<Object> queue = MinMaxPriorityQueue.orderedBy(comparator)
.expectedSize(size)
.create();
for (int i = 0; i < size; ++i) {
queue.offer(kryo.readClassAndObject(input));
}
return queue;
}
}
Here is how you could use it:
class MinMaxPriorityQueueSerializerTester {
public static void main(String[] args) {
MinMaxPriorityQueueSerializerTester tester = new MinMaxPriorityQueueSerializerTester();
MinMaxPriorityQueue<Integer> inQueue = MinMaxPriorityQueue.<Integer>orderedBy(Comparator.reverseOrder())
.create(Arrays.asList(5, 2, 7, 2, 4));
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
tester.serialize(outputStream, inQueue);
ByteArrayInputStream inputStream = new ByteArrayInputStream(outputStream.toByteArray());
#SuppressWarnings("unchecked")
MinMaxPriorityQueue<Integer> outQueue = (MinMaxPriorityQueue<Integer>) tester.deserialize(inputStream);
System.out.println(inQueue);
System.out.println(outQueue);
}
private final Kryo kryo;
public MinMaxPriorityQueueSerializerTester() {
this.kryo = new Kryo();
registerMinMaxSerializer();
allowForClassesWithoutNoArgConstructor(); // needed to serialize Ordering
}
private void registerMinMaxSerializer() {
kryo.addDefaultSerializer(MinMaxPriorityQueue.class, new MinMaxPriorityQueueSerializer());
}
private void allowForClassesWithoutNoArgConstructor() {
((Kryo.DefaultInstantiatorStrategy) kryo.getInstantiatorStrategy())
.setFallbackInstantiatorStrategy(new StdInstantiatorStrategy());
}
public void serialize(OutputStream out, MinMaxPriorityQueue<?> queue) {
try (Output output = new Output(out)) {
kryo.writeObject(output, queue);
}
}
public MinMaxPriorityQueue<?> deserialize(InputStream in) {
try (Input input = new Input(in)) {
return kryo.readObject(input, MinMaxPriorityQueue.class);
}
}
}
I finally give up and tried to use a different Data Structure and make it Serializable with java.io.Serializable.
This Data Structure is an IntervalHeap implemented here, I just made it Serializable in my project.
All works correctly now.
Project Reactor has this factory method for creating a push/pull Producer<T>.
http://projectreactor.io/docs/core/release/reference/#_hybrid_push_pull_model
Is there any such thing in RxJava-2?
If not, what would be the recommended way (without actually implemementing reactive specs interfaces from scratch) to create such beast that can handle the push/pull model?
EDIT: as requested I am giving an example of the API I am trying to use...
private static class API
{
CompletableFuture<Void> getT(Consumer<Object> consumer) {}
}
private static class Callback implements Consumer<Object>
{
private API api;
public Callback(API api) { this api = api; }
#Override
public void accept(Object o)
{
//do stuff with o
//...
//request for another o
api.getT(this);
}
}
public void example()
{
API api = new API();
api.getT(new Callback(api)).join();
}
So it's call back based, which will get one item and from within you can request for another one. the completable future flags no more items.
Here is an example of a custom Flowable that turns this particular API into an RxJava source. Note however that in general, the API peculiarities in general may not be possible to capture with a single reactive bridge design:
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.atomic.*;
import java.util.function.*;
import org.reactivestreams.*;
import io.reactivex.Flowable;
import io.reactivex.internal.subscriptions.EmptySubscription;
import io.reactivex.internal.util.BackpressureHelper;
public final class SomeAsyncApiBridge<T> extends Flowable<T> {
final Function<? super Consumer<? super T>,
? extends CompletableFuture<Void>> apiInvoker;
final AtomicBoolean once;
public SomeAsyncApiBridge(Function<? super Consumer<? super T>,
? extends CompletableFuture<Void>> apiInvoker) {
this.apiInvoker = apiInvoker;
this.once = new AtomicBoolean();
}
#Override
protected void subscribeActual(Subscriber<? super T> s) {
if (once.compareAndSet(false, true)) {
SomeAsyncApiBridgeSubscription<T> parent =
new SomeAsyncApiBridgeSubscription<>(s, apiInvoker);
s.onSubscribe(parent);
parent.moveNext();
} else {
EmptySubscription.error(new IllegalStateException(
"Only one Subscriber allowed"), s);
}
}
static final class SomeAsyncApiBridgeSubscription<T>
extends AtomicInteger
implements Subscription, Consumer<T>, BiConsumer<Void, Throwable> {
/** */
private static final long serialVersionUID = 1270592169808316333L;
final Subscriber<? super T> downstream;
final Function<? super Consumer<? super T>,
? extends CompletableFuture<Void>> apiInvoker;
final AtomicInteger wip;
final AtomicLong requested;
final AtomicReference<CompletableFuture<Void>> task;
static final CompletableFuture<Void> TASK_CANCELLED =
CompletableFuture.completedFuture(null);
volatile T item;
volatile boolean done;
Throwable error;
volatile boolean cancelled;
long emitted;
SomeAsyncApiBridgeSubscription(
Subscriber<? super T> downstream,
Function<? super Consumer<? super T>,
? extends CompletableFuture<Void>> apiInvoker) {
this.downstream = downstream;
this.apiInvoker = apiInvoker;
this.requested = new AtomicLong();
this.wip = new AtomicInteger();
this.task = new AtomicReference<>();
}
#Override
public void request(long n) {
BackpressureHelper.add(requested, n);
drain();
}
#Override
public void cancel() {
cancelled = true;
CompletableFuture<Void> curr = task.getAndSet(TASK_CANCELLED);
if (curr != null && curr != TASK_CANCELLED) {
curr.cancel(true);
}
if (getAndIncrement() == 0) {
item = null;
}
}
void moveNext() {
if (wip.getAndIncrement() == 0) {
do {
CompletableFuture<Void> curr = task.get();
if (curr == TASK_CANCELLED) {
return;
}
CompletableFuture<Void> f = apiInvoker.apply(this);
if (task.compareAndSet(curr, f)) {
f.whenComplete(this);
} else {
curr = task.get();
if (curr == TASK_CANCELLED) {
f.cancel(true);
return;
}
}
} while (wip.decrementAndGet() != 0);
}
}
#Override
public void accept(Void t, Throwable u) {
if (u != null) {
error = u;
task.lazySet(TASK_CANCELLED);
}
done = true;
drain();
}
#Override
public void accept(T t) {
item = t;
drain();
}
void drain() {
if (getAndIncrement() != 0) {
return;
}
int missed = 1;
long e = emitted;
for (;;) {
for (;;) {
if (cancelled) {
item = null;
return;
}
boolean d = done;
T v = item;
boolean empty = v == null;
if (d && empty) {
Throwable ex = error;
if (ex == null) {
downstream.onComplete();
} else {
downstream.onError(ex);
}
return;
}
if (empty || e == requested.get()) {
break;
}
item = null;
downstream.onNext(v);
e++;
moveNext();
}
emitted = e;
missed = addAndGet(-missed);
if (missed == 0) {
break;
}
}
}
}
}
Test and example source:
import java.util.concurrent.*;
import java.util.function.Consumer;
import org.junit.Test;
public class SomeAsyncApiBridgeTest {
static final class AsyncRange {
final int max;
int index;
public AsyncRange(int start, int count) {
this.index = start;
this.max = start + count;
}
public CompletableFuture<Void> next(Consumer<? super Integer> consumer) {
int i = index;
if (i == max) {
return CompletableFuture.completedFuture(null);
}
index = i + 1;
CompletableFuture<Void> cf = CompletableFuture
.runAsync(() -> consumer.accept(i));
CompletableFuture<Void> cancel = new CompletableFuture<Void>() {
#Override
public boolean cancel(boolean mayInterruptIfRunning) {
cf.cancel(mayInterruptIfRunning);
return super.cancel(mayInterruptIfRunning);
}
};
return cancel;
}
}
#Test
public void simple() {
AsyncRange r = new AsyncRange(1, 10);
new SomeAsyncApiBridge<Integer>(
consumer -> r.next(consumer)
)
.test()
.awaitDone(500, TimeUnit.SECONDS)
.assertResult(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
}
}
This is something that looks that is working using Reactor's Flux.create(). I changed the API a bit.
public class FlowableGenerate4
{
private static class API
{
private ExecutorService es = Executors.newFixedThreadPool(1);
private CompletableFuture<Void> done = new CompletableFuture<>();
private AtomicInteger stopCounter = new AtomicInteger(10);
public boolean isDone()
{
return done.isDone();
}
public CompletableFuture<Void> getT(Consumer<Object> consumer)
{
es.submit(() -> {
try {
Thread.sleep(100);
} catch (Exception e) {
}
if (stopCounter.decrementAndGet() < 0)
done.complete(null);
else
consumer.accept(new Object());
});
return done;
}
}
private static class Callback implements Consumer<Object>
{
private API api;
private FluxSink<Object> sink;
public Callback(API api, FluxSink<Object> sink)
{
this.api = api;
this.sink = sink;
}
#Override
public void accept(Object o)
{
sink.next(o);
if (sink.requestedFromDownstream() > 0 && !api.isDone())
api.getT(this);
else
sink.currentContext().<AtomicBoolean>get("inProgress")
.set(false);
}
}
private Publisher<Object> reactorPublisher()
{
API api = new API();
return
Flux.create(sink -> {
sink.onRequest(n -> {
//if it's in progress already, do nothing
//I understand that onRequest() can be called asynchronously
//regardless if the previous call demand has been satisfied or not
if (!sink.currentContext().<AtomicBoolean>get("inProgress")
.compareAndSet(false, true))
return;
//else kick off calls to API
api.getT(new Callback(api, sink))
.whenComplete((o, t) -> {
if (t != null)
sink.error(t);
else
sink.complete();
});
});
}).subscriberContext(
Context.empty().put("inProgress", new AtomicBoolean(false)));
}
#Test
public void test()
{
Flowable.fromPublisher(reactorPublisher())
.skip(5)
.take(10)
.blockingSubscribe(
i -> System.out.println("onNext()"),
Throwable::printStackTrace,
() -> System.out.println("onComplete()")
);
}
}
I have simple counter actor implemented in java:
public class CounterJavaActor extends UntypedActor {
int count = 0;
#Override
public void onReceive(Object message) throws Exception {
if (message.equals("incr")) {
count += 1;
} else if (message.equals("get")) {
sender().tell(count, self());
}
}
}
In courses on coursera "Functional reactive programming in scala", I saw functional impementation of counter:
/**
* Advantages:
* state change is explicit
* state is scoped to current behaviour
*/
class CounterScala extends Actor{
def counter(n: Int) : Receive = {
case "incr" => context.become(counter(n+1))
case "get" => sender ! n
}
def receive = counter(0)
}
Upd:
My problem, that in java i can't make recourse functional call like in scala counter(n+1). What it means:
public class CounterJava8Actor extends AbstractActor {
//counter(0) in scala
private PartialFunction<Object, BoxedUnit> counter;
private int n = 0;
public CounterJava8Actor() {
counter =
ReceiveBuilder.
matchEquals("get", s -> {
sender().tell(n, self());
}).
matchEquals("inc", s -> {
//become(counter(n+1) in scala
context().become(counter);
}).build();
receive(counter);
}
}
It is possible to implement it in functional style with java?
According to docs you can use become/unbecome in java 8
http://doc.akka.io/docs/akka/snapshot/java/lambda-actors.html#become-unbecome
here is the sample code copied from there
public class HotSwapActor extends AbstractActor {
private PartialFunction<Object, BoxedUnit> angry;
private PartialFunction<Object, BoxedUnit> happy;
public HotSwapActor() {
angry =
ReceiveBuilder.
matchEquals("foo", s -> {
sender().tell("I am already angry?", self());
}).
matchEquals("bar", s -> {
context().become(happy);
}).build();
happy = ReceiveBuilder.
matchEquals("bar", s -> {
sender().tell("I am already happy :-)", self());
}).
matchEquals("foo", s -> {
context().become(angry);
}).build();
receive(ReceiveBuilder.
matchEquals("foo", s -> {
context().become(angry);
}).
matchEquals("bar", s -> {
context().become(happy);
}).build()
);
}
}
Or you can use UntypedActor like explained in the docs here
http://doc.akka.io/docs/akka/snapshot/java/untyped-actors.html
public class Manager extends UntypedActor {
public static final String SHUTDOWN = "shutdown";
ActorRef worker = getContext().watch(getContext().actorOf(
Props.create(Cruncher.class), "worker"));
public void onReceive(Object message) {
if (message.equals("job")) {
worker.tell("crunch", getSelf());
} else if (message.equals(SHUTDOWN)) {
worker.tell(PoisonPill.getInstance(), getSelf());
getContext().become(shuttingDown);
}
}
Procedure<Object> shuttingDown = new Procedure<Object>() {
#Override
public void apply(Object message) {
if (message.equals("job")) {
getSender().tell("service unavailable, shutting down", getSelf());
} else if (message instanceof Terminated) {
getContext().stop(getSelf());
}
}
};
}
To know how to add parameter to Procedure you can see this answer:
Akka/Java getContext().become with parameter?
and here is actual solution with java 8
private PartialFunction<Object, BoxedUnit> counter(final int n) {
return ReceiveBuilder.
matchEquals("get", s -> {
sender().tell(n, self());
}).
matchEquals("inc", s -> {
context().become(counter(n + 1));
}).build();
}
public CounterJava8Actor() {
receive(counter(0));
}
UPDATE FROM 2015-10-30
based on Roland Kuhn Awnser:
Akka Streams is using asynchronous message passing between Actors to
implement stream processing stages. Passing data across an
asynchronous boundary has an overhead that you are seeing here: your
computation seems to take only about 160ns (derived from the
single-threaded measurement) while the streaming solution takes
roughly 1µs per element, which is dominated by the message passing.
Another misconception is that saying “stream” implies parallelism: in
your code all computation runs sequentially in a single Actor (the map
stage), so no benefit can be expected over the primitive
single-threaded solution.
In order to benefit from the parallelism afforded by Akka Streams you
need to have multiple processing stages that each perform tasks of
1µs per element, see also the docs.
I did some changes. My code now looks like:
object MultiThread {
implicit val actorSystem = ActorSystem("Sys")
implicit val materializer = ActorMaterializer()
var counter = 0
var oldProgess = 0
//RunnableFlow: in -> flow -> sink
val in = Source(() => Iterator.continually((1254785478l, "name", 48, 23.09f)))
val flow = Flow[(Long, String, Int, Float)].map(p => SharedFunctions.transform2(SharedFunctions.transform(p)))
val tupleToEvent = Flow[(Long, String, Int, Float)].map(SharedFunctions.transform)
val eventToFactorial = Flow[Event].map(SharedFunctions.transform2)
val eventChef: Flow[(Long, String, Int, Float), Int, Unit] = Flow() { implicit builder =>
import FlowGraph.Implicits._
val dispatchTuple = builder.add(Balance[(Long, String, Int, Float)](4))
val mergeEvents = builder.add(Merge[Int](4))
dispatchTuple.out(0) ~> tupleToEvent ~> eventToFactorial ~> mergeEvents.in(0)
dispatchTuple.out(1) ~> tupleToEvent ~> eventToFactorial ~> mergeEvents.in(1)
dispatchTuple.out(2) ~> tupleToEvent ~> eventToFactorial ~> mergeEvents.in(2)
dispatchTuple.out(3) ~> tupleToEvent ~> eventToFactorial ~> mergeEvents.in(3)
(dispatchTuple.in, mergeEvents.out)
}
val sink = Sink.foreach[Int]{
v => counter += 1
oldProgess = SharedFunctions.printProgress(oldProgess, SharedFunctions.maxEventCount, counter,
DateTime.now.getMillis - SharedFunctions.startTime.getMillis)
if(counter == SharedFunctions.maxEventCount) endAkka()
}
def endAkka() = {
val duration = new Duration(SharedFunctions.startTime, DateTime.now)
println("Time: " + duration.getMillis + " || Data: " + counter)
actorSystem.shutdown
actorSystem.awaitTermination
System.exit(-1)
}
def main(args: Array[String]) {
println("MultiThread started: " + SharedFunctions.startTime)
in.via(flow).runWith(sink)
// in.via(eventChef).runWith(sink)
}
}
I not sure if I get something totally wrong, but still my implementation with akka-streams is much slower (now even slower as before) but what I found out is: If I increase the work for example by doing some division the implementation with akka-streams gets faster. So If I get it right (correct me otherwise) it seems there is too much overhead in my example. So you only get a benefit from akka-streams if the code has to do heavy work?
I'm relatively new in both scala & akka-stream. I wrote a little test project which creates some events until a counter has reached a specific number. For each event the factorial for one field of the event is being computed. I implemented this twice. One time with akka-stream and one time without akka-stream (single threaded) and compared the runtime.
I didn't expect that: When I create a single event the runtime of both programs are nearly the same. But if I create 70,000,000 events the implementation without akka-streams is much faster. Here are my results (the following data is based on 24 measurements):
Single event without akka-streams: 403 (+- 2)ms
Single event with akka-streams: 444 (+-13)ms
70Mio events without akka-streams: 11778 (+-70)ms
70Mio events with akka-steams: 75424(+-2959)ms
So my Question is: What is going on? Why is my implementation with akka-stream slower?
here my code:
Implementation with Akka
object MultiThread {
implicit val actorSystem = ActorSystem("Sys")
implicit val materializer = ActorMaterializer()
var counter = 0
var oldProgess = 0
//RunnableFlow: in -> flow -> sink
val in = Source(() => Iterator.continually((1254785478l, "name", 48, 23.09f)))
val flow = Flow[(Long, String, Int, Float)].map(p => SharedFunctions.transform2(SharedFunctions.transform(p)))
val sink = Sink.foreach[Int]{
v => counter += 1
oldProgess = SharedFunctions.printProgress(oldProgess, SharedFunctions.maxEventCount, counter,
DateTime.now.getMillis - SharedFunctions.startTime.getMillis)
if(counter == SharedFunctions.maxEventCount) endAkka()
}
def endAkka() = {
val duration = new Duration(SharedFunctions.startTime, DateTime.now)
println("Time: " + duration.getMillis + " || Data: " + counter)
actorSystem.shutdown
actorSystem.awaitTermination
System.exit(-1)
}
def main(args: Array[String]) {
import scala.concurrent.ExecutionContext.Implicits.global
println("MultiThread started: " + SharedFunctions.startTime)
in.via(flow).runWith(sink).onComplete(_ => endAkka())
}
}
Implementation without Akka
object SingleThread {
def main(args: Array[String]) {
println("SingleThread started at: " + SharedFunctions.startTime)
println("0%")
val i = createEvent(0)
val duration = new Duration(SharedFunctions.startTime, DateTime.now());
println("Time: " + duration.getMillis + " || Data: " + i)
}
def createEventWorker(oldProgress: Int, count: Int, randDate: Long, name: String, age: Int, myFloat: Float): Int = {
if (count == SharedFunctions.maxEventCount) count
else {
val e = SharedFunctions.transform((randDate, name, age, myFloat))
SharedFunctions.transform2(e)
val p = SharedFunctions.printProgress(oldProgress, SharedFunctions.maxEventCount, count,
DateTime.now.getMillis - SharedFunctions.startTime.getMillis)
createEventWorker(p, count + 1, 1254785478l, "name", 48, 23.09f)
}
}
def createEvent(count: Int): Int = {
createEventWorker(0, count, 1254785478l, "name", 48, 23.09f)
}
}
SharedFunctions
object SharedFunctions {
val maxEventCount = 70000000
val startTime = DateTime.now
def transform(t : (Long, String, Int, Float)) : Event = new Event(t._1 ,t._2,t._3,t._4)
def transform2(e : Event) : Int = factorial(e.getAgeYrs)
def calculatePercentage(totalValue: Long, currentValue: Long) = Math.round((currentValue * 100) / totalValue)
def printProgress(oldProgress : Int, fileSize: Long, currentSize: Int, t: Long) = {
val cProgress = calculatePercentage(fileSize, currentSize)
if (oldProgress != cProgress) println(s"$oldProgress% | $t ms")
cProgress
}
private def factorialWorker(n1: Int, n2: Int): Int = {
if (n1 == 0) n2
else factorialWorker(n1 -1, n2*n1)
}
def factorial (n : Int): Int = {
factorialWorker(n, 1)
}
}
Implementation Event
/**
* Autogenerated by Avro
*
* DO NOT EDIT DIRECTLY
*/
#SuppressWarnings("all")
#org.apache.avro.specific.AvroGenerated
public class Event extends org.apache.avro.specific.SpecificRecordBase implements org.apache.avro.specific.SpecificRecord {
public static final org.apache.avro.Schema SCHEMA$ = new org.apache.avro.Schema.Parser().parse("{\"type\":\"record\",\"name\":\"Event\",\"namespace\":\"week2P2\",\"fields\":[{\"name\":\"timestampMS\",\"type\":\"long\"},{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"ageYrs\",\"type\":\"int\"},{\"name\":\"sizeCm\",\"type\":\"float\"}]}");
public static org.apache.avro.Schema getClassSchema() { return SCHEMA$; }
#Deprecated public long timestampMS;
#Deprecated public CharSequence name;
#Deprecated public int ageYrs;
#Deprecated public float sizeCm;
/**
* Default constructor. Note that this does not initialize fields
* to their default values from the schema. If that is desired then
* one should use <code>newBuilder()</code>.
*/
public Event() {}
/**
* All-args constructor.
*/
public Event(Long timestampMS, CharSequence name, Integer ageYrs, Float sizeCm) {
this.timestampMS = timestampMS;
this.name = name;
this.ageYrs = ageYrs;
this.sizeCm = sizeCm;
}
public org.apache.avro.Schema getSchema() { return SCHEMA$; }
// Used by DatumWriter. Applications should not call.
public Object get(int field$) {
switch (field$) {
case 0: return timestampMS;
case 1: return name;
case 2: return ageYrs;
case 3: return sizeCm;
default: throw new org.apache.avro.AvroRuntimeException("Bad index");
}
}
// Used by DatumReader. Applications should not call.
#SuppressWarnings(value="unchecked")
public void put(int field$, Object value$) {
switch (field$) {
case 0: timestampMS = (Long)value$; break;
case 1: name = (CharSequence)value$; break;
case 2: ageYrs = (Integer)value$; break;
case 3: sizeCm = (Float)value$; break;
default: throw new org.apache.avro.AvroRuntimeException("Bad index");
}
}
/**
* Gets the value of the 'timestampMS' field.
*/
public Long getTimestampMS() {
return timestampMS;
}
/**
* Sets the value of the 'timestampMS' field.
* #param value the value to set.
*/
public void setTimestampMS(Long value) {
this.timestampMS = value;
}
/**
* Gets the value of the 'name' field.
*/
public CharSequence getName() {
return name;
}
/**
* Sets the value of the 'name' field.
* #param value the value to set.
*/
public void setName(CharSequence value) {
this.name = value;
}
/**
* Gets the value of the 'ageYrs' field.
*/
public Integer getAgeYrs() {
return ageYrs;
}
/**
* Sets the value of the 'ageYrs' field.
* #param value the value to set.
*/
public void setAgeYrs(Integer value) {
this.ageYrs = value;
}
/**
* Gets the value of the 'sizeCm' field.
*/
public Float getSizeCm() {
return sizeCm;
}
/**
* Sets the value of the 'sizeCm' field.
* #param value the value to set.
*/
public void setSizeCm(Float value) {
this.sizeCm = value;
}
/** Creates a new Event RecordBuilder */
public static Event.Builder newBuilder() {
return new Event.Builder();
}
/** Creates a new Event RecordBuilder by copying an existing Builder */
public static Event.Builder newBuilder(Event.Builder other) {
return new Event.Builder(other);
}
/** Creates a new Event RecordBuilder by copying an existing Event instance */
public static Event.Builder newBuilder(Event other) {
return new Event.Builder(other);
}
/**
* RecordBuilder for Event instances.
*/
public static class Builder extends org.apache.avro.specific.SpecificRecordBuilderBase<Event>
implements org.apache.avro.data.RecordBuilder<Event> {
private long timestampMS;
private CharSequence name;
private int ageYrs;
private float sizeCm;
/** Creates a new Builder */
private Builder() {
super(Event.SCHEMA$);
}
/** Creates a Builder by copying an existing Builder */
private Builder(Event.Builder other) {
super(other);
if (isValidValue(fields()[0], other.timestampMS)) {
this.timestampMS = data().deepCopy(fields()[0].schema(), other.timestampMS);
fieldSetFlags()[0] = true;
}
if (isValidValue(fields()[1], other.name)) {
this.name = data().deepCopy(fields()[1].schema(), other.name);
fieldSetFlags()[1] = true;
}
if (isValidValue(fields()[2], other.ageYrs)) {
this.ageYrs = data().deepCopy(fields()[2].schema(), other.ageYrs);
fieldSetFlags()[2] = true;
}
if (isValidValue(fields()[3], other.sizeCm)) {
this.sizeCm = data().deepCopy(fields()[3].schema(), other.sizeCm);
fieldSetFlags()[3] = true;
}
}
/** Creates a Builder by copying an existing Event instance */
private Builder(Event other) {
super(Event.SCHEMA$);
if (isValidValue(fields()[0], other.timestampMS)) {
this.timestampMS = data().deepCopy(fields()[0].schema(), other.timestampMS);
fieldSetFlags()[0] = true;
}
if (isValidValue(fields()[1], other.name)) {
this.name = data().deepCopy(fields()[1].schema(), other.name);
fieldSetFlags()[1] = true;
}
if (isValidValue(fields()[2], other.ageYrs)) {
this.ageYrs = data().deepCopy(fields()[2].schema(), other.ageYrs);
fieldSetFlags()[2] = true;
}
if (isValidValue(fields()[3], other.sizeCm)) {
this.sizeCm = data().deepCopy(fields()[3].schema(), other.sizeCm);
fieldSetFlags()[3] = true;
}
}
/** Gets the value of the 'timestampMS' field */
public Long getTimestampMS() {
return timestampMS;
}
/** Sets the value of the 'timestampMS' field */
public Event.Builder setTimestampMS(long value) {
validate(fields()[0], value);
this.timestampMS = value;
fieldSetFlags()[0] = true;
return this;
}
/** Checks whether the 'timestampMS' field has been set */
public boolean hasTimestampMS() {
return fieldSetFlags()[0];
}
/** Clears the value of the 'timestampMS' field */
public Event.Builder clearTimestampMS() {
fieldSetFlags()[0] = false;
return this;
}
/** Gets the value of the 'name' field */
public CharSequence getName() {
return name;
}
/** Sets the value of the 'name' field */
public Event.Builder setName(CharSequence value) {
validate(fields()[1], value);
this.name = value;
fieldSetFlags()[1] = true;
return this;
}
/** Checks whether the 'name' field has been set */
public boolean hasName() {
return fieldSetFlags()[1];
}
/** Clears the value of the 'name' field */
public Event.Builder clearName() {
name = null;
fieldSetFlags()[1] = false;
return this;
}
/** Gets the value of the 'ageYrs' field */
public Integer getAgeYrs() {
return ageYrs;
}
/** Sets the value of the 'ageYrs' field */
public Event.Builder setAgeYrs(int value) {
validate(fields()[2], value);
this.ageYrs = value;
fieldSetFlags()[2] = true;
return this;
}
/** Checks whether the 'ageYrs' field has been set */
public boolean hasAgeYrs() {
return fieldSetFlags()[2];
}
/** Clears the value of the 'ageYrs' field */
public Event.Builder clearAgeYrs() {
fieldSetFlags()[2] = false;
return this;
}
/** Gets the value of the 'sizeCm' field */
public Float getSizeCm() {
return sizeCm;
}
/** Sets the value of the 'sizeCm' field */
public Event.Builder setSizeCm(float value) {
validate(fields()[3], value);
this.sizeCm = value;
fieldSetFlags()[3] = true;
return this;
}
/** Checks whether the 'sizeCm' field has been set */
public boolean hasSizeCm() {
return fieldSetFlags()[3];
}
/** Clears the value of the 'sizeCm' field */
public Event.Builder clearSizeCm() {
fieldSetFlags()[3] = false;
return this;
}
#Override
public Event build() {
try {
Event record = new Event();
record.timestampMS = fieldSetFlags()[0] ? this.timestampMS : (Long) defaultValue(fields()[0]);
record.name = fieldSetFlags()[1] ? this.name : (CharSequence) defaultValue(fields()[1]);
record.ageYrs = fieldSetFlags()[2] ? this.ageYrs : (Integer) defaultValue(fields()[2]);
record.sizeCm = fieldSetFlags()[3] ? this.sizeCm : (Float) defaultValue(fields()[3]);
return record;
} catch (Exception e) {
throw new org.apache.avro.AvroRuntimeException(e);
}
}
}
}
Akka Streams is using asynchronous message passing between Actors to implement stream processing stages. Passing data across an asynchronous boundary has an overhead that you are seeing here: your computation seems to take only about 160ns (derived from the single-threaded measurement) while the streaming solution takes roughly 1µs per element, which is dominated by the message passing.
Another misconception is that saying “stream” implies parallelism: in your code all computation runs sequentially in a single Actor (the map stage), so no benefit can be expected over the primitive single-threaded solution.
In order to benefit from the parallelism afforded by Akka Streams you need to have multiple processing stages that each perform tasks of >1µs per element, see also the docs.
In addition to Roland's explanation, which I agree with fully, it should be understood that akka Streams are not just a concurrent programming framework. Streams also provide back pressure which means Events are only generated by the Source when there is demand to process them in the Sink. This communication of demand adds some overhead at each processing step.
Therefore your single-thread and multi-thread comparison is not "apples-to-apples".
If you want raw multi-threaded execution performance then Futures/Actors are a better way to go.