I am writing a library that provides a distributed algorithm. The idea being that existing applications can add the library to use the algorithm. The algorithm is in a library module and it abstracts the actual transmission of data over the network behind a trait. An application that uses the algorithm has to provide the actual network transport code. In code it looks something like the following:
// library is really a separate project not a single object
object Library {
// handle to a remote server
trait RemoteProcess
// concrete server need to know how to actually send to a real remote process
trait Server {
def send(client: RemoteProcess, msg: String)
}
}
// the application uses the library and provides the concrete transport
object AkkaDemoApplication {
// concreate ref is a m wrapper to an actor ref in demo app
case class ConcreteRemoteProcess(ref: akka.actor.ActorRef) extends Library.RemoteProcess
class AkkaServer extends Library.Server {
// **WARNING** this wont compile its here to make my question
override def send(client: ConcreteRemoteProcess, msg: String): Unit = client.ref ! msg
}
}
A couple of options I have considered:
Have the signature of the AkkaServer method overload the library trait method then perform an unsafe cast to ConcreteRemoteProcess. Yuk!
Have the signature of the AkkaServer method overload the library trait method then pattern match on the RemoteProcesss argument give a ConcreteRemoteProcess. This is no better than an unsafe cast in terms of blowing up at runtime if the wrong thing is passed.
Make the library server generic in terms of the RemoteProcess.
An example of option 3 looks like:
object Library {
trait Server[RemoteProcess] {
def send(client: RemoteProcess, msg: String)
}
}
object Application {
class AkkaServer extends Library.Server[ActorRef] {
override def send(client: ActorRef, msg: String): Unit = client ! msg
}
}
I tried option 3 and it worked but the generic type ended up be stamped on just about every type throughout the entire library module. There was then a lot of covariant and contravariant hassles to get the algorithmic code to compile. Simply to get compile time certainty at the one integration point the cognitive overhead was very large. Visually the library code is dominated by the generic signatures as though understanding that is critical to understanding the library when in fact it's a total distraction to understanding the library logic.
So using the genetic works and gave me compile time certainly but now I wished I had gone with the option 2 (the pattern match) with the excuse "it would fail fast at startup if someone got it wrong lets keep it simple".
Am I missing some Scala feature or idiom here to get compile time certainty without the cognitive overhead of a "high touch" generic that all the library code touches but ignores?
Edit I have considered that perhaps my code library is badly factored such that a refactor could move the generic to the boundaries. Yet the library has already been refactored for testability and that breakup into testable responsibilities is part of the problem of the generic getting smeared around the codebase. Hence my question is: in general is there another technique I don't know about to avoid a generic to provide a concrete implementation to an abstract API?
I think you are coupling your algorithm and Akka too closely. Further more, I assume the server to send data to the remote client that performs some operation and sends back the result
Answer
Why not
object Library {
// handle to a remote server
trait RemoteProcessInput
trait RemoteProcessResult
// concrete server need to know how to actually send to a real remote process and how to deal with the result
trait Server {
def handle(clientData: RemoteProcessInput) : Future[RemoteProcessResult]
}
}
A concrete implementation provides the implementation with Akka
object Application {
class AkkaServerImpl(system: ActorSystem)
extends Library.Server {
override def handle(clientData: RemoteProcessInput)
: ActorRef, msg: String): Future[RemoteProcessResult] = {
// send data to client and expect result
// you can distinguish target and msg from the concrete input
val ref : ActorRef = ??? // (resolve client actor)
val msg = ??? // (create your message based on concrete impl)
val result = ref ? msg // using ask pattern here
// alternatively have an actor living on server side that sends msgs and receives the clients results, triggered by handle method
result
}
}
}
Related
I have an actor (actually a persistent actor) that, in response to a message (command), needs to make an HTTP call. The question now is how do I deal with this HTTP call in unit test?
Normally I would use the combination of DI and mocking to inject a mock implementation of the HTTP service while testing. But I am not sure if this is how to approach the problem in Akka? Even if this is the how to approach it in Akka, I am not sure how to go about doing the injecting and mocking in testing.
Any thoughts on this? What is the idiomatic way for testing Actors that perform IO operations (HTTP calls, writing to DB etc).
PS: I am using Akka Typed.
My personal belief is that you should avoid IO operations in Actors if at all possible (see this presentation for more details).
That being said, I’m sure there are people who would disagree and you probably shouldn’t listen to me :)
Here is how I would go about mocking it for a test.
Create a trait that represents your http call.
trait Client {
def makeHttpCall(input: Int): Future[Output]
}
(You will want to create a class of some kind that implements this trait and test it separately from your actor.)
Pass an instance of that trait into your actor using its constructor/apply method.
def apply(client: Client): Behavior[...] = {
// use your client inside of your behavior
}
In your test create a mock instance of Client and pass that in to your actor. You can mock your Client using a mocking library (ScalaMock or Mockito) or you can actually just create a test implementation of your trait with relative ease:
class TestClient extends Client {
def makeHttpCall(input: Int): Future[Output] =
Future.successful(Output(...))
}
Note: all of the names for classes and methods that I chose are just placeholders. You should, of course, choose more specific names based on the context you are working in.
In particular situations, you need to have some utility methods that are required across different classes. To solve this situation, you create an Util object wherein you place all these methods
object AggregatorUtil {
def aggregateValues(list : List[BigDecimal]) = //some logic...
}
// Import everything in the Utilities object
import AggregatorUtil._
and then import whichever members of util are required in your class. However, the downside to this is that, as all your methods are inside the singleton object and it becomes tricky to mock the object and unit test methods of the class that use utility methods.
To solve this problem again, the only solution that came to mind was Extracting the functionality out to a trait and then mocking the trait.
Please let me know if there is any other approach for handling and testing of util methods and which one is a rather cleaner approach.
Thanks in advance !!!
Note: -I am using scalatest and mockito in my project.
If you need to mock, putting this all in a mocked-out trait is the way forward. If mocking is unnecessary though, avoid it. Mocking unnecessarily is... unnecessary. You'll just be wasting time and effort for something which provides no additional value.
Mocking is best used when you have complex functionality or functionality in other files which you want to treat as a black box and just assume it works as expected (you'd then typically unit test this stuff separately). If you can avoid it and use the functions' actual functionality though, you'll get a much more realistic view of what your application does and will spot new bugs/breaking changes quicker (if you have mocked out functionality and forget to update your mocks, you might not any spot new bugs you introduce).
A good example of when mocking is necessary is when you're mocking calls to a database in a MVC application (e.g. a Scala Play microservice). You obviously don't want to have to run an actual database when testing your code, so you'd typically mock out your connector layer and return dummy/mocked data from your connector functions.
An example of something you wouldn't mock is something like:
trait MyTrait {
def toInt(str: String): Int
}
val mockedTrait = mock[MyTrait]
when(mockedTrait.toInt(eq("3")).thenReturn(3)
It's a bit of a silly example, but I think it explains the point clearly - doing something like this would be ridiculous. Mocking isn't always the answer.
I mostly mock with Test-Implementation which I find more readable and you don't have to learn a mocking framework.
Here an example:
The Interface:
trait DataRepo {
def persist(data: DataObject): Future[DataObject]
def idents(): Future[List[String]]
def insertData(dataCont: DataObject): Future[Int]
...
}
The Mocked Interface:
object DataRepoMock extends DataRepo {
def persist(data: DataObject): Future[DataObject] = ??? // only implement when needed
def idents(): Future[List[String]] = Future.successful((0 to 10).map(_=>Random.nextInt(100)))
def insertData(dataCont: DataObject): Future[Int] = Future.successful(Random.nextInt(100))
...
}
You can also use all the Scala goodies, like Pattern Matching, to make your Mock react differently to input.
Here is an example that this is not just used by myself;):
EPFLx: scala-reactiveX see Lecture 2.5 Testing Actor Systems:
def fakeGetter(url:String, depth: Int):Props =
Props(new Getter(url, depth){
override def webClient: WebClient = FakeWebClient
})
Let's say I start out with a Akka Persistence system like this:
case class MyMessage(x: Int)
class MyProcessor extends Processor {
def receive = {
case Persistent(m # MyMessage) => m.x
//...
}
}
And then someday I change it to this:
case class MyMessage(x: Int, y: Int)
class MyProcessor extends Processor {
def receive = {
case Persistent(m # MyMessage) => m.x + m.y
//...
}
}
After I deploy my new system, when the instance of MyProcessor tries to restore its state, the journaled messages will be of the former case class. Because it is expecting the latter type, it will throw an OnReplayFailure, rendering the processor useless. Question is: if we were to assume an absent y can equal 0 (or whatever) is there a best practice to overcome this? For example, perhaps using an implicit to convert from the former message to the latter on recovery?
Akka uses Java serialization by default and says that for a long-term project we should use a proper alternative. This is because Java serialization is very difficult to evolve over time. Akka recommends using either Google Protocol Buffers, Apache Thrift or Apache Avro.
With Google Protocol Buffers, for example, in your case you'd be writing something like:
if (p.hasY) p.getY else 0
Akka explains all that in a nice article (admittedly it's not very Google-able):
http://doc.akka.io/docs/akka/current/scala/persistence-schema-evolution.html
and even explains your particular use case of adding a new field to an existing message type:
http://doc.akka.io/docs/akka/current/scala/persistence-schema-evolution.html#Add_fields
A blog post the Akka documentation recommends for a comparison of the different serialization toolkits:
http://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html
Update: I've rewritten it in wonky-style and posted it on github - I would appreciate some feedback: https://github.com/drozzy/parallel-discrete-event-akka/tree/master/src/main/scala
Can anyone help me re-write "Parallel discrete event simulation" example from the "Chapter 32: Actors and Concurrency" from Programming in Scala, 2nd, by Martin Odersky?
It was written originally in Scala actors, but translating to Akka I encouter a lot of problems.
Some examples
Actor inheritance (p. 708)
Code like this:
trait Simulant extends Actor
class Wire extends Simulant
I have no idea how to translate to Akka, since in my understanding we never directly instantiate them.
Main Loop
The author constantly uses loops in actors, and I have no idea what they are about:
def act() {
loop {
if (running && busySimulants.isEmpty)
advance()
reactToOneMessage()
}
}
Types
Mostly though, I am struggling with strong types -- it seems Akka, due to requiring ActorRefs prevents me from depending on a specific type of an actor. For example, in the book, the following:
trait Simulant extends Actor {
val clock: Clock
...
depends on a strongly-typed actor Clock. And then it is simply "instantiated" in the implementing actor:
class Wire(name: String, init: Boolean) extends Simulant {
def this(name: String) { this(name, false) }
def this() { this("unnamed") }
val clock = Circuit.this.clock
In my implementation I have something along the lines of:
trait Simulant extends Actor {
val clock: ActorRef
...
and I have no idea how to "instantiate" it. What I have now (untested) is:
class Wire(val clock:ActorRef, name:String, init: Boolean) extends Actor{
def this(clock:ActorRef, name:String) {this(clock, name, false)}
def this(clock:ActorRef){this(clock, "unnamed")}
so I just end up draggin the actor refs around the constructors!
Hooking up components
How do I hook up components to each other? For example, hook-up an AndGate to a Wire - such that any time a signal on the wire changes it sends a message to the gate.
I do it by sending Add messages (i.e. AndGate sends Add to Wire so that wire can add it to the list of its subscribers), and so have to wait for all of them to arrive before starting the simulation. Is there any way to avoid that (the waiting)? (In original implementation the with Scala Actors, some actors were just accessed from global scope, and sometimes actors called other actor's methods directly!)
Source Code
The source code of the example can be found in full at the following url:
http://booksites.artima.com/programming_in_scala_2ed/examples/html/ch32.html
under the heading: 32.6 A longer example: Parallel discrete event simulation
P.S.: I'm new to akka, so forgive my ignorance.
I can not provide any migrated source code, but this link might help you:
actors migration guide.
Some comments:
Actor Inheritance
You can do this in Akka, but class Wire has to implement the receive method.
Main Loop
In Akka you implement the receive method instead of the main loop.
Types
You can use an ActorRef as constructor parameter, but it has to be created (and started) before calling the constructor, e.g. with context.system.actorOf(...).
And there is something called Typed Actors in Akka.
I also strongly encourage you to have a look at the documentation.
EDIT
I had a (quick) look at the source code, these are my findings:
In Scala it is not so common as in Java (and not enforced) that only one public class exists per file (although it can improve compile speed).
Demo.scala: Line 11, Use var instead of val
Constructor initialization (e.g. in FullAdder, Gate, ...): You should be careful about
that, because the constructor is executed every time the actor is restarted; maybe using
the preStart method would be better.
FullAdder, HalfAdder: An actor who doesn't react to messages (or only returning Unit) is a
strange thing in my opinion. Maybe you find another solution for constructing an adder.
Clock.advance: Using return is not good scala style (and I believe it doesn't work
in this case). Use else instead.
I know it's not directly possible to serialize a function/anonymous class to the database but what are the alternatives? Do you know any useful approach to this?
To present my situation: I want to award a user "badges" based on his scores. So I have different types of badges that can be easily defined by extending this class:
class BadgeType(id:Long, name:String, detector:Function1[List[UserScore],Boolean])
The detector member is a function that walks the list of scores and return true if the User qualifies for a badge of this type.
The problem is that each time I want to add/edit/modify a badge type I need to edit the source code, recompile the whole thing and re-deploy the server. It would be much more useful if I could persist all BadgeType instances to a database. But how to do that?
The only thing that comes to mind is to have the body of the function as a script (ex: Groovy) that is evaluated at runtime.
Another approach (that does not involve a database) might be to have each badge type into a jar that I can somehow hot-deploy at runtime, which I guess is how a plugin-system might work.
What do you think?
My very brief advice is that if you want this to be truly data-driven, you need to implement a rules DSL and an interpreter. The rules are what get saved to the database, and the interpreter takes a rule instance and evaluates it against some context.
But that's overkill most of the time. You're better off having a little snippet of actual Scala code that implements the rule for each badge, give them unique IDs, then store the IDs in the database.
e.g.:
trait BadgeEval extends Function1[User,Boolean] {
def badgeId: Int
}
object Badge1234 extends BadgeEval {
def badgeId = 1234
def apply(user: User) = {
user.isSufficientlyAwesome // && ...
}
}
You can either have a big whitelist of BadgeEval instances:
val weDontNeedNoStinkingBadges = Map(
1234 -> Badge1234,
5678 -> Badge5678,
// ...
}
def evaluator(id: Int): Option[BadgeEval] = weDontNeedNoStinkingBadges.get(id)
def doesUserGetBadge(user: User, id: Int) = evaluator(id).map(_(user)).getOrElse(false)
... or if you want to keep them decoupled, use reflection:
def badgeEvalClass(id: Int) = Class.forName("com.example.badge.Badge" + id + "$").asInstanceOf[Class[BadgeEval]]
... and if you're interested in runtime pluggability, try the service provider pattern.
You can try and use Scala Continuations - they can give you the ability to serialize the computation and run it at later time or even on another machine.
Some links:
Continuations
What are Scala continuations and why use them?
Swarm - Concurrency with Scala Continuations
Serialization relates to data rather than methods. You cannot serialize functionality because it is a class file which is designed to serialize that and object serialization serializes the fields of an object.
So like Alex says, you need a rule engine.
Try this one if you want something fairly simple, which is string based, so you can serialize the rules as strings in a database or file:
http://blog.maxant.co.uk/pebble/2011/11/12/1321129560000.html
Using a DSL has the same problems unless you interpret or compile the code at runtime.