Possible to offload route completion DSL to actors in akka-http? - scala

I have become interested in the akka-http implementation but one thing that strikes me as kind of an anti-pattern is the creation of a DSL for all routing, parsing of parameters, error handling and so on. The examples given in the documents are trivial in the extreme. However I saw a route of a real product on the market and it was a massive 10k line file with mesting many levels deep and a ton of business logic in the route. Real world systems ahave to deal with users passing bad parameters, not having the right permissions and so on so the simple DSL explodes fast in real life. To me the optimal solution would be to hand off the route completion to actors, each with the same api who will then do what is needed to complete the route. This would spread out the logic and enable maintainable code but after hours I have been unable to manage it. With the low level API I can pass off the HttpRequest and handle it the old way but that leaves me out of most of the tools in the DSL. So is there a way I could pass something to an actor that would enable it to continue the DSL at that point, handling route specific stuff? I.e. I am talking about something like this:
class MySlashHandler() extends Actor {
def receive = {
case ctxt: ContextOfSomeKind =>
decodeRequest {
// unmarshal with in-scope unmarshaller
entity(as[Order]) { order =>
sender ! "Order received"
}
context.stop(self)
}
}
val route =
pathEndOrSingleSlash {
get { ctxt =>
val actor = actorSystem.actorOf(Props(classOf[MySlashHandler]))
complete(actor ? ctxt)
}
}
Naturally that wont even compile. Despite my best efforts i haven't found a type for ContextOfSomeKind or how to re-enter the DSL once I am inside the actor. It could be this isnt possible. If not I dont think I like the DSL because it encourages what I would consider horrible programming methodology. Then the only problem with the low level API is getting access to the entity marshallers but I would rather do that then make a massive app in a single source file.

Offload Route Completion
Answering your question directly: a Route is nothing more than a function. The definition being:
type Route = (RequestContext) => Future[RouteResult]
Therefore you can simply write a function that does what you are look for, e.g. sends the RequestContext to an Actor and gets back the result:
class MySlashHandler extends Actor {
val routeHandler = (_ : RequestContext) => complete("Actor Complete")
override def receive : Receive = {
case requestContext : RequestContext => routeHandler(requestContext) pipeTo sender
}
}
val actorRef : ActorRef = actorSystem actorOf (Props[MySlashHandler])
val route : Route =
(requestContext : RequestContext) => (actorRef ? requestContext).mapTo[RouteResult]
Actors Don't Solve Your Problem
The problem you are trying to deal with is the complexity of the real world and modelling that complexity in code. I agree this is an issue, but an Actor isn't your solution. There are several reasons to avoid Actors for the design solution you seek:
There are compelling arguments against putting business logic in
Actors.
Akka-http uses akka-stream under the hood. Akka stream uses Actors under the hood. Therefore, you are trying to escape a DSL based on composable Actors by using Actors. Water isn't usually the solution for a drowning person...
The akka-http DSL provides a lot of compile time checks that are washed away once you revert to the non-typed receive method of an Actor. You'll get more run time errors, e.g. dead letters, by using Actors.
Route Organization
As stated previously: a Route is a simple function, the building block of scala. Just because you saw an example of a sloppy developer keeping all of the logic in 1 file doesn't mean that is the only solution.
I could write all of my application code in the main method, but that doesn't make it good functional programming design. Similarly, I could write all of my collection logic in a single for-loop, but I usually use filter/map/reduce.
The same organizing principles apply to Routes. Just from the most basic perspective you could break up the Route logic according to method type:
//GetLogic.scala
object GetLogic {
val getRoute = get {
complete("get received")
}
}
//PutLogic.scala
object PutLogic {
val putRoute = put {
complete("put received")
}
}
Another common organizing principle is to keep your business logic separate from your Route logic:
object BusinessLogic {
type UserName = String
type UserId = String
//isolated business logic
val dbLookup(userId : UserId) : UserName = ???
val businessRoute = get {
entity(as[String]) { userId => complete(dbLookup(userId)) }
}
}
These can then all be combined in your main method:
val finalRoute : Route =
GetLogic.getRoute ~ PutLogic.putRoute ~ BusinessLogic.businessRoute
The routing DSL can be misleading because it sort of looks like magic at times, but underneath it's just plain-old functions which scala can organize and isolate just fine...

I encountered a problem like this last week as well. Eventually I ended up at this blog and decided to go the same way as described there.
I created a custom directive which makes it possible for me to pass request contexts to Actors.
def imperativelyComplete(inner: ImperativeRequestContext => Unit): Route = { ctx: RequestContext =>
val p = Promise[RouteResult]()
inner(new ImperativeRequestContext(ctx, p))
p.future
}
Now I can use this in my Routes file like this:
val route =
put {
imperativelyComplete { ctx =>
val actor = actorSystem.actorOf(Props(classOf[RequestHandler], ctx))
actor ! HandleRequest
}
}
With my RequestHandler Actor looking like the following:
class RequestHandler(ctx: ImperativeRequestContext) extends Actor {
def receive: Receive = {
case handleRequest: HandleRequest =>
someActor ! DoSomething() // will return SomethingDone to the sender
case somethingDone: SomethingDone =>
ctx.complete("Done handling request")
context.stop(self)
}
}
I hope this brings you into the direction of finding a better solution. I am not sure if this solution should be the way to go, but up until now it works out really well for me.

Related

Appropriate Future Handling in Akka Actors Typed

What is the proper way to handle Futures from inside an Akka (typed) Actor?
For example, assume there is an Actor OrderActor that receives Commands to place orders... which it does by making an http call to an external service. Since these are http calls to an external service, Futures are involved. So, what is the right way to handle that Future from within the Actor.
I read something about the pipeTo pattern. Is that what needs to happen here or something else?
class OrderActor(context: ActorContext[OrderCommand], orderFacade: OrderFacade)
extends AbstractBehavior[OrderCommand](context) {
context.log.info("Order Actor started")
override def onMessage(msg: OrderCommand): Behavior[OrderCommand] = {
msg match {
case PlaceOrder(
referenceId: OrderReferenceId,
ticker: Ticker,
quantity: Int,
replyTo: ActorRef[OrderResult]
) =>
orderFacade
.placeOrder(ticker, quantity) //this returns a Future
.map(res => {
//transform result
//book keeping / notification (affects state)
replyTo ! transformed
//Can/Should we map like this? I tried adding a log statement in here, but I never see it... and the replyTo doesnt seem to get the message.
})
this
It's generally best to avoid doing Future transformations (map, flatMap, foreach, etc.) inside an actor. There's a distinct risk that some mutable state within the actor isn't what you expect it to be when the transformation runs. In Akka Classic, perhaps the most pernicious form of this would result in sending a reply to the wrong actor.
Akka Typed (especially in the functional API) reduces a lot of the mutable state which could cause trouble, but it's still generally a good idea to pipe the Future as a message to the actor.
So if orderFacade.placeOrder results in a Future[OrderResponse], you might add subclasses of OrderCommand like this
// also include fields from the PlaceOrder which will be useful
case class OrderResponseIs(resp: OrderResponse, replyTo: ActorRef[OrderResult]) extends OrderCommand
// TODO include fields
case class OrderFailed() extends OrderCommand
And then pipe the Future to yourself with:
import scala.util.{ Failure, Success }
context.pipeToSelf(orderFacade.placeOrder) {
case Success(resp) => OrderResponseIs(resp, replyTo)
case Failure(_) => OrderFailed()
}
You then have to handle those messages:
case OrderResponseIs(resp, replyTo) =>
// transform resp
val transformed = ???
replyTo ! transformed
this
case OrderFailed() =>
context.log.warning("Stuff is broken")
this
There's not actually much overhead in doing this versus map and friends (both will typically involve scheduling a task for asynchronous execution on the dispatcher).

How to pass an ExecutionContext to spray route or better to avoid ask and use tell

As I am gathering experiences with Spray.io I see that there is more ways how to solve the task. I found a various opinions which were sometimes contradictory. So I would like to ask more experienced users to share their view.
Essentially having a REST api and separated services layer all assembled via cake pattern as follows:
object Main extends App with BootedCore with ServiceActors with RestApi with Web
ServiceActors are holding an ActorRefs to various services and RestApi holds Spray-routes. Service actors replies with case classes.
The question is how to connect those two layers?
** Possibility 1 - Use akka ask pattern**
import scala.concurrent.ExecutionContext.Implicits.global
//TODO: Excecution context - this is not scalable
val adServiceRoute: Route = {
path("service" / "import" / "status") {
get {
respondWithMediaType(`text/plain`) {
complete {
adImporterService.ask(GetImportStatus)(1 second).mapTo[StatusOfImport]
}
}
}
}
}
implicit val importStatusMarshaller: Marshaller[StatusOfImport] =
Marshaller.of[StatusOfImport](ContentTypes.`text/plain`) { (value, contentType, ctx) =>
val string = "Hello marshalled status"
ctx.marshalTo(HttpEntity(contentType, string))
}
I have found some Cons to this solution which I would like to get clarified.
- various blogs clams that ask pattern is best to avoid if possible, some example reasoning: e.g. it hides underlying errors etc.
- not sure on which ExcecutionContext to run. Import global is definitively the correct one. What I have available is ActorSystem. Is it common to configure separated dispatcher just for this purpose?
Possibility 2 - use tell
I tried to rewrite the previous one but wasn't able to make it work
val adServiceRoute: Route = {
path("service" / "ad" / "import" / "status") {
get {
respondWithMediaType(`text/plain`) (getServiceStatus)
}
}
}
implicit val importStatusMarshaller: Marshaller[StatusOfImport] =
Marshaller.of[StatusOfImport](ContentTypes.`text/plain`) { (value, contentType, ctx) =>
val string = "Hello marshalled status"
ctx.marshalTo(HttpEntity(contentType, string))
}
def getServiceStatus(ctx: RequestContext): Unit = {
adImporterService.tell(GetImportStatus , ctx.responder)
}
The issue here is that adImporterService's response is StatusOfImport which responder doesn't know how to handle. Creating an actor-per-request just for handling the response in order to transform it to HttpResponse seems to me as overkill.
I am clearly missing something important here.
Could someone more experienced clarify ?
I think you can use spray's onComplete directive. For example:
onComplete(adImporterService.ask(GetImportStatus).mapTo[StatusOfImport]) {
case Success(v) => complete(v)
case Failure(e) => ...
}

Scala Actors: Returning a Future of a type other than Any.

I am working my way through a book on Scala Actors, and I am running into a bit of a syntactical hangup. In practice, I tend to assign my variables and function definitions as such:
val v: String = "blahblahblah"
def f(n: Int): Int = n+1
including the (return)type of the item after its name. While I know this is not necessary, I have grown comfortable with this convention and find that it makes the code more easily understood by myself.
That being said, observe the below example:
class Server extends Actor {
def act() = {
while (true) {
receive {
case Message(string) => reply("Good,very good.")
}
}
}
}
def sendMsg(m: Message, s: Server): Future[String] = {
s !! m
}
The above code produces an error at compile time, complaining that the server returned a Future[Any], as opposed to a Future[String]. I understand that this problem can be circumvented by removing the return type from sendMsg:
def sendMsg(m: Message,s: Server) = s !! m
However, this is not consistant with my style. Is there a way that I can specify the type of Future that the server generates (as opposed to Future[Any])?
Your problem is a lot deeper than just style: you get a Future[Any] because the compiler cannot statically know better—with the current Akka actors as well as with the now deprecated scala.actors. In the absence of compile-time checks you need to resort to runtime checks instead, as idonnie already commented:
(actorRef ? m).mapTo[String]
This will chain another Future to the original one which is filled either with a String result, a ClassCastException if the actor was naughty, or with a TimeoutException if the actor did not reply, see the Akka docs.
There might be a way out soon, I’m working on an Akka extension to include statically typed channels, but that will lead to you having to write your code a little differently, with more type annotations.

Parallel file processing in Scala

Suppose I need to process files in a given folder in parallel. In Java I would create a FolderReader thread to read file names from the folder and a pool of FileProcessor threads. FolderReader reads file names and submits the file processing function (Runnable) to the pool executor.
In Scala I see two options:
create a pool of FileProcessor actors and schedule a file processing function with Actors.Scheduler.
create an actor for each file name while reading the file names.
Does it make sense? What is the best option?
Depending on what you're doing, it may be as simple as
for(file<-files.par){
//process the file
}
I suggest with all my energies to keep as far as you can from the threads. Luckily we have better abstractions which take care of what's happening below, and in your case it appears to me that you do not need to use actors (while you can) but you can use a simpler abstraction, called Futures. They are a part of Akka open source library, and I think in the future will be a part of the Scala standard library as well.
A Future[T] is simply something that will return a T in the future.
All you need to run a future, is to have an implicit ExecutionContext, which you can derive from a java executor service. Then you will be able to enjoy the elegant API and the fact that a future is a monad to transform collections into collections of futures, collect the result and so on. I suggest you to give a look to http://doc.akka.io/docs/akka/2.0.1/scala/futures.html
object TestingFutures {
implicit val executorService = Executors.newFixedThreadPool(20)
implicit val executorContext = ExecutionContext.fromExecutorService(executorService)
def testFutures(myList:List[String]):List[String]= {
val listOfFutures : Future[List[String]] = Future.traverse(myList){
aString => Future{
aString.reverse
}
}
val result:List[String] = Await.result(listOfFutures,1 minute)
result
}
}
There's a lot going on here:
I am using Future.traverse which receives as a first parameter which is M[T]<:Traversable[T] and as second parameter a T => Future[T] or if you prefer a Function1[T,Future[T]] and returns Future[M[T]]
I am using the Future.apply method to create an anonymous class of type Future[T]
There are many other reasons to look at Akka futures.
Futures can be mapped because they are monad, i.e. you can chain Futures execution :
Future { 3 }.map { _ * 2 }.map { _.toString }
Futures have callback: future.onComplete, onSuccess, onFailure, andThen etc.
Futures support not only traverse, but also for comprehension
Ideally you should use two actors. One for reading the list of files, and one for actually reading the file.
You start the process by simply sending a single "start" message to the first actor. The actor can then read the list of files, and send a message to the second actor. The second actor then reads the file and processes the contents.
Having multiple actors, which might seem complicated, is actually a good thing in the sense that you have a bunch of objects communicating with eachother, like in a theoretical OO system.
Edit: you REALLY shouldn't be doing doing concurrent reading of a single file.
I was going to write up exactly what #Edmondo1984 did except he beat me to it. :) I second his suggestion in a big way. I'll also suggest that you read the documentation for Akka 2.0.2. As well, I'll give you a slightly more concrete example:
import akka.dispatch.{ExecutionContext, Future, Await}
import akka.util.duration._
import java.util.concurrent.Executors
import java.io.File
val execService = Executors.newCachedThreadPool()
implicit val execContext = ExecutionContext.fromExecutorService(execService)
val tmp = new File("/tmp/")
val files = tmp.listFiles()
val workers = files.map { f =>
Future {
f.getAbsolutePath()
}
}.toSeq
val result = Future.sequence(workers)
result.onSuccess {
case filenames =>
filenames.foreach { fn =>
println(fn)
}
}
// Artificial just to make things work for the example
Thread.sleep(100)
execContext.shutdown()
Here I use sequence instead of traverse, but the difference is going to depend on your needs.
Go with the Future, my friend; the Actor is just a more painful approach in this instance.
But if use actors, what's wrong with that?
If we have to read / write to some property file. There is my Java example. But still with Akka Actors.
Lest's say we have an actor ActorFile represents one file. Hm.. Probably it can not represent One file. Right? (would be nice it could). So then it represents several files like PropertyFilesActor then:
Why would not use something like this:
public class PropertyFilesActor extends UntypedActor {
Map<String, String> filesContent = new LinkedHashMap<String, String>();
{ // here we should use real files of cource
filesContent.put("file1.xml", "");
filesContent.put("file2.xml", "");
}
#Override
public void onReceive(Object message) throws Exception {
if (message instanceof WriteMessage) {
WriteMessage writeMessage = (WriteMessage) message;
String content = filesContent.get(writeMessage.fileName);
String newContent = content + writeMessage.stringToWrite;
filesContent.put(writeMessage.fileName, newContent);
}
else if (message instanceof ReadMessage) {
ReadMessage readMessage = (ReadMessage) message;
String currentContent = filesContent.get(readMessage.fileName);
// Send the current content back to the sender
getSender().tell(new ReadMessage(readMessage.fileName, currentContent), getSelf());
}
else unhandled(message);
}
}
...a message will go with parameter (fileName)
It has its own in-box, accepting messages like:
WriteLine(fileName, string)
ReadLine(fileName, string)
Those messages will be storing into to the in-box in the order, one after antoher. The actor would do its work by receiving messages from the box - storing/reading, and meanwhile sending feedback sender ! message back.
Thus, let's say if we write to the property file, and send showing the content on the web page. We can start showing page (right after we sent message to store a data to the file) and as soon as we received the feedback, update part of the page with a data from just updated file (by ajax).
Well, grab your files and stick them in a parallel structure
scala> new java.io.File("/tmp").listFiles.par
res0: scala.collection.parallel.mutable.ParArray[java.io.File] = ParArray( ... )
Then...
scala> res0 map (_.length)
res1: scala.collection.parallel.mutable.ParArray[Long] = ParArray(4943, 1960, 4208, 103266, 363 ... )

Can I transform this asynchronous java network API into a monadic representation (or something else idiomatic)?

I've been given a java api for connecting to and communicating over a proprietary bus using a callback based style. I'm currently implementing a proof-of-concept application in scala, and I'm trying to work out how I might produce a slightly more idiomatic scala interface.
A typical (simplified) application might look something like this in Java:
DataType type = new DataType();
BusConnector con = new BusConnector();
con.waitForData(type.getClass()).addListener(new IListener<DataType>() {
public void onEvent(DataType t) {
//some stuff happens in here, and then we need some more data
con.waitForData(anotherType.getClass()).addListener(new IListener<anotherType>() {
public void onEvent(anotherType t) {
//we do more stuff in here, and so on
}
});
}
});
//now we've got the behaviours set up we call
con.start();
In scala I can obviously define an implicit conversion from (T => Unit) into an IListener, which certainly makes things a bit simpler to read:
implicit def func2Ilistener[T](f: (T => Unit)) : IListener[T] = new IListener[T]{
def onEvent(t:T) = f
}
val con = new BusConnector
con.waitForData(DataType.getClass).addListener( (d:DataType) => {
//some stuff, then another wait for stuff
con.waitForData(OtherType.getClass).addListener( (o:OtherType) => {
//etc
})
})
Looking at this reminded me of both scalaz promises and f# async workflows.
My question is this:
Can I convert this into either a for comprehension or something similarly idiomatic (I feel like this should map to actors reasonably well too)
Ideally I'd like to see something like:
for(
d <- con.waitForData(DataType.getClass);
val _ = doSomethingWith(d);
o <- con.waitForData(OtherType.getClass)
//etc
)
If you want to use a for comprehension for this, I'd recommend looking at the Scala Language Specification for how for comprehensions are expanded to map, flatMap, etc. This will give you some clues about how this structure relates to what you've already got (with nested calls to addListener). You can then add an implicit conversion from the return type of the waitForData call to a new type with the appropriate map, flatMap, etc methods that delegate to addListener.
Update
I think you can use scala.Responder[T] from the standard library:
Assuming the class with the addListener is called Dispatcher[T]:
trait Dispatcher[T] {
def addListener(listener: IListener[T]): Unit
}
trait IListener[T] {
def onEvent(t: T): Unit
}
implicit def dispatcher2Responder[T](d: Dispatcher[T]):Responder[T] = new Responder[T} {
def respond(k: T => Unit) = d.addListener(new IListener[T] {
def onEvent(t:T) = k
})
}
You can then use this as requested
for(
d <- con.waitForData(DataType.getClass);
val _ = doSomethingWith(d);
o <- con.waitForData(OtherType.getClass)
//etc
) ()
See the Scala wiki and this presentation on using Responder[T] for a Comet chat application.
I have very little Scala experience, but if I were implementing something like this I'd look to leverage the actor mechanism rather than using callback listener classes. Actors were made for asynchronous communication, they nicely separate those different parts of your app for you. You can also have them send messages to multiple listeners.
We'll have to wait for a "real" Scala programmer to flesh this idea out, though. ;)