How to pass results from one source stream to another - scala

I have a method that processes a Source and returns. I am trying to modify it but can't seem to be able to return the same thing:
Original
def originalMethod[as: AS, mat: MAT, ec: EC](checkType: String)
: Flow[ByteString, MyValidation[MyClass], NotUsed]{
collectStuff
.map { ts =>
val errors = MyEngine.checkAll(ts.code)
(ts, errors)
}
.map { x =>
x._2
.leftMap(xs => {
addInformation(x._1, xs.toList)
})
.toEither
}
}
I am modifying by using another source and pass result of that to the original source and yet return the same thing:
def calculate[T: AS: MAT](source: Source[T, NotUsed]): Future[Seq[T]] =
{
source.runWith(Sink.seq)
}
def modifiedMethod[as: AS, mat: MAT, ec: EC](checkType: String, mySource: Source[LoanApplicationRegister, NotUsed])
: Flow[ByteString, MyValidation[MyClass], NotUsed]{
for {
calc <- calculate(mySource)
orig <- collectStuff
.map { ts =>
val errors = MyEngine.checkAll(ts.code, calc)
(ts, errors)
}
.map { x =>
x._2
.leftMap(xs => {
addInformation(x._1, xs.toList)
})
.toEither
}
}
yield {
orig
}
}
But I'm getting compilation error Expression of type Future[Nothing] doesn't conform to existing type Flow[ByteString, MyValidation[MyClass]
How can I return Flow[ByteString, MyValidation[MyClass] in my modifiedMethod just like the originalMethod was

for { calc <- calculate(mySource)}
yield {
collectStuff
.map { ts =>
val errors = MyEngine.checkAll(ts.code, calc)
(ts, errors)
}
.map { x =>
x._2
.leftMap(xs => {
addInformation(x._1, xs.toList)
})
.toEither
}
}
would give you a Future[Flow[ByteString, MyValidation[MyClass], NotUsed]] instead of Future[Nothing]
but if you want to remove the Future you'd need to Await somewhere for it (either when you call calculate (and then you don't need the for) or after it. Usually, that's not the way to use Futures

Related

How to get Left from a method that returns Future?

def myMethod(myType: String) :Future[Future[Either[List[MyError], MyClass]]] {
for {
first <- runWithSeq(firstSource)
}
yield {
runWithSeq(secondSource)
.map {s ->
val mine = MyClass(s.head, lars)
val errors = myType match {
case "all" => Something.someMethod(mine)
}
(s, errors)
}
.map { x =>
x._2.leftMap(xs => {
addInfo(x._1.head, xs.toList)
}).toEither
}
}
}
for {
myStuff <- myMethod("something")
} yield {
myStuff.collect {
case(Left(errors), rowNumber) =>
MyCaseClass(errors, None) //compilation error here
}
}
I get compilation error on MyCaseClass that expected: List[MyError], found: Any
The signature of MyCaseClass is:
case class MyCaseClass(myErrors: List[ValidationError])
How can I fix this such that I can correctly call MyCaseClass inside the yield?
Your code example doesn't make much sense, and doesn't compile, but if runWithSeq() returns a Future then you should be able to eliminate the double Future return type like so.
for {
_ <- runWithSeq(firstSource)
scnd <- runWithSeq(secondSource)
} yield { ...
Your example is pretty hard to paste and fix
Abstact example for this
Class C may be whatever you want
def test(testval: Int)(implicit ec: ExecutionContext): Future[Future[Either[String, Int]]] = {
Future(Future{
if (testval % 2 == 0) Right(testval) else Left("Smth wrong")
})
}
implicit class FutureEitherExt[A, B](ft: Future[Either[A, B]]) {
def EitherMatch[C](f1: A => C, f2: B => C)(implicit ec: ExecutionContext): Future[C] = {
ft.map {
case Left(value) => f1(value)
case Right(value) => f2(value)
}
}
}
val fl: Future[Either[String, Int]] = test(5).flatten
val result: Future[String] = fl.EitherMatch(identity, _.toString)

How to create an Either[S,T] where S or T could be Unit?

private def responseValidationFlow[T](responsePair: ResponsePair)(implicit evidence: FromByteStringUnmarshaller[T]) = responsePair match {
case (Success(response), _) => {
response.entity.dataBytes
.via(Framing.delimiter(ByteString("\n"), maximumFrameLength = 8192))
.mapAsyncUnordered(Runtime.getRuntime.availableProcessors()) { body =>
if (response.status == OK) {
val obj: Future[T] = Unmarshal(body).to[T]
obj.foreach(x => log.debug("Received {}: {}.", x.getClass.getSimpleName, x))
obj.map(Right(_))
} else {
val reason = body.utf8String
log.error("Non 200 response status: {}, body: {}.", response.status.intValue(), reason)
Future.successful(reason)
.map(Left(_))
}
}
}
case (Failure(t), _) => {
Source.single(Left(t.getMessage))
}
}
What I’d like to do is parameterize both sides of the Either. That’s not hard to do, but what I’m having trouble with is creating a Left or Right that doesn’t have a value. In that case, the body should be consumed fully and discarded. I tried using ClassTags, but the compiler thinks that the type is Any, not S or T. An sample invocation of this a method would look like responseValidationFlow[String, Unit] producing an Source[Either[String, Unit]]
I believe, you can just define an implicit unmarshaller to Unit in scope:
implicit val unitUnmarshaller: FromByteStringUnmarshaller[Unit] =
Unmarshaller.strict(_ => ())
Based on what #Kolmar suggested, here is the working code.
private def responseValidationFlow[L, R](responsePair: ResponsePair)(
implicit ev1: FromByteStringUnmarshaller[L], ev2: FromByteStringUnmarshaller[R]
): Source[Either[L, R], Any] = {
responsePair match {
case (Success(response), _) => {
response.entity.dataBytes
.via(Framing.delimiter(ByteString("\n"), maximumFrameLength = 8192))
.mapAsyncUnordered(Runtime.getRuntime.availableProcessors()) { body =>
if (response.status == OK) {
val obj: Future[R] = Unmarshal(body).to[R]
obj.foreach(x => log.debug("Received {}.", x.getClass.getSimpleName))
obj.map(Right(_))
} else {
log.error("Non 200 response status: {}.", response.status.intValue())
Unmarshal(body).to[L]
.map(Left(_))
}
}
}
case (Failure(t), _) => {
log.error(t, "Request failed.")
Source.empty
}
}
}
If the method is invoked like this responseValidationFlow[Status, Unit], then a FromByteStringUnmarshaller[Unit] is made available at call site. The compiler uses the implicit evidence to find the required Unmarshaller.

Scala promise does not work if I encapsulate it in a commodity function

I have this code that I use in a spray handler
get {
def callService = {
val p = Promise[Option[DocumentRef]]
val fut = p.future
archiveService.getByHash(ZeroHash, {
result => p success result
})
fut
}
onComplete(OnCompleteFutureMagnet(callService)){
case Success(docRef) => {
val doc = docRef map {
x => x.title
} getOrElse "nothing"
complete("Done with " + doc)
}
case Failure(ex) => complete("error ${ex.getMessage}")
}
}
so I had the bright idea of writing the following function to encapsulate the work done to create a future out of a promise:
def callback2Future[T](funToCall: (T => Unit) => Any): Future[T] = {
val p = Promise[T]
val resultFuture = p future
def callbacklistener(arg: T): Unit = {
arg: T => p success arg
}
funToCall(callbacklistener)
resultFuture
}
And restructure the onComplete as:
onComplete(OnCompleteFutureMagnet(callback2Future(archiveService.getByHash(ZeroHash, _: Option[DocumentRef] => Unit)))) {
case Success(docRef) => {
...
}
In the original implementation with callservice, it works great (with great throughput too), with the callback2Future implementation I get a forever wait and it eventually times out. They seem the same to me, can anyone spot the error?
I believe that your problem is due to the infamous auto-Unit feature of Scala. Your function:
def callbacklistener(arg: T): Unit = {
arg: T => p success arg
}
will probably be interpreted as:
def callbacklistener(arg: T): Unit = {
{ arg: T => p success arg }
()
}
What you really want is probably:
def callbacklistener(arg: T): Unit = p success arg
To be clear, in your implementation you are defining a function callbackListener with return type Unit; in the body of this function you have an expression, { arg: T => p success arg }, whose value is of type T => Unit and is discarded; the Scala compiler will then put a free () in your code as the return type of the callbackListener is supposed to be Unit.

Can this be made prettier? Matching string to case class field

Sorry about the title, please edit it to be more descriptive if you can!
Is there a way to generalize this with scala? I have quite a few fields that can be filtered against, and this is just plain ugly! The problem I ran against was matching parameter name against the case class field, can it be done in a more general way, without this much code duplication?
get("/MostClicked") { request =>
val res = MongoDbOps.findMostClicked()
val res1 = request.params.get("source") match {
case None => res
case Some(f) => res.filter(_.source == f)
}
val res2 = request.params.get("category") match {
case None => res1
case Some(f) => res1.filter(_.category == f)
}
// more of the same...
render.plain {
res2.toJson.prettyPrint
}.toFuture
}
You can try either of the two approaches below.
case class MostClicked(
source: String,
category: String,
rating: String)
object MongoDbOps {
def findMostClicked() = List[MostClicked]()
}
class Request {
val params = Map[String, String]()
}
def get(path: String)(f: Request => String) = {
f(new Request)
}
First is to use a List of matchers and then apply them sequentially using foldLeft:
get("/MostClicked") { request =>
val res = MongoDbOps.findMostClicked()
val kfun = List(
"source" -> ((x: MostClicked, y: String) => x.source == y),
"category" -> ((x: MostClicked, y: String) => x.category == y),
"rating" -> ((x: MostClicked, y: String) => x.rating == y))
val r = kfun.foldLeft(res) { (x, y) =>
request.params.get(y._1)
.map(f => res.filter(y._2(_, f)))
.getOrElse(x)
}
r.toString
// more of the same...
render.plain {
r.toJson.prettyPrint
}.toFuture
}
Or simply make it more readable:
get("/MostClicked") { request =>
val res = MongoDbOps.findMostClicked()
val res1 = request.params.get("source")
.map(f => res.filter(_.source == f))
.getOrElse(res)
val res2 = request.params.get("category")
.map(f => res.filter(_.category == f))
.getOrElse(res1)
val res3 = request.params.get("rating")
.map(f => res.filter(_.rating == f))
.getOrElse(res2)
// more of the same...
render.plain {
res3.toJson.prettyPrint
}.toFuture
}
I had to break it down to parts and work out the types, thus the smaller methods.
It works in my simple experiment and gets rid of the duplication where it can, but might not be as readable?
Note that unless you get into reflection, you still need to create the name of the parameter and how it should be filtered.
This looks to be the same as tuxdna's answer except with types to increase readability and maintainability
SETUP
case class Request(params: Map[String, String])
case class Result(category: String, source: String)
type Filterer = (Result, String) => Boolean
case class FilterInfo(paramName: String, filterer: Filterer)
type Analyzer = FilterInfo => List[Result]
val request = Request(Map("source"->"b"))
EXTRACTION METHODS
def reduce(filterInfos: List[FilterInfo], results: List[Result]) = {
filterInfos.foldLeft(results) { (currentResult, filterInfo) =>
request.params.get(filterInfo.paramName)
.map(filterVal => currentResult.filter(filterInfo.filterer(_, filterVal)))
.getOrElse(currentResult)
}
}
USAGE
val filterInfos = List(
FilterInfo("source", (result, filterVal) => result.source == filterVal),
FilterInfo("category", (result, filterVal) => result.category == filterVal))
val res = List(Result("a","a"), Result("b", "b"))
reduce(filterInfos, res)
Used in your example it would be more like this:
get("/MostClicked") { request =>
val res = MongoDbOps.findMostClicked()
val filterInfos = List(
FilterInfo("source", (result, filterVal) => result.source == filterVal),
FilterInfo("category", (result, filterVal) => result.category == filterVal))
val finalResult = reduce(filterInfos, res)
render.plain {
finalResult.toJson.prettyPrint
}.toFuture
}

What is best way to wrap blocking Try[T] in Future[T] in Scala?

Here is the problem, I have a library which has a blocking method return Try[T]. But since it's a blocking one, I would like to make it non-blocking using Future[T]. In the future block, I also would like to compute something that's depend on the origin blocking method's return value.
But if I use something like below, then my nonBlocking will return Future[Try[T]] which is less convince since Future[T] could represent Failure[U] already, I would rather prefer propagate the exception to Future[T] is self.
def blockMethod(x: Int): Try[Int] = Try {
// Some long operation to get an Int from network or IO
throw new Exception("Network Exception") }
}
def nonBlocking(x: Int): Future[Try[Int]] = future {
blockMethod(x).map(_ * 2)
}
Here is what I tried, I just use .get method in future {} block, but I'm not sure if this is the best way to do that.
def blockMethod(x: Int): Try[Int] = Try {
// Some long operation to get an Int from network or IO
throw new Exception("Network Exception") }
}
def nonBlocking(x: Int): Future[Int] = future {
blockMethod(x).get * 2
}
Is this correct way to do that? Or there is a more scala idiomatic way to convert t Try[T] to Future[T]?
Here's an example that doesn't block, note that you probably want to use your own execution context and not scala's global context:
import scala.util._
import scala.concurrent._
import scala.concurrent.duration._
import ExecutionContext.Implicits.global
object Main extends App {
def blockMethod(x: Int): Try[Int] = Try {
// Some long operation to get an Int from network or IO
Thread.sleep(10000)
100
}
def tryToFuture[A](t: => Try[A]): Future[A] = {
future {
t
}.flatMap {
case Success(s) => Future.successful(s)
case Failure(fail) => Future.failed(fail)
}
}
// Initiate long operation
val f = tryToFuture(blockMethod(1))
println("Waiting... 10 seconds to complete")
// Should return before 20 seconds...
val res = Await.result(f, 20 seconds)
println(res) // prints 100
}
In my opinion: Try & Future is a monadic construction and idiomatic way to is monadic composition (for-comprehension):
That you need to define monad transformer for Future[Try[_]] (code for your library):
case class FutureT[R](run : Future[Try[R]])(implicit e: ExecutionContext) {
def map[B](f : R => B): FutureT[B] = FutureT(run map { _ map f })
def flatMap[B](f : R => FutureT[B]): FutureT[B] = {
val p = Promise[Try[B]]()
run onComplete {
case Failure(e) => p failure e
case Success(Failure(e)) => p failure e
case Success(Success(v)) => f(v).run onComplete {
case Failure(e) => p failure e
case Success(s) => p success s
}
}
FutureT(p.future)
}
}
object FutureT {
def futureTry[R](run : => Try[R])(implicit e: ExecutionContext) =
new FutureT(future { run })
implicit def toFutureT[R](run : Future[Try[R]]) = FutureT(run)
implicit def fromFutureT[R](futureT : FutureT[R]) = futureT.run
}
and usage example:
def blockMethod(x: Int): Try[Int] = Try {
Thread.sleep(5000)
if(x < 10) throw new IllegalArgumentException
else x + 1
}
import FutureT._
// idiomatic way :)
val async = for {
x <- futureTry { blockMethod(15) }
y <- futureTry { blockMethod(25) }
} yield (x + y) * 2 // possible due to using modan transformer
println("Waiting... 10 seconds to complete")
val res = Await.result(async, 20 seconds)
println(res)
// example with Exception
val asyncWithError = for {
x <- futureTry { blockMethod(5) }
y <- futureTry { blockMethod(25) }
} yield (x + y) * 2 // possible due to using modan transformer
// Can't use Await because will get exception
// when extract value from FutureT(Failure(java.lang.IllegalArgumentException))
// no difference between Failure produced by Future or Try
asyncWithError onComplete {
case Failure(e) => println(s"Got exception: $e.msg")
case Success(res) => println(res)
}
// Output:
// Got exception: java.lang.IllegalArgumentException.msg