Related
I have multiple map functions running over the same data and I'd like to have them run in a single pass. I'm looking for a generic way to do this.
val fruits: Seq[String] = Seq("apple", "banana", "cherry")
def mapF(s: String): Char = s.head
def reduceF(c1: Char, c2: Char): Char = if(c1 > c2) c1 else c2
def mapG(s: String): Int = s.length
def reduceG(i1: Int, i2: Int): Int = i1 + i2
val largestStartingChar = fruits.map(mapF).reduce(reduceF)
val totalStringLength = fruits.map(mapG).reduce(reduceG)
I'd like to reduce the number of passes over fruits. I can make this generic for two maps and reduces like this:
def productMapFunction[A, B, C](f: A=>B, g: A=>C): A => (B, C) = {
x => (f(x), g(x))
}
def productReduceFunction[T, U](f: (T, T)=>T, g: (U, U) => U):
((T,U), (T,U)) => (T, U) = {
(tu1, tu2) => (f(tu1._1, tu2._1), g(tu1._2, tu2._2))
}
val xMapFG = productMapFunction(mapF, mapG)
val xReduceFG = productReduceFunction(reduceF, reduceG)
val (largestStartingChar2, totalStringLength2) =
fruits.map(xMapFG).reduce(xReduceFG))
I'd like to do this even more generically, with an arbitrary number of map and reduce functions, but I'm not sure how to proceed, or if this is possible.
Following solution uses Cats 2 and custom type MapReduce.
Reducing operation can be specified by function reduce: (O, O) => O
or cats reducer: Semigroup[O].
Multiple MapReduce objects can be combined into one by Apply instance provided by implicit def mapReduceApply[I]
import cats._
import cats.implicits._
trait MapReduce[I, O] {
type R
def reducer: Semigroup[R]
def map: I => R
def mapResult: R => O
def apply(input: Seq[I]): O = mapResult(input.map(map).reduce(reducer.combine))
}
object MapReduce {
def apply[I, O, _R](_reducer: Semigroup[_R], _map: I => _R, _mapResult: _R => O): MapReduce[I, O] =
new MapReduce[I, O] {
override type R = _R
override def reducer = _reducer
override def map = _map
override def mapResult = _mapResult
}
def apply[I, O](map: I => O)(implicit r: Semigroup[O]): MapReduce[I, O] =
MapReduce[I, O, O](r, map, identity)
def apply[I, O](map: I => O, reduce: (O, O) => O): MapReduce[I, O] = {
val reducer = new Semigroup[O] {
override def combine(x: O, y: O): O = reduce(x, y)
}
MapReduce(map)(reducer)
}
implicit def mapReduceApply[I] =
new Apply[({type F[X] = MapReduce[I, X]})#F] {
override def map[A, B](f: MapReduce[I, A])(fn: A => B): MapReduce[I, B] =
MapReduce(f.reducer, f.map, f.mapResult.andThen(fn))
override def ap[A, B](ff: MapReduce[I, (A) => B])(fa: MapReduce[I, A]): MapReduce[I, B] =
MapReduce(ff.reducer product fa.reducer,
i => (ff.map(i), fa.map(i)),
(t: (ff.R, fa.R)) => ff.mapResult(t._1)(fa.mapResult(t._2))
)
}
}
object MultiMapReduce extends App {
val fruits: Seq[String] = Seq("apple", "banana", "cherry")
def mapF(s: String): Char = s.head
def reduceF(c1: Char, c2: Char): Char = if (c1 > c2) c1 else c2
val biggestFirsChar = MapReduce(mapF, reduceF)
val totalChars = MapReduce[String, Int](_.length) // (Semigroup[Int]) reduce by _ + _
def count[A] = MapReduce[A, Int](_ => 1)
val multiMapReduce = (biggestFirsChar, totalChars, count[String]).mapN((_, _, _))
println(multiMapReduce(fruits))
val sum = MapReduce[Double, Double](identity)
val average = (sum, count[Double]).mapN(_ / _)
println(sum(List(1, 2, 3, 4)))
println(average(List(1, 2, 3, 4)))
}
Runnable version is also available on GitHub.
Interesting question!
I don't know of any such implementation in the standard library or even scalaz/cats.
It's not very surprising because if your list is not very large you can just perform map-reduces sequentially and I'm not even sure that overhead of constructing lots of intermediate objects would be smaller than overhead of traversing the list several times.
And if the list is potentially doesn't fit into memory you should be using one of the streaming libraries (fs2/zio-streams/akka-streams)
Although if your input was Iterator instead of List, such functionality would be useful.
There is an interesting article about this problem:
https://softwaremill.com/beautiful-folds-in-scala/
tldr:
Map-reduce workflow could be formalized as follows:
trait Fold[I, O] {
type M
def m: Monoid[M]
def tally: I => M
def summarize: M => O
}
In your case I = List[A], tally = list => list.map(mapF), summarize = list => list.reduce(reduceF) .
To run a map-reduce on a list using an instance of fold you need to run
fold.summarize(fold.tally(list))
You can define combine operation on them:
def combine[I, O1, O2](f1: Fold[I, O1], f2: Fold[I, O2]): Fold[I, (O1, O2)]
Using combine few times would give you what you want:
combine(combine(f1, f2), f3): Fold[I, ((O1, O2), O3)]
I think you're just trying to reinvent transducers. It's been a while since I have used Scala, but there's at least one implementation.
I'm playing around with implementing a tagless final DSL & interpreter in Scala, based on this blog post written in Haskell.
I can get an example running - see code below, but I don't quite understand why I need testVal(Interp)(Interp). If I only supply a single Interp argument, then I get the following compile errors:
Error:(29, 24) could not find implicit value for evidence parameter of type Test.Expr[Test.Id]
val x = testVal(Interp)
Error:(29, 24) not enough arguments for method testVal: (implicit evidence$1: Test.Expr[Test.Id])Test.Id[Int].
Unspecified value parameter evidence$1.
val x = testVal(Interp)
Is there a simple way to eliminate one of the Interp arguments?
object Test {
trait Expr[F[_]] {
def const(i: Int): F[Int]
def lam[A, B](f: F[A] => F[B]): F[A => B]
def app[A, B](f: F[A => B], a: F[A]): F[B]
def add(x: F[Int], y: F[Int]): F[Int]
}
type Id[A] = A
object Interp extends Expr[Id] {
override def const(i: Int): Id[Int] = i
override def lam[A, B](f: Id[A] => Id[B]): Id[A => B] = f
override def app[A, B](f: Id[A => B], a: Id[A]): Id[B] = f(a)
override def add(x: Id[Int], y: Id[Int]): Id[Int] = x + y
}
def testVal[F[_]: Expr](f: Expr[F]): F[Int] =
f.app(
f.lam[Int, Int](
x => f.add(x, f.const(1))),
f.const(10)
)
def main(args: Array[String]): Unit = {
// val x = testVal(Interp) -- won't compile
val x = testVal(Interp)(Interp)
println(x)
}
}
The syntax
def f[X: Y](args: Types): Res = { ... }
is a shortcut for
def f[X](args: Types)(implicit yx: Y[X]): Res = { ... }
so if you write
def testVal[F[_]: Expr](f: Expr[F]): F[Int] = { ... }
then it's the same as if you wrote
def testVal[F[_]](f: Expr[F])(implicit redundant: Expr[F]): F[Int] = { ... }
but you obviously don't need the same Expr[F] twice.
The signature should be either
def testVal[F[_]: Expr]: F[Int]
or
def testVal[F[_]](implicit f: Expr[F]): F[Int]
but not both at the same time.
Here is a full example, which also shows how to get the f using implicitly in the case that you decide to use the F: Expr variant (which does not assign a name to the implicit argument):
import scala.language.higherKinds
object Test {
trait Expr[F[_]] {
def const(i: Int): F[Int]
def lam[A, B](f: F[A] => F[B]): F[A => B]
def app[A, B](f: F[A => B], a: F[A]): F[B]
def add(x: F[Int], y: F[Int]): F[Int]
}
type Id[A] = A
object Interp extends Expr[Id] {
override def const(i: Int): Id[Int] = i
override def lam[A, B](f: Id[A] => Id[B]): Id[A => B] = f
override def app[A, B](f: Id[A => B], a: Id[A]): Id[B] = f(a)
override def add(x: Id[Int], y: Id[Int]): Id[Int] = x + y
}
def testVal[F[_]: Expr]: F[Int] = {
implicit val f = implicitly[Expr[F]]
f.app(
f.lam[Int, Int](
x => f.add(x, f.const(1))),
f.const(10)
)
}
def main(args: Array[String]): Unit = {
val x = testVal(Interp)
println(x)
}
}
Moreover, if you make Interp itself implicit, then you can omit all argument lists when invoking testVal, and instead write just
val x = testVal // no arguments at all.
I know I can traverse Lists
import cats.instances.list._
import cats.syntax.traverse._
def doMagic(item: A): M[B] = ???
val list: List[A] = ???
val result: M[List[B]] = list.traverse(doMagic)
And I can convert a Seq back and forth to List
val seq: Seq[A] = ???
val result: M[Seq[B]] = seq.toList.traverse(doMagic).map(_.toSeq)
But can I also traverse Seq without the boilerplate?
val seq: Seq[A] = ???
val result: M[Seq[B]] = seq.traverse(doMagic)
Or what's an easy way to get an instance of Traverse[Seq]?
Cats does not provide typeclass instances for Seq, so besides implementing it yourself you're stuck with the conversion.
As to why, there's an ongoing discussion in an (somewhat old) Cats issue. To sum it up, you won't know enough about Seq underlying characteristics to make sure some of the typeclasses instances laws hold.
EDIT : Nevermind, it exists now, see linked thread
As of cats 2.3, support for immutable.Seq is now built in. See "Where are implicit instances for Seq?" on the FAQ or this PR where the functionality was added.
If you are absolutely sure that the conversion from all Seq to List will always succeed in your code, you can simply transfer the Traverse structure from List to Seq over an (pseudo-)isomorphism:
def traverseFromIso[F[_], Z[_]]
(forward: F ~> Z, inverse: Z ~> F)
(implicit zt: Traverse[Z])
: Traverse[F] = new Traverse[F] {
def foldLeft[A, B](fa: F[A], b: B)(f: (B, A) ⇒ B): B = zt.foldLeft(forward(fa), b)(f)
def foldRight[A, B](fa: F[A], lb: Eval[B])(f: (A, Eval[B]) => Eval[B]): Eval[B] =
zt.foldRight(forward(fa), lb)(f)
def traverse[G[_], A, B]
(fa: F[A])
(f: (A) ⇒ G[B])
(implicit appG: Applicative[G])
: G[F[B]] = {
(zt.traverse(forward(fa))(f)(appG)).map(zb => inverse(zb))
}
}
This isn't really an isomorphism, because the conversion from Seq to List can fail badly (e.g. if the sequence is infinite). What it does is simply converting Seq to List back and forth, and forwarding all method calls to those of Traverse[List].
Now you can use this method to build an instance of Traverse[Seq]:
implicit val seqTraverse: Traverse[Seq] = traverseFromIso(
new FunctionK[Seq, List] { def apply[X](sx: Seq[X]): List[X] = sx.toList },
new FunctionK[List, Seq] { def apply[X](lx: List[X]): Seq[X] = lx }
)
Full code snippet (compiles with scala 2.12.4 and cats 1.0.1):
import cats._
import cats.implicits._
import cats.arrow.FunctionK
import scala.language.higherKinds
object TraverseFromIso {
// This method can build you a `Traversable[Seq]` from
// an `Traversable[List]` and a pair of polymorphic conversion
// functions:
def traverseFromIso[F[_], Z[_]]
(forward: F ~> Z, inverse: Z ~> F)
(implicit zt: Traverse[Z])
: Traverse[F] = new Traverse[F] {
def foldLeft[A, B](fa: F[A], b: B)(f: (B, A) ⇒ B): B = zt.foldLeft(forward(fa), b)(f)
def foldRight[A, B](fa: F[A], lb: Eval[B])(f: (A, Eval[B]) => Eval[B]): Eval[B] =
zt.foldRight(forward(fa), lb)(f)
def traverse[G[_], A, B]
(fa: F[A])
(f: (A) ⇒ G[B])
(implicit appG: Applicative[G])
: G[F[B]] = {
(zt.traverse(forward(fa))(f)(appG)).map(zb => inverse(zb))
}
}
// A little demo
def main(args: Array[String]): Unit = {
// To instantiate a `Traverse[Seq]`, we have to provide
// two natural transformations (from List to Seq and back):
implicit val seqTraverse: Traverse[Seq] = traverseFromIso(
new FunctionK[Seq, List] { def apply[X](sx: Seq[X]): List[X] = sx.toList },
new FunctionK[List, Seq] { def apply[X](lx: List[X]): Seq[X] = lx }
)
// do stuff with `Traversable[Seq]` here
}
}
we are creating our own OptionT of cats for understanding, how monads are work and monads transformation flow. While creating our own custom monad getting some of the errors. First thing below is our code:
case class WhateverOpt[W[_], A] (value: W[Option[A]]) {
def map[B] (f: A => B) (implicit M: Monad[W]): WhateverOpt[W, B] =
WhateverOpt(M.map(value)(_.map(f)))
def flatMap[B] (f: A => WhateverOpt[W, B]) (implicit M: Monad[W]): WhateverOpt[W, B] =
WhateverOpt(M.flatMap(value)(optA => optA match {
case Some(v) => f(v).value
}))
}
implicit val optionTMonad = new Monad[Option] {
override def map[A, B](fa: Option[A])(f: A => B): Option[B] = fa.map(f)
override def flatMap[A, B](fa: Option[A])(f: A => Option[B]): Option[B] = fa.flatMap(f)
}
val optionResult = for {
user <- WhateverOpt(repository.getUserOption(1))
addres <- WhateverOpt(repository.getAddressOption(user))
} yield addres.city
Below are the points, where we stuck:
How to handle None case in WhateverOpt flatMap method?
When executing the code, getting runtime error:
Error:(26, 12) could not find implicit value for parameter M: usercases.mtransfomer.Monad[scala.concurrent.Future]
addres <- WhateverOpt(repository.getAddressOption(user))
We are not sure about the error because we are creating optionTMonad implicit and by default, all are in the same scope. How can we resolve these two issues?
Update
Full code is available on Github branch https://github.com/harmeetsingh0013/fp_scala/blob/master/src/main/scala/usercases/mtransfomer/Example5.scala
About how to deal with None:
case class WhateverOpt[W[_], A] (value: W[Option[A]]) {
def map[B] (f: A => B) (implicit M: Monad[W]): WhateverOpt[W, B] =
WhateverOpt(M.map(value)(_.map(f)))
def flatMap[B]
(f: A => WhateverOpt[W, B])
(implicit wMonad: Monad[W])
: WhateverOpt[W, B] = {
WhateverOpt(wMonad.flatMap(value) { (oa: Option[A]) =>
oa match {
case None => wMonad.pure(None)
case Some(a) => f(a).value
}
})
}
}
Imagine for a second that W is Future. Then the above code says:
wait until the wrapped value yields a result oa of type Option[A]
If oa turns out to be None, then there is nothing we can do, because we cannot obtain any instances of type A in order to call f. Therefore, immediately return None. The immediately return is the pure-method of the Future-monad, so for the general case we have to invoke wMonad.pure(None).
If oa yields Some(a), we can give this a to f, and then immediately unpack it to get to the value of type W[Option[B]].
Once we have the W[Option[B]] (whether empty or not), we can wrap it into WhateverOpt and return from the flatMap method.
I assume that you wanted to reimplement Monad[Option] just for fun (it's already in the library (the catsStdInstancesForOption thing is a CommutativeMonad), but here is how you could re-build it:
implicit val optionTMonad = new Monad[Option] {
override def map[A, B](fa: Option[A])(f: A => B): Option[B] = fa.map(f)
def flatMap[A, B](fa: Option[A])(f: A => Option[B]): Option[B] = fa.flatMap(f)
def pure[A](a: A): Option[A] = Some(a)
def tailRecM[A, B](a: A)(f: (A) => Option[Either[A, B]]): Option[B] = {
f(a) match {
case Some(Left(nextA)) => tailRecM(nextA)(f)
case Some(Right(res)) => Some(res)
case None => None
}
}
}
Notice that 1.0.1 requires to implement pure and tailRecM, and does not provide default implementations for that.
I don't want to say much about the necessary imports for future, but the latest version has cats.instances.future which provides a Monad instance. Check this again, because it seems as if you are using a different version of cats (your version didn't complain because of the missing tailRecM in your Option-monad).
How to handle None case in WhateverOpt flatMap method?
This answer is already explained by #Gabriele Petronella and #Andrey Tyukin with details.
When executing the code, getting runtime error: Error:(26, 12) could
not find implicit value for parameter M:
usercases.mtransfomer.Monad[scala.concurrent.Future] addres <-
WhateverOpt(repository.getAddressOption(user))
This error occurs, because In WhateverOpt constructor we know that our value is W[Option[A]], where Option is already defined and handle by the code, but repository.getUserOption(1) return Future[Option[User]] where Future is handled by generic parameter W and in that, case, we need to define, how to handle monads for Future. For resolving that issue, we need to implement new Monad[Future] rather than, new Monad[Option] as below:
case class WhateverOpt[W[_], A] (value: W[Option[A]]) {
def map[B] (f: A => B) (implicit M: Monad[W]): WhateverOpt[W, B] =
WhateverOpt(M.map(value)(_.map(f)))
def flatMap[B] (f: A => WhateverOpt[W, B]) (implicit M: Monad[W]): WhateverOpt[W, B] =
WhateverOpt(M.flatMap(value)(optA => optA match {
case Some(v) => f(v).value
case None => M.pure(None)
}))
}
implicit val futureMonad = new Monad[Future] {
override def pure[A](a: A): Future[A] = Future.successful(a)
override def map[A, B](fa: Future[A])(f: A => B): Future[B] = fa.map(f)
override def flatMap[A, B](fa: Future[A])(f: A => Future[B]): Future[B] = fa.flatMap(f)
}
val optionResult: WhateverOpt[Future, String] = for {
user <- WhateverOpt(repository.getUserOption(1))
addres <- WhateverOpt(repository.getAddressOption(user))
} yield addres.city
I am not sure about my disription, which I mention in answere, but current my assumptions are this and for me above code is working fine. For a complete example, please click on the GitHub repo, which is mention in the question.
How to handle None case in WhateverOpt flatMap method?
When you flatMap over None you return None. When you flatMap over WhateverOpt[W[_], B] you want to return the pure of it, which in your code would be M.pure(None).
When executing the code, getting runtime error: Error:(26, 12) could not find implicit value for parameter M: usercases.mtransfomer.Monad[scala.concurrent.Future]
addres <- WhateverOpt(repository.getAddressOption(user))
That's a compile-time error (not a runtime one) and it's due to the missing instance of Monad[Future]. In order to get an instance of Monad[Future] in scope with cats, you can do:
import cats.instances.future._
import scala.concurrent.ExecutionContext.Implicits.global
Also, you can avoid declaring your own Monad[Option] by importing it from cats with
import cats.instances.option._
I am playing around with the basic Kleisli example
import cats.data.Kleisli
import cats.FlatMap
val parse = Kleisli[Option, String, Int](
(s: String) => try {
Some(s.toInt)
} catch {
case _: NumberFormatException => None
}
)
val reciprocal = Kleisli[Option, Int, Double](
(i: Int) =>
if (i == 0)
None
else
Some(1.0 / i))
val c = reciprocal.compose(parse)
and two things entagle me
why do I need to specify explicitly the types [Option,String,Int]? The example code omits them and it must have worked this way too, I guess...
to evaluate c like c("5") do I really need to give a FlatMap[Option]by hand or is there an implicit import I am missing?
I can add
implicit val optionFlatmap = new FlatMap[Option] {
override def flatMap[A, B](fa: Option[A])(f: (A) => Option[B]): Option[B] = fa.flatMap(f)
override def map[A, B](fa: Option[A])(f: (A) => B): Option[B] = fa.map(f)
}
to evaluate c("5"), but isn't there a default implementation for this thing?