I have the following method that I would like to apply fold operation on:
def rec(id: String, elems: Seq[(String, MyCase)]) = {
elems.fold(Seq.empty[(String, Seq[String])] { elem =>
....
}
}
What I do not get is the type of the elem is Nothing and I do not understand why it should be! Any clues?
You are missing the closing parentheses before the {, that's why your IDE, probably, thinks, the type is Nothing.
Also, you are, probably, looking for foldLeft, not fold (the first parameter of the latter must match the type of elements of the sequence).
Now the (simplified) signature of .foldLeft on Seq[A] is:
foldLeft[B](b: B)(f: (B,A) => B)
As you can see, it takes a function, that transforms a Tuple2 into the type of the first parameter. The first element of the tuple has the same type as the first param, the second element is the same type as the elements of the sequence.
In your example, B is Seq[(String, Seq[String])], and the sequence elements are (String, MyCase). The type of input to the function would therefore take a horribly looking tuple like this:
(Seq[(String, Seq[String])], (String, MyCase))
This is caused by you are not only want to fold, you also you want to map the tuple, see the fold method signature:
def fold[A1 >: A](z: A1)(op: (A1, A1) => A1): A1 = foldLeft(z)(op)
The input type and outtype must be same: A1,
so if want to map, you maybe want to try foldLeft:
def foldLeft[B](z: B)(op: (B, A) => B): B =
There is a generics for output type B without bounding to A.
also we can find the fold source code is calling:
def fold[A1 >: A](z: A1)(op: (A1, A1) => A1): A1 = foldLeft(z)(op)
use foldLeft maybe like:
elements.fold(Seq.empty[(String, Seq[String])])((a, b) => a ++ Seq((b._1, Seq[String]())))
2nd EDIT:
I'm an idiot. A fold must return the same value as the input type.
Replace all the below folds with foldLefts for example and it works.
You can't transform the data type in a standard fold.
It was compiling for me but I didn't notice the return type was useless.
The second parameter in a fold is a function:
(ResultType, SingleElement) => Result Type
In this case
(Seq[(String, Seq[String])], Seq[(String, MyCase)]) => Seq[(String, Seq[String])]
Your code only has one input on the second parameter so the compiler doesn't know what it is. So it should look something like:
elems.foldLeft(Seq.empty[(String, Seq[MyCase])] {(zeroSeq, nextElem) =>
//zeroSeq: Seq.empty[(String, Seq[MyCase]
//nextElem: (String, MyCase)
}
EDIT:
The following for example compiles:
case class MyCase(x: String)
val elems: Seq[(String, MyCase)] = Seq(("Hi", MyCase("B")))
elems.foldLeft(Seq.empty[(String, Seq[MyCase])]){(z, s) =>
z
}
Related
2 different examples, the first one works:
import cats.syntax.either._
val e = 10.asRight[String]
def i2s(i:Int):String = i.toString
e.map(i => List(i2s(i))) //using explicit parameter
e.map(List(i2s(_))) //using no-name _ parameter
Now the same example with Option is not compiled:
e.map(Option(i2s(_)))
The error:
Error:(27, 15) type mismatch;
found : Option[Int => String]
required: Int => ?
e.map(Option(i2s(_)))
With explicit parameter it works fine:
e.map(i => Option(i2s(i)))
In both cases apply method is invoked with List and Option. List.apply signature:
def apply[A](xs: A*): List[A] = ???
Option.apply signature:
def apply[A](x: A): Option[A]
Please explain the difference.
Both of your List examples compile but they don't mean the same thing and don't produce the same results.
e.map(i => List(i2s(i))) //res0: scala.util.Either[String,List[String]] = Right(List(10))
e.map(List(i2s(_))) //java.lang.IndexOutOfBoundsException: 10
The 1st is easy to understand, so what's going on with the 2nd?
What's happening is that you're using eta expansion to create an Int => String function from the i2s() method. You then populate a List with that single function as the only element in the list, and then try to retrieve the value at index 10, which doesn't exist, thus the exception.
If you change the 1st line to val e = 0.asRight[String] then the exception goes away because something does exist at index 0, the function that was just put in there.
This compiles because a List instance will accept an Int as a parameter (via the hidden apply() method), but an Option instance does not have an apply() method that takes an Int (*) so that can't be compiled.
(*) The Option object does have an apply() method, but that's a different animal.
There are multiple things at play here as to why your first example with List[A] works. First, let's look at the expansion that happens on the expression:
val res: Either[String, Int => String] =
e.map[Int => String](List.apply[Int => String](((x$1: Int) => FlinkTest.this.i2s(x$1))));
Notice two things:
The expansion of the lambda expression happens inside List.apply, and perhaps not as you expected, for it to be outside of List.apply, like this:
e.map(i => List(i2s(i))
The return type from .map is somehow not Either[String, List[Int => String]], but Either[String, Int => String]. This is due to the fact that in it's hierarchy chain, List[A] extends PartialFunction[Int, A], thus allowing it to transform the result into a function type.
This doesn't work for Option[A], as it doesn't extend PartialFunction anywhere in it's type hierarchy.
The key takeaway here is that the expansion of the lambda expression doesn't work as you expect, as List(i2s(_)) expands to List(i2s(x => i2s(x)) and not List(i => i2s(i)). For more on underscore expansion, see What are all the uses of an underscore in Scala?
For the following method series of transforms the Intellij IDE seems to "understand" what is going on: it does not note any errors/warnings and correctly shows the data types:
val recsWithNames = recsWithNamesAndCnts.map(_._1)
.reduceLeft{ case (dfCum: DataFrame,dfNew: DataFrame) => dfCum.union(dfNew)}
Here is the type inference:
However it does not compile:
Error:(426, 68) missing parameter type for expanded function
The argument types of an anonymous function must be fully known. (SLS 8.5)
Expected type was: (?, org.apache.spark.sql.DataFrame) => ?
val recsWithNames = recsWithNamesAndCnts.map(_._1).reduceLeft{ case (dfCum: DataFrame,dfNew: DataFrame) => dfCum.union(dfNew)}
This has been a bit of a pattern with reduceLeft so I typically end up converting to foldLeft. In particular the following does work:
val dfs = recsWithNamesAndCnts.map(_._1)
val recsWithNames = dfs.tail.foldLeft(dfs.head){ case (dfCum: DataFrame,dfNew: DataFrame) => dfCum.union(dfNew)}
But maybe in this case someone might provide some insight on the particular nuances of reduceLeft leading to this error.
reduceLeft and foldLeft have the following signatures:
def reduceLeft[B >: A](op: (B, A) => B): B
def foldLeft[B](z: B)(op: (B, A) => B): B
Both involve an op that takes a Function2, hence both will work fine without the case match:
(dfCum: DataFrame, dfNew: DataFrame) => dfCum.union(dfNew)
// Or, shorthanded to:
_ union _
On the other hand, case (dfCum, dfNew) => dfCum.union(dfNew) is a Function1 (in particular, a partial function of Tuple2). The compiler is able to interpret it and infer type B in foldLeft, but not in reduceLeft (my guess is due to B >: A). It'll work if you help the compiler a litte:
reduceLeft[DataFrame]{ case (dfCum, dfNew) => dfCum.union(dfNew) }
I took the scala odersky course and thought that the function that Flatmap takes as arguments , takes an element of Monad and returns a monad of different type.
trait M[T] {
def flatMap[U](f: T => M[U]): M[U]
}
On Monad M[T] , the return type of function is also the same Monad , the type parameter U might be different.
However I have seen examples on internet , where the function returns a completely different Monad. I was under impression that return type of function should the same Monad. Can someone simplify the below to explain how flapmap results in the actual value instead of Option in the list.
Is the List not a Monad in Scala.
val l= List(1,2,3,4,5)
def f(x:int) = if (x>2) Some(x) else None
l.map(x=>f(x))
//Result List[Option[Int]] = List( None , None , Some(3) , Some(4) , Some(5))
l.flatMap(x=>f(x))
//Result: List(3,4,5)
Let's start from the fact that M[T]is not a monad by itself. It's a type constructor. It becomes a monad when it's associated with two operators: bind and return (or unit). There are also monad laws these operators must satisfy, but let's omit them for brevity. In Haskell the type of bind is:
class Monad m where
...
(>>=) :: m a -> (a -> m b) -> m b
where m is a type constructor. Since Scala is OO language bind will look like (first argument is self):
trait M[T] {
def bind[U](f: T => M[U]): M[U]
}
Here M === m, T === a, U === b. bind is often called flatMap. In a pure spherical world in a vacuum that would be a signature of flatMap in OO language. Scala is a very practical language, so the real signature of flatMap for List is:
final def flatMap[B, That](f: (A) ⇒ GenTraversableOnce[B])(implicit bf: CanBuildFrom[List[A], B, That]): That
It's not bind, but will work as a monadic bind if you provide f in the form of (A) => List[B] and also make sure that That is List[B]. On the other hand Scala is not going to watch your back if you provide something different, but will try to find some meaningful conversion (e.g. CanBuildFrom or something else) if it exists.
UPDATE
You can play with scalac flags (-Xlog-implicits, -Xlog-implicit-conversions) to see what's happening:
scala> List(1).flatMap { x => Some(x) }
<console>:1: inferred view from Some[Int] to scala.collection.GenTraversableOnce[?] via scala.this.Option.option2Iterable[Int]: (xo: Option[Int])Iterable[Int]
List(1).flatMap { x => Some(x) }
^
res1: List[Int] = List(1)
Hmm, perhaps confusingly, the signature you gave is not actually correct, since it's really (in simplified form):
def flatMap[B](f: (A) ⇒ GenTraversableOnce[B]): Traversable[B]
Since the compiler is open-source, you can actually see what it's doing (with its full signature):
def flatMap[B, That](f: A => GenTraversableOnce[B])(implicit bf: CanBuildFrom[Repr, B, That]): That = {
def builder = bf(repr) // extracted to keep method size under 35 bytes, so that it can be JIT-inlined
val b = builder
for (x <- this) b ++= f(x).seq
b.result
}
So you can see that there is actually no requirement that the return type of f be the same as the return type of flatMap.
The flatmap found in the standard library is a much more general and flexible method than the monadic bind method like the flatMap from Odersky's example.
For example, the full signature of flatmap on List is
def flatMap[B, That](f: (A) ⇒ GenTraversableOnce[B])(implicit bf: CanBuildFrom[List[A], B, That]): That
Instead of requiring the function passed into flatmap to return a List, it is able to return any GenTraversableOnce object, a very generic type.
flatmap then uses the implicit CanBuildFrom mechanism to determine the appropriate type to return.
So when you use flatmap with a function that returns a List, it is a monadic bind operation, but it lets you use other types as well.
The following bind(>>=) code, in Haskell, does not compile:
ghci> [[1]] >>= Just
<interactive>:38:11:
Couldn't match type ‘Maybe’ with ‘[]’
Expected type: [t] -> [[t]]
Actual type: [t] -> Maybe [t]
In the second argument of ‘(>>=)’, namely ‘Just’
In the expression: [[1]] >>= Just
But, in Scala, it does actually compile and run:
scala> List( List(1) ).flatMap(x => Some(x) )
res1: List[List[Int]] = List(List(1))
Haskell's >>= signature is:
>>= :: Monad m => m a -> (a -> m b) -> m b
So, in [[1]] >>= f, f's type should be: a -> [b].
Why does the Scala code compile?
As #chi explained Scala's flatMap is more general than the Haskell's >>=. The full signature from the Scala docs is:
final def flatMap[B, That](f: (A) ⇒ GenTraversableOnce[B])(implicit bf: CanBuildFrom[List[A], B, That]): That
This implicit isn't relevant for this specific problem, so we could as well use the simpler definition:
final def flatMap[B](f: (A) ⇒ GenTraversableOnce[B]): List[B]
There is only one Problem, Option is no subclass of GenTraversableOnce, here an implicit conversion comes in. Scala defines an implicit conversion from Option to Iterable which is a subclass of Traversable which is a subclass of GenTraversableOnce.
implicit def option2Iterable[A](xo: Option[A]): Iterable[A]
The implicit is defined in the companion object of Option.
A simpler way to see the implicit at work is to assign a Option to an Iterable val:
scala> val i:Iterable[Int] = Some(1)
i: Iterable[Int] = List(1)
Scala uses some defaulting rules, to select List as the implementation of Iterable.
The fact that you can combine different subtypes of TraversableOnce with monad operations comes from the implicit class MonadOps:
implicit class MonadOps[+A](trav: TraversableOnce[A]) {
def map[B](f: A => B): TraversableOnce[B] = trav.toIterator map f
def flatMap[B](f: A => GenTraversableOnce[B]): TraversableOnce[B] = trav.toIterator flatMap f
def withFilter(p: A => Boolean) = trav.toIterator filter p
def filter(p: A => Boolean): TraversableOnce[A] = withFilter(p)
}
This enhances every TraversableOnce with the methods above. The subtypes are free to define more efficient versions on there own, these will shadow the implicit definitions. This is the case for List.
Quoting from the Scala reference for List
final def flatMap[B](f: (A) ⇒ GenTraversableOnce[B]): List[B]
So, flatMap is more general than Haskell's (>>=), since it only requires the mapped function f to generate a traversable type, not necessarily a List.
I'm experimenting with currying/tupled as an alternative means to pass parameters to a function. In this journey, I'm facing some type difficulties.
I'd like to use implicit conversions to transform a tuple of parameters to the target type. This is what I expect:
Given (REPL example):
case class Cont(value:String) // a simplified container class
val paramTuple = (Cont("One"),Cont("2"))
def operate(s:String, i:Int): String = s+i // my target operation
implicit def contToString(c:Cont): String = c.value
implicit def contToInt(c:Cont): Int = c.value.toInt
//This works - obviously
operate(paramTuple._1, paramTuple._2)
>res4: String = One2
//This is what I want, but doesn't work
(operate _).tupled(params)
<console>:14: error: type mismatch;
found : (Cont, Cont)
required: (String, Int)
Q1: Why the tuple conversion doesn't work? (I'm expecting that an answer to that goes in the lines of: What's required is an implicit conversion between Tuple2[Cont,Cont] to Tuple2[String,Int] but that does not scale up)
Q2: What are my options to have such conversion working?
Keep in mind that I'd like to have a solution for an N-tuple, so defining a specific Tuple2 to Tuple2 is not really a good solution in this case. AFAIK, the magnet pattern works similarly, so I hope there's a way.
Q1: Why the tuple conversion doesn't work? (I'm expecting that an answer to that goes in the lines of: What's required is an implicit conversion between Tuple2[Cont,Cont] to Tuple2[String,Int] but that does not scale up)
Yes, you've got that right.
What are my options to have such conversion working?
You could do it this way:
implicit def liftImplicitTuple2[A, B, A1, B1](tuple: (A, B))
(implicit f1: A => A1, f2: B => B1): (A1, B1) =
(f1(tuple._1), f2(tuple._2))
implicit def liftImplicitTuple3[A, B, C, A1, B1, C1](tuple: (A, B, C))
(implicit f1: A => A1, f2: B => B1, f3: C => C1): (A1, B1, C1) =
(f1(tuple._1), f2(tuple._2), f3(tuple._3))
// etc for tuples as large as you need