Scala foreach strange behaviour - scala

I want to iterate over a list of values using a beautiful one-liner in Scala.
For example, this one works well:
scala> val x = List(1,2,3,4)
x: List[Int] = List(1, 2, 3, 4)
scala> x foreach println
1
2
3
4
But if I use the placeholder _, it gives me an error:
scala> x foreach println(_ + 1)
<console>:6: error: missing parameter type for expanded function ((x$1) =>x$1.$plus(1))
x foreach println(_ + 1)
^
Why is that? Can't compiler infer type here?

This:
x foreach println(_ + 1)
is equivalent to this:
x.foreach(println(x$1 => x$1 + 1))
There's no indication as to what might be the type of x$1, and, to be honest, it doesn't make any sense to print a function.
You obviously (to me) meant to print x$0 + 1, where x$0 would the the parameter passed by foreach, instead. But, let's consider this... foreach takes, as a parameter, a Function1[T, Unit], where T is the type parameter of the list. What you are passing to foreach instead is println(_ + 1), which is an expression that returns Unit.
If you wrote, instead x foreach println, you'd be passing a completely different thing. You'd be passing the function(*) println, which takes Any and returns Unit, fitting, therefore, the requirements of foreach.
This gets slightly confused because of the rules of expansion of _. It expands to the innermost expression delimiter (parenthesis or curly braces), except if they are in place of a parameter, in which case it means a different thing: partial function application.
To explain this better, look at these examples:
def f(a: Int, b: Int, c: Int) = a + b + c
val g: Int => Int = f(_, 2, 3) // Partial function application
g(1)
Here, we applies the second and third arguments to f, and returned a function requiring just the remaining argument. Note that it only worked as is because I indicated the type of g, otherwise I'd have to indicate the type of the argument I was not applying. Let's continue:
val h: Int => Int = _ + 1 // Anonymous function, expands to (x$1: Int => x$1 + 1)
val i: Int => Int = (_ + 1) // Same thing, because the parenthesis are dropped here
val j: Int => Int = 1 + (_ + 1) // doesn't work, because it expands to 1 + (x$1 => x$1 + 1), so it misses the type of `x$1`
val k: Int => Int = 1 + ((_: Int) + 1) // doesn't work, because it expands to 1 + (x$1: Int => x$1 + 1), so you are adding a function to an `Int`, but this operation doesn't exist
Let discuss k in more detail, because this is a very important point. Recall that g is a function Int => Int, right? So, if I were to type 1 + g, would that make any sense? That's what was done in k.
What confuses people is that what they really wanted was:
val j: Int => Int = x$1 => 1 + (x$1 + 1)
In other words, they want the x$1 replacing _ to jump to outside the parenthesis, and to the proper place. The problem here is that, while it may seem obvious to them what the proper place is, it is not obvious to the compiler. Consider this example, for instance:
def findKeywords(keywords: List[String], sentence: List[String]) = sentence.filter(keywords contains _.map(_.toLowerCase))
Now, if we were to expand this to outside the parenthesis, we would get this:
def findKeywords(keywords: List[String], sentence: List[String]) = (x$1, x$2) => sentence.filter(keywords contains x$1.map(x$2.toLowerCase))
Which is definitely not what we want. In fact, if the _ did not get bounded by the innermost expression delimiter, one could never use _ with nested map, flatMap, filter and foreach.
Now, back to the confusion between anonymous function and partial application, look here:
List(1,2,3,4) foreach println(_) // doesn't work
List(1,2,3,4) foreach (println(_)) // works
List(1,2,3,4) foreach (println(_ + 1)) // doesn't work
The first line doesn't work because of how operation notation works. Scala just sees that println returns Unit, which is not what foreachexpects.
The second line works because the parenthesis let Scala evaluate println(_) as a whole. It is a partial function application, so it returns Any => Unit, which is acceptable.
The third line doesn't work because _ + 1 is anonymous function, which you are passing as a parameter to println. You are not making println part of an anonymous function, which is what you wanted.
Finally, what few people expect:
List(1,2,3,4) foreach (Console println _ + 1)
This works. Why it does is left as an exercise to the reader. :-)
(*) Actually, println is a method. When you write x foreach println, you are not passing a method, because methods can't be passed. Instead, Scala creates a closure and passes it. It expands like this:
x.foreach(new Function1[Any,Unit] { def apply(x$1: Any): Unit = Console.println(x$1) })

The underscore is a bit tricky. According to the spec, the phrase:
_ + 1
is equivalent to
x => x + 1
Trying
x foreach println (y => y + 1)
yields:
<console>:6: error: missing parameter type
x foreach println (y => y + 1)
If you add some types in:
x foreach( println((y:Int) => y + 1))
<console>:6: error: type mismatch;
found : Unit
required: (Int) => Unit
x foreach( println((y:Int) => y + 1))
The problem is that you are passing an anonymous function to println and it's not able to deal with it. What you really want to do (if you are trying to print the successor to each item in the list) is:
x map (_+1) foreach println

scala> for(x <- List(1,2,3,4)) println(x + 1)
2
3
4
5

There is a strange limitation in Scala for the nesting depth of expressions with underscore. It's well seen on the following example:
scala> List(1) map(1+_)
res3: List[Int] = List(2)
scala> Some(1) map (1+(1+_))
<console>:5: error: missing parameter type for expanded function ((x$1) => 1.+(x$1))
Some(1) map (1+(1+_))
^
Looks like a bug for me.

Welcome to Scala version 2.8.0.Beta1-prerelease (Java HotSpot(TM) Client VM, Java 1.6.0_17).
Type in expressions to have them evaluated.
Type :help for more information.
scala> val l1 = List(1, 2, 3)
l1: List[Int] = List(1, 2, 3)
scala>
scala> l1.foreach(println(_))
1
2
3

Related

How is the placeholder different to an explicit parameter in lambda functions? [duplicate]

I want to iterate over a list of values using a beautiful one-liner in Scala.
For example, this one works well:
scala> val x = List(1,2,3,4)
x: List[Int] = List(1, 2, 3, 4)
scala> x foreach println
1
2
3
4
But if I use the placeholder _, it gives me an error:
scala> x foreach println(_ + 1)
<console>:6: error: missing parameter type for expanded function ((x$1) =>x$1.$plus(1))
x foreach println(_ + 1)
^
Why is that? Can't compiler infer type here?
This:
x foreach println(_ + 1)
is equivalent to this:
x.foreach(println(x$1 => x$1 + 1))
There's no indication as to what might be the type of x$1, and, to be honest, it doesn't make any sense to print a function.
You obviously (to me) meant to print x$0 + 1, where x$0 would the the parameter passed by foreach, instead. But, let's consider this... foreach takes, as a parameter, a Function1[T, Unit], where T is the type parameter of the list. What you are passing to foreach instead is println(_ + 1), which is an expression that returns Unit.
If you wrote, instead x foreach println, you'd be passing a completely different thing. You'd be passing the function(*) println, which takes Any and returns Unit, fitting, therefore, the requirements of foreach.
This gets slightly confused because of the rules of expansion of _. It expands to the innermost expression delimiter (parenthesis or curly braces), except if they are in place of a parameter, in which case it means a different thing: partial function application.
To explain this better, look at these examples:
def f(a: Int, b: Int, c: Int) = a + b + c
val g: Int => Int = f(_, 2, 3) // Partial function application
g(1)
Here, we applies the second and third arguments to f, and returned a function requiring just the remaining argument. Note that it only worked as is because I indicated the type of g, otherwise I'd have to indicate the type of the argument I was not applying. Let's continue:
val h: Int => Int = _ + 1 // Anonymous function, expands to (x$1: Int => x$1 + 1)
val i: Int => Int = (_ + 1) // Same thing, because the parenthesis are dropped here
val j: Int => Int = 1 + (_ + 1) // doesn't work, because it expands to 1 + (x$1 => x$1 + 1), so it misses the type of `x$1`
val k: Int => Int = 1 + ((_: Int) + 1) // doesn't work, because it expands to 1 + (x$1: Int => x$1 + 1), so you are adding a function to an `Int`, but this operation doesn't exist
Let discuss k in more detail, because this is a very important point. Recall that g is a function Int => Int, right? So, if I were to type 1 + g, would that make any sense? That's what was done in k.
What confuses people is that what they really wanted was:
val j: Int => Int = x$1 => 1 + (x$1 + 1)
In other words, they want the x$1 replacing _ to jump to outside the parenthesis, and to the proper place. The problem here is that, while it may seem obvious to them what the proper place is, it is not obvious to the compiler. Consider this example, for instance:
def findKeywords(keywords: List[String], sentence: List[String]) = sentence.filter(keywords contains _.map(_.toLowerCase))
Now, if we were to expand this to outside the parenthesis, we would get this:
def findKeywords(keywords: List[String], sentence: List[String]) = (x$1, x$2) => sentence.filter(keywords contains x$1.map(x$2.toLowerCase))
Which is definitely not what we want. In fact, if the _ did not get bounded by the innermost expression delimiter, one could never use _ with nested map, flatMap, filter and foreach.
Now, back to the confusion between anonymous function and partial application, look here:
List(1,2,3,4) foreach println(_) // doesn't work
List(1,2,3,4) foreach (println(_)) // works
List(1,2,3,4) foreach (println(_ + 1)) // doesn't work
The first line doesn't work because of how operation notation works. Scala just sees that println returns Unit, which is not what foreachexpects.
The second line works because the parenthesis let Scala evaluate println(_) as a whole. It is a partial function application, so it returns Any => Unit, which is acceptable.
The third line doesn't work because _ + 1 is anonymous function, which you are passing as a parameter to println. You are not making println part of an anonymous function, which is what you wanted.
Finally, what few people expect:
List(1,2,3,4) foreach (Console println _ + 1)
This works. Why it does is left as an exercise to the reader. :-)
(*) Actually, println is a method. When you write x foreach println, you are not passing a method, because methods can't be passed. Instead, Scala creates a closure and passes it. It expands like this:
x.foreach(new Function1[Any,Unit] { def apply(x$1: Any): Unit = Console.println(x$1) })
The underscore is a bit tricky. According to the spec, the phrase:
_ + 1
is equivalent to
x => x + 1
Trying
x foreach println (y => y + 1)
yields:
<console>:6: error: missing parameter type
x foreach println (y => y + 1)
If you add some types in:
x foreach( println((y:Int) => y + 1))
<console>:6: error: type mismatch;
found : Unit
required: (Int) => Unit
x foreach( println((y:Int) => y + 1))
The problem is that you are passing an anonymous function to println and it's not able to deal with it. What you really want to do (if you are trying to print the successor to each item in the list) is:
x map (_+1) foreach println
scala> for(x <- List(1,2,3,4)) println(x + 1)
2
3
4
5
There is a strange limitation in Scala for the nesting depth of expressions with underscore. It's well seen on the following example:
scala> List(1) map(1+_)
res3: List[Int] = List(2)
scala> Some(1) map (1+(1+_))
<console>:5: error: missing parameter type for expanded function ((x$1) => 1.+(x$1))
Some(1) map (1+(1+_))
^
Looks like a bug for me.
Welcome to Scala version 2.8.0.Beta1-prerelease (Java HotSpot(TM) Client VM, Java 1.6.0_17).
Type in expressions to have them evaluated.
Type :help for more information.
scala> val l1 = List(1, 2, 3)
l1: List[Int] = List(1, 2, 3)
scala>
scala> l1.foreach(println(_))
1
2
3

Why can't a variable be assigned placeholder in function literal?

I'm having trouble understanding underscores in function literals.
val l = List(1,2,3,4,5)
l.filter(_ > 0)
works fine
l.filter({_ > 0})
works fine
l.filter({val x=1; 1+_+3 > 0}) // ie you can have multiple statements in your function literal and use the underscore not just in the first statement.
works fine
And yet:
l.filter({val x=_; x > 0})
e>:1: error: unbound placeholder parameter
l.filter({val x=_; x > 0})
I can't assign the _ to a variable, even though the following is legal function literal:
l.filter(y => {val x=y; x > 0})
works fine.
What gives? Is my 'val x=_' getting interpreted as something else? Thanks!
Actually, you have to back up a step.
You are misunderstanding how the braces work.
scala> val is = (1 to 5).toList
is: List[Int] = List(1, 2, 3, 4, 5)
scala> is map ({ println("hi") ; 2 * _ })
hi
res2: List[Int] = List(2, 4, 6, 8, 10)
If the println were part of the function passed to map, you'd see more greetings.
scala> is map (i => { println("hi") ; 2 * i })
hi
hi
hi
hi
hi
res3: List[Int] = List(2, 4, 6, 8, 10)
Your extra braces are a block, which is some statements followed by a result expression. The result expr is the function.
Once you realize that only the result expr has an expected type that is the function expected by map, you wouldn't think to use underscore in the preceding statements, since a bare underscore needs the expected type to nail down what the underscore means.
That's the type system telling you that your underscore isn't in the right place.
Appendix: in comments you ask:
how can I use the underscore syntax to bind the parameter of a
function literal to a variable
Is this a "dumb" question, pardon the expression?
The underscore is so you don't have to name the parameter, then you say you want to name it.
One use case might be: there are few incoming parameters, but I'm interested in naming only one of them.
scala> (0 /: is)(_ + _)
res10: Int = 15
scala> (0 /: is) { case (acc, i) => acc + 2 * i }
res11: Int = 30
This doesn't work, but one may wonder why. That is, we know what the fold expects, we want to apply something with an arg. Which arg? Whatever is left over after the partially applied partial function.
scala> (0 /: is) (({ case (_, i) => _ + 2 * i })(_))
or
scala> (0 /: is) (({ case (_, i) => val d = 2 * i; _ + 2 * d })(_))
SLS 6.23 "placeholder syntax for anonymous functions" mentions the "expr" boundary for when you must know what the underscore represents -- it's not a scope per se. If you supply type ascriptions for the underscores, it will still complain about the expected type, presumably because type inference goes left to right.
The underscore syntax is mainly user for the following replacement:
coll.filter(x => { x % 2 == 0});
coll.filter(_ % 2 == 0);
This can only replace a single parameter. This is the placeholder syntax.
Simple syntactic sugar for a lambda.
In the breaking case you are attempting null initialization/defaulting.
For primitive types with init conventions:
var x: Int = _; // x will be 0
The general case:
var y: List[String] = _; // y is null
var z: Any = _; // z = null;
To get pedantic, it works because null is a ref to the only instance of scala.Null, a sub-type of any type, which will always satisfy the type bound because of covariance. Look HERE.
A very common usage scenario, in ScalaTest:
class myTest extends FeatureTest with GivenWhenThen with BeforeAndAfter {
var x: OAuthToken = _;
before {
x = someFunctionThatReturnsAToken;
}
}
You can also see why you shouldn't use it with val, since the whole point is to update the value after initialization.
The compiler won't even let you, failing with: error: unbound placeholder parameter.
This is your exact case, the compiler thinks you are defaulting, a behaviour undefined for vals.
Various constraints, such as timing or scope make this useful.
This is different from lazy, where you predefine the expression that will be evaluated when needed.
For more usages of _ in Scala, look HERE.
Because in this two cases underscore (_) means two different things. In case of a function it's a syntactic sugar for lambda function, your l.filter(_ > 0) later desugares into l.filter(x => x > 0). But in case of a var it has another meaning, not a lambda function, but a default value and this behavior is defined only for var's:
class Test {
var num: Int = _
}
Here num gonna be initialized to its default value determined by its type Int. You can't do this with val cause vals are final and if in case of vars you can later assign them some different values, with vals this has no point.
Update
Consider this example:
l filter {
val x = // compute something
val z = _
x == z
}
According to your idea, z should be bound to the first argument, but how scala should understand this, or you you have more code in this computation and then underscore.
Update 2
There is a grate option in scala repl: scala -Xprint:type. If you turn it on and print your code in (l.filter({val x=1; 1+_+3 > 0})), this what you'll see:
private[this] val res1: List[Int] = l.filter({
val x: Int = 1;
((x$1: Int) => 1.+(x$1).+(3).>(0))
});
1+_+3 > 0 desugares into a function: ((x$1: Int) => 1.+(x$1).+(3).>(0)), what filter actually expects from you, a function from Int to Boolean. The following also works:
l.filter({val x=1; val f = 1+(_: Int)+3 > 0; f})
cause f here is a partially applied function from Int to Boolean, but underscore isn't assigned to the first argument, it's desugares to the closes scope:
private[this] val res3: List[Int] = l.filter({
val x: Int = 1;
val f: Int => Boolean = ((x$1: Int) => 1.+((x$1: Int)).+(3).>(0));
f
});

Type mismatch with map and flatMap

While trying to play with Options in scala I have come across this peculiar problem.
I started off with creating a List[Option[Int]] as follows:
scala> List(Some(1),Some(2),None,Some(3))
res0: List[Option[Int]] = List(Some(1), Some(2), None, Some(3))
Then I tried to map an addition to 1 over the entries of the list in res0 as follows:
scala> res0 map (_ + 1)
This gave me the error:
<console>:9: error: type mismatch;
found : Int(1)
required: String
res0 map (_ + 1)
^
Then I tried flatMapping an addition over the entries as follows:
scala> res0 flatMap (_ + 1)
This gave me the same error:
<console>:9: error: type mismatch;
found : Int(1)
required: String
res0 flatMap (_ + 1)
^
But something like res0.flatMap(r => r) works just fine with a result of:
res9: List[Int] = List(1, 2, 3)
Can anybody tell me why adding the entry to 1 would fail for both map and flatMap?
The first two things you tried failed because you are trying to add an Option to an Int, and that's not possible.
The weird error message happens because Scala assumes, since Option doesn't have a + method, that you are trying String concatenation, but you'd have to either add an Option to a String, or a String to an Option, and you are doing neither, hence the error message.
In the last case, you are not trying to add anything, you are simply returning Option as is, hence no error message.
To increment all values that are not None, you need to map also each Option element of the list, like so:
scala> res0.map(_.map(_ + 1))
res1: List[Option[Int]] = List(Some(2), Some(3), None, Some(4))
If you want to filter out the Nones, you would indeed use a flatMap:
scala> res0.flatMap(_.map(_ + 1))
res2: List[Int] = List(2, 3, 4)
Both the function given to flatMap as well as the function given to map take a value of the list's element type - in this case Option[Int]. But your function _ + 1 expects an Int, not an Option[Int], so you can't use it as the argument to either map or flatMap in this case. Further the function given to flatMap should return an iterable¹, but your function would return a number.
This will do what you want: res0 flatMap (_ map (_ + 1)). Here the function given to flatMap takes an Option[Int] and returns an Option[Int] by calling map on the option. flatMap then takes the options returned by the function and concatenates them.
¹ Technically a GenTraversableOnce.
You try to invokee.+(1) on every element e in a List[Option[Int]], but + is not a function declared by Option[_]. String concatenation, however, would be possible (I assume there is an implicit from Any to String), but only if the second argument were a string as well (not sure why the implicit whose existence I assumed isn't considered here).
You can overcome this problem by doing working with a default value as suggested by #korefn, or "hide" the differentiation between Some(x) and None in another invocation of map, namely by
map(_.map(_ + 1))
They are failing because the types are wrong and the compiler correctly states it as such.
The map case fails because map expects a function A => B. In your code, A => B is really Int => Int which will not work because calling map on your list means that A is actually Option[Int].
Furthermore, flatMap expects a function of the form A => F[B]. So you will get your answer if you did res0 flatMap { o => o map { a => a + 1 } }. This basically is the expansion of:
for {
element <- res0 // o above
value <- element // a above
} yield value + 1
Try to use get to extract the value from Some[Int] to Int allowing for calculation value + 1 ie:
res0 map{_.getOrElse(0) + 1}
As pointed out by #Sepp2k you could alternatively use a collect to avoid having a default for None
res0 collect {case Some(x) => x + 1 }

how to simplify scala's function literal like this?

I'm new to scala and trying to write a function literal that check whether a given integer is odd or not.
my first attempt is:
val isOdd = (x:Int) => (x & 1) == 1
it works great, and, since the parameter x only appears once within this function literal, I'm tempted to use the "_" notation to simplify it further, like this:
val isOdd = ((_:Int) & 1 ) == 1
however this time the compiler complains :
warning: comparing a fresh object using `==' will always yield false
val isOdd = ((_:Int) & 1 ) == 1
what does this warning mean? why does the compiler recognize ((_ :Int) & 1) as fresh object rather than a bitwise operation that results in a value? is there any way to write this function literal using the "_" notation?
The problem is basically that Scala needs to tell the difference between
val isOdd = ((_:Int) & 1 ) == 1
where you want everything to the right of the equals sign to be a lambda, and
val result = collection.map( _ + 1 )
where you want only the stuff inside the parentheses to be a lambda
Scala has decided that when you use the underscore to create a lambda, that it's going to pick the innermost set of parentheses as the boundaries of that lambda. There's one exception: (_:Int) doesn't count as the innermost parentheses because its purpose is only to group they type declaration with the _ placeholder.
Hence:
val isOdd = ((_:Int) & 1 ) == 1
^^^^^^^^^^^^^^
this is the lambda
val result = collection.map( _ + 1 )
^^^^^^^
this is the lambda
val result = collection.map(( _ + 1) / 2)
^^^^^^^^
this is the lambda
and the compiler can't infer the type of the _
val result = somemap.map(( _ + 1) / 2 * _)
^^^^^^^^
this is an inner lambda with one parameter
and the compiler can't infer the type of the _
^^^^^^^^^^^^^^^^^
this is an outer lambda with one parameter
This last case lets you do things like
_.map(_ + 1)
and have that get translated into
x => x.map( y=> y + 1 )
Only slightly cheating:
val isOdd = (_: Int) % 2 == 1
:-)
There you go:
val isOdd = ((_: Int) & 1) andThen (1 ==)
What Scala is doing is this:
it sees ((_:Int) & 1 ) and creates an object of type (Int) => Int, that is, a function.
it then applies the comparison operator == to compare this function to the value 1
A function is not equal to the value 1. Therefore the result is false, so your code is equivalent to:
val isOdd = false
What you could do is create another anonymous function that does the == 1 part of your computation. This is ugly:
val isOdd = ((_: Int) & 1)(_: Int) == 1
This is equivalent to the more verbose (and perhaps easier to understand):
val isOdd = (x: Int) => 1 == ((_: Int) & 1)(x)
A different approach
val isOdd = (_:Int).&(1) == 1

Unexpected Scala pattern matching syntax

I had a List of Scala tuples like the following:
val l = List((1,2),(2,3),(3,4))
and I wanted to map it in a list of Int where each item is the sum of the Ints in a the corresponding tuple. I also didn't want to use to use the x._1 notation so I solved the problem with a pattern matching like this
def addTuple(t: (Int, Int)) : Int = t match {
case (first, second) => first + second
}
var r = l map addTuple
Doing that I obtained the list r: List[Int] = List(3, 5, 7) as expected. At this point, almost by accident, I discovered that I can achieve the same result with an abbreviated form like the following:
val r = l map {case(first, second) => first + second}
I cannot find any reference to this syntax in the documentation I have. Is that normal? Am I missing something trivial?
See Section 8.5 of the language reference, "Pattern Matching Anonymous Functions".
An anonymous function can be defined by a sequence of cases
{case p1 =>b1 ... case pn => bn }
which appear as an expression without a prior match. The expected type of such an expression must in part be defined. It must be either scala.Functionk[S1, ..., Sk, R] for some k > 0, or scala.PartialFunction[S1, R], where the argument type(s) S1, ..., Sk must be fully determined, but the result type R may be undetermined.
The expected type deternines whether this is translated to a FunctionN or PartialFunction.
scala> {case x => x}
<console>:6: error: missing parameter type for expanded function ((x0$1) => x0$1 match {
case (x # _) => x
})
{case x => x}
^
scala> {case x => x}: (Int => Int)
res1: (Int) => Int = <function1>
scala> {case x => x}: PartialFunction[Int, Int]
res2: PartialFunction[Int,Int] = <function1>
{case(first, second) => first + second} is treated as a PartialFunction literal. See examples in "Partial Functions" section here: http://programming-scala.labs.oreilly.com/ch08.html or section 15.7 of Programming in Scala.
Method map accepts a function. In your first example you create a function, assign it to a variable, and pass it to the map method. In the second example you pass your created function directly, omitting assigning it to a variable. You are doing just the same thing.