Getting to know scala better, I came across a behaviour I cannot explain.
The below code works fine:
def triple(x: Double) = 3 * x
Array(3.14,1.42,3.9).map(triple)
However, If I call the scala ceil function in math library then I will need to pass an _ for it to work
def valueAtOneQuarter(f: (Double)=> Double) = f(0.25)
valueAtOneQuarter(sqrt _)
What is special about the _ in this context from the function call in the earlier piece of code.
The underscore is actually expanded to a function.
So sqrt _ gets turned into the function a => sqrt(a).
You will notice that this expanded function matches the parameter type f of the valueatonequarter method.
In more general terms, the underscore is sometimes needed to flag to the compiler to turn the method (a method is declared using def) into a function (methods and functions are similar but not the same thing). The compiler will attempt to automatically convert the method to a function for you but in some cases it needs additional pointers (like an explicit type declaration or _). See here for a full explanation of eta expansion and partial functions: https://medium.com/#sinisalouc/on-method-invocations-or-what-exactly-is-eta-expansion-1019b37e010c
A method and a function are not the same in Scala.
You define a method this way:
def method(x:Double):Double = ...
But you define a function using this way:
val func = (x: Double):Double => {...}
when you pass a function as a parameter to a method it must be a function, but not a method.
So you must use underscore to make a function from a method.
Sometimes Scala uses “Eta Expansion” capability to automatically convert the method into a function. But in some cases you must do it manually
Answering since the existing answers don't really explain when _ is needed, just "sometimes" or "in some cases".
A very important concept to understand in Scala is expected type. In your first example the expected type for argument of map is Double => B with some unknown B; in the second it's Double => Double. When the expected type is a function type and a method name is passed, it'll be automatically converted to a function. So, as comments say, the second example works without _. It's only necessary where there is no expected type, say
val f = sqrt _
There can also be issues when the method is overloaded, but there just adding _ generally won't work either; instead, you'll specify the argument type to show which method is used, e.g. max(_: Int, _: Int) or max(_: Double, _: Double).
Thanks seems _ has different meaning in different context in scala
Yes, quite a few: What are all the uses of an underscore in Scala?
Related
I have created an UDF in Scala and when I was trying to register this UDF with just function name it was showing me error.
Not Working
def IPConvertUDF = spark.udf.register("IPConvertUDF", IPConvert)
Error
error: missing argument list for method IPConvert
Unapplied methods are only converted to functions when a function type is expected.
You can make this conversion explicit by writing `IPConvert _` or `IPConvert(_)` instead of `IPConvert`.
def IPConvertUDF = spark.udf.register("IPConvertUDF", IPConvert)
And so I added extra _ after method name and it worked.
Works perfectly
def IPConvertUDF = spark.udf.register("IPConvertUDF", IPConvert _)
Would someone be able to explain be what is the meaning of extra _ after the method name?
Well the short answer is, you are trying to pass a method where a function is expected as an argument. Methods are not functions.
Let's dig a bit deeper.
Lets try with simple add function first
val add:(Int,Int) => Int = (val1,val2) => val1+val2
spark.udf.register("add",add)
The above code compiled without any error. The reason is add is a function.
Now lets try the same add as a method
def add(val1:Int,val2:Int): Int ={
val1+val2
}
spark.udf.register("add",add)
Now you get an error saying error: missing argument list for method add.
As I mentioned, register(..) is expecting function and methods cannot be passed as arguments.
_ is a shorthand for partially applied function.In other words , add method is converted into partially applied function and that's the reason the error has disappeared.
spark.udf.register("add",add _)
I'm having trouble understand the signature of partially applied function for function literal. For example, consider the following function and method definitions that work the same way:
def meth(x: Int): Unit = ()
val func = (x: Int) => ()
Now create a partially applied function out of each of the method/function above. Notice the signatures of the resulting function:
val f = meth _ // Int => Unit
val g = func _ // () => Int => Unit
I don't understand why the signature of func _ includes a () => part, which makes g must be called like g()(1) rather than g(1).
For example, consider the following function and method definitions that work the same way:
Methods and functions are not the same in Scala. They're in fact very different. Methods are not first-class objects, whereas functions are. That is, you'll never have a variable whose type is a method type.
As a consequence of this, let's look at the types without the underscores.
val f = meth
val g = func
Since meth is a method, it's going to be called. So the first line will try to call meth() with no arguments, an error. func, however, is an ordinary value, so there's no trouble. We'll just have another alias to func called g.
val f = meth _
val g = func _
In order to allow us to treat methods as objects, Scala introduces the _ syntax. When used with a method, it converts that method into a function object. So meth _ is a special method syntax which turns it into a function. On the other hand, func _ is taking a value (remember, functions are objects, just like strings or anything else) and lifting it to be a function object. The "easiest" way to do that is to add an empty argument list, so that's exactly what Scala does.
tl;dr Putting _ after a name turns the thing into a function object. If the thing is a method, it doesn't apply it. If the thing is a value (including but not limited to a function), it produces a new function object.
Method with ONLY implicit parameter
scala> def test1 (implicit i:Int )= Option(i)
test1: (implicit i: Int)Option[Int]
In trying to convert test1 into a function as shown below throws following error. I must be missing something obvious here?
scala> def test2(implicit i:Int) = test1 _
<console>:8: error: type mismatch;
found : Option[Int]
required: ? => ?
def test2(implicit i:Int) = test1 _
When using implicit parameter you have one in the scope. Like this:
object test{
implicit var i = 10
def fun(implicit i: Int) = println(i)
}
test.fun
And if you want to make an option isntance of something you should use Some()
Because you have an implicit Int in scope when defining test2, it's applied to test1, so you really end up with test1(i) _, which makes no sense to the compiler.
If it did compile, test2 would return the same function corresponding to test1 independent of the argument. If that's what you actually want, you can fix it by being more explicit: test1(_: Int) or { x: Int => test1(x) } instead of test1 _.
There are several different possible answers here depending on what it is exactly you are trying to achieve. If you meant test2 to be a behaviorally identical method to test1 then there is no need to use the underscore (and indeed you cannot!).
def test3(implicit i: Int) = test1(i)
If you meant test2 to be a behaviorally identical function to test1, then you need to do the following:
val test4 = test1(_: Int)
Note that I am using a val instead of a def here, because when using the underscore, I am turning test1 which was originally a method into an instance of a class (specifically one which extends Function2, i.e. a function). Also note that test4 no longer has implicit parameters as it is a function (and to the best of my knowledge functions cannot have implicit parameters) whereas test1 was a method.
Why you need to do this as opposed to just test1 _ turns out to be pretty complex...
TLDR: Scala is finnicky about the difference between foo(_) and foo _, methods are different from functions, and implicit resolution takes higher precedence over eta-expansion
Turning a method into a function object by way of an anonymous function in Scala is known as "eta-expansion." In general eta-expansion comes from the lambda calculus and refers to turning a term into its equivalent anonymous function (e.g. thing becomes x => thing(x)) or in the language of the lambda calculus, adding an abstraction over a term.
Note that without eta-expansion, programming in Scala would be absolutely torturous because methods are not functions. Therefore they cannot be directly passed as arguments to other methods or other functions. Scala tries to convert methods into functions via eta-expansion whenever possible, so as to give the appearance that you can pass methods to other methods.
So why doesn't the following work?
val test5 = test1 _ // error: could not find implicit value for parameter i: Int
Well let's see what would happen if we tried test4 without a type signature.
val test6 = test1(_) // error: missing parameter type for expanded function ((x$1) => test1(x$1))
Ah ha, different errors!
This is because test1(_) and test1 _ are subtly different things. test1(_) is partially applying the method test1 while test1 _ is a direction to perform eta-expansion on test1 until it is fully applied.
For example,
math.pow(_) // error: not enough arguments for method pow: (x: Double, y: Double)Double.
math.pow _ // this is fine
But what about the case where we have just one argument? What's the difference between partial application and eta-expansion of just one abstraction? Not really all that much. One way to view it is that eta-expansion is one way of implementing partial application. Indeed the Scala Spec seems to be silent on the difference between foo(_) and foo _ for a single parameter method. The only difference that matters for us is that the expansion that occurs in foo(_) always "binds" tighter than foo _, indeed it seems to bind as closely as method application does.
That's why test1(_) gives us an error about types. Partial application is treated analogously to normal method application. Because normal method application must always take place before implicit resolution, otherwise we could never replace an implicit value with our own value, partial application takes place before implicit resolution. It just so happens that the way Scala implements partial application is via eta-expansion and therefore we get the expansion into the anonymous function which then complains about the lack of a type for its argument.
My reading of Section 6.26 of the Scala Spec suggests that there is an ordering to the resolution of various conversions. In particular it seems to list resolving implicits as coming before eta-expansion. Indeed, for fully-applied eta-expansion, it would seem that this would necessarily be the case since Scala functions cannot have implicit parameters (only its methods can).
Hence, in the case of test5 as #Alexey says, when we are explicitly telling Scala to eta-expand test1 with test1 _, the implicit resolution takes place first, and then eta-expansion tries to take place, which fails because after implicit resolution Scala's typechecker realizes we have an Option not a method.
So that's why we need test1(_) over test1 _. The final type annotation test1(_: Int) is needed because Scala's type inference isn't robust enough to determine that after eta-expanding test1, the only possible type you can give to the anonymous function is the same as the type signature for the method. In fact, if we gave the type system enough hints, we can get away with test1(_).
val test7: Int => Option[Int] = test1(_) // this works
val test8: Int => Option[Int] = test1 _ // this still doesn't
I'm working on a DSL for a experimental library I'm building in Scala, and I've run into some vexing peculiarities of Scala's type inference as it pertains to lambda expression arguments that don't seem to be covered in the book Programming In Scala.
In my library, I have a trait, Effect[-T], that is used to represent temporary modifiers that can be applied to an object of type T. I have a object, myEffects, that has a method named += that accepts an argument of type Effect[PlayerCharacter]. Lastly, I have a generic method, when[T], that is used for constructing conditional effects by accepting a conditional expression and another effect as an argument. The signature is as follows:
def when[T](condition : T => Boolean) (effect : Effect[T]) : Effect[T]
When I call the "when" method with the above signature, passing it's result to the += method, it is unable to infer the type of the argument to the lambda expression.
myEffects += when(_.hpLow()) (applyModifierEffect) //<-- Compiler error
If I combine the arguments of "when" into a single parameter list, Scala is able to infer the type of the lambda expression just fine.
def when[T](condition : T => Boolean, effect : Effect[T]) : Effect[T]
/* Snip */
myEffects += when(_.hpLow(), applyModifierEffect) //This works fine!
It also works if I remove the second parameter entirely.
def when[T](condition : T => Boolean) : Effect[T]
/* Snip */
myEffects += when(_.hpLow()) //This works too!
However, for aesthetic reasons, I really want the the arguments to be passed to the "when" method as separate parameter lists.
My understanding from section 16.10 of Programming in Scala is that the compiler first looks at whether the method type is known, and if so it uses that to infer the expected type of it's arguments. In this case, the outermost method call is +=, which accepts an argument of type Effect[PlayerCharacter]. Since the return type of when[T] is Effect[T], and the method to which the result is being passed expects an argument of type Effect[PlayerCharacter], it can infer that T is PlayerCharacter, and therefore the type of the lambda expression passed as the first argument to "when" is PlayerCharacter => Boolean. This seems to be how it works when the arguments are provided in one parameter list, so why does breaking the arguments into two parameter lists break it?
I'm relatively new to Scala myself, and I don't have a lot of detailed technical knowledge of how the type inference works. So best take this with a grain of salt.
I think the difference is that the compiler is having trouble proving to itself that the two Ts are the same in condition : T => Boolean and effect : Effect[T], in the two-parameter-list version.
I believe when you have multiple parameter lists (because Scala views this as defining a method which returns a function that consumes the next parameter list) the compiler treats the parameter lists one at a time, not all together as in the single parameter list version.
So in this case:
def when[T](condition : T => Boolean, effect : Effect[T]) : Effect[T]
/* Snip */
myEffects += when(_.hpLow(), applyModifierEffect) //This works fine!
The type of applyModifierEffect and the required parameter type of myEffects += can help constrain the parameter type of _.hpLow(); all the Ts must be PlayerCharacter. But in the following:
myEffects += when(_.hpLow()) (applyModifierEffect)
The compiler has to figure out the type of when(_.hpLow()) independently, so that it can check if it's valid to apply it to applyModifierEffect. And on its own, _.hpLow() doesn't provide enough information for the compiler to deduce that this is when[PlayerCharacter](_.hpLow()), so it doesn't know that the return type is a function of type Effect[PlayerCharacter] => Effect[PlayerCharacter], so it doesn't know that it's valid to apply that function in that context. My guess would be that the type inference just doesn't connect the dots and figure out that there is exactly one type that avoids a type error.
And as for the other case that works:
def when[T](condition : T => Boolean) : Effect[T]
/* Snip */
myEffects += when(_.hpLow()) //This works too!
Here the return type of when and its parameter type are more directly connected, without going through the parameter type and the return type of the extra function created by currying. Since myEffects += requires an Effect[PlayerCharacter], the T must be PlayerCharacter, which has a hpLow method, and the compiler is done.
Hopefully someone more knowledgeable can correct me on the details, or point out if I'm barking up the wrong tree altogether!
I am a little confused because in my view, none of the versions you say work should, and indeed I cannot make any of them work.
Type inferences works left to right from one parameter list (not parameter) to the next. Typical example is method foldLeft in collections (say Seq[A])
def foldLeft[B] (z: B)(op: (B, A) => B): B
The type of z will makes B known, so op can be written without specifying B (nor A which is known from the start, type parameter of this).
If the routine was written as
def foldLeft[B](z: B, op: (B,A) => B): B
or as
def foldLeft[B](op: (B,A) => B)(z: B): B
it would not work, and one would have to make sure op type is clear, or to pass the B explicity when calling foldLeft.
In your case, I think the most pleasant to read equivalent would be to make when a method of Effect, (or make it look like one with an implicit conversion) you would then write
Effects += applyModifierEffect when (_.hpLow())
As you mentionned that Effect is contravariant, when signature is not allowed for a method of Effect (because of the T => Boolean, Function is contravariant in its first type parameter, and the condition appears as a parameter, so in contravariant position, two contravariants makes a covariant), but it can still be done with an implicit
object Effect {
implicit def withWhen[T](e: Effect[T])
= new {def when(condition: T => Boolean) = ...}
}
Why, in Scala, given:
a = List(1, 2, 3, 4)
def f(x : String) = { x }
does
a.map(_.toString)
work, but
a.map(f(_.toString))
give the error
missing parameter type for expanded function ((x$1) => x$1.toString)
Well... f() takes a String as a parameter. The construct _.toString has type A <: Any => String. The function f() expects a type of String, so the example above does not type check. It seems that Scala is friendly in this case and gives the user another chance. The error message means: "By my type inference algorithms this does not compile. Put the types in and it might, if it's something I can't infer."
You would have to write the anonymous function longhand in this case, i.e. a.map(n => f(n.toString)). This is not a limitation of type inference, but of the wildcard symbol. Basically, when you write a.map(f(_.toString)), the _.toString gets expanded into an anonymous function inside the closest brackets it can find, otherwise this would lead to enormous ambiguity. Imagine something like f(g(_.toString)). Does this mean f(g(x => x.toString)) or f(x => g(x.toString))? Worse ambiguities would arise for multiple nested function calls. The Scala type checker therefore takes the most logical solution, as described above.
Nitpick: the first line of your code should be val a = List(1,2,3,4) :).