Consider the following definition in Scala:
val f = ((_: Int) + 1).toString()
The code assigns to f the string representation of the function literal _ + 1, which is quite natural, except that this is not i want. i intended to define a function that accepts an int argument, increments it by 1, and returns its string format.
To disambiguate, i have to write a lambda expression with explicit parameters:
val g = (x: Int) => (x + 1).toString()
So can i conclude that the placeholder syntax is not suitable for complex function literals?
Or is there some rule that states the scope of the function literal? It seems placeholders cannot be nested in parentheses(except the ones needed for defining its type) within the function literal
Yes, you are right in thinking that placeholder syntax is not useful beyond simple function literals.
For what it's worth, the particular case you mentioned can be written with a conjunction of placeholder syntax and function composition.
scala> val f = ((_: Int) + 1) andThen (_.toString)
f: Int => java.lang.String = <function1>
scala> f(34)
res14: java.lang.String = 35
It seems placeholders cannot be nested in parentheses(except the ones needed for defining its type) within the function literal
This is correct. See the rules for the placeholder syntax and a use case example.
Related
In Scala, if you have an expression containing an underscore, this is an anonymous function with the expression as its body and the underscore as as its parameter, e.g. 2*_ is the anonymous function that doubles its argument. But how far does the function body extend? I'm missing a clear rule here that disambiguates cases like e.g. the following (tested with the Scala 2.11.7 REPL):
scala> (_: Int)+2-1 // function body up to 1 - OK
res7: Int => Int = <function1>
scala> ((_: Int)+2)-1 // function body up to 2, - applied to function is an error
<console>:11: error: value - is not a member of Int => Int
((_: Int)+2)-1
^
The definition is given in http://www.scala-lang.org/files/archive/spec/2.11/06-expressions.html#placeholder-syntax-for-anonymous-functions, and it's... not that simple.
An expression e of syntactic category Expr binds an underscore section u, if the following two conditions hold: (1) e properly contains u, and (2) there is no other expression of syntactic category Expr which is properly contained in e and which itself properly contains u.
If an expression e binds underscore sections u_1 , \ldots , u_n, in this order, it is equivalent to the anonymous function (u'_1, ... u'_n) => e' where each u_i' results from u_i by replacing the underscore with a fresh identifier and e' results from e by replacing each underscore section u_i by u_i'.
And if you look at the grammar in the beginning of the section, (_: Int)+2 in (_: Int)+2-1 is not an Expr, but in ((_: Int)+2)-1 it is.
((_: Int)+2)-1 // function body up to 2, - applied to function is an error
error: value - is not a member of Int => Int
((_: Int)+2)-1
The error message from the compiler is sensible. Your additional parens have created a function literal that adds '2' to a wildcard/placeholder parameter. The compiler reads your code to mean that you have a this function value and you are trying to subtract '1' from it.
And this doesn't make sense. You can subract '1' from other numbers, but certainly not a function value. Thus, the compiler is telling you that it doesn't make sense to subtract one from a function value. Or, in compiler terms, a function of type Int => Int doesn't have a '-' function.
value - is not a member of Int => Int
Understanding this error message requires that you know that all operators in Scala ( -, *, +, etc ) are implemented as methods of types. If you look at the Scala API docs for Int, you'll see that it defines a long list of methods with common mathematical and logical operator symbols as function names.
I am working on spark and not an expert in scala. I have got the two variants of map function. Could you please explain the difference between them.?
first variant and known format.
first variant
val.map( (x,y) => x.size())
Second variant -> This has been applied on tuple
val.map({case (x, y) => y.toString()});
The type of val is RDD[(IntWritable, Text)]. When i tried with first function, it gave error as below.
type mismatch;
found : (org.apache.hadoop.io.IntWritable, org.apache.hadoop.io.Text) ⇒ Unit
required: ((org.apache.hadoop.io.IntWritable, org.apache.hadoop.io.Text)) ⇒ Unit
When I added extra parenthesis it said,
Tuples cannot be directly destructured in method or function parameters.
Well you say:
The type of val is RDD[(IntWritable, Text)]
so it is a tuple of arity 2 with IntWritable and Text as components.
If you say
val.map( (x,y) => x.size())
what you're doing is you are essentially passing in a Function2, a function with two arguments to the map function. This will never compile because map wants a function with one argument. What you can do is the following:
val.map((xy: (IntWritable, Text)) => xy._2.toString)
using ._2 to get the second part of the tuple which is passed in as xy (the type annotation is not required but makes it more clear).
Now the second variant (you can leave out the outer parens):
val.map { case (x, y) => y.toString() }
this is special scala syntax for creating a PartialFunction that immediately matches on the tuple that is passed in to access the x and y parts. This is possible because PartialFunction extends from the regular Function1 class (Function1[A,B] can be written as A => B) with one argument.
Hope that makes it more clear :)
I try this in repl:
scala> val l = List(("firstname", "tom"), ("secondname", "kate"))
l: List[(String, String)] = List((firstname,tom), (secondname,kate))
scala> l.map((x, y) => x.size)
<console>:9: error: missing parameter type
Note: The expected type requires a one-argument function accepting a 2-Tuple.
Consider a pattern matching anonymous function, `{ case (x, y) => ... }`
l.map((x, y) => x.size)
maybe can give you some inspire.
Your first example is a function that takes two arguments and returns a String. This is similar to this example:
scala> val f = (x:Int,y:Int) => x + y
f: (Int, Int) => Int = <function2>
You can see that the type of f is (Int,Int) => Int (just slightly changed this to be returning an int instead of a string). Meaning that this is a function that takes two Int as arguments and returns an Int as a result.
Now the second example you have is a syntactic sugar (a shortcut) for writing something like this:
scala> val g = (k: (Int, Int)) => k match { case (x: Int, y: Int) => x + y }
g: ((Int, Int)) => Int = <function1>
You see that the return type of function g is now ((Int, Int)) => Int. Can you spot the difference? The input type of g has two parentheses. This shows that g takes one argument and that argument must be a Tuple[Int,Int] (or (Int,Int) for short).
Going back to your RDD, what you have is an Collection of Tuple[IntWritable, Text] so the second function will work, whereas the first one will not work.
I read a project's source code, found:
val sampleMBR = inputMBR.map(_._2).sample
inputMBR is a tuple.
the function map's definition is :
map[U classTag](f:T=>U):RDD[U]
it seems that map(_._2) is the shorthand for map(x => (x._2)).
Anyone can tell me rules of those shorthand ?
The _ syntax can be a bit confusing. When _ is used on its own it represents an argument in the anonymous function. So if we working on pairs:
map(_._2 + _._2) would be shorthand for map(x, y => x._2 + y._2). When _ is used as part of a function name (or value name) it has no special meaning. In this case x._2 returns the second element of a tuple (assuming x is a tuple).
collection.map(_._2) emits a second component of the tuple. Example from pure Scala (Spark RDDs work the same way):
scala> val zipped = (1 to 10).zip('a' to 'j')
zipped: scala.collection.immutable.IndexedSeq[(Int, Char)] = Vector((1,a), (2,b), (3,c), (4,d), (5,e), (6,f), (7,g), (8,h), (9,i), (10,j))
scala> val justLetters = zipped.map(_._2)
justLetters: scala.collection.immutable.IndexedSeq[Char] = Vector(a, b, c, d, e, f, g, h, i, j)
Two underscores in '_._2' are different.
First '_' is for placeholder of anonymous function; Second '_2' is member of case class Tuple.
Something like:
case class Tuple3 (_1: T1, _2: T2, _3: T3)
{...}
I have found the solutions.
First the underscore here is as placeholder.
To make a function literal even more concise, you can use underscores
as placeholders for one or more parameters, so long as each parameter
appears only one time within the function literal.
See more about underscore in Scala at What are all the uses of an underscore in Scala?.
The first '_' is referring what is mapped to and since what is mapped to is a tuple you might call any function within the tuple and one of the method is '_2' so what below tells us transform input into it's second attribute.
I'm trying to have curried apply and update methods like this:
def apply(i: Int)(j: Int) = matrix(i)(j)
def update(i: Int, j: Int, value: Int) =
new Matrix(n, m, (x, y) => if ((i,j) == (x,y)) value else matrix(x)(y))
Apply method works correctly, but update method complains:
scala> matrix(2)(1) = 1
<console>:16: error: missing arguments for method apply in class Matrix;
follow this method with `_' if you want to treat it as a partially applied function
matrix(2)(1) = 1
Calling directly update(2)(1)(1) works, so it is a conversion to update method that doesn't work properly. Where is my mistake?
The desugaring of assignment syntax into invocations of update maps the concatenation of a single argument list on the LHS of the assignment with the value on the RHS of the assignment to the first parameter block of the update method definition, irrespective of how many other parameter blocks the update method definition has. Whilst this transformation in a sense splits a single parameter block into two (one on the LHS, one on the RHS of the assignment), it will not further split the left parameter block in the way that you want.
I also think you're mistaken about the example of the explicit invocation of update that you show. This doesn't compile with the definition of update that you've given,
scala> class Matrix { def update(i: Int, j: Int, value: Int) = (i, j, value) }
defined class Matrix
scala> val m = new Matrix
m: Matrix = Matrix#37176bc4
scala> m.update(1)(2)(3)
<console>:10: error: not enough arguments for method update: (i: Int, j: Int, value: Int)(Int, Int, Int).
Unspecified value parameters j, value.
m.update(1)(2)(3)
^
I suspect that during your experimentation you actually defined update like so,
scala> class Matrix { def update(i: Int)(j: Int)(value: Int) = (i, j, value) }
defined class Matrix
The update desugaring does apply to this definition, but probably not in the way that you expect: as described above, it only applies to the first argument list, which leads to constructs like,
scala> val m = new Matrix
m: Matrix = Matrix#39741f43
scala> (m() = 1)(2)(3)
res0: (Int, Int, Int) = (1,2,3)
Here the initial one-place parameter block is split to an empty parameter block on the LHS of the assignment (ie. the ()) and a one argument parameter block on the RHS (ie. the 1). The remainder of the parameter blocks from the original definition then follow.
If you're surprised by this behaviour you won't be the first.
The syntax you're after is achievable via a slightly different route,
scala> class Matrix {
| class MatrixAux(i : Int) {
| def apply(j : Int) = 23
| def update(j: Int, value: Int) = (i, j, value)
| }
|
| def apply(i: Int) = new MatrixAux(i)
| }
defined class Matrix
scala> val m = new Matrix
m: Matrix = Matrix#3af30087
scala> m(1)(2) // invokes MatrixAux.apply
res0: Int = 23
scala> m(1)(2) = 3 // invokes MatrixAux.update
res1: (Int, Int, Int) = (1,2,3)
My guess is, that it is simply not supported. Probably not due to an explicit design decision, because I don't see why it shouldn't work in principle.
The translation concerned with apply, i.e., the one performed when converting m(i)(j) into m.apply(i, j) seems to be able to cope with currying. Run scala -print on your program to see the code resulting from the translation.
The translation concerned with update, on the other hand, doesn't seem to be able to cope with currying. Since the error message is missing arguments for method apply, it even looks as if the currying confuses the translator such that it tries to translate m(i)(j) = v into m.apply, but then screws up the number of required arguments. scala -print unfortunately won't help here, because the type checker terminates the translation too early.
Here is what the language specs (Scala 2.9, "6.15 Assignments") say about assignments. Since currying is not mentioned, I assume that it is not explicitly supported. I couldn't find the corresponding paragraph for apply, but I guess it is purely coincidental that currying works there.
An assignment f(args) = e with a function application to the left of
the ‘=’ operator is interpreted as f.update(args, e), i.e. the
invocation of an update function defined by f.
Being new to Scala (2.9.1), I have a List[Event] and would like to copy it into a Queue[Event], but the following Syntax yields a Queue[List[Event]] instead:
val eventQueue = Queue(events)
For some reason, the following works:
val eventQueue = Queue(events : _*)
But I would like to understand what it does, and why it works? I already looked at the signature of the Queue.apply function:
def apply[A](elems: A*)
And I understand why the first attempt doesn't work, but what's the meaning of the second one? What is :, and _* in this case, and why doesn't the apply function just take an Iterable[A] ?
a: A is type ascription; see What is the purpose of type ascriptions in Scala?
: _* is a special instance of type ascription which tells the compiler to treat a single argument of a sequence type as a variable argument sequence, i.e. varargs.
It is completely valid to create a Queue using Queue.apply that has a single element which is a sequence or iterable, so this is exactly what happens when you give a single Iterable[A].
This is a special notation that tells the compiler to pass each element as its own argument, rather than all of it as a single argument. See here.
It is a type annotation that indicates a sequence argument and is mentioned as an "exception" to the general rule in section 4.6.2 of the language spec, "Repeated Parameters".
It is useful when a function takes a variable number of arguments, e.g. a function such as def sum(args: Int*), which can be invoked as sum(1), sum(1,2) etc. If you have a list such as xs = List(1,2,3), you can't pass xs itself, because it is a List rather than an Int, but you can pass its elements using sum(xs: _*).
For Python folks:
Scala's _* operator is more or less the equivalent of Python's *-operator.
Example
Converting the scala example from the link provided by Luigi Plinge:
def echo(args: String*) =
for (arg <- args) println(arg)
val arr = Array("What's", "up", "doc?")
echo(arr: _*)
to Python would look like:
def echo(*args):
for arg in args:
print "%s" % arg
arr = ["What's", "up", "doc?"]
echo(*arr)
and both give the following output:
What's
up
doc?
The Difference: unpacking positional parameters
While Python's *-operator can also deal with unpacking of positional parameters/parameters for fixed-arity functions:
def multiply (x, y):
return x * y
operands = (2, 4)
multiply(*operands)
8
Doing the same with Scala:
def multiply(x:Int, y:Int) = {
x * y;
}
val operands = (2, 4)
multiply (operands : _*)
will fail:
not enough arguments for method multiply: (x: Int, y: Int)Int.
Unspecified value parameter y.
But it is possible to achieve the same with scala:
def multiply(x:Int, y:Int) = {
x*y;
}
val operands = (2, 4)
multiply _ tupled operands
According to Lorrin Nelson this is how it works:
The first part, f _, is the syntax for a partially applied function in which none of the arguments have been specified. This works as a mechanism to get a hold of the function object. tupled returns a new function which of arity-1 that takes a single arity-n tuple.
Futher reading:
stackoverflow.com - scala tuple unpacking