Simple question, and sorry if this is a stupid question as I am just beginning in scala. I am getting a type mismatch error that says:
found : (AnyRef, org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable) => List[Object]
required: ((AnyRef, org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable)) => scala.collection.GenTraversableOnce[?]
But according to this post (I have a Scala List, how can I get a TraversableOnce?), a scala.collection.immutable.List is an Iterable and therefore also a GenTraversableOnce. And yet this error seems to indicate otherwise. And furthermore, when I actually look at the link in the accepted answer of that post, I don't see any reference to the word "traversable".
If the problem has to do with my inner class not being correct, then I have to say this error is extremely uninformative, since requiring that the inner class be of type "?" is obviously a vacuous statement ... Any help in understanding this would be appreciated.
Function2[X, Y, Z] is not the same thing as Function1[(X, Y), Z].
Compare these two definitions:
val f: ((Int, Int)) => Int = xy => xy._1 + xy._2
val f: (Int, Int) => Int = (x, y) => x + y
The first could also be written with a pattern-matching, that first decomposes the tuple:
val f: ((Int, Int)) => Int = { case (x, y) => x + y }
This is exactly what the error message asks you to do: provide an unary function that takes a tuple as argument, not a binary function. Note that there is the tupled-method, that does exactly that.
The return types of the functions are mostly irrelevant here, the compiler doesn't get to unify them, because it fails on the types of the inputs.
Also related:
Same story with eta-expansions: Why does my implementation of Haskell snd not compile in Scala
Related
Given a string I want to create a map that for each character in a string will give the number of times the character occurs in a string. The following function makes a map from character to a list of Strings.
def wordOccurrences(w: String) = {
val lower = w.toLowerCase.toList
lower.groupBy(t => t)
}
Now I wanted to alter the last line to:
lower.groupBy(t => t) map ( (x,y) => x -> y.length)
But it doesn't work, can someone explain why and how to fix it?
For mapping purposes, a Map[K, V] is an Iterable[(K, V)] (notice the extra pair of parentheses, identifying a tuple type), meaning that when you map over it you have pass a function that goes from (K, V) to your target type.
What you are doing, however, is passing a function that takes two independent arguments, rather then a single tuple argument.
The difference can be seen by inspecting the types of these two functions in the Scala shell:
scala> :t (a: Int, b: Int) => a + b
(Int, Int) => Int
scala> :t (p: (Int, Int)) => p._1 + p._2
((Int, Int)) => Int
Notice how the former takes two arguments while the latter takes a single tuple.
What you can do is pass a function which decomposes the tuple so that you can bind the components of the tuple independently:
lower.groupBy(t => t) map { case (x, y) => x -> y.length }
or alternatively pass a function which uses the tuple without deconstructing it
lower.groupBy(t => t) map (p => p._1 -> p._2.length)
Note
Dotty, which is the current project Scala's original author Martin Odersky is working on and that will probably become Scala 3, supports the syntax you are proposing, calling the feature function arity adaptation. This has been discussed, along with other feature, in Odersky's 2016 Keynote at Scala eXchange, "From DOT to Dotty" (here the video taped at 2017 Voxxed Days CERN).
You can use
lower.groupBy(t => t).mapValues(_.length)
I am working on spark and not an expert in scala. I have got the two variants of map function. Could you please explain the difference between them.?
first variant and known format.
first variant
val.map( (x,y) => x.size())
Second variant -> This has been applied on tuple
val.map({case (x, y) => y.toString()});
The type of val is RDD[(IntWritable, Text)]. When i tried with first function, it gave error as below.
type mismatch;
found : (org.apache.hadoop.io.IntWritable, org.apache.hadoop.io.Text) ⇒ Unit
required: ((org.apache.hadoop.io.IntWritable, org.apache.hadoop.io.Text)) ⇒ Unit
When I added extra parenthesis it said,
Tuples cannot be directly destructured in method or function parameters.
Well you say:
The type of val is RDD[(IntWritable, Text)]
so it is a tuple of arity 2 with IntWritable and Text as components.
If you say
val.map( (x,y) => x.size())
what you're doing is you are essentially passing in a Function2, a function with two arguments to the map function. This will never compile because map wants a function with one argument. What you can do is the following:
val.map((xy: (IntWritable, Text)) => xy._2.toString)
using ._2 to get the second part of the tuple which is passed in as xy (the type annotation is not required but makes it more clear).
Now the second variant (you can leave out the outer parens):
val.map { case (x, y) => y.toString() }
this is special scala syntax for creating a PartialFunction that immediately matches on the tuple that is passed in to access the x and y parts. This is possible because PartialFunction extends from the regular Function1 class (Function1[A,B] can be written as A => B) with one argument.
Hope that makes it more clear :)
I try this in repl:
scala> val l = List(("firstname", "tom"), ("secondname", "kate"))
l: List[(String, String)] = List((firstname,tom), (secondname,kate))
scala> l.map((x, y) => x.size)
<console>:9: error: missing parameter type
Note: The expected type requires a one-argument function accepting a 2-Tuple.
Consider a pattern matching anonymous function, `{ case (x, y) => ... }`
l.map((x, y) => x.size)
maybe can give you some inspire.
Your first example is a function that takes two arguments and returns a String. This is similar to this example:
scala> val f = (x:Int,y:Int) => x + y
f: (Int, Int) => Int = <function2>
You can see that the type of f is (Int,Int) => Int (just slightly changed this to be returning an int instead of a string). Meaning that this is a function that takes two Int as arguments and returns an Int as a result.
Now the second example you have is a syntactic sugar (a shortcut) for writing something like this:
scala> val g = (k: (Int, Int)) => k match { case (x: Int, y: Int) => x + y }
g: ((Int, Int)) => Int = <function1>
You see that the return type of function g is now ((Int, Int)) => Int. Can you spot the difference? The input type of g has two parentheses. This shows that g takes one argument and that argument must be a Tuple[Int,Int] (or (Int,Int) for short).
Going back to your RDD, what you have is an Collection of Tuple[IntWritable, Text] so the second function will work, whereas the first one will not work.
This question already has answers here:
Iterate Over a tuple
(4 answers)
Closed 8 years ago.
In scala, we can get an iterator over a tuple as follows
val t = (1, 2)
val it = t.productIterator
and even
it.foreach( x => println(x.isInstanceOf[Int]) )
returns true, we cannot do simple operations on the iterator values without using asInstanceOf[Int], since
it.foreach( x => println(x+1) )
returns an error: type mismatch; found : Int(1) required: String
I understand the issue with Integer vs. Int, but still the validity of isInstanceOf[Int] is somewhat confusing.
What is the best way to do these operations over tuples? Notice that the tuple can have a mix of types like integers with doubles, so converting to a list might not always work.
A tuple does not have to be homogenous and the compiler didn't try to apply magic type unification across the elements1. Take (1, "hello") as an example of such a a heterogenous tuple (Tuple2[Int,String]).
This means that x is typed as Any (not Int!). Try it.foreach( (x: Int) => println(x) ), with the original tuple, to get a better error message indicating that the iterator is not unified over the types of the tuple elements (it is an Iterators[Any]). The error reported should be similar to:
error: type mismatch;
found : (Int) => Unit
required: (Any) => ?
(1, 2).productIterator.foreach( (x: Int) => println(x) )
In this particular case isInstanceOf[Int] can be used to refine the type - from the Any that the type-system gave us - because we know, from manual code inspection, that it will "be safe" with the given tuple.
Here is another look at the iterators/types involved:
(1, 2) // -> Tuple2[Int,Int]
.productIterator // -> Iterator[Any]
.map(_.asInstanceOf[Int]) // -> Iterator[Int]
.foreach(x => println(x+1))
While I would recommend treating tuples as finite sets of homogenous elements and not a sequence, the same rules can be used as when dealing with any Iterator[Any] such as using pattern matching (e.g. match) that discriminates by the actual object type. (In this case the code is using an implicit PartialFunction.)
(1, "hello").productIterator
.foreach {
case s: String => println("string: " + s)
case i: Int => println("int: " + i)
}
1 While it might be possible to make the compiler unify the types in this scenario, it sounds like a special case that requires extra work for minimal gain. Normally sequences like lists - not tuples - are used for homogenous elements and the compiler/type-system correctly gives us a good refinement for something like List(1,2) (which is typed as List[Int] as expected).
There is another type HList, that is like tuple and List all-in-one. See shapeless.
I think, you can get close to what you want:
import shapeless._
val t = 1 :: 2 :: HNil
val lst = t.toList
lst.foreach( x => println(x+1) )
Not sure how to properly formulate the question, there is a problem with currying in the merge sort example from Scala by Example book on page 69. The function is defined as follows:
def msort[A](less: (A, A) => Boolean)(xs: List[A]): List[A] = {
def merge(xs1: List[A], xs2: List[A]): List[A] =
if (xs1.isEmpty) xs2
else if (xs2.isEmpty) xs1
else if (less(xs1.head, xs2.head)) xs1.head :: merge(xs1.tail, xs2)
else xs2.head :: merge(xs1, xs2.tail)
val n = xs.length/2
if (n == 0) xs
else merge(msort(less)(xs take n), msort(less)(xs drop n))
}
and then there is an example of how to create other functions from it by currying:
val intSort = msort((x : Int, y : Int) => x < y)
val reverseSort = msort((x:Int, y:Int) => x > y)
however these two lines give me errors about insufficient number of arguments. And if I do like this:
val intSort = msort((x : Int, y : Int) => x < y)(List(1, 2, 4))
val reverseSort = msort((x:Int, y:Int) => x > y)(List(4, 3, 2))
it WILL work. Why? Can someone explain? Looks like the book is really dated since it is not the first case of such an inconsistance in its examples. Could anyone point to something more real to read? (better a free e-book).
My compiler (2.9.1) agrees, there seems to be an error here, but the compiler does tell you what to do:
error: missing arguments for method msort in object $iw;
follow this method with `_' if you want to treat it as a partially applied function
So, this works:
val intSort = msort((x : Int, y : Int) => x < y) _
Since the type of intSort is inferred, the compiler doesn't otherwise seem to know whether you intended to partially apply, or whether you missed arguments.
The _ can be omitted when the compiler can infer from the expected type that a partially applied function is what is intended. So this works too:
val intSort: List[Int] => List[Int] = msort((x: Int, y: Int) => x < y)
That's obviously more verbose, but more often you will take advantage of this without any extra boilerplate, for example if msort((x: Int, y: Int) => x < y) were the argument to a function where the parameter type is already known to be List[Int] => List[Int].
Edit: Page 181 of the current edition of the Scala Language Specification mentions a tightening of rules for implicit conversions of partially applied methods to functions since Scala 2.0. There is an example of invalid code very similar to the one in Scala by Example, and it is described as "previously legal code".
I had a List of Scala tuples like the following:
val l = List((1,2),(2,3),(3,4))
and I wanted to map it in a list of Int where each item is the sum of the Ints in a the corresponding tuple. I also didn't want to use to use the x._1 notation so I solved the problem with a pattern matching like this
def addTuple(t: (Int, Int)) : Int = t match {
case (first, second) => first + second
}
var r = l map addTuple
Doing that I obtained the list r: List[Int] = List(3, 5, 7) as expected. At this point, almost by accident, I discovered that I can achieve the same result with an abbreviated form like the following:
val r = l map {case(first, second) => first + second}
I cannot find any reference to this syntax in the documentation I have. Is that normal? Am I missing something trivial?
See Section 8.5 of the language reference, "Pattern Matching Anonymous Functions".
An anonymous function can be defined by a sequence of cases
{case p1 =>b1 ... case pn => bn }
which appear as an expression without a prior match. The expected type of such an expression must in part be defined. It must be either scala.Functionk[S1, ..., Sk, R] for some k > 0, or scala.PartialFunction[S1, R], where the argument type(s) S1, ..., Sk must be fully determined, but the result type R may be undetermined.
The expected type deternines whether this is translated to a FunctionN or PartialFunction.
scala> {case x => x}
<console>:6: error: missing parameter type for expanded function ((x0$1) => x0$1 match {
case (x # _) => x
})
{case x => x}
^
scala> {case x => x}: (Int => Int)
res1: (Int) => Int = <function1>
scala> {case x => x}: PartialFunction[Int, Int]
res2: PartialFunction[Int,Int] = <function1>
{case(first, second) => first + second} is treated as a PartialFunction literal. See examples in "Partial Functions" section here: http://programming-scala.labs.oreilly.com/ch08.html or section 15.7 of Programming in Scala.
Method map accepts a function. In your first example you create a function, assign it to a variable, and pass it to the map method. In the second example you pass your created function directly, omitting assigning it to a variable. You are doing just the same thing.