Related
Let's say I have following code:
val xs: List[Int] = List(1, 2, 3)
val ys: List[Int] = List(4, 5, 6)
val zs: List[Int] = xs.appended(ys)
The last line does not compile with an error:
Error:(162, 33) type mismatch; found : List[Int] required: Int val
zs: List[Int] = xs.appended(ys)
If I remove the explicit type declaration, then the code compiles, but the real problem is that
the error message appears in a recursive function where I would like to pass appended list as a parameter of type List[Int], so removing the explicit type is not an option.
According to scaladoc appended method takes only one argument, not an entire list. So the following examples will compile:
xs.appended(ys(0))
for(x <- xs) yield ys appended x
or appendAll:
xs appendAll ys
ys :++ xs
P.S.: Note, that appending to the list is not optimal, as it's time is proportional to the size of the list, prefer prepend instead:
ys ::: xs
According scala documentation appended method accepting just one element, not collection. And zs type after removing explicit types will be List[Any]:
val xs = List(1, 2, 3)
val ys = List(4, 5, 6)
val zs: List[Any] = xs.appended(ys) // List(1, 2, 3, List(4, 5, 6))
it compiles, but result will be List(1, 2, 3, List(4, 5, 6))
You can use method appendedAll to do that you want or just concatenate lists using concat or ++ operator :
val xs = List(1, 2, 3)
val ys = List(4, 5, 6)
val zs: List[Int] = xs ++ ys // List(1, 2, 3, 4, 5, 6)
val as: List[Int] = xs.appendedAll(ys) // List(1, 2, 3, 4, 5, 6)
val bs: List[Int] = xs.concat(ys) // List(1, 2, 3, 4, 5, 6)
1. val xs: List[Int] = List(1, 2, 3)
2. val ys: List[Int] = List(4, 5, 6)
3. val zs: List[Int] = xs.appended(ys)
The third line is a problem until you have the type declaration. Because when you compile your code compiler is not going to infer the type of the variable zs and it will expect the output of xs.appended(ys) to be a List[Int] which is not the case because xs is List[Int] now if you want to add an element in this list you can do xs.append(1) or any other integer but you are trying to insert List[Int] which is not Int.
Now when you remove the type declaration from line 3 it compile successfully because now compiler will infer the type of the variable zs and if you will see on REPL it will say the of this variable zs is List[Any].
Now if you want to add list into a list and get a flatten result you can simply use
val zs: List[Int] = xs ::: ys
If you will see the scala docs here
this is the signature of appended:
final def:+[B >: A](elem: B): List[B]
:+ is Alias for appended
:++ is Alias for appendedAll
As we can see from the signature appended function takes a parameter of type B and return List[B] in your case B is Int and you are trying to add List[Int].
I hope it clears why you are getting the compilation error.
How can I merge two lists / Seqs so it takes 1 element from list 1, then 1 element from list 2, and so on, instead of just appending list 2 at the end of list 1?
E.g
[1,2] + [3,4] = [1,3,2,4]
and not [1,2,3,4]
Any ideas? Most concat methods I've looked at seem to do to the latter and not the former.
Another way:
List(List(1,2), List(3,4)).transpose.flatten
So maybe your collections aren't always the same size. Using zip in that situation would create data loss.
def interleave[A](a :Seq[A], b :Seq[A]) :Seq[A] =
if (a.isEmpty) b else if (b.isEmpty) a
else a.head +: b.head +: interleave(a.tail, b.tail)
interleave(List(1, 2, 17, 27)
,Vector(3, 4)) //res0: Seq[Int] = List(1, 3, 2, 4, 17, 27)
You can do:
val l1 = List(1, 2)
val l2 = List(3, 4)
l1.zip(l2).flatMap { case (a, b) => List(a, b) }
Try
List(1,2)
.zip(List(3,4))
.flatMap(v => List(v._1, v._2))
which outputs
res0: List[Int] = List(1, 3, 2, 4)
Also consider the following implicit class
implicit class ListIntercalate[T](lhs: List[T]) {
def intercalate(rhs: List[T]): List[T] = lhs match {
case head :: tail => head :: (rhs.intercalate(tail))
case _ => rhs
}
}
List(1,2) intercalate List(3,4)
List(1,2,5,6,6,7,8,0) intercalate List(3,4)
which outputs
res2: List[Int] = List(1, 3, 2, 4)
res3: List[Int] = List(1, 3, 2, 4, 5, 6, 6, 7, 8, 0)
For example, if I have a list of List(1,2,1,3,2), and I want to remove only one 1, so the I get List(2,1,3,2). If the other 1 was removed it would be fine.
My solution is:
scala> val myList = List(1,2,1,3,2)
myList: List[Int] = List(1, 2, 1, 3, 2)
scala> myList.patch(myList.indexOf(1), List(), 1)
res7: List[Int] = List(2, 1, 3, 2)
But I feel like I am missing a simpler solution, if so what am I missing?
surely not simpler:
def rm(xs: List[Int], value: Int): List[Int] = xs match {
case `value` :: tail => tail
case x :: tail => x :: rm(tail, value)
case _ => Nil
}
use:
scala> val xs = List(1, 2, 1, 3)
xs: List[Int] = List(1, 2, 1, 3)
scala> rm(xs, 1)
res21: List[Int] = List(2, 1, 3)
scala> rm(rm(xs, 1), 1)
res22: List[Int] = List(2, 3)
scala> rm(xs, 2)
res23: List[Int] = List(1, 1, 3)
scala> rm(xs, 3)
res24: List[Int] = List(1, 2, 1)
you can zipWithIndex and filter out the index you want to drop.
scala> val myList = List(1,2,1,3,2)
myList: List[Int] = List(1, 2, 1, 3, 2)
scala> myList.zipWithIndex.filter(_._2 != 0).map(_._1)
res1: List[Int] = List(2, 1, 3, 2)
The filter + map is collect,
scala> myList.zipWithIndex.collect { case (elem, index) if index != 0 => elem }
res2: List[Int] = List(2, 1, 3, 2)
To remove first occurrence of elem, you can split at first occurance, drop the element and merge back.
list.span(_ != 1) match { case (before, atAndAfter) => before ::: atAndAfter.drop(1) }
Following is expanded answer,
val list = List(1, 2, 1, 3, 2)
//split AT first occurance
val elementToRemove = 1
val (beforeFirstOccurance, atAndAfterFirstOccurance) = list.span(_ != elementToRemove)
beforeFirstOccurance ::: atAndAfterFirstOccurance.drop(1) // shouldBe List(2, 1, 3, 2)
Resource
How to remove an item from a list in Scala having only its index?
How should I remove the first occurrence of an object from a list in Scala?
List is immutable, so you can’t delete elements from it, but you can filter out the elements you don’t want while you assign the result to a new variable:
scala> val originalList = List(5, 1, 4, 3, 2)
originalList: List[Int] = List(5, 1, 4, 3, 2)
scala> val newList = originalList.filter(_ > 2)
newList: List[Int] = List(5, 4, 3)
Rather than continually assigning the result of operations like this to a new variable, you can declare your variable as a var and reassign the result of the operation back to itself:
scala> var x = List(5, 1, 4, 3, 2)
x: List[Int] = List(5, 1, 4, 3, 2)
scala> x = x.filter(_ > 2)
x: List[Int] = List(5, 4, 3)
I am new to spark programming and scala and i am not able to understand the difference between map and flatMap.
I tried below code as i was expecting both to work but got error.
scala> val b = List("1","2", "4", "5")
b: List[String] = List(1, 2, 4, 5)
scala> b.map(x => (x,1))
res2: List[(String, Int)] = List((1,1), (2,1), (4,1), (5,1))
scala> b.flatMap(x => (x,1))
<console>:28: error: type mismatch;
found : (String, Int)
required: scala.collection.GenTraversableOnce[?]
b.flatMap(x => (x,1))
As per my understanding flatmap make Rdd in to collection for String/Int Rdd.
I was thinking that in this case both should work without any error.Please let me know where i am making the mistake.
Thanks
You need to look at how the signatures defined these methods:
def map[U: ClassTag](f: T => U): RDD[U]
map takes a function from type T to type U and returns an RDD[U].
On the other hand, flatMap:
def flatMap[U: ClassTag](f: T => TraversableOnce[U]): RDD[U]
Expects a function taking type T to a TraversableOnce[U], which is a trait Tuple2 doesn't implement, and returns an RDD[U]. Generally, you use flatMap when you want to flatten a collection of collections, i.e. if you had an RDD[List[List[Int]] and you want to produce a RDD[List[Int]] you can flatMap it using identity.
map(func) Return a new distributed dataset formed by passing each element of the source through a function func.
flatMap(func) Similar to map, but each input item can be mapped to 0 or more output items (so func should return a Seq rather than a single item).
The following example might be helpful.
scala> val b = List("1", "2", "4", "5")
b: List[String] = List(1, 2, 4, 5)
scala> b.map(x=>Set(x,1))
res69: List[scala.collection.immutable.Set[Any]] =
List(Set(1, 1), Set(2, 1), Set(4, 1), Set(5, 1))
scala> b.flatMap(x=>Set(x,1))
res70: List[Any] = List(1, 1, 2, 1, 4, 1, 5, 1)
scala> b.flatMap(x=>List(x,1))
res71: List[Any] = List(1, 1, 2, 1, 4, 1, 5, 1)
scala> b.flatMap(x=>List(x+1))
res75: scala.collection.immutable.Set[String] = List(11, 21, 41, 51) // concat
scala> val x = sc.parallelize(List("aa bb cc dd", "ee ff gg hh"), 2)
scala> val y = x.map(x => x.split(" ")) // split(" ") returns an array of words
scala> y.collect
res0: Array[Array[String]] = Array(Array(aa, bb, cc, dd), Array(ee, ff, gg, hh))
scala> val y = x.flatMap(x => x.split(" "))
scala> y.collect
res1: Array[String] = Array(aa, bb, cc, dd, ee, ff, gg, hh)
Map operation return type is U where as flatMap return type is TraversableOnce[U](means collections)
val b = List("1", "2", "4", "5")
val mapRDD = b.map { input => (input, 1) }
mapRDD.foreach(f => println(f._1 + " " + f._2))
val flatmapRDD = b.flatMap { input => List((input, 1)) }
flatmapRDD.foreach(f => println(f._1 + " " + f._2))
map does a 1-to-1 transformation, while flatMap converts a list of lists to a single list:
scala> val b = List(List(1,2,3), List(4,5,6), List(7,8,90))
b: List[List[Int]] = List(List(1, 2, 3), List(4, 5, 6), List(7, 8, 90))
scala> b.map(x => (x,1))
res1: List[(List[Int], Int)] = List((List(1, 2, 3),1), (List(4, 5, 6),1), (List(7, 8, 90),1))
scala> b.flatMap(x => x)
res2: List[Int] = List(1, 2, 3, 4, 5, 6, 7, 8, 90)
Also, flatMap is useful for filtering out None values if you have a list of Options:
scala> val c = List(Some(1), Some(2), None, Some(3), Some(4), None)
c: List[Option[Int]] = List(Some(1), Some(2), None, Some(3), Some(4), None)
scala> c.flatMap(x => x)
res3: List[Int] = List(1, 2, 3, 4)
This question already has answers here:
In Scala 2, type inference fails on Set made with .toSet?
(3 answers)
Closed 8 years ago.
Strange as it may seem, this does not work:
scala> (1 to 6).toSet map (_ / 2)
<console>:8: error: missing parameter type for expanded function ((x$1) => x$1.$div(2))
(1 to 6).toSet map (_ / 2)
^
However, using to[Set] instead of toSet does:
scala> (1 to 6).to[Set] map (_ / 2)
res0: scala.collection.immutable.Set[Int] = Set(2, 0, 3, 1)
Huh. o_O
Also consider that this works:
scala> val s = (1 to 6).toSet; s map (_ / 2)
s: scala.collection.immutable.Set[Int] = Set(5, 1, 6, 2, 3, 4)
res1: scala.collection.immutable.Set[Int] = Set(2, 0, 3, 1)
As Range.Inclusive is a first-order type, as suggested by #AlexIv, keep in mind that this also won't work with List[Int]:
scala> List(1, 2, 3, 4, 5, 6).toSet map (_ / 2)
<console>:8: error: missing parameter type for expanded function ((x$1) => x$1.$
div(2))
List(1, 2, 3, 4, 5, 6).toSet map (_ / 2)
^
And as previously, this works:
scala> val s = List[Int](1, 2, 3, 4, 5, 6).toSet; s map (_ / 2)
s: scala.collection.immutable.Set[Int] = Set(5, 1, 6, 2, 3, 4)
res3: scala.collection.immutable.Set[Int] = Set(2, 0, 3, 1)
Edit: duplicate of Type inference fails on Set made with .toSet?
The typer phase (scala -Xprint:typer) hides the answer:
private[this] val res7: <error> = Predef.intWrapper(1).to(6).toSet[B].map[B, That]({
((x: Nothing) => Predef.identity[Nothing](x))
})();
(1 to 6) returns a Range.Inclusive, which is a first-order type and not a type constructor, it's not parameterized, but Set[A] expects/requires you to provide it some type and returns you a type. When you call toSet, scalac expects some type, cause Inclusive doesn't have toSet method, it's inherited from TraversableOnce and is a generic method, so you need explicitly provide some type:
(1 to 6).toSet[Int].map(identity)
res0: scala.collection.immutable.Set[Int] = Set(5, 1, 6, 2, 3, 4)
toBuffer also doesn't work, other conversions works perfectly and this two methods have a similar implementation:
def toBuffer[B >: A]: mutable.Buffer[B] = to[ArrayBuffer].asInstanceOf[mutable.Buffer[B]]
def toSet[B >: A]: immutable.Set[B] = to[immutable.Set].asInstanceOf[immutable.Set[B]]