Related
I'm trying to add element to a List[String] while omitting annoying parenthesis. I tried this:
object Main extends App {
val l = List("fds")
val xs1: List[String] = l.+:("123") // ok
val xs2: List[String] = l +: "123" // compile-error
}
DEMO
Why is omitting parenthesis causing compile-error? These assignments look the same to me. What is the difference?
It's happening because of right associative methods.
scala> val l = List("abc")
l: List[String] = List(abc)
scala> "efg" +: l
res3: List[String] = List(efg, abc)
Read more here What good are right-associative methods in Scala?
Error case
scala> val l = List(1, 2, 3)
l: List[Int] = List(1, 2, 3)
scala> 4 +: l
res1: List[Int] = List(4, 1, 2, 3)
scala> l +: 1
<console>:13: error: value +: is not a member of Int
l +: 1
^
Because +: is right associative. Method +: is getting invoked on Int instead of list
In order to make it work we can explicitly invoke method on list without the special operator syntax
scala> val l = List(1, 2, 3)
l: List[Int] = List(1, 2, 3)
scala> l.+:(1)
res4: List[Int] = List(1, 1, 2, 3)
Above case works because its normal method invocation.
I am new to spark programming and scala and i am not able to understand the difference between map and flatMap.
I tried below code as i was expecting both to work but got error.
scala> val b = List("1","2", "4", "5")
b: List[String] = List(1, 2, 4, 5)
scala> b.map(x => (x,1))
res2: List[(String, Int)] = List((1,1), (2,1), (4,1), (5,1))
scala> b.flatMap(x => (x,1))
<console>:28: error: type mismatch;
found : (String, Int)
required: scala.collection.GenTraversableOnce[?]
b.flatMap(x => (x,1))
As per my understanding flatmap make Rdd in to collection for String/Int Rdd.
I was thinking that in this case both should work without any error.Please let me know where i am making the mistake.
Thanks
You need to look at how the signatures defined these methods:
def map[U: ClassTag](f: T => U): RDD[U]
map takes a function from type T to type U and returns an RDD[U].
On the other hand, flatMap:
def flatMap[U: ClassTag](f: T => TraversableOnce[U]): RDD[U]
Expects a function taking type T to a TraversableOnce[U], which is a trait Tuple2 doesn't implement, and returns an RDD[U]. Generally, you use flatMap when you want to flatten a collection of collections, i.e. if you had an RDD[List[List[Int]] and you want to produce a RDD[List[Int]] you can flatMap it using identity.
map(func) Return a new distributed dataset formed by passing each element of the source through a function func.
flatMap(func) Similar to map, but each input item can be mapped to 0 or more output items (so func should return a Seq rather than a single item).
The following example might be helpful.
scala> val b = List("1", "2", "4", "5")
b: List[String] = List(1, 2, 4, 5)
scala> b.map(x=>Set(x,1))
res69: List[scala.collection.immutable.Set[Any]] =
List(Set(1, 1), Set(2, 1), Set(4, 1), Set(5, 1))
scala> b.flatMap(x=>Set(x,1))
res70: List[Any] = List(1, 1, 2, 1, 4, 1, 5, 1)
scala> b.flatMap(x=>List(x,1))
res71: List[Any] = List(1, 1, 2, 1, 4, 1, 5, 1)
scala> b.flatMap(x=>List(x+1))
res75: scala.collection.immutable.Set[String] = List(11, 21, 41, 51) // concat
scala> val x = sc.parallelize(List("aa bb cc dd", "ee ff gg hh"), 2)
scala> val y = x.map(x => x.split(" ")) // split(" ") returns an array of words
scala> y.collect
res0: Array[Array[String]] = Array(Array(aa, bb, cc, dd), Array(ee, ff, gg, hh))
scala> val y = x.flatMap(x => x.split(" "))
scala> y.collect
res1: Array[String] = Array(aa, bb, cc, dd, ee, ff, gg, hh)
Map operation return type is U where as flatMap return type is TraversableOnce[U](means collections)
val b = List("1", "2", "4", "5")
val mapRDD = b.map { input => (input, 1) }
mapRDD.foreach(f => println(f._1 + " " + f._2))
val flatmapRDD = b.flatMap { input => List((input, 1)) }
flatmapRDD.foreach(f => println(f._1 + " " + f._2))
map does a 1-to-1 transformation, while flatMap converts a list of lists to a single list:
scala> val b = List(List(1,2,3), List(4,5,6), List(7,8,90))
b: List[List[Int]] = List(List(1, 2, 3), List(4, 5, 6), List(7, 8, 90))
scala> b.map(x => (x,1))
res1: List[(List[Int], Int)] = List((List(1, 2, 3),1), (List(4, 5, 6),1), (List(7, 8, 90),1))
scala> b.flatMap(x => x)
res2: List[Int] = List(1, 2, 3, 4, 5, 6, 7, 8, 90)
Also, flatMap is useful for filtering out None values if you have a list of Options:
scala> val c = List(Some(1), Some(2), None, Some(3), Some(4), None)
c: List[Option[Int]] = List(Some(1), Some(2), None, Some(3), Some(4), None)
scala> c.flatMap(x => x)
res3: List[Int] = List(1, 2, 3, 4)
I want to group elements of a list such as :
val lst = List(1,2,3,4,5)
On transformation it should return a new list as:
val newlst = List(List(1), List(1,2), List(1,2,3), List(1,2,3,4), Lis(1,2,3,4,5))
You can do it this way:
lst.inits.toList.reverse.tail
(1 to lst.size map lst.take).toList should do it.
Not as pretty or short as others, but gotta have some tail recursion for the soul:
def createFromElements(list: List[Int]): List[List[Int]] = {
#tailrec
def createFromElements(l: List[Int], p: List[List[Int]]): List[List[Int]] =
l match {
case x :: xs =>
createFromElements(xs, (p.headOption.getOrElse(List()) ++ List(x)) :: p)
case Nil => p.reverse
}
createFromElements(list, Nil)
}
And now:
scala> createFromElements(List(1,2,3,4,5))
res10: List[List[Int]] = List(List(1), List(1, 2), List(1, 2, 3), List(1, 2, 3, 4), List(1, 2, 3, 4, 5))
Doing a foldLeft seems to be more efficient, though ugly:
(lst.foldLeft((List[List[Int]](), List[Int]()))((x,y) => {
val z = x._2 :+ y;
(x._1 :+ z, z)
}))._1
I have an iterator of Options, and would like to find the first member that is:
Some
and meets a predicate
What's the best idiomatic way to do this?
Also: If an exception is thrown along the way, I'd like to ignore it and move on to the next member
optionIterator find { case Some(x) if predicate(x) => true case _ => false }
As for ignoring exceptions… Is it the predicate that could throw? 'Cause that's not really wise. Nonetheless…
optionIterator find {
case Some(x) => Try(predicate(x)) getOrElse false
case _ => false
}
Adding a coat of best and idiomatic to the paint job:
scala> val vs = (0 to 10) map { case 3 => None case i => Some(i) }
vs: scala.collection.immutable.IndexedSeq[Option[Int]] = Vector(Some(0), Some(1), Some(2), None, Some(4), Some(5), Some(6), Some(7), Some(8), Some(9), Some(10))
scala> def p(i: Int) = if (i % 2 == 0) i > 5 else ???
p: (i: Int)Boolean
scala> import util._
import util._
scala> val it = vs.iterator
it: Iterator[Option[Int]] = non-empty iterator
scala> it collectFirst { case Some(i) if Try(p(i)) getOrElse false => i }
res2: Option[Int] = Some(6)
Getting the first even number over five that doesn't blow up the test.
Assuming that you can wrap your predicate so that any error returns false:
iterator.flatMap(x => x).find(yourSafePredicate)
flatMap takes a collection of collections (which an iterable of Option is as Option and Either are considered collections with a max size of one) and transforms it into a single collection:
scala> for { x <- 1 to 3; y <- 1 to x } yield x :: y :: Nil
res30: IndexedSeq[List[Int]] = Vector(List(1, 1), List(2, 1), List(2, 2), List(3, 1), List(3, 2), List(3, 3))
scala> res30.flatMap(x => x)
res31: IndexedSeq[Int] = Vector(1, 1, 2, 1, 2, 2, 3, 1, 3, 2, 3, 3)
find returns the first entry in your iterable that matches a predicate as an Option or None if there is no match:
scala> (1 to 10).find(_ > 3)
res0: Option[Int] = Some(4)
scala> (1 to 10).find(_ == 11)
res1: Option[Int] = None
Some sample data
scala> val l = Seq(Some(1),None,Some(-7),Some(8))
l: Seq[Option[Int]] = List(Some(1), None, Some(-7), Some(8))
Using flatMap on a Seq of Options will produce a Seq of defined values, all the None's will be discarded
scala> l.flatMap(a => a)
res0: Seq[Int] = List(1, -7, 8)
Then use find on the sequence - you will get the first value, that satisfies the predicate. Pay attention, that found value is wrapped as Option, cause find should be able to return valid value (None) value in case of "not found" situation.
scala> l.flatMap(a => a).find(_ < 0)
res1: Option[Int] = Some(-7)
As far as I know it is "OK" way for the Scala.
Might be more idiomatic way is to use collect / collectFirst on the Seq ...
scala> l.collectFirst { case a#Some(x) if x < 0 => a }
res2: Option[Some[Int]] = Some(Some(-7))
Pay attention that here we have Some(Some(-7)) because the collectFind should have chance to produce "not found" value, so here 1st Some - from collectFirst, the 2nd Some - from the source elements of Seq of Option's.
You can flatten the Some(Some(-7)) if you need the values in your hand.
scala> l.collectFirst({ case a#Some(x) if x < 0 => a }).flatten
res3: Option[Int] = Some(-7)
If nothing found - you will have the None
scala> l.collectFirst({ case a#Some(x) if x < -10 => a }).flatten
res9: Option[Int] = None
I wish to add a list of tuples of integers i.e. given an input list of tuples of arity k, produce a tuple of arity k whose fields are sums of corresponding fields of the tuples in the list.
Input
List( (1,2,3), (2,3,-3), (1,1,1))
Output
(4, 6, 1)
I was trying to use foldLeft, but I am not able to get it to compile. Right now, I am using a for loop, but I was looking for a more concise solution.
This can be done type safely and very concisely using shapeless,
scala> import shapeless._, syntax.std.tuple._
import shapeless._
import syntax.std.tuple._
scala> val l = List((1, 2, 3), (2, 3, -1), (1, 1, 1))
l: List[(Int, Int, Int)] = List((1,2,3), (2,3,-1), (1,1,1))
scala> l.map(_.toList).transpose.map(_.sum)
res0: List[Int] = List(4, 6, 3)
Notice that unlike solutions which rely on casts, this approach is type safe, and any type errors are detected at compile time rather than at runtime,
scala> val l = List((1, 2, 3), (2, "foo", -1), (1, 1, 1))
l: List[(Int, Any, Int)] = List((1,2,3), (2,foo,-1), (1,1,1))
scala> l.map(_.toList).transpose.map(_.sum)
<console>:15: error: could not find implicit value for parameter num: Numeric[Any]
l.map(_.toList).transpose.map(_.sum)
^
scala> val tuples = List( (1,2,3), (2,3,-3), (1,1,1))
tuples: List[(Int, Int, Int)] = List((1,2,3), (2,3,-3), (1,1,1))
scala> tuples.map(t => t.productIterator.toList.map(_.asInstanceOf[Int])).transpose.map(_.sum)
res0: List[Int] = List(4, 6, 1)
Type information is lost when calling productIterator on Tuple3 so you have to convert from Any back to an Int.
If the tuples are always going to contain the same type I would suggest using another collection such as List. The Tuple is better suited for disparate types. When you have the same types and don't lose the type information by using productIterator the solution is more elegant.
scala> val tuples = List(List(1,2,3), List(2,3,-3), List(1,1,1))
tuples: List[List[Int]] = List(List(1, 2, 3), List(2, 3, -3), List(1, 1, 1))
scala> tuples.transpose.map(_.sum)
res1: List[Int] = List(4, 6, 1)
scala> val list = List( (1,2,3), (2,3,-3), (1,1,1))
list: List[(Int, Int, Int)] = List((1,2,3), (2,3,-3), (1,1,1))
scala> list.foldRight( (0, 0, 0) ){ case ((a, b, c), (a1, b1, c1)) => (a + a1, b + b1, c + c1) }
res0: (Int, Int, Int) = (4,6,1)