I have two ArrayList(List[String]) e.g:
ArrayList1: ArrayList1:
List(a, b, c) List(1, 2, 3)
List(d, e, f) List(4, 5, 6)
List(g,h, i) List(7,8, 9)
I want to use scala to join in parallel such as doing this
val ArrayList12 = ArrayList1 ++ ArrayList2
but this is not what I want since the ArrayList2 got added at the end of ArrayList1
Please I will appreciate help using scala to have a parallel join in this form:
List(a, b, c,1,2,3)
List(d, e, f,4,5,6)
List(g,h,i,7,8,9)
You can use zip.
val arrayList12 = arrayList1.zip(arrayList2).map(tuple => tuple._1 ++ tuple._2)
zip returns a List that combines each element of the first and the second list into a Tuple2, the resulting type is Seq[(List[A],List[Int])].
Then we can map that list to to a flat structure with map to concatenate the two lists in the Tuple.
Try this:
scala> List(List("a","b","c"), List("d", "e", "f"))
res1: List[List[String]] = List(List(a, b, c), List(d, e, f))
scala> List(List(1,2,3),List(4,5,6))
res2: List[List[Int]] = List(List(1, 2, 3), List(4, 5, 6))
scala> res1.zip(res2)
res3: List[(List[String], List[Int])] = List((List(a, b, c),List(1, 2, 3)), (List(d, e, f),List(4, 5, 6)))
which gives a List of tuples. You can then map() across these, adding the two elements of each tuple:
scala> res1.zip(res2).map(x => x._1 ++ x._2)
res5: List[List[Any]] = List(List(a, b, c, 1, 2, 3), List(d, e, f, 4, 5, 6))
Just another way of skinning the cat:
val arrayList1 = List(List("a","b","c"), List("d", "e", "f"), List("g","h", "i"))
val arrayList2 =List(List("1","2","3"), List("4", "5", "6"), List("7", "8", "9"))
List(arrayList1, arrayList2).transpose.map(_.flatten)
// List(List(a, b, c, 1, 2, 3), List(d, e, f, 4, 5, 6), List(g, h, i, 7, 8, 9))
But if I were doing it for real, I'd use Brian Agnew's answer
Related
I'm trying to write a function that takes multiple lists as input and returns a list of strings representation of every combination between those lists.
Sample Input :
val integers = List(1,2,3,4)
val characters = List('a','b','c')
val strings = List("apple","grapefruit")
Sample Output :
("1-a-apple", "1-a-grapefruit", .....)
Here is what I have so far :
def main(args: Array[String]): Unit = {
val integers = List(1,2,3,4)
val characters = List('a','b','c')
val strings = List("apple","grapefruit")
val lists = List(integers, characters, strings)
println(generator(lists).toString())
}
//using list api
def generator(x: List[List[Any]]): List[List[Any]] = x match {
case Nil => List(Nil)
case h :: _ => h.flatMap(i => generator(x.tail).map(i :: _))
}
And here is the output of my code :
List(List(1, a, apple), List(1, a, grapefruit), List(1, b, apple),
List(1, b, grapefruit), List(1, c, apple), List(1, c, grapefruit),
List(2, a, apple), List(2, a, grapefruit), List(2, b, apple), List(2,
b, grapefruit), List(2, c, apple), List(2, c, grapefruit), List(3, a,
apple), List(3, a, grapefruit), List(3, b, apple), List(3, b,
grapefruit), List(3, c, apple), List(3, c, grapefruit), List(4, a,
apple), List(4, a, grapefruit), List(4, b, apple), List(4, b,
grapefruit), List(4, c, apple), List(4, c, grapefruit))
So my questions are :
1)How can I get the output to be a list of strings instead of list of list of any?
2)How can I use a for comprehension instead of the list api?
val integers = List(1,2,3,4)
val characters = List('a','b','c')
val strings = List("apple","grapefruit")
for {
int <- integers
chars <- characters
string <- strings
} yield { s"${int}-${chars}-${string}" }
That code outputs
val res4: List[String] = List(1-a-apple, 1-a-grapefruit, 1-b-apple, 1-b-grapefruit, 1-c-apple, 1-c-grapefruit, 2-a-apple, 2-a-grapefruit, 2-b-apple, 2-b-grapefruit, 2-c-apple, 2-c-grapefruit, 3-a-apple, 3-a-grapefruit, 3-b-apple, 3-b-grapefruit, 3-c-apple, 3-c-grapefruit, 4-a-apple, 4-a-grapefruit, 4-b-apple, 4-b-grapefruit, 4-c-apple, 4-c-grapefruit)
Here's I'm using string interpolation (e.g. s" variable is $var1") to turn the items into strings.
Two newbie questions.
It seems that for comprehension knows about Options and can skip automatically None and unwrap Some, e.g.
val x = Map("a" -> List(1,2,3), "b" -> List(4,5,6), "c" -> List(7,8,9))
val r = for {map_key <- List("WRONG_KEY", "a", "b", "c")
map_value <- x get map_key } yield map_value
outputs:
r: List[List[Int]] = List(List(1, 2, 3), List(4, 5, 6), List(7, 8, 9))
Where do the Options go? Can someone please shed some light on how does this work? Can we always rely on this behaviour?
The second things is why this does not compile?
val x = Map("a" -> List(1,2,3), "b" -> List(4,5,6), "c" -> List(7,8,9))
val r = for {map_key <- List("WRONG_KEY", "a", "b", "c")
map_value <- x get map_key
list_value <- map_value
} yield list_value
It gives
Error:(57, 26) type mismatch;
found : List[Int]
required: Option[?]
list_value <- map_value
^
Looking at the type of the first example, I am not sure why we need to have an Option here?
For comprehensions are converted into calls to sequence of map or flatMap calls. See here
Your for loop is equivalent to
List("WRONG_KEY", "a", "b", "c").flatMap(
map_key => x.get(map_key).flatMap(map_value => map_value)
)
flatMap in Option is defined as
#inline final def flatMap[B](f: A => Option[B]): Option[B]
So it is not allowed to pass List as argument as you are notified by compiler.
I think the difference is due to the way for comprehensions are expanded into map() and flatMap method calls within the Seq trait.
For conciseness, lets define some variables:
var keys = List("WRONG_KEY", a, b, c)
Your first case is equivalent to:
val r = keys.flatMap(x.get(_))
whereas your second case is equivalent to:
val r= keys.flatMap(x.get(_).flatMap{ case y => y })
I think the issue is that Option.flatMap() should return an Option[], which is fine in the first case, but is not consistent in the second case with what the x.get().flatMap is passed, which is a List[Int].
These for-comprehension translation rules are explained in further detail in chapter 7 of "Programming Scala" by Wampler & Payne.
Maybe this small difference, setting parenthesis and calling flatten, makes it clear:
val r = for {map_key <- List("WRONG_KEY", "a", "b", "c")
| } yield x get map_key
r: List[Option[List[Int]]] = List(None, Some(List(1, 2, 3)), Some(List(4, 5, 6)), Some(List(7, 8, 9)))
val r = (for {map_key <- List("WRONG_KEY", "a", "b", "c")
| } yield x get map_key).flatten
r: List[List[Int]] = List(List(1, 2, 3), List(4, 5, 6), List(7, 8, 9))
That's equivalent to:
scala> List("WRONG_KEY", "a", "b", "c").map (x get _)
res81: List[Option[List[Int]]] = List(None, Some(List(1, 2, 3)), Some(List(4, 5, 6)), Some(List(7, 8, 9)))
scala> List("WRONG_KEY", "a", "b", "c").map (x get _).flatten
res82: List[List[Int]] = List(List(1, 2, 3), List(4, 5, 6), List(7, 8, 9))
The intermediate value (map_key) vanished as _ in the second block.
You are mixing up two different monads (List and Option) inside the for statement. This sometimes works as expected, but not always. In any case, you can trasform options into lists yourself:
for {
map_key <- List("WRONG_KEY", "a", "b", "c")
list_value <- x get map_key getOrElse Nil
} yield list_value
I have a queue of List[Any], but somehow dequeue doesn't return correct result and it doesn't remove the item from the queue either. Any idea why this happened?
Here is an example:
scala> val a = scala.collection.immutable.Queue(List(1, "A", "B", 987), List(2, "C", "D", 456), List(3, "E", "F", 123))
a: scala.collection.immutable.Queue[List[Any]] = Queue(List(1, A, B, 987), List(2, C, D, 456), List(3, E, F, 123))
scala> a.dequeue
res5: (List[Any], scala.collection.immutable.Queue[List[Any]]) = (List(1, A, B, 987),Queue(List(2, C, D, 456), List(3, E, F, 123)))
scala> a
res6: scala.collection.immutable.Queue[List[Any]] = Queue(List(1, A, B, 987), List(2, C, D, 456), List(3, E, F, 123))
You used an immutable queue, so dequeue does not modify the original queue but rather returns a modified one together with the head element as tuple:
val (elem, newQ) = oldQ.dequeue
For example, you can feed the tail queue into a recursive call for processing. Something like this:
def pq(q: Queue[List[Any]]): Unit = {
q.dequeue match {
case (head, tail) =>
println(head + "..."); // ... process head element ...
if (tail.nonEmpty) pq(tail)
}
}
I have a rdd of list of strings, like the following:
(['a','b','c'],['1','2','3','4'],['e','f'],...)
now I want to get the list consists of all pairwise combinations in each innerlist, like the following:
(('a','b'),('a','c'),('b','c'),('1','2'),('1','3'),('1','4'),('2','3'),'('2','4'),('3','4'),('e','f'),...)
How to do that?
You can use flatMap with List.combinations:
val rdd = sc.parallelize(Seq(List("a", "b", "c"), List("1", "2", "3", "4"), List("e", "f")))
// rdd: org.apache.spark.rdd.RDD[List[String]] = ParallelCollectionRDD[0] at parallelize at <console>:24
rdd.flatMap(list => list.combinations(2)).collect()
// res1: Array[List[String]] = Array(List(a, b), List(a, c), List(b, c), List(1, 2), List(1, 3), List(1, 4), List(2, 3), List(2, 4), List(3, 4), List(e, f))
If I have unknown number of Set[Int] (or List[Int]) as an input and want to combine
i don't know size of input List[Int] for which I need to produce these tuples as a final result, what's the best way to achieve this? My code looks like below.
Ok. Since combine(xs) yields a List[List[Any]] and you have a :: combine(xs) you just insert a into the the List of all combinations. You want to combine a with each element of the possible combinations. That lead me to this solution.
You can also generalize it to lists:List[List[T]] because when you combine from lists:List[List[Int]] you will get a List[List[Int]].
def combine[T](lists: List[List[T]]): List[List[T]] = lists match {
case Nil => lists
case x :: Nil => for(a <- x) yield List(a) //needed extra case because if comb(xs) is Nil in the for loop, it won't yield anything
case x :: xs => {
val comb = combine(xs) //since all further combinations are constant, you should keep it in a val
for{
a <- x
b <- comb
} yield a :: b
}
}
Tests:
val first = List(7, 3, 1)
val second = List(2, 8)
val third = List("a","b")
combine(List(first, second))
//yields List(List(7, 2), List(7, 8), List(3, 2), List(3, 8), List(1, 2), List(1, 8))
combine(List(first, second, third))
//yields List(List(7, 2, a), List(7, 2, b), List(7, 8, a), List(7, 8, b), List(3, 2, a), List(3, 2, b), List(3, 8, a), List(3, 8, b), List(1, 2, a), List(1, 2, b), List(1, 8, a), List(1, 8, b))
I think you can also generalize this to work with other collections than List, but then you can't use pattern-match this easily and you have to work via iterators.