Flatten an element in spark scala - scala

Question

I have data like this in an RDD:
RDD[((Int, Int, Int), ((Int, Int), Int))]
as:
(((9,679,16),((2,274),1)), ((250,976,13),((2,218),1)))
I want output as :
((9,679,16,2,274,1),(250,976,13,2,218,1))
After Joining 2 rdds with:
val joinSale = salesTwo.join(saleFinal)
I got that result set. I tried the following code.
joinSale.flatMap(x => x).take(100).foreach(println)
I have tried map/flatMap but couldn't do it. Any ideas how to implement a scenario like this ? Thanks in advance ..

Answer 1

You can do this with pattern matching in scala. Simply wrap your tuple modification logic within a map similar to the below:
val mappedJoinSale = joinSale.map { case ((a, b, c), ((d, e), f)) => (a, b, c, d, e, f) }
Using your example, we have:
scala> val example = sc.parallelize(Array(((9,679,16),((2,274),1)), ((250,976,13),((2,218),1))))
example: org.apache.spark.rdd.RDD[((Int, Int, Int), ((Int, Int), Int))] = ParallelCollectionRDD[0] at parallelize at <console>:12
scala> val mapped = example.map { case ((a, b, c), ((d, e), f)) => (a, b, c, d, e, f) }
mapped: org.apache.spark.rdd.RDD[(Int, Int, Int, Int, Int, Int)] = MappedRDD[1] at map at <console>:14
scala> mapped.take(2).foreach(println)
...
(9,679,16,2,274,1)
(250,976,13,2,218,1)

Answer 2

You could also create generic tuple flattener using marvelous shapeless library as follows:
import shapeless._
import shapeless.ops.tuple
trait LowLevelFlatten extends Poly1 {
implicit def anyFlat[T] = at[T](x => Tuple1(x))
}
object concat extends Poly2 {
implicit def atTuples[T1, T2](implicit prepend: tuple.Prepend[T1, T2]): Case.Aux[T1, T2, prepend.Out] =
at[T1,T2]((t1,t2) => prepend(t1,t2))
}
object flatten extends LowLevelFlatten {
implicit def tupleFlat[T, M](implicit
mapper: tuple.Mapper.Aux[T, flatten.type, M],
reducer: tuple.LeftReducer[M, concat.type]
): Case.Aux[T, reducer.Out] =
at[T](t => reducer(mapper(t)))
}
Now in any code where import shapeless._ exists you could use it as
joinSale.map(flatten)