rdd.sortByKey gives wrong result - scala

Question

I copied sortByKey's body and renamed to sortByKey2, but they give different results. Why the first result is wrong here? This was run in eclipse. I restarted eclipse and still got the wrong result.
package test.spark
import org.apache.spark.sql.SparkSession
object RddTests {
var spark = SparkSession.builder().appName("rdd-test").master("local[*]")
.enableHiveSupport()
.getOrCreate()
val sc = spark.sparkContext
def main(args: Array[String]) {
//mapValues
//combineWithKey
//foldByKey
sortByKey
sortByKey2
}
def sortByKey() {
val people = List(("Mobin", 2), ("Mobin", 1), ("Lucy", 2), ("Amy", 1), ("Lucy", 3), ("Lucy", 1))
val rdd = sc.parallelize(people)
val sortByKeyRDD = rdd.sortByKey()
println;println("sortByKeyRDD")
sortByKeyRDD.foreach(println)
}
def sortByKey2() {
val people = List(("Mobin", 2), ("Mobin", 1), ("Lucy", 2), ("Amy", 1), ("Lucy", 3), ("Lucy", 1))
val rdd = sc.parallelize(people)
val sortByKeyRDD = rdd.sortByKey()
println;println("sortByKeyRDD2")
sortByKeyRDD.foreach(println)
}
}
The output is:
[Stage 0:> (0 + 0) / 4]
sortByKeyRDD
(Mobin,2)
(Mobin,1)
(Amy,1)
(Lucy,2)
(Lucy,3)
(Lucy,1)
sortByKeyRDD2
(Amy,1)
(Mobin,2)
(Mobin,1)
(Lucy,2)
(Lucy,3)
(Lucy,1)

Answer 1

foreach does not guarantee that the elements will be processed in any particular order. If you do sortByKeyRDD.collect.foreach(println) you will see the results in order, although this assumes that your data will fit in driver memory.
As noted in the sortByKey documentation:
Calling collect or save on the resulting RDD will return or output an ordered list of records
[EDIT] Using toLocalIterator instead of collect limits the driver memory requirement to the largest single partition. Thanks to user8371915 for pointing that out in a comment.

Answer 2

It is important to understand here how methods like foreach() or sortByKey() works in spark.
When you try to sort your data and like to print the output using foreach(System.out::println), the driver distribute this method to each partition (i.e. node in case of cluster OR multiple threads in case of single machine). So each partition execute the foreach locally. This means you will not see the output that you want to see.
Possible solution that people suggest, though not right solution in Bigdata,
sortByKeyRDD.coalesce(1).foreach(System.out::println);
or
sortByKeyRDD.collect().forEach(System.out::println);
Above solution is just for understanding purpose, I do not recommend to use it. If your data is large, it might give you out of memory exception as it try to collect all the data at driver for printing the output.