how to use achieve below requirement using spark RDD - scala

Question

I am really new for Spark. would you please help below require..
I have below source file. first field is name, second field is group id. I need to count how many group the name has, and list all the groups and count.
abc 1
abc 2
abc 3
xyz 1
xyz 3
def 2
def 4
lmn 6
I want to get below ex
name dept count
abc 1,2,3 3
xyz 1,3 2
def 2,4 2
lmn 6 1
thanks in advance.

Answer 1

Assuming you have a CSV file. So , first create dataframe using following steps.
import org.apache.spark.sql.types._
import org.apache.spark.Row
val members = sc.textFile("member").Map(lines => lines.split(",")).map(a => Row(a(0),a(1)))
val rddStruct = new StructType(Array(StructField("name", StringType, nullable = true),StructField("depart", StringType, nullable = true)))
val df = sqlContext.createDataFrame(members,rddStruct)
To achieve the output, following steps can be followed.
Apply a groupBy function can collect all departments as a set
val df2 = df.groupBy("Name").agg(collect_set("Depart").as("Depart"))
df2.show
+---+-----------+
| Name| Depart|
+---+-----------+
|lmn| [6]|
|def| [2, 4]|
|abc|[1, 2, 3]|
|xyz| [1, 2]|
+---+---------+
Then apply a size function on the Depart column to get the count.
val df3 = df2.withColumn("Count", size(df2("Depart")))
df3.show
+---+---------+-----+
| Name| Depart|Count|
+---+---------+-----+
|lmn| [6]| 1|
|def| [2, 4]| 2|
|abc|[1, 2, 3]| 3|
|xyz| [1, 2]| 2|
+---+---------+-----+
If result required should be sorted in descending order than you can apply a orderBy function on the above output.
val df4 = df3.orderBy(desc("Count"))
df4.show
+---+---------+-----+
| Name| Depart|Count|
+---+---------+-----+
|abc|[1, 2, 3]| 3|
|def| [2, 4]| 2|
|xyz| [1, 2]| 2|
|lmn| [6]| 1|
+---+---------+-----+
About structType you can read here

Answer 2

You can make it simple using RDD transformation:
scala> var rdd = sc.textFile("/data_test1")
scala> rdd.map(x => x.split(" ")).
map(x => (x(0), x(1))).
groupByKey().
map(x => (x._1, x._2.toSet.mkString(","),x._2.size)).
toDF("name", "dept", "count").show()
Output:
+----+-----+-----+
|name| dept|count|
+----+-----+-----+
| abc|1,2,3| 3|
| lmn| 6| 1|
| def| 2,4| 2|
| xyz| 1,3| 2|
+----+-----+-----+