how to map an DataFrame to an EdgeRDD - scala

Question

I have a DataFrame like:
val data = sc.parallelize(Array((1,10,10,7,7),(2,7,7,7,8),(3, 5,5,6,8))).toDF("id","col1","col2","col3","col4")
What I want to do is to create an EdgeRDD where two ids share a link if they share the same value in at least one of the columns
id col1 col2 col3 col4
1 10 10 7 7
2 7 7 7 8
3 5 5 6 8
then node 1 and 2 have an undirected link 1--2, because they share a common value in col3.
For the same reason, node 2 and 3 share an undirected link because they share a common value in col4
I know how to resolve this in a ugly way (but I have way too many columns to adopt this strategy in my real case)
val data2 = data.withColumnRenamed("id", "idd").withColumnRenamed("col1", "col1d").withColumnRenamed("col2", "col2d").withColumnRenamed("col3", "col3d").withColumnRenamed("col4", "col4d")
val res = data.join(data2, data("id") < data2("idd")
&& (data("col1") === data2("col1d")
|| data("col2") === data2("col2d")
|| data("col3") === data2("col3d")
|| data("col4") === data2("col4d")))
//> res : org.apache.spark.sql.DataFrame = [id: int, col1: int, col2: int, col
//| 3: int, col4: int, idd: int, col1d: int, col2d: int, col3d: int, col4d: int
//| ]
res.show //> +---+----+----+----+----+---+-----+-----+-----+-----+
//| | id|col1|col2|col3|col4|idd|col1d|col2d|col3d|col4d|
//| +---+----+----+----+----+---+-----+-----+-----+-----+
//| | 1| 10| 10| 7| 7| 2| 7| 7| 7| 8|
//| | 2| 7| 7| 7| 8| 3| 5| 5| 6| 8|
//| +---+----+----+----+----+---+-----+-----+-----+-----+
//|
val links = EdgeRDD.fromEdges(res.map(row => Edge(row.getAs[Int]("id").toLong, row.getAs[Int]("idd").toLong, "indirect")))
//> links : org.apache.spark.graphx.impl.EdgeRDDImpl[String,Nothing] = EdgeRDD
//| Impl[27] at RDD at EdgeRDD.scala:42
links.foreach(println) //> Edge(1,2,indirect)
//| Edge(2,3,indirect)
how to resolve this for much more columns?

Answer 1

Do you mean something like this?
val expr = data.columns.diff(Seq("id"))
.map(c => data(c) === data2(s"${c}d"))
.reduce(_ || _)
data.join(data2, data("id") < data2("idd") && expr)
You can also use aliases
import org.apache.spark.sql.functions.col
val expr = data.columns.diff(Seq("id"))
.map(c => col(s"d1.$c") === col(s"d2.$c"))
.reduce(_ || _)
data.alias("d1").join(data.alias("d2"), col("d1.id") < col("d2.id") && expr)
You can easily follow each of this by as simple select ($ is equivalent to col but requires an import of sqlContext.implicits.StringToColumn)
.select($"id".cast("long"), $"idd".cast("long"))
or
.select($"d1.id".cast("long"), $"d2.id".cast("long"))
and a pattern matching:
.rdd.map { case Row(src: Long, dst: Long) => Edge(src, dst, "indirect") }
Just note that logical disjunctions like this one cannot be optimized and are expanded to a Cartesian product followed by a filter. If you want to avoid you can try to approach this problem in different ways.
Lets start with reshaping data from wide to long:
val expr = explode(array(data.columns.tail.map(
c => struct(lit(c).alias("column"), col(c).alias("value"))
): _*))
val long = data.withColumn("tmp", expr)
.select($"id", $"tmp.column", $"tmp.value")
This will give us a DataFrame with a following schema:
long.printSchema
// root
// |-- id: integer (nullable = false)
// |-- column: string (nullable = false)
// |-- value: integer (nullable = false)
With data like this you have multiple choices including optimized join:
val pairs = long.as("long1")
.join(long.as("long2"),
$"long1.column" === $"long2.column" && // Optimized
$"long1.value" === $"long2.value" && // Optimized
$"long1.id" < $"long2.id" // Not optimized - filtered after sort-merge join
)
// Select only ids
.select($"long1.id".alias("src"), $"long2.id".alias("dst"))
// And keep distict
.distinct
pairs.show
// +---+---+
// |src|dst|
// +---+---+
// | 1| 2|
// | 2| 3|
// +---+---+
This can further improved by using different hashing techniques to avoid a large number of record generated by explode.
You can also think about this problem as bipartite graph where observations belong to on category of nodes and property-value pairs to another.
sealed trait CustomNode
case class Record(id: Long) extends CustomNode
case class Property(name: String, value: Int) extends CustomNode
With this as a starting point you can use long to generate edges of the following type:
Record -> Property
and solve this problem using GraphX directly by searching for paths like
Record -> Property <- Record
Hint: Collect neighbors for each property and propagate back.
Same as before you should consider using hashing or buckets to reduce limit a number of the generated Property nodes.