Graph processing using GraphX in Apache Spark + big data - scala

Question

I have created a Graph using node and edges data in Spark application. Now, I want to come up with the adjacency list for the created graph. How can I achieve this ?
I have written following code to read csv files for node and edge data and creating the Graph.
val grapha = sc.textFile("graph.csv")
val getgdata = grapha.map(line=>line.split(","))
val node1 = getgdata.map(line=>(line(3).toLong,(line(0)))).distinct
val node2 = getgdata.map(line=>(line(4).toLong,(line(1)))).distinct
// This is node list of a graph.
val nodes = node1.union(node2).distinct
//This is edge list.
val routes = getgdata.map(line=>
(Edge(line(3).toLong,line(4).toLong,line(2)))).distinct
// now create graph using Graph library
val graphx = Graph(nodes,routes)
Now I need to see adjacency list for each node from this graph. How can I do it using scala?

Answer 1

Looking at your code, I am assuming that your graph.csv looks like following,
node_1, node_2, node_1_relation_node_2, 1, 2
node_1, node_3, node_1_relation_node_3, 1, 3
node_2, node_3, node_2_relation_node_3, 2, 3
Now, you can read this into an RDD as follows,
val graphData = sc.textFile("graph.csv").map(line => line.split(","))
Now, you to create your graph you need two things,
RDD of Vertices,
val verticesRdd = graphData.flatMap(line => List(
(line(3), line(0)),
(line(4), line(1))
)).distinct
RDD of Edges,
val edgesRdd = graphData.map(line => Edge(line(3), line(4), line(2))).distinct
Now, you can create your graph as,
val graph = Graph(verticesRdd, edgesRdd)
But if you just need adjacency list, you can obtain it just from the graphData as following,
val adjacencyRdd = graphData
.flatMap(line => {
val v1 = line(3).toLong
val v2 = line(4).toLong
List(
(v1, v2),
(v2, v1)
)
)
.aggregateByKey(Set.empty[Long])(
{ case (adjacencySet, vertexId) => adjacencySet + vertexId }
{ case (adjacencySet1, adjacencySet2) => adjacencySet1 ++ adjacencySet2 }
)
.map({ case (vertexId, adjacencySet) => (vertextId, adjacencySet.toList) })