Join two pipelinedRDDs - pyspark

Question

I am trying to join two pipelinedRDDs using .join() in pyspart jupyter notebook
First RDD:
primaryType.take(5)
['DECEPTIVE PRACTICE',
'CRIM SEXUAL ASSAULT',
'BURGLARY',
'THEFT',
'CRIM SEXUAL ASSAULT']
Second RDD:
districts.take(5)
['004', '022', '008', '003', '001']
Join RDDs:
rdd_joined = primaryType.join(districts)
rdd_joined.take(5)
Output:
[]
What am I donig wrong here?

Answer 1

There should be some unique key to join both the rdds, so use rdd.zipWithIndex() to create indexes for both the rdds and then try to join them
districts.take(5)
['004', '022', '008', '003', '001']
primaryType.take(5)
['DECEPTIVE PRACTICE',
'CRIM SEXUAL ASSAULT',
'BURGLARY',
'THEFT',
'CRIM SEXUAL ASSAULT']
districts=districts.zipWithIndex()
districts.take(5)
[('004', 0), ('022', 1), ('008', 2), ('003', 3), ('001', 4)]
districts=districts.map(lambda (x,y):(y,x))
primaryType=primaryType.zipWithIndex()
primaryType=primaryType.map(lambda (x,y):(y,x))
primaryType.join(districts).map(lambda (x,y):y).take(5)
[('DECEPTIVE PRACTICE', '004'), ('CRIM SEXUAL ASSAULT', '001'), ('CRIM SEXUAL ASSAULT', '022'), ('BURGLARY', '008'), ('THEFT', '003')]