Pyspark: Passing full dictionary to each task - pyspark

Question

PySpark: I want to pass my custom dictionary which contains the distances of several locations to each task in Pyspark as for each row in my rdd, I need to calculate the distances from each location and every location in dictionary and take the minimum distance. broadcast didnt solve my problem.
Example:
dict = {(a,3),(b,6),(c,2)}
RDD :
(location1, 5)
(location2, 9)
(location3, 8)
Output: (location1,1)
(location2,3)
(location3,2)
Please help and thanks

Answer 1

A broadcast variable will definitely solve your problem in this case, though you could also just pass the dictionary (or list--see below) in your map function. Whether using a broadcast variable is worthwhile depends on the size of the object.
First of all, since all you want is the minimum distance, it looks like you don't care about the keys of the dictionary, just the values. If that list is sorted, it will make it possible to find the minimum distance efficiently.
>>> d = {'a': 3, 'b': 6, 'c': 2}
>>> locations = sorted(d.itervalues())
>>> rdd = sc.parallelize([('location1', 5), ('location2', 9), ('location3', 8)])
Now define a function to find the minimum distance using bisect.bisect. We make a function of a single element from the general function using functools.partial to fix the second argument.
>>> from functools import partial
>>> from bisect import bisect
>>> def find_min_distance(loc, locations):
... ind = bisect(locations, loc)
... if ind == len(locations):
... return loc - locations[-1]
... elif ind == 0:
... return locations[0] - loc
... else:
... left_dist = loc - locations[ind - 1]
... right_dist = locations[ind] - loc
... return min(left_dist, right_dist)
>>> mapper = partial(find_min_distance, locations=locations)
>>> rdd.mapValues(mapper).collect()
[('location1', 1), ('location2', 3), ('location3', 2)]
To do this instead with a broadcast variable:
>>> locations_bv = sc.broadcast(locations)
>>> def mapper(loc):
... return find_min_distance(loc, locations_bv.value)
...
>>> rdd.mapValues(mapper).collect()
[('location1', 1), ('location2', 3), ('location3', 2)]