pyspark is hanging while attempting to make URL requests in parallel - scala

Question

I have an rdd of URLs and I want to make url requests in parallel. I am running get_menus with an rdd of url strings. Pyspark hangs when trying to collect(), take(), etc. on items.
def format_json(response, slug):
clean_response = str(response)[:-1]
clean_response += ", 'slug': '" + slug + "'}\n" # ', \"store-slug\": \"' + slug +'\"}'
clean_response = clean_response.replace("None", "null").replace("True", 'true').replace("False", "false")
return clean_response
def get_all(url, slug):
response = requests.get(url)
response = json.loads(response).get('data', {})
clean_response = format_json(response, slug)
return clean_response
def get_items(rdd_of_urls):
get_items_udf = udf(lambda x: get_all(x), StringType())
items = rdd_of_urls.map(get_items_udf)
return items
I see many GC allocation errors but don't know how to interpret them:
2019-08-21T18:40:04.666+0000: [GC (Allocation Failure) 2019-08-21T18:40:04.666+0000: [ParNew: 34369K->1837K(36864K), 0.0029907 secs] 127948K->95416K(186848K), 0.0030414 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2019-08-21T18:40:04.951+0000: [GC (Allocation Failure) 2019-08-21T18:40:04.951+0000: [ParNew: 34605K->2576K(36864K), 0.0036551 secs] 128184K->96155K(186848K), 0.0037126 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]