JSONDecodeError when connecting to Dataverse with Databricks - pyspark

Question

When executing the code listed below to access Azure Dataverse Endpoint with Databricks I get the following error
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The code is as follows:
list_data = call_dataverse_endpoint(f'https://{my_dv_orgName}.crm11.dynamics.com/api/data/v9.2/{my_dv_entity}')
The full error message is as follows:
JSONDecodeError Traceback (most recent call last)
<command-2234217625858083> in <module>
----> 1 list_data = call_dataverse_endpoint(f'https://{my_dv_orgName}.crm11.dynamics.com/api/data/v9.2/{my_dv_entity}')
<command-3028378077557710> in call_dataverse_endpoint(endpoint)
16
17 # Loop through the responses until odata.nextLink is gone.
---> 18 while "#odata.nextLink" in response.json():
19 # Append the data returned by the endpoint to the list
20 data.extend(response.json()["value"])
/databricks/python/lib/python3.8/site-packages/requests/models.py in json(self, **kwargs)
898 # used.
899 pass
--> 900 return complexjson.loads(self.text, **kwargs)
901
902 #property
/usr/lib/python3.8/json/__init__.py in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
355 parse_int is None and parse_float is None and
356 parse_constant is None and object_pairs_hook is None and not kw):
--> 357 return _default_decoder.decode(s)
358 if cls is None:
359 cls = JSONDecoder
/usr/lib/python3.8/json/decoder.py in decode(self, s, _w)
335
336 """
--> 337 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
338 end = _w(s, end).end()
339 if end != len(s):
/usr/lib/python3.8/json/decoder.py in raw_decode(self, s, idx)
353 obj, end = self.scan_once(s, idx)
354 except StopIteration as err:
--> 355 raise JSONDecodeError("Expecting value", s, err.value) from None
356 return obj, end
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The function that is being called is as follows:
def call_dataverse_endpoint(endpoint):
# Empty list
data = []
# Headers
headers = {
"Authorization": f"Bearer {my_dv_accessToken}",
"Accept": "application/json",
"Content-Type": "application/json; charset=utf-8"
}
# Initial request
response = requests.get(endpoint, headers=headers)
#print(print(f''' Fetching from initial url: {endpoint} ''' ))
# Loop through the responses until odata.nextLink is gone.
while "#odata.nextLink" in response.json():
# Append the data returned by the endpoint to the list
data.extend(response.json()["value"])
# Request the odata.nextLink URL
response = requests.get(response.json()["#odata.nextLink"], headers=headers)
#print(f''' Fetching from successive url: {response.json()["#odata.nextLink"]} ''' )
# Append nextLink response data
data.extend(response.json()["value"])
# Return
return data
Any thoughts?