How to send a list as parameter in databricks notebook task? - rest

Question

I am using Databricks Resi API to create a job with notebook_task in an existing cluster and getting the job_id in return.
Then I am calling the run-now api to trigger the job.
In this step, I want to send a list as argument via the notebook_params, which throws an error saying "Expected non-array for field value".
Is there any way I can send a list as an argument to the job?
I have tried sending the list argument in base_params as well with same error.
user_json={
"name": job_name,
"existing_cluster_id": cluster_id,
"notebook_task": {
"notebook_path": notebook_path
},
"email_notifications":{
"on_failure":[email_id]
},
"max_retries": 0,
"timeout_seconds": 3600
}
response=requests.post('https://<databricks_uri>/2.0/jobs/create',headers=head,json=user_json,timeout=5, verify=False)
job_id=response.json()['job_id']
json_job={"job_id":job_id,"notebook_params":{"name":"john doe","my_list":my_list}}
response = requests.post('https://<databricks_uri>/2.0/jobs/run-now', headers=head, json=json_job, timeout=200, verify=False)

Answer 1

Not found any native solution yet, but my solution was to pass the list as a string and parse it back out on the other side:
json_job={"job_id":job_id,
"notebook_params":{
"name":"john doe",
"my_list":"spam,eggs"
}
}
Then in databricks:
my_list=dbutils.widgets.get("my_list")
my_list=my_list.split(",")
With appropriate care around special characters or e.g. conversion to numeric types.
If the objects in the list are more substantial, then sending them as a file to dbfs using the CLI or API before running the job may be another option to explore.

Answer 2

Hi may be I'm bit late but found a better solution.
Step 1:
Use JSON.stringyfy() in the console of any browser to convert your value(object, array, JSON etc) into string
Ex:
Now use this value in the body of URL
In Databricks notebook convert string to JSON using python json module.
Hope this helps