Run same Databricks notebook for different arguments concurrently? - pyspark

Question

The following code (not mine) is able to run NotebookA and NotebookB concurrently. I need some help to figure out how to pass multiple arguments to the same notebooks.
I want to pass this list of arguments to each notebook:
args = {}
args["arg1"] = "some value"
args["arg2"] = "another value"
If I wanted to pass the arguments above to each of the running notebooks, what will I need to amend in the code below?
Here is the working code:
from multiprocessing.pool import ThreadPool
pool = ThreadPool(10)
inputs = [("NotebookA", "NotebookB") ]
run_in_parallel = lambda x: dbutils.notebook.run(x, 1800)
from concurrent.futures import ThreadPoolExecutor, wait
pool = ThreadPoolExecutor(3)
results = []
with ThreadPoolExecutor(3) as pool:
for x in inputs:
results.extend(pool.map(run_in_parallel, list(x)))

Answer 1

The dbutils.notebook.run accepts the 3rd argument as well, this is a map of parameters (see documentation for more details). So in your case, you'll need to change definition of the run_in_parallel to something like this:
run_in_parallel = lambda x: dbutils.notebook.run(x, 1800, args)
and the rest of the code should be the same.
If you'll want to pass different arguments to different notebooks, then you'll need to have a list of tuples, and pass this list to a map, like this:
data = [('notebook 1', {'arg1':'abc'}), ('notebook2', {'arg1': 'def', 'arg2': 'jkl'})]
...
run_in_parallel = lambda x: dbutils.notebook.run(x[0], 1800, x[1])
with ThreadPoolExecutor(3) as pool:
results.extend(pool.map(run_in_parallel, data))