Managing Scala dependencies in Databricks notebooks - scala

Question

I'm a new dev on a big Scala project where all the code is stored as notebook and run inside Databricks Clusters...
Each notebook defines classes and methods, and we have 'Main' notebooks which have very few lines of codes, but execute all needed Scala notebooks (i.e. nearly all the notebooks in this project) in cells such as %run ./myPackage/Foo. Then these 'Main' notebooks have one little Scala code cell like this:
import com.bar.foo.Main
Main.main()
Furthermore, each notebook imports the package it needs, as Scala instructions import com.bar.foo.MyClass.
I find this really annoying:
If I move one notebook I must update all the %run path/Notebook commands inside all my main notebooks/test notebooks
I feel that it's redundant to run notebooks inside the main notebooks and import the package inside all the other notebooks.
Do you know another workflow? Is there a simpler way to work with multiple Scala notebooks inside Databricks?

Answer 1

I think that these issues occur when users and companies consider notebooks a replacement of software engineering principles. The software world in order to address these issues created and uses extensively the design patterns which is hard (if not impossible) to apply them with notebooks. Therefore, I think that users shouldn't handle notebooks as a tool to develop their end-user solutions. The main role of the notebooks used to be for prototyping and ML testing therefore by definition they are not suitable for cases where modularity and scalability are important factors.
As for your case and presuming that the usage of the notebooks is unavoidable I would suggest to minimize the usage of notebooks and start organising your code into JAR libraries. That would be useful if the notebooks share a significant part of the code between them.
Let's consider for instance the case where the notebook N1 and N2 are both using the notebooks N3 and N4. You could then place the implementation of N3 and N4 into a JAR, let's call it common_lib.jar and then make common_lib.jar available to both N1 and N2 by attaching it to the cluster where they run (assuming that you run a notebook job). By following this approach you achieve:
Better modularity since you completely separate the functionality of your notebooks. Also for each job/notebook you can attach the exact dependencies to the cluster avoiding redundant dependencies that occurs because of the difficulty to separate your notebook application into modules.
More maintainable code. Eventually you should have one final notebook per module that imports the dependencies as you would do in a common scala application avoiding the complex hierarchy that is required by calling multiple notebooks.
More scalable code. The notebooks provide a poor interface dbutils.widget.text(...) and dbutils.widget.get(...) is definitely much less that what you can achieve with scala/java.
More testable code. You should know by now that with notebooks is very hard to implement proper unit or integration testing. By having the main implementation into a jar you could execute unit testing as you would do with any scala/java application.
UPDATE
One solution for your case (refactoring to JAR libraries is not possible) would be to organise the notebooks into modules where each one will use an _includes_ file responsible for all the dependencies of the module. The _includes_ file could look as the next snippet:
%run "myproject/lib/notebook_1"
%run "myproject/lib/notebook_3"
...
Now let's assume that notebooks X1 and X2 they share the same dependencies myproject/lib/notebook_1 and myproject/lib/notebook_3 in order to use the mentioned dependencies you should just place the _includes_ file under the same folder and execute:
%run "_includes_"
in the first cell of the X1 and/or X2 notebook. In this manner you have a common way to include all the dependencies of your project and you avoid cases where you need to copy/paste all the includes repeatedly.
This doesn't provide an automated way to check and include the correct path of the dependencies in your project although it could be a significant improvement. By the way I am not aware of such a automated way for going through the files and changing the imports dynamically. One way though is to write an external custom script. Although this script it shouldn't be invoked through your job.
Note: you must ensure that the hierarchy of the dependencies is well defined and you don't have any circular dependencies.