This about running Spark/Scala from a Zeppelin notebook.
In order to better modularize and reorganize code, I need to import existing Scala classes, packages or functions into the notebook, preferably skipping creating a jar file (much the same as in PySpark).
Something like:
import myclass
where 'myclass' is implemented in a .scala file. Probably this source code needs to reside in a specific location for Zeppelin.
Currently there's no such feature in zeppelin.
The only way of doing what you propose is to add a jar to Spark's classpath jars. At least, that's how I'm using it.
I wouldn't recommend the practice of importing scala classes from somewhere in a .scala file. That code should be packaged and made available for all workers, such as in all cluster workers and master.
Related
I have fair knowledge of scala and i use it in databricks for my data engineering needs. I want to create some customer libraries that i can use in all other notebooks. Here is what i'm looking for
create a scala notebook helperfunctions.scala which will have functions like ParseUrl(), GetUrl() etc
Deploy these libraries on databricks cluster
Call these libraries from another notebook using 'import from helperfunctions as fn' and use the functions
Can you give me an idea about how to get started? What does databricks offer?
I'd suggest not using notebooks as imports.
You can compile and package your functions as a JAR from plain JVM code using your preferred tooling, then upload it to something like JitPack or GitHub Packages, which you can then import your utilities as a Maven reference like other Spark dependencies
Im working on a project of frequent item sets, and I use the Algorithm FP-Growth, I depend on the version developed in Scala-Spark
https://github.com/apache/spark/blob/v2.1.0/mllib/src/main/scala/org/apache/spark/mllib/fpm/FPGrowth.scala
I need to modify this code and recompile it to have a jar file that I can include it to Spark-shell , and call its functions in spark
the problem s that spark-shell is un interpreter , and it finds errors in this file, Ive tried Sbt with eclipse but it did not succeded .
what i need is compiler that can use the last version of scala and spark-shel libraries to compile this file to jar file.
Got your question now!
All you need to do is add dependency jars(scala, java, etc.,) with respect to the machine you are going to use you own jar. Later on add the jars to spark-shell and you can use it like below,
spark-shell --jars your_jar.jar
Follow this steps:
check out Spark repository
modify files to want to modify
build project
run ./dev/make-distribution.sh script, which is inside Spark repository
run Spark Shell from your Spark distribution
I built a jar copy of Spark from https://github.com/apache/spark.git with some code modifications.
To call this jar into jupyter with Spark 1.5.1 (Scala 2.10) kernel, i used the %AddJar magic which looks like this:
%AddJar file:/Directory/To/filename.jar
My problem now is that whenever I try to call
import org.apache.spark.mllib.recommendation.ALS
the kernel calls the default implementation inside the kernel. Is there a way to call what's on my jar file instead?
The solution I came up with is this,
Use the same version of Spark as the one in the Spark kernel (In my case, Spark 1.5.1) so that the new jar file is compatible with the kernel.
Find the location of the particular edited jar file and replace it with the modified code. (In my case, I edited the mllib module so I had to find the mllib jar file and replace with the new one.)
Tip: Keep the original jar file in case the new code breaks.
I am new to spark but am trying to do some development. I am following "Reducing Build Times" instructions from the spark developer page. After creating the normal assembly I have written some classes that are dependent on one specific jar. I test my package in the spark-shell in which I have been able to include my jar by using defining SPARK_CLASSPATH, but the problem lies in actually compiling my code. What I want to achieve is to include that jar when compiling my added package (with build/sbt compile). Could I do that by adding a path to my jar in build/sbt file or sbt-launch-lib.bash, and if so how?
(Side note: I do not want to yet include the jar in the assembly because as I go I make some changes to it, and so it would be inconvenient. I am using Spark 1.4)
Any help is appreciated!
Based on the answer in the comments above, it looks like you are trying to add your jar as a dependency to the the mllib project as you do development on mllib itself. You can accomplish this by modifying the pom.xml file in the mllib directory within the Spark distribution.
You can find instructions on how to add a local file as a dependency here - http://blog.valdaris.com/post/custom-jar/. I haven't used this approach myself to including local file as a dependency, but I think it should work.
I am just getting started with Spark, so downloaded the for Hadoop 1 (HDP1, CDH3) binaries from here and extracted it on a Ubuntu VM. Without installing Scala, I was able to execute the examples in the Quick Start guide from the Spark interactive shell.
Does Spark come included with Scala? If yes, where are the libraries/binaries?
For running Spark in other modes (distributed), do I need to install Scala on all the nodes?
As a side note, I observed that Spark has one of the best documentation around open source projects.
Does Spark come included with Scala? If yes, where are the libraries/binaries?
The project configuration is placed in project/ folder. I my case here it is:
$ ls project/
build.properties plugins.sbt project SparkBuild.scala target
When you do sbt/sbt assembly, it downloads appropriate version of Scala along with other project dependencies. Checkout the folder target/ for example:
$ ls target/
scala-2.9.2 streams
Note that Scala version is 2.9.2 for me.
For running Spark in other modes (distributed), do I need to install Scala on all the nodes?
Yes. You can create a single assembly jar as described in Spark documentation
If your code depends on other projects, you will need to ensure they are also present on the slave nodes. A popular approach is to create an assembly jar (or “uber” jar) containing your code and its dependencies. Both sbt and Maven have assembly plugins. When creating assembly jars, list Spark itself as a provided dependency; it need not be bundled since it is already present on the slaves. Once you have an assembled jar, add it to the SparkContext as shown here. It is also possible to submit your dependent jars one-by-one when creating a SparkContext.
Praveen -
checked now the fat-master jar.
/SPARK_HOME/assembly/target/scala-2.9.3/spark-assembly_2.9.3-0.8.0-incubating-hadoop1.0.4.jar
this jar included with all the scala binaries + spark binaries.
you are able to run because this file is added to your CLASSPAH when you run spark-shell
check here : run spark-shell > http:// machine:4040 > environment > Classpath Entries
if you downloaded pre build spark , then you don't need to have scala in nodes, just this file in CLASSAPATH in nodes is enough.
note: deleted the last answer i posted, cause it may mislead some one. sorry :)
You do need Scala to be available on all nodes. However, with the binary distribution via make-distribution.sh, there is no longer a need to install Scala on all nodes. Keep in mind the distinction between installing Scala, which is necessary to run the REPL, and merely packaging Scala as just another jar file.
Also, as mentioned in the file:
# The distribution contains fat (assembly) jars that include the Scala library,
# so it is completely self contained.
# It does not contain source or *.class files.
So Scala does indeed come along for the ride when you use make-distribution.sh.
From spark 1.1 onwards, there is no SparkBuild.scala
You ahve to make your changes in pom.xml and build using Maven