I am just getting started with Spark, so downloaded the for Hadoop 1 (HDP1, CDH3) binaries from here and extracted it on a Ubuntu VM. Without installing Scala, I was able to execute the examples in the Quick Start guide from the Spark interactive shell.
Does Spark come included with Scala? If yes, where are the libraries/binaries?
For running Spark in other modes (distributed), do I need to install Scala on all the nodes?
As a side note, I observed that Spark has one of the best documentation around open source projects.
Does Spark come included with Scala? If yes, where are the libraries/binaries?
The project configuration is placed in project/ folder. I my case here it is:
$ ls project/
build.properties plugins.sbt project SparkBuild.scala target
When you do sbt/sbt assembly, it downloads appropriate version of Scala along with other project dependencies. Checkout the folder target/ for example:
$ ls target/
scala-2.9.2 streams
Note that Scala version is 2.9.2 for me.
For running Spark in other modes (distributed), do I need to install Scala on all the nodes?
Yes. You can create a single assembly jar as described in Spark documentation
If your code depends on other projects, you will need to ensure they are also present on the slave nodes. A popular approach is to create an assembly jar (or “uber” jar) containing your code and its dependencies. Both sbt and Maven have assembly plugins. When creating assembly jars, list Spark itself as a provided dependency; it need not be bundled since it is already present on the slaves. Once you have an assembled jar, add it to the SparkContext as shown here. It is also possible to submit your dependent jars one-by-one when creating a SparkContext.
Praveen -
checked now the fat-master jar.
/SPARK_HOME/assembly/target/scala-2.9.3/spark-assembly_2.9.3-0.8.0-incubating-hadoop1.0.4.jar
this jar included with all the scala binaries + spark binaries.
you are able to run because this file is added to your CLASSPAH when you run spark-shell
check here : run spark-shell > http:// machine:4040 > environment > Classpath Entries
if you downloaded pre build spark , then you don't need to have scala in nodes, just this file in CLASSAPATH in nodes is enough.
note: deleted the last answer i posted, cause it may mislead some one. sorry :)
You do need Scala to be available on all nodes. However, with the binary distribution via make-distribution.sh, there is no longer a need to install Scala on all nodes. Keep in mind the distinction between installing Scala, which is necessary to run the REPL, and merely packaging Scala as just another jar file.
Also, as mentioned in the file:
# The distribution contains fat (assembly) jars that include the Scala library,
# so it is completely self contained.
# It does not contain source or *.class files.
So Scala does indeed come along for the ride when you use make-distribution.sh.
From spark 1.1 onwards, there is no SparkBuild.scala
You ahve to make your changes in pom.xml and build using Maven
Related
I'm trying to figure out how to build scala from a source code archive. I see a build.sbt file but if I don't have scala installed, so how to do build scala?
I also see a Gemfile, implying that there are Ruby bindings. I checked the README.md but there isn't any information there sadly.
I don't know what to do to start building.
I'm assuming that you're talking about Scala 2 https://github.com/scala/scala and not Scala 3 https://github.com/lampepfl/dotty since you mentioned Gemfile (https://github.com/scala/scala/blob/2.13.x/Gemfile).
The Gemfile is for Travis CI. So you can ignore it.
If you can see build.sbt then in order to build a project you need JVM and sbt installed, not Scala
https://docs.scala-lang.org/getting-started/index.html
I checked the README.md but there isn't any information there sadly.
Actually, there is all necessary information in README:
https://github.com/scala/scala#using-the-sbt-build
sbt dist/mkBin generates runner scripts (scala, scalac, etc) in build/quick/bin
sbt dist/mkPack creates a build in the Scala distribution format in build/pack
this tool (https://scala-cli.virtuslab.org/) will let you easily download a working scala compiler & associated tools. they, in turn, will make it possible to build a compiler from the source tree. i'm assuming you've cloned https://github.com/lampepfl/dotty (scala 3)?
I'm building a job in Scala to run on a Flink Cluster, that will store data in AWS S3, and I've some problems related to dependencies.
I've checked most of the question previously asked here, and to fix this I needed to add flink-s3-fs-hadoop-1.9.1.jar jar files to the $FLINK_HOME/plugins in order to run my job successfully:
My question is, should this be detected as been inside the fatjar generate by sbt assembly ? The files are inside the jar, but for some reason the Flink Cluster can't see them.
I know that in the documentation says that flink-s3-fs-hadoop-1.9.1.jar should download to $FLINK_HOME/plugins folder.
Filesystems cannot be bundled in the user-jar, they must be present either in /lib or /plugins.
The components that use filesystems aren't necessarily aware of the user-jar.
Im working on a project of frequent item sets, and I use the Algorithm FP-Growth, I depend on the version developed in Scala-Spark
https://github.com/apache/spark/blob/v2.1.0/mllib/src/main/scala/org/apache/spark/mllib/fpm/FPGrowth.scala
I need to modify this code and recompile it to have a jar file that I can include it to Spark-shell , and call its functions in spark
the problem s that spark-shell is un interpreter , and it finds errors in this file, Ive tried Sbt with eclipse but it did not succeded .
what i need is compiler that can use the last version of scala and spark-shel libraries to compile this file to jar file.
Got your question now!
All you need to do is add dependency jars(scala, java, etc.,) with respect to the machine you are going to use you own jar. Later on add the jars to spark-shell and you can use it like below,
spark-shell --jars your_jar.jar
Follow this steps:
check out Spark repository
modify files to want to modify
build project
run ./dev/make-distribution.sh script, which is inside Spark repository
run Spark Shell from your Spark distribution
I am new to spark but am trying to do some development. I am following "Reducing Build Times" instructions from the spark developer page. After creating the normal assembly I have written some classes that are dependent on one specific jar. I test my package in the spark-shell in which I have been able to include my jar by using defining SPARK_CLASSPATH, but the problem lies in actually compiling my code. What I want to achieve is to include that jar when compiling my added package (with build/sbt compile). Could I do that by adding a path to my jar in build/sbt file or sbt-launch-lib.bash, and if so how?
(Side note: I do not want to yet include the jar in the assembly because as I go I make some changes to it, and so it would be inconvenient. I am using Spark 1.4)
Any help is appreciated!
Based on the answer in the comments above, it looks like you are trying to add your jar as a dependency to the the mllib project as you do development on mllib itself. You can accomplish this by modifying the pom.xml file in the mllib directory within the Spark distribution.
You can find instructions on how to add a local file as a dependency here - http://blog.valdaris.com/post/custom-jar/. I haven't used this approach myself to including local file as a dependency, but I think it should work.
As far as I understand, I have multiple versions/installs of Scala to be able to access it via Eclipse, bash/OS-X shell, and for SBT:
one version of Scala as supplied with the Scala IDE;
the Scala binaries to be able to run it from within a shell; and,
Scala as part of SBT.
Is my understanding correct? If so, is there any way to run with just the one version/install for all uses?
Is my understanding correct?
No. You don't "install" Scala. You just have multiple versions of Executable Jar file of scala-compiler, scala-library etc. The version that you have on your PATH is the one that seems installed but its nothing more than running a jar file.
TO run on a specific version, just add the scala jars to the classpath of your project. If you are using SBT, you can specify the scalaVersion in your build.sbt and it will add the proper Jar to the classpath