Scala is a must for Spark? - scala

Question

I am new to Spark. In its docs, it says It is available in either Scala or Python.
And some blogs says spark depends on scala (for instance, http://cn.soulmachine.me/blog/20130614/). Therefore, I am wondering : Is scala a must for Spark? (Do I have to install scala first due to the dependency?)

Answer 1

The API of Scala has the following language bindings:
Scala
Java
Python
Scala as a natural fit, since it supports strongly functional programming, which is obvious benefical in the area of Big Data. Most tutorials and coding snippets, which you find on the net, are written in Scala.
Concerning the runtimne depenendencies please have a look at
the project download page
"Spark runs on Java 6+ and Python 2.6+. For the Scala API, Spark 1.2.0 uses Scala 2.10. You will need to use a compatible Scala version (2.10.x)."

Answer 2

Java is a must for Spark + many other transitive dependencies (scala compiler is just a library for JVM). PySpark just connects remotely (by socket) to the JVM using Py4J (Python-Java interoperation). Py4J is included in PySpark.
PySpark requires Python 2.6 or higher. PySpark applications are
executed using a standard CPython interpreter in order to support
Python modules that use C extensions. We have not tested PySpark with
Python 3 or with alternative Python interpreters, such as PyPy or
Jython.
All of PySpark’s library dependencies, including Py4J, are bundled
with PySpark and automatically imported.
Standalone PySpark applications should be run using the bin/pyspark
script, which automatically configures the Java and Python environment
using the settings in conf/spark-env.sh or .cmd. The script
automatically adds the bin/pyspark package to the PYTHONPATH.
https://spark.apache.org/docs/0.9.1/python-programming-guide.html - this instruction shows how to build and run all this with Scala/Java Build Tool (SBT), which will download all dependencies (including scala) automatically from remote repository. Yo can also use Maven.
If you don't want Java on your machine - you can start it on any other and configure PySpark for using it (by SparkConf().setMaster).
So, you need Java for master node with Spark itself (and all java-dependencies like scala), and Python 2.6 for py-client