I'm trying to build Apache Spark 1.4.0 with maven, the build never ends; it freezes and no error is shown.
I have no idea why but it freezes on this line :
[INFO] --- scala-maven-plugin:3.2.0:compile (scala-compile-first) # spark-launcher_2.10 ---
I try to update scala (2.11.4) and Maven (3.0.5) and it's the same.
Are you building it the way they suggest it?
Which is by running the included mvn command (see README).
Related
I have downloaded the spark source from apache site, then I built the source using maven.
spark - version 1.6.3
hadoop - version 2.7.3
scala - version 2.10.4
I have used below command for build the project
.build/mvn -Pyarn -Phadoop-2.7 -Dhadoop.version=2.7.3 -Phive -Phive-thriftserver -Dscala-2.10 -DskipTests clean package
I have tried with version - Phadoop-2.4, 2.6 but every time I am getting error at the time of hive build --
Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (scala-compile-first) on project spark-hive_2.10: wrap: scala.reflect.internal.MissingRequirementError: object scala.runtime in compiler mirror not found. -> [Help 1]
We are creating a customize version of Spark since we are changing some lines of code from ALS.scala. We build the customize spark version using
mvn command:
./make-distribution.sh --name custom-spark --tgz -Psparkr -Phadoop-2.6 -Phive -Phive-thriftserver -Pyarn
However, upon using the customized version of Spark, we run into this error:
Do you guys have some idea on what causes the error and how we might solve the issue?
I am actually using a jar file in the local machine by building them using sbt: sbt compile then sbt clean package and putting the jar file here: /Users/user/local/kernel/kernel-0.1.5-SNAPSHOT/lib.
However in the hadoop environment, the installation is different. Thus, I use maven to build spark and that's where the error comes in. I am thinking that this error might be dependent on using maven to build spark as there are some reports like this:
https://issues.apache.org/jira/browse/SPARK-2075
or probably on building spark assembly files
I have (multiple times) tried to install SBT-0.13.8 from the SBT download page via the SBT-0.13.8-MSI button and I always end up getting an SBT version which shows the following output
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
[info] Loading project definition from C:\Users\Tina\Desktop\example\project\project
error: error while loading CharSequence, class file 'C:\Program Files\Java\jre1.
8.0_20\lib\rt.jar(java/lang/CharSequence.class)' is broken (bad constant pool tag 15 at byte 1470)
[error] Type error in expression
Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore? i
[warn] Ignoring load failure: no project loaded.
> about
[info] This is sbt 0.12.4
[info] No project is currently loaded
[info] sbt, sbt plugins, and build definitions are using Scala 2.9.2
SBT produces an error message and (!) shows itself as version 0.12.4.!!!
I really made sure that I have no other version of SBT installed and I even made a reboot before re-installing it but nothing changed. SBT files have a creation date of March, 21, 2015. This seems to be the newest version but why does this version show up as 0.12.4 and does not work with JDK1.8?
Looks like the directory you're in is wrong, you want to launch sbt from C:\Users\Tina\Desktop\example\project not C:\Users\Tina\Desktop\example\project\project (note 1 project less).
The second problem you're having is you're trying to use sbt 0.12.4 with Java 8. sbt 0.12.4 uses (as can be seen from the output) Scala 2.9.2 which doesn't work with Java 8. I would recommend migrating to sbt 0.13.8, but if not you could always downgrade to Java 7.
Edit after comments below
The progfun Coursera course download is still configured with sbt
0.12.4 which you're having trouble building because you are using Java 8.
I had a quick go at trying to see if upgrading the project to 0.13.8 would work (FYI it involves editing project/build.properties), but then plugins need to be upgraded, then meta-build code needed to be changed because of namespace changes..
Unless you're willing to experiment more and work you're way through the various changes required to use latest sbt and latest Java (which is what I would do if this were for myself), I would recommend as an alternative to:
Downgrade to Java 7
Notify the relevant people involved in running the course that the source material should be updated as it demonstrably does not work on latest, stable Java.
If your default Java is Java 8 but you want to switch to Java 7 for this out-of-date project setup, you need to create an .sbtopts file at the root of the project containing (note: on separate lines!):
-java-home
C:\Users\Tina\PATH\TO\YOUR\JAVA\HOME
I've been trying to compile Apache spark with scala-2.11.1 (the latest version at the time). However, each time I try it ends up compiling everything to scala-2.10.*. I don't understand why.
The official documentation suggests that we use maven for compilation after switching to 2.11 using script in the dev/ folder.
What if I wanted to use sbt instead?
You need to enable scala-2.11 profile
>sbt -Dscala-2.11=true
sbt> compile
I'm new to Scala and Lift and SBT. I've cloned Ecommerce example from Lift's sample applications.
Now I want to simply run the application.
Documentation says, that container:start should start the app. But I'm getting error:
> container:start
[error] No action named 'container:start' exists.
[info] Execute 'help' for a list of commands or 'actions' for a list of available project actions and methods.
[info]
[info] Total time: 0 s, completed 25.05.2013 16:47:05
I guess that maybe it's because version difference.
[info] using LiftProject with sbt 0.7.5 and Scala 2.7.7
How can I start this application ?
And if it's caused by version difference is it possible to upgrade the application to latest sbt and scala version ?
Update
I've tried to use jetty-start and received another error:
> jetty-start
[error] No action named 'jetty-start' exists.
[info] Execute 'help' for a list of commands or 'actions' for a list of available project actions and methods.
I've also found another similar command jetty-run from sbt help. When I run it I'm getting a huge number of compilation errors. It seems like my scala 2.9.2 cannot compile the code.
This project, as far as I see, uses the old SBT version. The old way to start the web application is:
./sbt
> jetty-start
if you want to reload the app on source change, use ~ jetty-start
Note that there's a far more up to date liftweb project example here: https://github.com/lift/lift_25_sbt
probably sbt did not download the requried jar files.
run the following command
> sbt
> update
now sbt should download the requried jar files
>jetty-run
now sbt should compile and run the web-app.