I want to contribute to apache spark. I am using Eclipse Scala IDE to fetch the source code from https://github.com/apache/spark.git but I am getting lot of errors. Can anyone please tell me the steps needed to build spark source code and modify it? I am able to successfully build it using mvn command line. As per the Spark Read me guide mvn -DskipTests clean package runs well in command line. but after importing to eclipse error starts. Do we need to do any changes to built it in eclipse compiler. The Apache Documentation is not very clear.
Related
I downloaded sample Gatling maven project from here . I am able to run test from command-line but I also would like to run it from Eclipse.
I imported project to Eclipse as existing maven project and I installed Scala IDE. The documentation says: "Simply launch the Engine class in your IDE." And the question is how I can run that Engine class? Should I create new configuration to run it? Could anybody explain it please?
I know this question has been asked many times before, but I couldn't find any for this specific case, and this up-to-date. How do I install the Eclipse MapReduce plugin with Hadoop 2.5.2? I've found multiple tutorials addressing this for other versions, and I know that it can be compiled from here, but that link, while it says 2.x, only says up to 2.4.1. I tried to compile it anyway using 2.5.2 using the following command:
ant jar -Dversion=2.4.1 -Dhadoop.version=2.5.2 -Declipse.home=/opt/eclipse -Dhadoop.home=/usr/local/hadoop -DBuildfile: build.xml
This attempted to compile, but then returned
Warning: Could not find file /usr/local/hadoop/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar to copy.
Which makes sense, because the version of this jar in Hadoop 2.5.2 is 1.9.13.
Then I tried to use version 2.5.2, even though it is not mentioned in the jar, since it says that they support 2.x.
ant jar -Dversion=2.5.2 -Dhadoop.version=2.5.2 -Declipse.home=/opt/eclipse -Dhadoop.home=/usr/local/hadoop -DBuildfile: build.xml
In this case, it said BUILD SUCCESSFUL, but there was no output - no jar produced, and no output printed under the 'compile:' and 'jar:' steps.
I even tried downloading the compiled jar from this guy's github, but of course that didn't work either - it didn't show up in Eclipse when I added the jar to the plugins folder.
Is there any way to use the plugin with Hadoop 2.5.2, or do I have to downgrade to 2.4.1? I am using Ubuntu 14.0.4, Hadoop 2.5.2, and Eclipse Luna, but can downgrade that if necessary.
I downloaded hadoop-eclipse-plugin-2.6.0.jar (at the time of writing this is the latest) from https://github.com/winghc/hadoop2x-eclipse-plugin/tree/master/release and pasted the jar file in the plugin folder of the Eclipse. It gave me "Map/Reduce" Wizard and I got the "Run on Hadoop" Option!! I am using Eclipse Luna.
you could use the "Hadoop" Wizard to define the HDFS Server.
If you just need to run the map-reduce jobs from eclipse, then you need to
Create a Java project in eclipse
Add the hadoop jar files in the project reference library
Mapreduce programs can be compiled and executed from eclipse.
Note: By default eclipse considers the local filesystem for input and output files.
I'm trying to build and modify hadoop 2.5.0 source code using eclipse.
I've followed all instructions in this tutorial
http://blog.cloudera.com/blog/2013/05/how-to-configure-eclipse-for-hadoop-contributions/
and this one too http://wiki.apache.org/hadoop/HowToContribute.
But still I don't understand, how to build and test my modification directly from eclipse?
or if there is no way to do that, what is the best way to achive my goal to modify, build, and run my own modified version of hadoop?
Note:
*I've already successfully build hadoop using maven via ubuntu terminal previously, and run the package successfully also.
*I'm new in java, eclipse, and hadoop development
Thanks.
Edit 2: I seem to have discovered the problem: the JAR files that were downloaded to my m2 repository are corrupt:
It seems to me that they could be causing the problems. But why are they corrupt? And how can I fix that?
Edit: I've ran mvn eclipse:clean eclipse:eclipse in the project, and it completed successfully... and introduced 5 additional errors:
On another PC I've created a Scala Maven project using the m2eclipse and Scala IDE plugins for Eclipse. I don't remember any errors. Then I pushed it to GitHub.
I then proceeded to clone the project on my laptop, but I was first greeted with this (error) message:
After selecting yes, what seemed to be the logical step, these errors appeared:
What's going on and how can I fix it?
(Linux, Elementary OS)
It is possibly a difference in Scala versions used. Scala IDE is usually based on specific version of Scala. Looks like your project pom.xml requires different version of Scala.
A solution is to either update your pom.xml to version of your IDE or upgrade the Scala IDE if possible.
The other possibility is that downloaded jars are corrupted.
Trying to get a lift project to run on windows, I've installed maven on my computer, then tried to use
mvn archetype:generate from cmd, then chose the lift basic one , but it always failed on the jetty:run part. Before that, I had to change a lot of the pom.xml file to even get a successful build (since the tutorial I was using was obsolete and it couldn't find the dependencies).
So I've decided to try with the eclipse plugin, thinking it should be easier.
I've installed the maven plugin for eclipse and created a project with the lift archetype.
The project was succesfully created, but I have all kinds of errors in the editor for missing ; and so on (I have scala ide installed on eclipse also and creating a normal scala project works and compiles/runs fine). I have also tried to include the scala-library.jar in the buildpath, with no change.
The odd part is that I can run the cmd and go to the project folder, then run mvn jetty:run and it will actually work (?!) Trying to run in eclipse with jetty:run as goal will give me ClassNotFoundException on the HelloWorld snippet.
So what might be wrong with my settings?
UPDATE: no luck whatsoever, after trying 3 different eclipse versions, installing the maven for scala plugin and following every tutorial I could find. It just simply feels all is outdated on the instructions:
this is how it looks after I include the scala libraries for eclipse on every project (after I create it from an archetype):
is this "mvn archetype:generate" up to date? I don't know if anybody guarantees it will work at all..
Anyway, the default way to at least try Lift and see it working is the following: https://github.com/lift/lift_25_sbt
It uses "Simple Build Tool" instead of maven, but maven should work, too. Also, you can easily read "build.sbt" to get all the dependencies.
Plugins for creating eclipse/intellij project definitions are included. (See the README of the demo project.)
The target to create eclipse project is "eclipse" or "eclipse with-sources=true".
The target to launch the app is "container:start".