I'm trying to build and modify hadoop 2.5.0 source code using eclipse.
I've followed all instructions in this tutorial
http://blog.cloudera.com/blog/2013/05/how-to-configure-eclipse-for-hadoop-contributions/
and this one too http://wiki.apache.org/hadoop/HowToContribute.
But still I don't understand, how to build and test my modification directly from eclipse?
or if there is no way to do that, what is the best way to achive my goal to modify, build, and run my own modified version of hadoop?
Note:
*I've already successfully build hadoop using maven via ubuntu terminal previously, and run the package successfully also.
*I'm new in java, eclipse, and hadoop development
Thanks.
Related
I want to contribute to apache spark. I am using Eclipse Scala IDE to fetch the source code from https://github.com/apache/spark.git but I am getting lot of errors. Can anyone please tell me the steps needed to build spark source code and modify it? I am able to successfully build it using mvn command line. As per the Spark Read me guide mvn -DskipTests clean package runs well in command line. but after importing to eclipse error starts. Do we need to do any changes to built it in eclipse compiler. The Apache Documentation is not very clear.
I downloaded sample Gatling maven project from here . I am able to run test from command-line but I also would like to run it from Eclipse.
I imported project to Eclipse as existing maven project and I installed Scala IDE. The documentation says: "Simply launch the Engine class in your IDE." And the question is how I can run that Engine class? Should I create new configuration to run it? Could anybody explain it please?
I want to import the Grinder(java load testing tool)into eclipse as a new project.
These are my environment:
Win7 64bit
JDK 1.6 32bit
eclipse-java-luna-SR2-win32
These are my steps:
Eclipse--import from git uri ,I used the git link that provided by official
Eclipse--File--Import--Maven--Existing Maven Projects,I use the directory which contain the source files that just downloaded by git
Then Eclipse started to build and handle the dependency,when the process is over,Eclipse gave the error info:
” No marketplace entries found to handle clojure-maven-plugin:1.3.15”
I don’t modify the pom file ,and use the default maven plugin of Eclipse:m2e.I don’t install the maven and other library.
I try to delete “C:\Users\g.m2\repository”,and redo these steps ,however,it doesn't work.
Plz help me,thanks!
Not sure what is the purpose. Are you planning to develop/enhance the grinder tool. If yes, then probably you should connect to its developers in the grinder site.
And if you are planning to use the tool and trying to configure eclipse for the scripting purpose, your should try Grinderstone. Note that it is outdated and works only with the specific version of software. Otherway, you can use pydev in the eclispe for scripting but you can't run there.
Hope it helps..
I know this question has been asked many times before, but I couldn't find any for this specific case, and this up-to-date. How do I install the Eclipse MapReduce plugin with Hadoop 2.5.2? I've found multiple tutorials addressing this for other versions, and I know that it can be compiled from here, but that link, while it says 2.x, only says up to 2.4.1. I tried to compile it anyway using 2.5.2 using the following command:
ant jar -Dversion=2.4.1 -Dhadoop.version=2.5.2 -Declipse.home=/opt/eclipse -Dhadoop.home=/usr/local/hadoop -DBuildfile: build.xml
This attempted to compile, but then returned
Warning: Could not find file /usr/local/hadoop/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar to copy.
Which makes sense, because the version of this jar in Hadoop 2.5.2 is 1.9.13.
Then I tried to use version 2.5.2, even though it is not mentioned in the jar, since it says that they support 2.x.
ant jar -Dversion=2.5.2 -Dhadoop.version=2.5.2 -Declipse.home=/opt/eclipse -Dhadoop.home=/usr/local/hadoop -DBuildfile: build.xml
In this case, it said BUILD SUCCESSFUL, but there was no output - no jar produced, and no output printed under the 'compile:' and 'jar:' steps.
I even tried downloading the compiled jar from this guy's github, but of course that didn't work either - it didn't show up in Eclipse when I added the jar to the plugins folder.
Is there any way to use the plugin with Hadoop 2.5.2, or do I have to downgrade to 2.4.1? I am using Ubuntu 14.0.4, Hadoop 2.5.2, and Eclipse Luna, but can downgrade that if necessary.
I downloaded hadoop-eclipse-plugin-2.6.0.jar (at the time of writing this is the latest) from https://github.com/winghc/hadoop2x-eclipse-plugin/tree/master/release and pasted the jar file in the plugin folder of the Eclipse. It gave me "Map/Reduce" Wizard and I got the "Run on Hadoop" Option!! I am using Eclipse Luna.
you could use the "Hadoop" Wizard to define the HDFS Server.
If you just need to run the map-reduce jobs from eclipse, then you need to
Create a Java project in eclipse
Add the hadoop jar files in the project reference library
Mapreduce programs can be compiled and executed from eclipse.
Note: By default eclipse considers the local filesystem for input and output files.
I've just finished installing Hadoop 0.20.2 under Cygwin on Windows 7 with Eclipse Helios (3.6). Hadoop is now fully started, and I'm trying to run a test application within a newly created MapReduce test project in Eclipse. I'm using the Hadoop 0.20.2 plugin from the Hadoop download.
The Map/Reduce Location perspective operates correctly, as does DFS Locations tree in the Package Explorer. However, when I right-click the driver, select 'Run As' > 'Run on Hadoop', nothing happens and no errors spawn on the Console (silent fail :(). I believe a dialog window should appear asking for config before it runs, but this is not happening.
There seems to be a few others with the same problem, but I've yet to find an answer that works. I've tried the 0.20.1 plugin (total fail). The following bug report seems to describe my issue, though I'm a bit of a newbie to all this, so could do with a hand / voice of experience to help out: https://issues.apache.org/jira/browse/MAPREDUCE-1280
The hadoop eclipse plugin bundled with the hadoop distribution is compatible with eclipse up to version 3.3. The JIRA-ticket MAPREDUCE-1280 contains a patch for running the plugin in eclipse 3.4 and upwards.
I just compiled the patched plugin with the fixes from the JIRA-ticket MAPREDUCE-1280. The file is attached to the ticket. You can find it here.
Simply remove the old plugin from your eclipse-installation and put the new version of the plugin into the dropins-folder of your eclipse-installation.
After upgrading from an older version of the plugin you will have to start eclipse with the "-clean" command line switch. Help on eclipse command line switches can be found here.
I don't know whether the plugin has been updated or not, but as far as I know, that one is out of maintenance for several previous releases.
One of the solution is that you should download the source code and try to re-compile the jar file for that plugin (for the latest version of eclipse), however I didn't try it so don't know whether it is working.
Maybe you can try to use Karmasphere.
askswOrder is correct that the Eclipse plugin has not seen much attention for quite a long time. The JIRA you reference does provide a fix, but it's only been applied to Hadoop 20.3 and above. One option would be to try to apply the patch to 20.2 and recompile, but that's asking quite a lot from a newbie. I'd second the suggestion to use Karmasphere; it's a great product for working with MapReduce and those gents have taken on the work of staying current with Hadoop releases.