I know this question has been asked many times before, but I couldn't find any for this specific case, and this up-to-date. How do I install the Eclipse MapReduce plugin with Hadoop 2.5.2? I've found multiple tutorials addressing this for other versions, and I know that it can be compiled from here, but that link, while it says 2.x, only says up to 2.4.1. I tried to compile it anyway using 2.5.2 using the following command:
ant jar -Dversion=2.4.1 -Dhadoop.version=2.5.2 -Declipse.home=/opt/eclipse -Dhadoop.home=/usr/local/hadoop -DBuildfile: build.xml
This attempted to compile, but then returned
Warning: Could not find file /usr/local/hadoop/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar to copy.
Which makes sense, because the version of this jar in Hadoop 2.5.2 is 1.9.13.
Then I tried to use version 2.5.2, even though it is not mentioned in the jar, since it says that they support 2.x.
ant jar -Dversion=2.5.2 -Dhadoop.version=2.5.2 -Declipse.home=/opt/eclipse -Dhadoop.home=/usr/local/hadoop -DBuildfile: build.xml
In this case, it said BUILD SUCCESSFUL, but there was no output - no jar produced, and no output printed under the 'compile:' and 'jar:' steps.
I even tried downloading the compiled jar from this guy's github, but of course that didn't work either - it didn't show up in Eclipse when I added the jar to the plugins folder.
Is there any way to use the plugin with Hadoop 2.5.2, or do I have to downgrade to 2.4.1? I am using Ubuntu 14.0.4, Hadoop 2.5.2, and Eclipse Luna, but can downgrade that if necessary.
I downloaded hadoop-eclipse-plugin-2.6.0.jar (at the time of writing this is the latest) from https://github.com/winghc/hadoop2x-eclipse-plugin/tree/master/release and pasted the jar file in the plugin folder of the Eclipse. It gave me "Map/Reduce" Wizard and I got the "Run on Hadoop" Option!! I am using Eclipse Luna.
you could use the "Hadoop" Wizard to define the HDFS Server.
If you just need to run the map-reduce jobs from eclipse, then you need to
Create a Java project in eclipse
Add the hadoop jar files in the project reference library
Mapreduce programs can be compiled and executed from eclipse.
Note: By default eclipse considers the local filesystem for input and output files.
Related
The download page at scala-ide.org for the pre-configured Scala IDE for Eclipse (4.0.0 Release) states that its content includes SBT 0.13.6. I downloaded the Mac OS X 64-bit version, and found that the /plugins directory indeed contains a jar file called org.scala-ide.sbt.full.library_0.13.6.v-2... Is this a replacement for having a separate installation of SBT on one's development machine, perhaps facilitating some tighter integration between the Scala Eclipse plugin and SBT? If so, how does it work?
Yes, it does indeed include sbt. It is used internally to control the build, notably the incremental compiler.
It isn't a replacement for having sbt installed on your machine, it isn't really accessible from outside scala-ide.
As for tighter integration, not at the minute.
The exact OS I'm using is RedHat 6.5 Enterprise, 32 bit. I believe Eclipse was installed using yum.
I have tried multiple URLs to try to install an SVN plugin using Eclipse's Help->Install New Software function, including this one: http://subclipse.tigris.org/update_1.10.x, which I found here: http://subclipse.tigris.org/servlets/ProjectProcess?pageID=p4wYuA
Eclipse installed the plugin successfully, but whenever I start Eclipse, the plugin is nowhere to be found. I tried starting Eclipse with the -clean option, and that isn't doing anything.
Therefore, from the site I linked above, I downloaded the .zip file of the plugin from here: http://subclipse.tigris.org/servlets/ProjectDocumentList?folderID=2240. The exact file I downloaded is site-1.8.22.zip.
I had this same problem in the past with Mercurial where installing from Help->Install New Software didn't work, so I downloaded that jar (for some reason there is only one jar for Mercurial but for SVN there are many plus some directories?). I then put that jar in /usr/share/eclipse/dropins/plugins (I had to create the plugins directory). That worked. However, when I put all of the SVN jars in there, SVN is not showing up in Eclipse. I have tried many different combinations of putting only some jars in there, creating folders, etc. Again, I have been using -clean when starting eclipse, and that's not making a difference. I truly think the plugin isn't being put in the right spot because I can move the Mercurial plugin I have installed in and out, and it shows that correctly in Eclipse each time. For reference, the Mercurial jar I'm using is: com.vectrace.MercurialEclipse_2.1.0.201304290948.jar
I also noticed that /etc/eclipse.ini has this line in it, which should say where to look for plugins: -Dorg.eclipse.equinox.p2.reconciler.dropins.directory=/usr/share/eclipse/dropins
So I have some questions:
Where do I put all of these SVN jars and folders that I downloaded?
Why does the Mercurial one only have one jar and the SVN have a whole bunch? I'm just curious why the developers would package these completely differently because maybe this means there's a single SVN jar out there that I could use and just have it work.
I also found a /usr/lib/eclipse folder, and there are also plugin and dropin folders in there. Why are there multiple Eclipse directories for plugins?
I believe I was using the wrong version of java. I downloaded the latest version of Eclipse, and that required me to update my java version in order to run it. I updated to 1.8 (was using 1.6). I think the latest Subclipse plugin requires something at least higher than 1.6. There were no warnings or error messages to let me know about this, so that's why it was difficult to figure out.
Once I started using the new Eclipse and Java 1.8, I was able to easily install and use Subsclipse using Help->Install New Software.
I'm trying to build and modify hadoop 2.5.0 source code using eclipse.
I've followed all instructions in this tutorial
http://blog.cloudera.com/blog/2013/05/how-to-configure-eclipse-for-hadoop-contributions/
and this one too http://wiki.apache.org/hadoop/HowToContribute.
But still I don't understand, how to build and test my modification directly from eclipse?
or if there is no way to do that, what is the best way to achive my goal to modify, build, and run my own modified version of hadoop?
Note:
*I've already successfully build hadoop using maven via ubuntu terminal previously, and run the package successfully also.
*I'm new in java, eclipse, and hadoop development
Thanks.
Edit 2: I seem to have discovered the problem: the JAR files that were downloaded to my m2 repository are corrupt:
It seems to me that they could be causing the problems. But why are they corrupt? And how can I fix that?
Edit: I've ran mvn eclipse:clean eclipse:eclipse in the project, and it completed successfully... and introduced 5 additional errors:
On another PC I've created a Scala Maven project using the m2eclipse and Scala IDE plugins for Eclipse. I don't remember any errors. Then I pushed it to GitHub.
I then proceeded to clone the project on my laptop, but I was first greeted with this (error) message:
After selecting yes, what seemed to be the logical step, these errors appeared:
What's going on and how can I fix it?
(Linux, Elementary OS)
It is possibly a difference in Scala versions used. Scala IDE is usually based on specific version of Scala. Looks like your project pom.xml requires different version of Scala.
A solution is to either update your pom.xml to version of your IDE or upgrade the Scala IDE if possible.
The other possibility is that downloaded jars are corrupted.
I'm using the Indigo Release of eclipse. When I check Window -> Preferences, Maven -> Installations, it confirms that it's using the Embedded (3.0.2/...) installation of Maven.
I want to be able to run maven commands from the command line (Windows Powershell), but when I type, e.g.,
PS C:\> mvn
no executable is found. I did a search for "mvn.exe" and found no results.
I suppose I could install a stand-alone version of Maven 3 and add that to my PATH, but I'd prefer to be using the same Maven installation for command line tasks as well as IDE (eclipse) tasks.
So, where is the embedded maven's mvn.exe hiding?
The simple answer is, cause it's embedded in Eclipse so you can't call it from console. If you like to do this you have to install Maven yourself.
There is no mvn.exe at all. The Maven distro comes with batch or shell scripts. Eclipse runs an embedded version which is called through an API.
remember, embedded maven is not configured to work out of eclipse environment (and terminal usage is out of STS usage).
we have to install maven.
and also note, there is no "mvn.exe"
follow the following to setup maven to work from terminal.
maven download link is here
download latest maven from here
extract it where ever you want.
i extracted as below (see the bin,boot,conf,... folders are shown below as in the image):
as this maven download is extracted from zip file (it is not setup), you will have to add this extracted path to "PATH" variable in windows environment.
follow this to set environment path:
now "mvn" command must work.
in my case, i use STS (spring tool suite)
and i had to restart STS for "mvn" to work in terminal in sts.
check it out with eclipse.