Using scala-eclipse for spark - eclipse

Could some please help me on how to use the scala-eclipse IDE for spark ?
I came across this link - http://syndeticlogic.net/?p=311. But I am unable to follow it.
I entered the command - mvn -Phadoop2 eclipse:clean eclipse:eclipse inside the spark directory after a long list of downloads it gave me some error. Please help. Thanks
Below is the error i received
Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM .......................... SUCCESS [5:22.386s]
[INFO] Spark Project Core ................................ SUCCESS [17:20.807s]
[INFO] Spark Project Bagel ............................... FAILURE [2.159s]
[INFO] Spark Project GraphX .............................. SKIPPED
[INFO] Spark Project ML Library .......................... SKIPPED
[INFO] Spark Project Streaming ........................... SKIPPED
[INFO] Spark Project Tools ............................... SKIPPED
[INFO] Spark Project Catalyst ............................ SKIPPED
[INFO] Spark Project SQL ................................. SKIPPED
[INFO] Spark Project Hive ................................ SKIPPED
[INFO] Spark Project REPL ................................ SKIPPED
[INFO] Spark Project Assembly ............................ SKIPPED
[INFO] Spark Project External Twitter .................... SKIPPED
[INFO] Spark Project External Kafka ...................... SKIPPED
[INFO] Spark Project External Flume ...................... SKIPPED
[INFO] Spark Project External ZeroMQ ..................... SKIPPED
[INFO] Spark Project External MQTT ....................... SKIPPED
[INFO] Spark Project Examples ............................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 25:15.115s
[INFO] Finished at: Wed May 07 15:27:51 GMT+05:30 2014
[INFO] Final Memory: 22M/81M
[INFO] ------------------------------------------------------------------------
[WARNING] The requested profile "hadoop2" could not be activated because it does not exist.
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) on project spark-bagel_2.10: Failed to resolve dependencies for one or more projects in the reactor. Reason: Missing:
[ERROR] ----------
[ERROR] 1) org.apache.spark:spark-core_2.10:jar:1.0.0-SNAPSHOT
[ERROR]
[ERROR] Try downloading the file manually from the project website.
[ERROR]
[ERROR] Then, install it using the command:
[ERROR] mvn install:install-file -DgroupId=org.apache.spark -DartifactId=spark-core_2.10 -Dversion=1.0.0-SNAPSHOT -Dpackaging=jar -Dfile=/path/to/file
[ERROR]
[ERROR] Alternatively, if you host your own repository you can deploy the file there:
[ERROR] mvn deploy:deploy-file -DgroupId=org.apache.spark -DartifactId=spark-core_2.10 -Dversion=1.0.0-SNAPSHOT -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
[ERROR]
[ERROR] Path to dependency:
[ERROR] 1) org.apache.spark:spark-bagel_2.10:jar:1.0.0-SNAPSHOT
[ERROR] 2) org.apache.spark:spark-core_2.10:jar:1.0.0-SNAPSHOT
[ERROR]
[ERROR] ----------
[ERROR] 1 required artifact is missing.
[ERROR]
[ERROR] for artifact:
[ERROR] org.apache.spark:spark-bagel_2.10:jar:1.0.0-SNAPSHOT
[ERROR]
[ERROR] from the specified remote repositories:
[ERROR] maven-repo (http://repo.maven.apache.org/maven2, releases=true, snapshots=false),
[ERROR] apache-repo (https://repository.apache.org/content/repositories/releases, releases=true, snapshots=false),
[ERROR] jboss-repo (https://repository.jboss.org/nexus/content/repositories/releases, releases=true, snapshots=false),
[ERROR] mqtt-repo (https://repo.eclipse.org/content/repositories/paho-releases, releases=true, snapshots=false),
[ERROR] cloudera-repo (https://repository.cloudera.com/artifactory/cloudera-repos, releases=true, snapshots=false),
[ERROR] apache.snapshots (http://repository.apache.org/snapshots, releases=false, snapshots=true),
[ERROR] central (http://repo.maven.apache.org/maven2, releases=true, snapshots=false)
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :spark-bagel_2.10

This is because there is no profile called hadoop2 in the pom.xml. The closest matches are hadoop-2.2,hadoop-2.3 etc.
You can run the following
mvn -Phadoop-2.2 eclipse:clean eclipse:eclipse
or you may run 'mvn help:all-profiles' to list all the profiles and use one from it

If you want to contribute to the Apache Spark project, then
Go to spark home and run sbt/sbt eclipse
In Scala IDE, Select File | Import | Existing Projects into Workspace.
Select root directory :MY_SPARK_HOME
Select Search for Nested Projects
Select the projects that you want
Do not select "Copy projects into workspace".
If you want to use the spark libraries in an application that you are using then,
- You can create a jar using the sbt/sbt assembly command and then add that jar as a library to your application project
Also refer to the eclipse documentation here at: https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark#ContributingtoSpark-Eclipse

Related

java.lang.RuntimeException: You must run the `stage` task before deploying your app when running `sbt stage deployHeroku`

I am trying to deploy my application to Heroku using sbt-nativepackager and sbt-heroku.
My code is available at https://github.com/hhimanshu/sbt101/tree/m5 (branch is m5)
When I run sbt stage deployHeroku, the application fails as below
➜ sbt101 git:(m5) ✗ sbt stage deployHeroku
[info] Loading global plugins from /Users/harit/.sbt/1.0/plugins
[info] Loading settings for project sbt101-build from plugins.sbt ...
[info] Loading project definition from /Users/harit/code/sc/sbt101/project
[info] Loading settings for project root from build.sbt ...
[info] Set current project to sbt101 (in build file:/Users/harit/code/sc/sbt101/)
[info] Packaging /Users/harit/code/sc/sbt101/api/target/scala-2.12/api_2.12-0.1.0-SNAPSHOT-sources.jar ...
[info] Done packaging.
[info] Wrote /Users/harit/code/sc/sbt101/api/target/scala-2.12/api_2.12-0.1.0-SNAPSHOT.pom
[info] Wrote /Users/harit/code/sc/sbt101/calculators/target/scala-2.12/calculators_2.12-0.1.0-SNAPSHOT.pom
[info] Main Scala API documentation to /Users/harit/code/sc/sbt101/api/target/scala-2.12/api...
[info] Compiling 1 Scala source to /Users/harit/code/sc/sbt101/api/target/scala-2.12/classes ...
model contains 3 documentable templates
[info] Done compiling.
[info] Main Scala API documentation successful.
[info] Packaging /Users/harit/code/sc/sbt101/api/target/scala-2.12/api_2.12-0.1.0-SNAPSHOT-javadoc.jar ...
[info] Packaging /Users/harit/code/sc/sbt101/api/target/scala-2.12/api_2.12-0.1.0-SNAPSHOT.jar ...
[info] Done packaging.
[info] Main Scala API documentation to /Users/harit/code/sc/sbt101/calculators/target/scala-2.12/api...
[info] Compiling 1 Scala source to /Users/harit/code/sc/sbt101/calculators/target/scala-2.12/classes ...
[info] Done packaging.
[warn] there was one feature warning; re-run with -feature for details
model contains 5 documentable templates
[warn] one warning found
[info] Main Scala API documentation successful.
[info] Packaging /Users/harit/code/sc/sbt101/calculators/target/scala-2.12/calculators_2.12-0.1.0-SNAPSHOT-javadoc.jar ...
[info] Done packaging.
[warn] there was one deprecation warning (since 2.11.0); re-run with -deprecation for details
[warn] there was one feature warning; re-run with -feature for details
[warn] two warnings found
[info] Done compiling.
[warn] Multiple main classes detected. Run 'show discoveredMainClasses' to see the list
[info] Packaging /Users/harit/code/sc/sbt101/calculators/target/scala-2.12/calculators_2.12-0.1.0-SNAPSHOT.jar ...
[info] Done packaging.
[success] Total time: 4 s, completed 10-May-2019 4:20:03 PM
[error] java.lang.RuntimeException: You must run the `stage` task before deploying your app!
[error] at com.heroku.sbt.SbtApp.packageType(SbtApp.scala:142)
[error] at com.heroku.sbt.SbtApp.prepare(SbtApp.scala:111)
[error] at com.heroku.sdk.deploy.App.deploy(App.java:60)
[error] at com.heroku.sbt.SbtApp.deploy(SbtApp.scala:98)
[error] at com.heroku.sbt.HerokuPlugin$autoImport$.$anonfun$baseHerokuSettings$1(HerokuPlugin.scala:53)
[error] at com.heroku.sbt.HerokuPlugin$autoImport$.$anonfun$baseHerokuSettings$1$adapted(HerokuPlugin.scala:26)
[error] at scala.Function1.$anonfun$compose$1(Function1.scala:44)
[error] at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:40)
[error] at sbt.std.Transform$$anon$4.work(System.scala:67)
[error] at sbt.Execute.$anonfun$submit$2(Execute.scala:269)
[error] at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16)
[error] at sbt.Execute.work(Execute.scala:278)
[error] at sbt.Execute.$anonfun$submit$1(Execute.scala:269)
[error] at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178)
[error] at sbt.CompletionService$$anon$2.call(CompletionService.scala:37)
[error] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[error] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error] at java.lang.Thread.run(Thread.java:748)
2019-05-10 16:20:03,606 Log4j2-TF-1-AsyncLogger[AsyncContext#cb644e]-1 ERROR Attempted to append to non-started appender heroku-logger
[error] (Compile / deployHeroku) You must run the `stage` task before deploying your app!
2019-05-10 16:20:03,607 Log4j2-TF-1-AsyncLogger[AsyncContext#cb644e]-1 ERROR Attempted to append to non-started appender heroku-logger
[error] Total time: 0 s, completed 10-May-2019 4:20:03 PM
However, using the Heroku toolbelt on command-line, I have been successful in deploying my app
➜ sbt101 git:(m5) ✗ git push heroku m5:master
The the app runs at https://h2-sbt101.herokuapp.com/rates
Can someone please help me understand what I may be missing?
I had the same problem. sbt deployHeroku looks for the directory target/universal/stage (see the source). However, it seems to look for it in the root project which may not be the one with the staged directory. For example, in the OP's log, it seems there are several projects called api and calculator. In my case, the correct one (containing the server code) was server.
So sbt stage server/deployHeroku worked for me.

Creating Maven multi-module project with Eclipse

I want to create a maven Multi-module project.
I want to create the Parent project using
mvn archetype:create -DgroupId=com.websystique.multimodule -DartifactId=parent-project
But I got this error:
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.203 s
[INFO] Finished at: 2017-05-02T19:57:19+02:00
[INFO] Final Memory: 9M/309M
[INFO] ------------------------------------------------------------------------
[ERROR] Could not find goal 'create' in plugin org.apache.maven.plugins:maven-archetype-plugin:3.0.1 among available goals crawl, create-from-project, generate, help, integration-test, jar, update-local-catalog -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoNotFoundException
Did you read the error message?
[ERROR] Could not find goal 'create' in plugin org.apache.maven.plugins:maven-archetype-plugin:3.0.1 among available goals crawl, create-from-project, generate, help, integration-test, jar, update-local-catalog -> [Help 1]
The error message tells you that the goal you've specified for the plugin (create) isn't one of the available options. It also lists the valid options for you:
crawl
create-from-project
generate
help
integration-test
jar
update-local-catalog
The goal you're looking for is generate. For example:
mvn archetype:generate -DgroupId=com.websystique.multimodule -DartifactId=parent-project
For more info: maven-archetype-plugin.

vaadin plugin for eclipse build failure on compile theme

Iam getting a build failure every time i try to compile my themes. The compilation seems to work. Iam just wondering about the console log which is saying(in extracts):
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 5.503 s
[INFO] Finished at: 2017-01-19T17:15:48+01:00
[INFO] Final Memory: 19M/303M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.vaadin:vaadin-maven-plugin:7.6.6:compile-theme (default-cli) on project admintool: Compiling theme "VAADIN/themes/nxo-fr" failed: Command [[
[ERROR] /bin/sh -c /opt/jdk/jdk1.8.0_91/jre/bin/java -classpath /home/cia/dev/projects/trunk/trunk/admintool/src/main/webapp:/home/cia/dev/projects/trunk/trunk/admintool/target/classes:/home/cia/dev/projects/trunk/trunk/admintool/src/main/java:/home/cia/dev/projects/trunk/trunk/admintool/src/main/resources:/home/cia/.m2/repository/com/sharis/bl/admintool-widgetset/7.6.7/admintool-widgetset-7.6.7.jar:/home/cia/.m2/repository/com/vaadin/vaadin-server/7.6.7/vaadin-server-7.6.7.jar:/home/cia/.m2/repository/com/vaadin/vaadin-sass-compiler/0.9.13/vaadin-sass-compiler-0.9.13.jar:/home/cia/.m2/repository/org/w3c/css/sac/1.3/sac-....................................................................1.5.10.jar com.vaadin.sass.SassCompiler /home/cia/dev/projects/trunk/trunk/admintool/src/main/webapp/VAADIN/themes/nxo-fr/styles.scss /home/cia/dev/projects/trunk/trunk/admintool/src/main/webapp/VAADIN/themes/nxo-fr/styles.css
[ERROR] ]] failed with status 2
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
I found the issue. The styles.scss file was missing in my older theme folders like runo etc.
So in each theme folder there must be a styles.scss for proper compilation.
Try to use a theme name which has no hyphens in it.
"VAADIN/themes/nxo-fr"

When trying to install jar in local repository using maven

I do not have this SFTP jar in my Nexus Repo & I have add it's dependency into the POM file but now i want to install this JAR in my local repository so that i can use it in my project. But getting access is denied even though i opened command prompt with Administrator rights.
C:\Windows\system32>mvn install:install-file -DgroupId=org.springframework.integration -DartifactId=spring-integration-sftp -Dversion=4.1.5 -Dpackaging=jar -Dfile=C:\Repo
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-install-plugin:2.4:install-file (default-cli) # standalone-pom
---
[INFO] Installing C:\Repo to C:\Users\t868396\.m2\repository\org\springframework
\integration\spring-integration-sftp\4.1.5\spring-integration-sftp-4.1.5.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.827 s
[INFO] Finished at: 2015-06-23T11:02:52+05:00
[INFO] Final Memory: 15M/64M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-install-plugin:2.4
:install-file (default-cli) on project standalone-pom: Error installing artifact
'org.springframework.integration:spring-integration-sftp:jar': Failed to instal
l artifact org.springframework.integration:spring-integration-sftp:jar:4.1.5: C:
\Repo (Access is denied) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e swit
ch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please rea
d the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionE
xception
Mostly likely the -Dfile=your.jar is not correct, check your argument (FYI it was omitted from your posting).
Here is an example:
mvn install:install-file -DlocalRepositoryPath=repo -DcreateChecksum=true
-Dfile=[your-jar] -Dpackaging=jar
-DgroupId=[...] -DartifactId=[...] -Dversion=[...]
See: Can I add jars to maven 2 build classpath without installing them?
And: https://maven.apache.org/general.html#importing-jars

I couldn't import 'Existing Maven Projects' of i-jetty

They said,
cd $IJETTY_HOME/console
mvn clean install
This will produce a war file in $IJETTY_HOME/console/webapp/target, and also the apk in $IJETTY_HOME/console/apk/target (although as we mentioned, you're probably not interested in that).
but I got following error (...oh, my god. I can not upload image ;0 )
**
[INFO] --- exec-maven-plugin:1.2:exec (generate-dex) # console ---
Unable to access jarfile ${env.ANDROID_HOME}/platform-tools/lib/dx.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] I-Jetty :: Console Parent ......................... SUCCESS [0.241s]
[INFO] I-Jetty :: Console Webapp ......................... FAILURE [2.431s]
[INFO] I-Jetty :: Console Android Installer .............. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4.122s
[INFO] Finished at: Thu Nov 08 19:33:47 KST 2012
[INFO] Final Memory: 14M/81M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2:exec (generate-dex) on project console: Command execution failed. Process exited with an error: 1(Exit value: 1) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :console**
and I tried 'Building with Eclipse'
finally, I got following error similar with the one.
on Import Maven Projects,
exec-maven-plugin:1.2:exec (1 errors)
maven-antrun-plugin:1.6:run(1 errors)
maven-depoendency-plugin:2.3:unpack-dependencies(1 error)
Help me, plz.
I installed maven plug-in 1.3 with eclipse on Mac OS X Lion.
finally, I solved it by myself.
I missed export android sdk path (export ANDROID_HOME=/opt/android/android-sdk-linux_86)
and I edited pom.xml
from
<artifactId>maven-android-plugin</artifactId>
<version>2.9.0-beta-5</version>
to
<artifactId>android-maven-plugin</artifactId>
<version>3.0.0</version>
thanks.