Building Apache Spark 2.1.0 from source fails - scala

Am trying to build Apache Spark 2.1.0 source, but get these errors below that baffle me...
Hadoop 2.8.0 was installed and is working
Scala 2.12.1 was installed in advance of executing the Spark install (Which seems to auto install Scala 2.11.8 ?!?)
My build line is:
build/mvn -Pyarn -Phadoop-2.7 -Dhadoop.version=2.7.0 -DskipTests clean package
Does anybody know why I get:
user#server:/usr/local/share/spark/spark-2.1.0$ sudo /usr/local/share/spark/spark-2.1.0/build/mvn -Pyarn -Phadoop-2.7 -Dhadoop.version=2.7.0 -DskipTests clean package
[sudo] password for user:
exec: curl --progress-bar -L https://downloads.typesafe.com/zinc/0.3.9/zinc-0.3.9.tgz
######################################################################## 100.0%
exec: curl --progress-bar -L https://downloads.typesafe.com/scala/2.11.8/scala-2.11.8.tgz
######################################################################## 100.0%
exec: curl --progress-bar -L https://www.apache.org/dyn/closer.lua?action=download&filename=/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
######################################################################## 100.0%
Using `mvn` from path: /usr/local/share/spark/spark-2.1.0/build/apache-maven-3.3.9/bin/mvn
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
[INFO] Scanning for projects...
Downloading: https://repo1.maven.org/maven2/org/apache/apache/14/apache-14.pom
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM for org.apache.spark:spark-parent_2.11:2.1.0: Could not transfer artifact org.apache:apache:pom:14 from/to central (https://repo1.maven.org/maven2): repo1.maven.org: Name or service not known and 'parent.relativePath' points at wrong local POM # line 22, column 11
#
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR] The project org.apache.spark:spark-parent_2.11:2.1.0 (/usr/local/share/spark/spark-2.1.0/pom.xml) has 1 error
[ERROR] Non-resolvable parent POM for org.apache.spark:spark-parent_2.11:2.1.0: Could not transfer artifact org.apache:apache:pom:14 from/to central (https://repo1.maven.org/maven2): repo1.maven.org: Name or service not known and 'parent.relativePath' points at wrong local POM # line 22, column 11: Unknown host repo1.maven.org: Name or service not known -> [Help 2]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException
I tested downloading manually (To see if that was causing the errors), which downloads without issues:
https://repo1.maven.org/maven2/org/apache/apache/14/apache-14.pom
I also tested accessing the URL below, which also shows content:
https://repo1.maven.org/maven2
Hope somebody smart knows how to solve this...

I found out what the issue was:
I had to configure our proxy settings in the SETTINGS.XML in the directory:
/usr/local/share/spark/spark-2.1.0/build/apache-maven-3.3.9/conf
After editing the file, the build went without any issues :)
Hope this helps someone else running into the same issue...
EDIT: Just to be extra clear, having a working proxy configuration in bash only is NOT SUFFICIENT for the Maven Build to succeed. I was able to download all files from bash manually, but Maven needed the proxy configuration in the SETTINGS.XML file present as well...

Related

Сan't run examples. Couldn't find artemis.home error

I get an error when I want to run any example through mvn verify
[ERROR] ********************************************************************************************
[ERROR] Could not locate suitable Artemis.home on either D:\Documents\IdeaProjects\activemq-artemis\examples\features\standard\security\..\..\..\.. or D:\Documents\IdeaProjects\activemq-artemis\examples\features\standard\security\..
\..\..\..\artemis-distribution\target\apache-artemis-2.18.0-SNAPSHOT-bin\apache-artemis-2.18.0-SNAPSHOT
[ERROR] Use the binary distribution or build the distribution before running the examples
[ERROR] ********************************************************************************************
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4.087 s
[INFO] Finished at: 2021-02-26T22:03:39+03:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.activemq:artemis-maven-plugin:2.18.0-SNAPSHOT:create (create) on project security: Couldn't find artemis.home -> [Help 1]
I tried to specify the path to the directory with artemis at startup, but it does not help mvn verify -Dartemis.home="D:\Documents\apache-artemis-2.17.0"
How to fix this error?
It appears you're running the example directly from the project source since it reports 2.18.0-SNAPSHOT. If that's the case then you should do what the ERROR message indicates:
Use the binary distribution or build the distribution before running the examples
You can get a snapshot binary distribution here or you can build the distribution from the source using mvn install -Prelease -DskipTests.
If you must point the example at a different home directory you can do so using the activemq.basedir system property, e.g.:
mvn verify -Dactivemq.basedir="D:\Documents\apache-artemis-2.17.0"
Or you can change the activemq.basedir property defined in the example's pom.xml.

Error Eclipse Kura Installation with ecilpse committers

I've installed eclipse Kura with eclipse committers on windows 10 machine, I used eclipse installer. However, it showed the following error.
I will appreciate if you suggest me anything. Thanks for your help.
1
[ERROR] Internal error: java.lang.RuntimeException: Failed to load p2 repository with ID 'p2-repo-equinox_3.11.1' from location file:////C:/Users/Maury/iot-kura-develop/git/kura/kura/org.eclipse.kura.api/../target-definition/equinox_3.11.1/repository/: URI has an authority component ->
...
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/InternalErrorException
Kura only supports OSX and Linux as official development environments. I would recommend running a Linux VM on your Windows host.

failed to compile dataflow sample

I tried the Google sample dataflow but I get the following error when I run maven install as described in the following link:
[ERROR] Failed to execute goal on project
google-cloud-dataflow-java-examples-all: Could not resolve
dependencies for project
com.google.cloud.dataflow:google-cloud-dataflow-java-examples-all:jar:manual_build:
Could not find artifact
com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:tests:manual_build
in central (http://repo.maven.apache.org/maven2) -> [Help 1] [ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with
the -e switch. [ERROR] Re-run Maven using the -X switch to enable full
debug logging. [ERROR] [ERROR] For more information about the errors
and possible solutions, please read the following articles: [ERROR]
[Help 1]
In fact I did mvn clean install and always get the error. It seams that the jar google-cloud-dataflow-java-sdk cannot be build.

Maven Error in Eclipse - BPMN2 Modeler

my problem is the following. I have Eclipse Kepler SR1 and imported an maven project out of the following git: http://git.eclipse.org/c/bpmn2/.git/
This is a metamodel for the BPMN 2.0. After that i wanted to do a maven install but when i do this the following Error pops up
[ERROR] Failed to execute goal org.eclipse.cbi.maven.plugins:eclipse-jarsigner-plugin:1.0.3:sign (sign) on project org.eclipse.bpmn2: Could not sign artifact org.eclipse.bpmn2:org.eclipse.bpmn2:eclipse-plugin:0.7.0-SNAPSHOT: Connection to http://build.eclipse.org:31338 refused: Connection timed out: connect -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :org.eclipse.bpmn2
Maven is correctly installed and i checked the superpom. There is no mistake in my opinion. I also updated the pluginRepository for plugin signin to https://repo.eclipse.org/content/repositories/cbi-releases/. Is there anyone who knows the mistake and can help me?
Best Regards
The first line of your error message contains a timeout for http://build.eclipse.org:31338 - that seems to me an internal address.
If I am not mistaken, this is used by the jar signer plug-in mentioned in the build script that is an eclipse.org-only service (for obvious reasons). Search for the followi
<groupId>org.eclipse.cbi.maven.plugins</groupId>
<artifactId>eclipse-jarsigner-plugin</artifactId>
As signing is an internal service, we in the EMF-IncQuery project execute signing only when a specific profile is selected, thus allowing building our project outside eclipse.org servers.

Installing Enclosure with Netbeans

I am having trouble installing Enclosure and getting it to work. I have followed this guide http://www.enclojure.org/gettingstarted and successfully installed Enclosure (I think). However, when I try to build the sample application (labrepl) I get a bunch of errors and a failed build. I haven't used Java in a long time and I've never used Netbeans, and the error doesn't seem very helpful with my limited knowledge of this domain. I'm using the latest Netbeans and the Enclosure URL from the guide. Since I am on Windows, I can't use git to clone the repo, so I'm not sure what to do from here.
Anyway, here are the error messages.
WARNING: You are running embedded Maven builds, some build may fail due to incompatibilities with latest Maven release.
To set Maven instance to use for building, click here.
Scanning for projects...
[#process-resources]
[resources:resources]
Using default encoding to copy filtered resources.
[#compile]
[ERROR]Transitive dependency resolution for scope: compile has failed for your project.
[ERROR]Error message: Missing:
[ERROR]----------
[ERROR]1) org.clojure:clojure-contrib:jar:1.2.0-master-SNAPSHOT
[ERROR] Try downloading the file manually from the project website.
[ERROR] Then, install it using the command:
[ERROR] mvn install:install-file -DgroupId=org.clojure -DartifactId=clojure-contrib -Dversion=1.2.0-master-SNAPSHOT -Dpackaging=jar -Dfile=/path/to/file
[ERROR] Alternatively, if you host your own repository you can deploy the file there:
[ERROR] mvn deploy:deploy-file -DgroupId=org.clojure -DartifactId=clojure-contrib -Dversion=1.2.0-master-SNAPSHOT -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
[ERROR] Path to dependency:
[ERROR] 1) labrepl:labrepl:jar:0.0.1
[ERROR] 2) org.clojure:clojure-contrib:jar:1.2.0-master-SNAPSHOT
[ERROR]----------
[ERROR]1 required artifact is missing.
[ERROR]for artifact:
[ERROR] labrepl:labrepl:jar:0.0.1
[ERROR]from the specified remote repositories:
[ERROR] central (http://repo1.maven.org/maven2),
[ERROR] clojars (http://clojars.org/repo/),
[ERROR] incanter (http://repo.incanter.org),
[ERROR] clojure-snapshots (http://build.clojure.org/snapshots),
[ERROR] clojure (http://build.clojure.org/releases),
[ERROR] clojure-releases (http://build.clojure.org/releases)
[ERROR]Group-Id: labrepl
[ERROR]Artifact-Id: labrepl
[ERROR]Version: 0.0.1
[ERROR]From file: C:\Users\chloey\Documents\NetBeansProjects\RelevanceLabRepl\pom.xml
------------------------------------------------------------------------
For more information, run with the -e flag
------------------------------------------------------------------------
BUILD FAILED
------------------------------------------------------------------------
Total time: 1 second
Finished at: Wed Jun 09 21:53:04 CDT 2010
Final Memory: 72M/172M
------------------------------------------------------------------------
Thanks all.
the missing artifact is the artifact itself. What command are you using to build it? Try mvn clean install on the command line if everything else fails.
I had the same problem so I did a clean install of Netbeans followed by and install of maven both to no avail.
When I went to download the missing artifact I noticed that clojure-contrib version 1.2.0 was not available for download. Changing the pom.xml to look for version 1.1.0 resulted in a successful build.
Open the pom.xml and change
<dependency>
<groupId>org.clojure</groupId>
<artifactId>clojure-contrib</artifactId>
<version>1.2.0-master-SNAPSHOT</version>
</dependency>
to
<dependency>
<groupId>org.clojure</groupId>
<artifactId>clojure-contrib</artifactId>
<version>1.1.0-master-SNAPSHOT</version>
</dependency>
By the looks of things some additional functionality within 1.2.0 is needed for labrepl not sure from where to download it.