Building Spark using Maven fails - scala

I have a spark program written in Scala and I'm trying to build it using Maven. However maven built fails without any obvious error. Here is the error message:
[INFO] Compiling 1 source files to D:\Scala-IDE\Workspace\ClassifierMaven\classifier\target\classes at 1438214677096
[ERROR] error: class file needed by package is missing.
[INFO] reference type ClassTag of package reflect refers to nonexisting symbol.
[ERROR] one error found
[INFO] Picked up _JAVA_OPTIONS: -Xmx2G
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 17.022 s
[INFO] Finished at: 2015-07-29T17:04:41-07:00
[INFO] Final Memory: 30M/879M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.scala-tools:maven-scala-plugin:2.15.0:compile (default) on project classifier: wrap: org.apache.commons.exec.ExecuteException: Process
exited with an error: 1(Exit value: 1) -> [Help 1]
what is the problem?
SOLUTION: the problem was missing a dependency in my POM. I added the dependency and now I'm getting a new error message. HEre is the error I'm getting now
[INFO] excludes = []
[INFO] D:\Scala-IDE\Workspace\ClassifierMaven\classifier\src\main\scala:-1: info: compiling
[INFO] Compiling 1 source files to D:\Scala-IDE\Workspace\ClassifierMaven\classifier\target\classes at 1438274292378
[INFO] java.lang.reflect.InvocationTargetException
[INFO] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[INFO] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[INFO] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[INFO] at java.lang.reflect.Method.invoke(Method.java:606)
[INFO] at org_scala_tools_maven_executions.MainHelper.runMain(MainHelper.java:161)
[INFO] at org_scala_tools_maven_executions.MainWithArgsInFile.main(MainWithArgsInFile.java:26)
[ERROR] Caused by: java.lang.AssertionError: assertion failed: List(object Byte, object Byte)
[INFO] at scala.tools.nsc.symtab.Symbols$Symbol.suchThat(Symbols.scala:1063)
[INFO] at scala.tools.nsc.symtab.Symbols$Symbol.companionModule0(Symbols.scala:1269)
[INFO] at scala.tools.nsc.symtab.Symbols$Symbol.companionModule(Symbols.scala:1277)
[INFO] at scala.tools.nsc.symtab.Symbols$Symbol.linkedClassOfClass(Symbols.scala:1296)
[INFO] at scala.tools.nsc.symtab.Definitions$definitions$.addModuleMethod$1(Definitions.scala:707)
[INFO] at scala.tools.nsc.symtab.Definitions$definitions$.initValueClasses(Definitions.scala:710)
[INFO] at scala.tools.nsc.symtab.Definitions$definitions$.init(Definitions.scala:787)
[INFO] at scala.tools.nsc.Global$Run.<init>(Global.scala:597)
[INFO] at scala.tools.nsc.Main$.process(Main.scala:107)
[INFO] at scala.tools.nsc.Main$.main(Main.scala:122)
[INFO] at scala.tools.nsc.Main.main(Main.scala)
[INFO] ... 6 more
[INFO] Picked up _JAVA_OPTIONS: -Xmx2G

It seems that you are missing the scala-reflect package from your class path. I'd suggest looking at some of the sample pom's (like learning spark, or some of the other databricks ones) and using one of those as a starting point for your build.

I got same issue, check your java version in your pom. it must be 1.8

Related

GWT - Error constructing Java AST

What could possibly be causing this error during compilation?
I have generated project from this archetype https://github.com/ArcBees/Arcbees-Archetypes and I have changed only thing - update GWT to 2.8.0 and GWTP to 1.5.3.
[INFO] [ERROR] An internal compiler exception occurred
[INFO] com.google.gwt.dev.jjs.InternalCompilerException: Error constructing Java AST
[INFO] at com.google.gwt.dev.jjs.impl.GwtAstBuilder.translateException(GwtAstBuilder.java:3099)
[INFO] at com.google.gwt.dev.jjs.impl.GwtAstBuilder$AstVisitor.endVisit(GwtAstBuilder.java:352)
[INFO] at org.eclipse.jdt.internal.compiler.ast.AllocationExpression.traverse(AllocationExpression.java:670)
[INFO] at org.eclipse.jdt.internal.compiler.ast.Assignment.traverse(Assignment.java:260)
[INFO] at org.eclipse.jdt.internal.compiler.ast.MethodDeclaration.traverse(MethodDeclaration.java:347)
[INFO] at org.eclipse.jdt.internal.compiler.ast.TypeDeclaration.traverse(TypeDeclaration.java:1379)
[INFO] at com.google.gwt.dev.jjs.impl.GwtAstBuilder.process(GwtAstBuilder.java:3058)
[INFO] at com.google.gwt.dev.javac.CompilationStateBuilder$CompileMoreLater$UnitProcessorImpl.process(CompilationStateBuilder.java:141)
[INFO] at com.google.gwt.dev.javac.JdtCompiler$CompilerImpl.process(JdtCompiler.java:384)
[INFO] at org.eclipse.jdt.internal.compiler.Compiler.compile(Compiler.java:470)
[INFO] at com.google.gwt.dev.javac.JdtCompiler.doCompile(JdtCompiler.java:985)
[INFO] at com.google.gwt.dev.javac.CompilationStateBuilder$CompileMoreLater.compile(CompilationStateBuilder.java:339)
[INFO] at com.google.gwt.dev.javac.CompilationStateBuilder.doBuildFrom(CompilationStateBuilder.java:580)
[INFO] at com.google.gwt.dev.javac.CompilationStateBuilder.buildFrom(CompilationStateBuilder.java:513)
[INFO] at com.google.gwt.dev.javac.CompilationStateBuilder.buildFrom(CompilationStateBuilder.java:499)
[INFO] at com.google.gwt.dev.cfg.ModuleDef.getCompilationState(ModuleDef.java:668)
[INFO] at com.google.gwt.dev.Precompile.precompile(Precompile.java:255)
[INFO] at com.google.gwt.dev.Precompile.precompile(Precompile.java:229)
[INFO] at com.google.gwt.dev.Precompile.precompile(Precompile.java:145)
[INFO] at com.google.gwt.dev.Compiler.run(Compiler.java:206)
[INFO] at com.google.gwt.dev.Compiler.run(Compiler.java:158)
[INFO] at com.google.gwt.dev.Compiler$1.run(Compiler.java:120)
[INFO] at com.google.gwt.dev.CompileTaskRunner.doRun(CompileTaskRunner.java:55)
[INFO] at com.google.gwt.dev.CompileTaskRunner.runWithAppropriateLogger(CompileTaskRunner.java:50)
[INFO] at com.google.gwt.dev.Compiler.main(Compiler.java:127)
[INFO] Caused by: java.lang.NullPointerException
[INFO] at com.google.gwt.dev.jjs.impl.ReferenceMapper.get(ReferenceMapper.java:99)
[INFO] at com.google.gwt.dev.jjs.impl.GwtAstBuilder$AstVisitor.pushNewExpression(GwtAstBuilder.java:2642)
[INFO] at com.google.gwt.dev.jjs.impl.GwtAstBuilder$AstVisitor.endVisit(GwtAstBuilder.java:350)
[INFO] ... 23 more
[INFO] [ERROR] at AbstractStringBuilder.java(137): new String(buffer)
[INFO] org.eclipse.jdt.internal.compiler.ast.AllocationExpression
[INFO] Shutting down PersistentUnitCache thread
The pom.xml generated GWTP archetype has an implicit dependency to gwt-dev-2.7.0, resulting in a version mismatch. Try adding an explicit dependency to 2.8.0:
<dependency>
<groupId>com.google.gwt</groupId>
<artifactId>gwt-dev</artifactId>
<version>2.8.0</version>
<scope>provided</scope>
</dependency>
A few years late, but I just ran into the same issue.
Got the same exception when trying to run GWT in Super Dev mode after adding Tomcat runtime to another project in my Eclipse workspace.
Adding a Tomcat Server to one of the projects will likely add it to other projects (Eclipse Bug) and this may cause dependency issues.
To resolve it:
Remove Tomcat dependency from all related projects build path, clean and rebuild.

Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2

While compiling the Maven project the following error occured:
[INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) # spark-streaming-flume-sink_2.10 ---
[WARNING] Zinc server is not available at port 3030 - reverting to normal incremental compile
[INFO] Using incremental compilation
[INFO] Compiling 6 Scala sources and 3 Java sources to /home/gorlec/Desktop/test/external/flume-sink/target/scala-2.10/classes...
[ERROR] /home/gorlec/Desktop/test/external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkAvroCallbackHandler.scala:47: identifier expected but 'with' found.
[ERROR] with Logging {
[ERROR] ^
[ERROR] one error found
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 7.992s
[INFO] Finished at: Fri Apr 15 17:44:33 CEST 2016
[INFO] Final Memory: 25M/350M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (scala-compile-first) on project spark-streaming-flume-sink_2.10: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed. CompileFailed ->
[Help 1]
[ERROR]
I removed the property <useZincServer>true</useZincServer> from pom.xml, and still the Logging error persists.
[INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) # spark-streaming-flume-sink_2.10 ---
[INFO] Using incremental compilation
[INFO] Compiling 6 Scala sources and 3 Java sources to /home/gorlec/Desktop/test/external/flume-sink/target/scala-2.10/classes...
[ERROR] /home/gorlec/Desktop/test/external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkAvroCallbackHandler.scala:47: identifier expected but 'with' found.
[ERROR] with Logging {
[ERROR] ^
[ERROR] one error found
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 5.814s
[INFO] Finished at: Fri Apr 15 17:41:00 CEST 2016
[INFO] Final Memory: 25M/335M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (scala-compile-first) on project spark-streaming-flume-sink_2.10: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed. CompileFailed ->
[Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
I checked that PATH and JAVA_HOME are defined in ~/.bashrc as follows:
export PATH=$PATH:/usr/lib/jvm/java-7-openjdk-amd64/bin
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
The only issue that I noticed is that echo $JAVA_HOME gives an empty output, though I did source ~/.bashrc.
Any help is highly appreciated.
The problem could be this [INFO] Using incremental compilation
In your pom.xml try to remove the line <recompileMode>incremental</recompileMode>
and then try again.
It is strange echo $JAVA_HOME gives an empty output.While compling the Spakr Source,I import the mvn clean package success porject into eclipse,I meet the same problem.And I found the solution here:
How to solve “Plugin execution not covered by lifecycle configuration” for Spring Data Maven Builds
I think you were compiling Spark with Scala 2.10. If so, you should do as follows.
cd /path/to/Spark
./dev/change-scala-version.sh 2.10
./build/mvn -Pyarn -Phadoop-2.4 -Dscala-2.10 -DskipTests clean package
Hope this helps.

Cygnus 0.10 Docker container

Following the guidelines provided in GITHub,
I have tried to get the jar files, but I'm getting the following errors.
sudo docker-compose -f ./docker/0.compose.jar-compiler.yml -p cygnus run --rm compiler
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building cygnus 0.10.0
[INFO] ------------------------------------------------------------------------
[WARNING] The artifact xml-apis:xml-apis:jar:1.2.01 has been relocated to xerces:xmlParserAPIs:jar:2.6.2
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) # cygnus ---
[INFO] Deleting /cygnus-compiler/target
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) # cygnus ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) # cygnus ---
[INFO] Compiling 44 source files to /cygnus-compiler/target/classes
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /cygnus-compiler/src/main/java/com/telefonica/iot/cygnus/backends/hdfs/HDFSBackendImpl.java:[40,7] error: HDFSBackendImpl is not abstract and does not override abstract method provisionHiveTable(FileFormat,String,String) in HDFSBackend
[INFO] 1 error
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.703 s
[INFO] Finished at: 2015-11-23T10:45:31+00:00
[INFO] Final Memory: 30M/352M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.3.2:compile (default-compile) on project cygnus: Compilation failure
[ERROR] /cygnus-compiler/src/main/java/com/telefonica/iot/cygnus/backends/hdfs/HDFSBackendImpl.java:[40,7] error: HDFSBackendImpl is not abstract and does not override abstract method provisionHiveTable(FileFormat,String,String) in HDFSBackend
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
I run this command from the root of the repository:
docker-compose -f ./docker/0.compose.jar-compiler.yml -p cygnus run --rm compiler
Can anybody help me out on this?

Test GWT composite component with ui binder using GwtTestCase?

I'm writing a simple test class extending GwtTestCase for a component created with UiBinder, and running with mvn gwt:test fails as below.
The problem seems similar to: How to test a gwt composite component with ui binder using GwtTestCase?; other tests for components that are not created with UiBinder run fine with the current project configuration.
The user.agent contains gecko1_8. I tried running from IntelliJ with -Dgwt.args="-war www-test -runStyle HtmlUnit:IE8" but I encountered another issue (net.sourceforge.htmlunit.corejs.javascript.EcmaError: TypeError: Cannot read property "__gwtDevModeHook:com.mypackage.MyEntryPoint.Junit").
[INFO] Caused by: java.lang.Exception: com.google.gwt.core.client.JavaScriptException: (null) #com.google.gwt.dom.client.DOMImplStandard::isOrHasChild(Lcom/google/gwt/dom/client/Node;Lcom/google/gwt/dom/client/Node;)([JavaScript object(29), JavaScript object(30)]): null
[INFO] at com.google.gwt.dev.shell.BrowserChannelServer.invokeJavascript(BrowserChannelServer.java:249)
[INFO] at com.google.gwt.dev.shell.ModuleSpaceOOPHM.doInvoke(ModuleSpaceOOPHM.java:136)
[INFO] at com.google.gwt.dev.shell.ModuleSpace.invokeNative(ModuleSpace.java:571)
[INFO] at com.google.gwt.dev.shell.ModuleSpace.invokeNativeBoolean(ModuleSpace.java:194)
[INFO] at com.google.gwt.dev.shell.JavaScriptHost.invokeNativeBoolean(JavaScriptHost.java:35)
[INFO] at com.google.gwt.dom.client.DOMImplStandard.isOrHasChild(DOMImplStandard.java)
[INFO] at com.google.gwt.dom.client.Node$.isOrHasChild$(Node.java:280)
[INFO] at com.google.gwt.user.client.ui.HTMLPanel.addAndReplaceElement(HTMLPanel.java:220)
[INFO] at com.google.gwt.user.client.ui.HTMLPanel.addAndReplaceElement(HTMLPanel.java:190)
[INFO] at com.mypackage.client.widgets.ScreenHeader_ScreenHeaderUiBinderImpl$Widgets.build_f_HTMLPanel1(ScreenHeader_ScreenHeaderUiBinderImpl.java:128)
[INFO] at com.mypackage.client.widgets.ScreenHeader_ScreenHeaderUiBinderImpl$Widgets.get_f_HTMLPanel1(ScreenHeader_ScreenHeaderUiBinderImpl.java:112)
[INFO] at com.mypackage.client.widgets.ScreenHeader_ScreenHeaderUiBinderImpl$Widgets.access$0(ScreenHeader_ScreenHeaderUiBinderImpl.java:111)
[INFO] at com.mypackage.client.widgets.ScreenHeader_ScreenHeaderUiBinderImpl.createAndBindUi(ScreenHeader_ScreenHeaderUiBinderImpl.java:33)
[INFO] at com.mypackage.client.widgets.ScreenHeader_ScreenHeaderUiBinderImpl.createAndBindUi(ScreenHeader_ScreenHeaderUiBinderImpl.java:1)
[INFO] at com.mypackage.client.widgets.ScreenHeader.<init>(ScreenHeader.java:38)
[INFO] at com.mypackage.client.widgets.ScreenHeaderGwtTest.testConstructor(ScreenHeaderGwtTest.java:12)
[INFO] at com.mypackage.client.widgets.__ScreenHeaderGwtTest_unitTestImpl.doRunTest(__ScreenHeaderGwtTest_unitTestImpl.java:7)
[INFO] at junit.framework.TestCase.runTest(TestCase.java:62)
[INFO] at com.google.gwt.junit.client.GWTTestCase.runBare(GWTTestCase.java:188)

How to set up a local DBpedia mirror

I am trying to set up a local DBpedia Information Extraction Framework, but there seems to have some problems which I cannot deal with. I followed the instructions on the official site using Intelli J, it went well until I get to the final step that tells "select DBpedia dump extraction ->Plugins -> scala ->scala:run", the error message shows below:
D:\program\jdk\bin\java -Dmaven.home=D:\program\apache-maven-3.0.5 -Dclassworlds.conf=D:\program\apache-maven-3.0.5\bin\m2.conf -Didea.launcher.port=7534 "-Didea.launcher.bin.path=D:\program\IntelliJ IDEA Community Edition 12.1.3\bin" -Dfile.encoding=UTF-8 -classpath "D:\program\apache-maven-3.0.5\boot\plexus-classworlds-2.4.jar;D:\program\IntelliJ IDEA Community Edition 12.1.3\lib\idea_rt.jar" com.intellij.rt.execution.application.AppMain org.codehaus.classworlds.Launcher --fail-fast --strict-checksums org.scala-tools:maven-scala-plugin:2.15.2:run
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building DBpedia Dump Extraction 3.8
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] >>> maven-scala-plugin:2.15.2:run (default-cli) # dump >>>
[INFO]
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) # dump ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory E:\code\Scala\extraction_framework\dump\src\main\resources
[INFO]
[INFO] --- maven-scala-plugin:2.15.2:compile (process-resources) # dump ---
[INFO] Checking for multiple versions of scala
[INFO] includes = [**/*.scala,**/*.java,]
[INFO] excludes = []
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) # dump ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-scala-plugin:2.15.2:compile (compile) # dump ---
[INFO] Checking for multiple versions of scala
[INFO] includes = [**/*.scala,**/*.java,]
[INFO] excludes = []
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) # dump ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory E:\code\Scala\extraction_framework\dump\src\test\resources
[INFO]
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) # dump ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-scala-plugin:2.15.2:testCompile (test-compile) # dump ---
[INFO] Checking for multiple versions of scala
[INFO] includes = [**/*.scala,**/*.java,]
[INFO] excludes = []
[WARNING] No source files found.
[INFO]
[INFO] org.dbpedia.extraction.dump.sql.Import
parsing \home\release\wikipedia\wikipedias.csv
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org_scala_tools_maven_executions.MainHelper.runMain(MainHelper.java:161)
at org_scala_tools_maven_executions.MainWithArgsInFile.main(MainWithArgsInFile.java:26)
Caused by: java.io.FileNotFoundException: \home\release\wikipedia\wikipedias.csv (ϵͳ�Ҳ���ָ����·����)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.(FileInputStream.java:138)
at scala.io.Source$.fromFile(Source.scala:91)
at scala.io.Source$.fromFile(Source.scala:76)
at org.dbpedia.extraction.util.WikiInfo$.fromFile(WikiInfo.scala:26)
at org.dbpedia.extraction.util.ConfigUtils$.parseLanguages(ConfigUtils.scala:43)
at org.dbpedia.extraction.dump.sql.Import$.main(Import.scala:26)
at org.dbpedia.extraction.dump.sql.Import.main(Import.scala)
... 6 more
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.104s
[INFO] Finished at: Fri May 24 18:34:48 CST 2013
[INFO] Final Memory: 10M/175M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.scala-tools:maven-scala-plugin:2.15.2:run (default-cli) on project dump: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: -10000(Exit value: -10000) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
Process finished with exit code 1
what should I do to fix it?
PS: I did it all on windows 7 OS, with intelli j 12 community version, maven 3 and jdk 7 installed. If any other info is needed, please let me know, thanks.