Build failing to include com.fasterxml.jackson.core - scala

I am working on a build file utilizing dependencies from scala and spark packages, etc, on Java 1.8 and Scala 2.10.5.
The build is throwing up on the jackson-bind package. I have added the entire pom and the exception I see:
Exception
central (https://repo.maven.apache.org/maven2, releases=true, snapshots=false)
[ERROR] : ClientProtocolException:
Invalid redirect URI: http://blog.goodstuff.im/repo-releases/com/fasterxml/jackson/core/jackson-databind/[2.7.0,)/jackson-databind-[2.7.0,).jar:
Illegal character in path at index 83: http://blog.goodstuff.im/repo-releases/com/fasterxml/jackson/core/jackson-databind/[2.7.0,)/jackson-databind-[2.7.0,).jar
Tried the exclusion tags, it is still failing with an error:
ClientProtocolException: Invalid redirect URI:
http://blog.goodstuff.im/repo-releases/com/fasterxml/jackson/core/jackson-databind/[2.7.0,)/jackson-databind-[2.7.0,).jar:
Illegal character in path at index 83:
http://blog.goodstuff.im/repo-releases/com/fasterxml/jackson/core/jackson-databind/[2.7.0,)/jackson-databind-[2.7.0,).jar
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
My pom.xml file:
<?xml version="1.0" encoding="utf-8" ?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.mycompany</groupId>
<artifactId>c-platform</artifactId>
<version>3.0.0-SNAPSHOT</version>
<name>${project.version} - ${project.artifactId}</name>
<packaging>pom</packaging>
<parent> <groupId>com.mycompany</groupId>
<artifactId>c-parent</artifactId>
<version>3.0.0-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>
<repositories>
<repository>
<id>Nexus</id>
<url>…-snapshots/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
<repository>
<id> Nexus shapshot</id>
<url>../nexus/content/groups/public</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
<repository>
<id>Cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
<repository>
<id>scala-tools.org</id>
<name>Scala-tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</repository>
<repository>
<id>maven2-repository.dev.java.net</id>
<name>Java.net Repository for Maven</name>
<url>http://download.java.net/maven/2/</url>
<layout>default</layout>
</repository>
</repositories>
<!-- list all dependencies - begin -->
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>
<dependency>
<groupId>com.maxmind.geoip2</groupId>
<artifactId>geoip2</artifactId>
<version>2.7.0</version>
</dependency>
<dependency>
<groupId>com.databricks</groupId>
<artifactId>spark-csv_2.10</artifactId>
<version>${spark.csv.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.10</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_2.10</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.module</groupId>
<artifactId>jackson-module-scala_2.11</artifactId>
<version>2.7.3</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.7.3</version>
<exclusions>
<exclusion>
<artifactId>jackson-annotations</artifactId>
<groupId>com.fasterxml.jackson.core</groupId>
</exclusion>
<exclusion>
<artifactId>jackson-core</artifactId>
<groupId>com.fasterxml.jackson.core</groupId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
<!-- list all dependencies - emd -->
<build>
<sourceDirectory>src/main/scala</sourceDirectory>
<testSourceDirectory>src/test/scala</testSourceDirectory>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.1.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<relocations>
<relocation>
<pattern>com.fasterxml.jackson.**</pattern>
<shadedPattern>com.shaded.fasterxml.jackson#1</shadedPattern>
</relocation>
</relocations>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<version>2.10</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
</goals>
<configuration>
<args>
<arg>-dependencyfile</arg>
<arg>${project.build.directory}/.scala_dependencies</arg>
</args>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.7.0</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<showDeprecation>true</showDeprecation>
<showWarnings>true</showWarnings>
</configuration>
</plugin>
</plugins>
</build>
</project>

It seems that jackson-databind 2.7.3 brings as transitive dependencies jackson-annotations and jackson-core 2.7.0 which might cause some dependency conflict in your build.
I couldn't reproduce exactly, your errors, since I don't have your entire pom. But I would suggest you fix the dependency conflict by:
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.7.3</version>
<exclusions>
<exclusion>
<artifactId>jackson-annotations</artifactId>
<groupId>com.fasterxml.jackson.core</groupId>
</exclusion>
<exclusion>
<artifactId>jackson-core</artifactId>
<groupId>com.fasterxml.jackson.core</groupId>
</exclusion>
</exclusions>
</dependency>
Now you jackon-bind wil miss "jackson-core" and jackson-annotation, but you could add them to the pom with the proper version (2.7.3) and apply the exclusion trick.

Related

Error while building Jar file in Intellij | Maven Plugins are missing?

I am getting below error while building Jar file. I am stuck with this issue from so long please advice.
Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.5.6:compile (default) on project Clinical_CDF: Execution default of goal net.alchim31.maven:scala-maven-plugin:4.5.6:compile failed: Compilation failed
below is my pom.xml, I am trying to clone my git branch in intellij, after that trying to create a code but before that I am not even able to resolve dependencies . below is the content/xml code from pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.optum.csgbdptenant.clinical.cdf</groupId>
<artifactId>Clinical_CDF</artifactId>
<version>4.0-SNAPSHOT</version>
<url>http://maven.apache.org</url>
<properties>
<java.version>1.8</java.version>
<scala.version>2.11.8</scala.version>
<scala.binary.version>2.11.8</scala.binary.version>
<spark.version>2.4.4.0-mapr-630</spark.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<repositories>
<repository>
<id>scala-tools.org</id>
<name>Scala-tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</repository>
<repository>
<id>mapr-releases</id>
<url>http://repository.mapr.com/maven/</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
<releases>
<enabled>true</enabled>
</releases>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>scala-tools.org</id>
<name>Scala-tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</pluginRepository>
</pluginRepositories>
<dependencies>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-nop</artifactId>
<version>1.7.12</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.4.4.1-mapr-630</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.4.4.1-mapr-630</version>
</dependency>
<dependency>
<groupId>net.liftweb</groupId>
<artifactId>lift-json_2.11</artifactId>
<version>3.4.1</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-csv</artifactId>
<version>1.4</version>
</dependency>
<!-- <dependency>-->
<!-- <groupId>org.apache.spark</groupId>-->
<!-- <artifactId>spark-hive_2.11</artifactId>-->
<!-- <version>2.4.4.0-mapr-630</version>-->
<!-- </dependency>-->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.7.0-mapr-1803-r1</version>
</dependency>
<!--<dependency>-->
<!--<groupId>org.scalatest</groupId>-->
<!--<artifactId>scalatest_2.11</artifactId>-->
<!--<version>3.0.1</version>-->
<!--<scope>test</scope>-->
<!--</dependency>-->
<dependency>
<groupId>org.scalactic</groupId>
<artifactId>scalactic_2.11</artifactId>
<version>3.0.5</version>
</dependency>
<dependency>
<groupId>org.scalatest</groupId>
<artifactId>scalatest_2.11</artifactId>
<version>3.0.5</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-compiler</artifactId>
<version>${scala.version}</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.github.nscala-time/nscala-time -->
<dependency>
<groupId>com.github.nscala-time</groupId>
<artifactId>nscala-time_2.11</artifactId>
<version>2.22.0</version>
</dependency>
<dependency>
<groupId>com.mapr.db</groupId>
<artifactId>maprdb-spark</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>2.1.0-mapr-1710</version>
<scope>provided</scope>
</dependency>
<!--<dependency>-->
<!--<groupId>org.apache.hbase</groupId>-->
<!--<artifactId>hbase</artifactId>-->
<!--<version>1.1.13.0-mapr-630</version>-->
<!--</dependency>-->
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch-spark-20_2.10</artifactId>
<version>6.2.3</version>
</dependency>
<!-- <dependency>-->
<!-- <groupId>org.apache.hive</groupId>-->
<!-- <artifactId>hive-hbase-handler</artifactId>-->
<!-- <version>2.3.6-mapr-1912</version>-->
<!-- </dependency>-->
<dependency>
<groupId>org.json4s</groupId>
<artifactId>json4s-native_2.11</artifactId>
<version>3.7.0-M2</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.hortonworks/shc-core -->
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-spark</artifactId>
<version>1.1.8-mapr-1904</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.4.1-mapr-1408</version>
</dependency>
<dependency>
<groupId>com.optum.csgbdptenant.Common</groupId>
<artifactId>Common</artifactId>
<version>6.4</version>
<scope>system</scope>
<systemPath>${project.basedir}/jars/Common-6.4.jar</systemPath>
</dependency>
<dependency>
<groupId>PBEEncryption_Optum</groupId>
<artifactId>PBEEncryption_Optum</artifactId>
<scope>system</scope>
<version>1.0</version>
<systemPath>${project.basedir}/jars/PBEEncryption_Optum.jar</systemPath>
</dependency>
<!-- <dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-jdbc</artifactId>
<version>2.0.0</version>
</dependency> -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<sourceDirectory>scripts/src/main/scala</sourceDirectory>
<resources>
<resource>
<directory>scripts/src/main/resources</directory>
</resource>
</resources>
<testSourceDirectory>scripts/src/test/scala</testSourceDirectory>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>3.1.2</version>
<configuration>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<classpathPrefix>lib/</classpathPrefix>
<mainClass>Acquire</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>4.5.6</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<includes>
<include>**/*TESTS*.*</include>
</includes>
</configuration>
<!--<configuration>-->
<!--<skipTests>false</skipTests>-->
<!--</configuration>-->
</plugin>
<plugin>
<groupId>org.scalatest</groupId>
<artifactId>scalatest-maven-plugin</artifactId>
<version>1.0</version>
<configuration>
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
<junitxml>.</junitxml>
<skipTests>false</skipTests>
<testFailureIgnore>false</testFailureIgnore>
<filereports>WDF TestSuite.txt</filereports>
</configuration>
<executions>
<execution>
<id>test</id>
<goals>
<goal>test</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
adding the log also
[INFO]
[INFO] ----------< com.optum.csgbdptenant.clinical.cdf:Clinical_CDF >----------
[INFO] Building Clinical_CDF 4.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) # Clinical_CDF ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory C:\Users\sbhatt56\IdeaProjects\CASE_DETAIL_FACT\scripts\src\main\resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) # Clinical_CDF ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- scala-maven-plugin:4.5.6:compile (default) # Clinical_CDF ---
[WARNING] Expected all dependencies to require Scala version: 2.11.8
[WARNING] net.liftweb:lift-json_2.11:3.4.1 requires scala version: 2.11.12
[WARNING] Multiple versions of scala libraries detected!
[INFO] Using incremental compilation using Mixed compile order
[INFO] Compiler bridge file: C:\Users\sbhatt56\.sbt\1.0\zinc\org.scala-sbt\org.scala-sbt-compiler-bridge_2.11-1.5.8-bin_2.11.8__52.0-1.5.8_20211211T222914.jar
[INFO] compiling 14 Scala sources to C:\Users\sbhatt56\IdeaProjects\CASE_DETAIL_FACT\target\classes ...
[ERROR] C:/Users/sbhatt56/IdeaProjects/CASE_DETAIL_FACT/scripts/src/main/scala/member_policy.scala:5: expected class or object definition
[ERROR] one error found
Code testing
object RDDWithSession {
def main(args: Array[String]): Unit = {
//val sparkSession=SparkSession.builder()
val sparkSession = SparkSession.builder()
.appName("RRDSparkSession")
.config("hive.metastore.uris","thrift://apsrp07754.uhc.com:11848")
.enableHiveSupport()
.master("local")
.getOrCreate()
var recordcounts = ""
var notifymsg = ""
val updateddate = sparkSession.sql(s"select distinct update_dt from clinical.mbr_cse_pol_ref SORT by date(update_dt) DESC LIMIT 3")
notifymsg = s"Updated Records : $updateddate"
}
Thanks for the help !!

Spark Scala : Failed to find data source: kafka

I'm trying to use an example from sparkByExamples.com .
But for some reason, spark program doesn't read data from kafka topic.
Code is here
Error msg below:
Exception in thread "main" org.apache.spark.sql.AnalysisException: Failed to find data source: kafka. Please deploy the application as per the deployment section of "Structured Streaming + Kafka Integration Guide".;
at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:666)
at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:194)
at com.sparkbyexamples.spark.streaming.kafka.json.SparkStreamingConsumerKafkaJson$.main(SparkStreaming.scala:22)
at com.sparkbyexamples.spark.streaming.kafka.json.SparkStreamingConsumerKafkaJson.main(SparkStreaming.scala)
Process finished with exit code 1
Basically, I have a kafka topic1 < data.json and reading from beginning.
I recently started spark + scala + kafka activities and would appreciate your help.
pom.xml:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<groupId>com.sparkbyexamples</groupId>
<modelVersion>4.0.0</modelVersion>
<artifactId>spark-scala-examples</artifactId>
<version>1.0-SNAPSHOT</version>
<inceptionYear>2008</inceptionYear>
<packaging>jar</packaging>
<properties>
<scala.version>2.12.12</scala.version>
<spark.version>3.0.0</spark.version>
</properties>
<repositories>
<repository>
<id>scala-tools.org</id>
<name>Scala-Tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>scala-tools.org</id>
<name>Scala-Tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</pluginRepository>
</pluginRepositories>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>
<dependency>
<groupId>org.specs</groupId>
<artifactId>specs</artifactId>
<version>1.2.5</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.thoughtworks.xstream</groupId>
<artifactId>xstream</artifactId>
<version>1.4.11</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.12</artifactId>
<version>${spark.version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.12</artifactId>
<version>${spark.version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.databricks</groupId>
<artifactId>spark-xml_2.11</artifactId>
<version>0.4.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-avro_2.12</artifactId>
<version>${spark.version}</version>
</dependency>
</dependencies>
<build>
<sourceDirectory>src/main/scala</sourceDirectory>
<resources><resource><directory>src/main/resources</directory></resource></resources>
<plugins>
<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
<configuration>
<scalaVersion>${scala.version}</scalaVersion>
<args>
<arg>-target:jvm-1.5</arg>
</args>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-eclipse-plugin</artifactId>
<configuration>
<downloadSources>true</downloadSources>
<buildcommands>
<buildcommand>ch.epfl.lamp.sdt.core.scalabuilder</buildcommand>
</buildcommands>
<additionalProjectnatures>
<projectnature>ch.epfl.lamp.sdt.core.scalanature</projectnature>
</additionalProjectnatures>
<classpathContainers>
<classpathContainer>org.eclipse.jdt.launching.JRE_CONTAINER</classpathContainer>
<classpathContainer>ch.epfl.lamp.sdt.launching.SCALA_CONTAINER</classpathContainer>
</classpathContainers>
</configuration>
</plugin>
</plugins>
</build>
<reporting>
<plugins>
<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<configuration>
<scalaVersion>${scala.version}</scalaVersion>
</configuration>
</plugin>
</plugins>
</reporting>
</project>
As the error indicates, you are missing the spark-sql-kafka-0-10 dependency
Refer docs - https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html
And the correct pom file - https://github.com/sparkbyexamples/spark-examples/blob/master/spark-streaming/pom.xml#L64
Unrelated, your spark-xml Scala version needs to be 2.12

java.lang.NoClassDefFoundError when running Scala-Maven project

Trying to run a new project for the first time. mvn install goes smoothly, but when I try to run it, I get:
Exception in thread "main" java.lang.NoClassDefFoundError: scala/Tuple2
Now, something is terribly wrong if it can't even find the Scala basics. To be honest, it doesn't recognize anything outside of the code itself.
I''m attaching my POM file, hope someone can help..
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.foo</groupId>
<artifactId>bar</artifactId>
<version>1.0.0</version>
<packaging>jar</packaging>
<name>Foo Bar</name>
<properties>
<app.main.class>com.foo.main</app.main.class>
<spark.version>1.6.1</spark.version>
<scala.version>2.10.4</scala.version>
<scala.binary.version>2.10</scala.binary.version>
</properties>
<pluginRepositories>
<pluginRepository>
<id>scala-tools.org</id>
<name>Scala-tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</pluginRepository>
</pluginRepositories>
<repositories>
<repository>
<id>maven-repo</id>
<name>Maven Repository</name>
<url>http://repo1.maven.apache.org/maven2</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
<repository>
<id>apache-repo</id>
<name>Apache release repo</name>
<url>https://github.com/adatao/mvnrepos/tree/master/releases/</url>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.binary.version}</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_${scala.binary.version}</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
</dependencies>
<build>
<extensions>
<extension>
<groupId>org.springframework.build</groupId>
<artifactId>aws-maven</artifactId>
<version>5.0.0.RELEASE</version>
</extension>
</extensions>
<sourceDirectory>src/main/scala</sourceDirectory>
<testSourceDirectory>src/test/scala</testSourceDirectory>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>2.4</version>
<configuration>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<classpathPrefix>lib/</classpathPrefix>
<mainClass>com.foo.main</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.0</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
<configuration>
<jvmArgs>
<jvmArg>-Xms256m</jvmArg>
<jvmArg>-Xmx2048m</jvmArg>
</jvmArgs>
</configuration>
</plugin>
</plugins>
</build>
</project>

Scala signature error for Scala module in IntelliJ Idea Maven project

Disclaimer: I am new to Scala and trying to create a sample Scala Maven project using simple scala archtype in IntelliJ IDEA. IntelliJ version is 14.1.2
Below is my pom file, I did change the Scala version to 2.11.6 from 2.7 which the archetype generates by default.
<!-- language: lang-xml -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>learn.rxscala</groupId>
<artifactId>rxscala-reactive-course-play</artifactId>
<version>1.0.0-SNAPSHOT</version>
<inceptionYear>2008</inceptionYear>
<properties>
<scala.version>2.11.6</scala.version>
<slf4j.version>1.7.5</slf4j.version>
<reactivex.version>0.23.0</reactivex.version>
<rx.scala.compat.version>2.11</rx.scala.compat.version>
<scala.async.version>0.9.2</scala.async.version>
</properties>
<repositories>
<repository>
<id>scala-tools.org</id>
<name>Scala-Tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>scala-tools.org</id>
<name>Scala-Tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</pluginRepository>
</pluginRepositories>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.10</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.specs</groupId>
<artifactId>specs</artifactId>
<version>1.2.5</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.reactivex</groupId>
<artifactId>rxscala_${rx.scala.compat.version}</artifactId>
<version>${reactivex.version}</version>
</dependency>
<dependency>
<groupId>org.scala-lang.modules</groupId>
<artifactId>scala-async_${rx.scala.compat.version}</artifactId>
<version>${scala.async.version}</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>${slf4j.version}</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>${slf4j.version}</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.17</version>
</dependency>
</dependencies>
<build>
<sourceDirectory>src/main/scala</sourceDirectory>
<testSourceDirectory>src/test/scala</testSourceDirectory>
<plugins>
<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
<configuration>
<scalaVersion>${scala.version}</scalaVersion>
<args>
<arg>-target:jvm-1.5</arg>
</args>
</configuration>
</plugin>
</plugins>
</build>
<reporting>
<plugins>
<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<configuration>
<scalaVersion>${scala.version}</scalaVersion>
</configuration>
</plugin>
</plugins>
</reporting>
</project>
On running Maven test, I keep getting following error -
[WARNING] error: error while loading JUnit4, Scala signature JUnit4 has wrong version
[WARNING] expected: 5.0
[WARNING] found: 4.1 in JUnit4.class
I am not sure how to fix this problem?
Just make sure you use the up-to-date scala-archetype-simple, because IDEA requires new scala-archetype-simple to work with. But by default IDEA does not provide the correct scala-srchetype-simple to choose, you need to type in the right one by yourself.
It is like.
groupId:net.alchim31.maven
artifactId:scala-archetype-simple
version:1.6
packaging:maven-archetype
I had this issue as well and while trying to correct the ver number manually I found that simple using terminal or CMD you can just build a new project.
Using the mvn commands ensures that your project is built with the correct ver.
mvn -B archetype:generate -DarchetypeGroupId=org.apache.maven.archetypes -DgroupId=com.mycompany.app -DartifactId=my-app
Reference: http://maven.apache.org/guides/getting-started/index.html

Unable to execute Spray Examples | Maven dependency error

I am just trying to have hands on Spray library however am not able to execute its examples due to maven dependency.
As provided in docs I have defined Spray repository as follows:
<repositories>
<repository>
<id>sprayrepo</id>
<name>spray repo</name>
<url>http://repo.spray.io/</url>
<layout>default</layout>
</repository>
</repositories>
Then I have defined its dependency as follows:
<dependency>
<groupId>io.spray</groupId>
<artifactId>spray-http</artifactId>
<version>1.1-M7</version>
</dependency>
<dependency>
<groupId>io.spray</groupId>
<artifactId>spray-httpx</artifactId>
<version>1.1-M7</version>
</dependency>
<dependency>
<groupId>io.spray</groupId>
<artifactId>spray-util</artifactId>
<version>1.1-M7</version>
</dependency>
<dependency>
<groupId>com.chuusai</groupId>
<artifactId>shapeless_2.9.2</artifactId>
<version>1.2.0</version>
</dependency>
</dependencies>
However when I try to execute to maven install, I face following error:
[ERROR] Failed to execute goal on project spray-test1: Could not
resolve dependencies for project
org.spray:spray-test1:jar:1.0-SNAPSHOT: Failed to collect dependencies
for [io.spray:spray-http:jar:1.1-M7 (compile),
io.spray:spray-httpx:jar:1.1-M7 (compile),
io.spray:spray-util:jar:1.1-M7 (compile),
com.chuusai:shapeless_2.9.2:jar:1.2.0 (compile)]: Failed to read
artifact descriptor for io.spray:spray-http:jar:1.1-M7: Could not
transfer artifact io.spray:spray-http:pom:1.1-M7 from/to sprayrepo
(http://repo.spray.io/): connection timed out to
http://repo.spray.io/io/spray/spray-http/1.1-M7/spray-http-1.1-M7.pom
-> [Help 1] [ERROR]
Any Pointers or suggestions will be highly appreciated.
Thanks
Try this:
<properties>
<maven.compiler.source>1.6</maven.compiler.source>
<maven.compiler.target>1.6</maven.compiler.target>
<encoding>UTF-8</encoding>
<scala.tools.version>2.10</scala.tools.version>
<scala.version>2.10.2</scala.version>
</properties>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>
<dependency>
<groupId>io.spray</groupId>
<artifactId>spray-can</artifactId>
<version>1.1-M8</version>
</dependency>
<!-- Test -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.specs2</groupId>
<artifactId>specs2_${scala.tools.version}</artifactId>
<version>1.13</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.scalatest</groupId>
<artifactId>scalatest_${scala.tools.version}</artifactId>
<version>2.0.M6-SNAP8</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<sourceDirectory>src/main/scala</sourceDirectory>
<testSourceDirectory>src/test/scala</testSourceDirectory>
<plugins>
<plugin>
<!-- see http://davidb.github.com/scala-maven-plugin -->
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.1.3</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
<configuration>
<args>
<arg>-make:transitive</arg>
<arg>-dependencyfile</arg>
<arg>${project.build.directory}/.scala_dependencies</arg>
</args>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.13</version>
<configuration>
<useFile>false</useFile>
<disableXmlReport>true</disableXmlReport>
<!-- If you have classpath issue like NoDefClassError,... -->
<!-- useManifestOnlyJar>false</useManifestOnlyJar -->
<includes>
<include>**/*Test.*</include>
<include>**/*Suite.*</include>
</includes>
</configuration>
</plugin>
</plugins>
</build>
<repositories>
<repository>
<id>SonatypeNexusSnapshots</id>
<name>Sonatype Nexus Snapshots</name>
<url>https://oss.sonatype.org/content/repositories/snapshots/</url>
</repository>
<repository>
<id>FuseSourceSnapshotRepository</id>
<name>FuseSource Snapshot Repository</name>
<url>http://repo.fusesource.com/nexus/content/repositories/snapshots/</url>
</repository>
<repository>
<id>ScalaToolsMaven2Repository</id>
<name>Scala-Tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases/</url>
</repository>
<repository>
<id>scala-tools.org</id>
<name>Scala-Tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</repository>
<repository>
<id>sprayrepo</id>
<name>spray repo</name>
<url>http://repo.spray.io/</url>
<layout>default</layout>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>scala-tools.org</id>
<name>Scala-Tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</pluginRepository>
</pluginRepositories>