trying to deploy war to heroku
using maven 3.2.5
mvn clean heroku:deploy-war
getting this exception:
[ERROR] Failed to execute goal com.heroku.sdk:heroku-maven-plugin:0.3.0:deploy-war (default-cli) on project MyApp: Failed to deploy application: There was an exception invoking the remote service: HTTP(404): https://api.heroku.com/apps/MyApp/config-vars -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.heroku.sdk:heroku-maven-plugin:0.3.0:deploy-war (default-cli) on project MyApp: Failed to deploy application
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:355)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:216)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:160)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoFailureException: Failed to deploy application
at com.heroku.sdk.maven.DeployWarMojo.execute(DeployWarMojo.java:58)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
... 19 more
Caused by: com.heroku.sdk.deploy.Curl$CurlException: There was an exception invoking the remote service: HTTP(404)
at com.heroku.sdk.deploy.Curl.handleResponse(Curl.java:94)
at com.heroku.sdk.deploy.Curl.get(Curl.java:22)
at com.heroku.sdk.deploy.App.getConfigVars(App.java:139)
at com.heroku.sdk.deploy.App.deploy(App.java:77)
at com.heroku.sdk.deploy.App.deploy(App.java:93)
at com.heroku.sdk.deploy.WarApp.deploy(WarApp.java:38)
at com.heroku.sdk.maven.DeployWarMojo.execute(DeployWarMojo.java:54)
... 21 more
Caused by: java.io.FileNotFoundException: https://api.heroku.com/apps/MyApp/config-vars
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1624)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
at com.heroku.sdk.deploy.Curl.handleResponse(Curl.java:90)
... 27 more
this is how my pom looks like:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.avi</groupId>
<artifactId>MyApp</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>war</packaging>
<description>My GoTickets Web App</description>
<inceptionYear>2014</inceptionYear>
<properties>
<scala.version>2.11.4</scala.version>
<spray.version>1.3.1</spray.version>
<akka.version>2.3.8</akka.version>
</properties>
<repositories>
<repository>
<id>scala-tools.org</id>
<name>Scala-Tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>scala-tools.org</id>
<name>Scala-Tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</pluginRepository>
</pluginRepositories>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.9</version>
<scope>test</scope>
</dependency>
<dependency>
<scope>test</scope>
<groupId>org.specs2</groupId>
<artifactId>specs2-core_2.11</artifactId>
<version>2.4.15</version>
</dependency>
<dependency>
<scope>test</scope>
<groupId>com.typesafe.akka</groupId>
<artifactId>akka-testkit_2.11</artifactId>
<version>${akka.version}</version>
</dependency>
<dependency>
<groupId>org.scalatest</groupId>
<artifactId>scalatest_2.11</artifactId>
<version>2.2.3</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.spray</groupId>
<artifactId>spray-can</artifactId>
<version>${spray.version}</version>
</dependency>
<dependency>
<groupId>io.spray</groupId>
<artifactId>spray-routing</artifactId>
<version>${spray.version}</version>
</dependency>
<dependency>
<groupId>io.spray</groupId>
<artifactId>spray-json_2.11</artifactId>
<version>${spray.version}</version>
</dependency>
<dependency>
<groupId>com.typesafe.akka</groupId>
<artifactId>akka-actor_2.11</artifactId>
<version>${akka.version}</version>
</dependency>
<dependency>
<groupId>com.typesafe.akka</groupId>
<artifactId>akka-slf4j_2.11</artifactId>
<version>${akka.version}</version>
</dependency>
<dependency>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-war-plugin</artifactId>
<version>2.5</version>
</dependency>
</dependencies>
<build>
<sourceDirectory>src/main/scala</sourceDirectory>
<testSourceDirectory>src/test/scala</testSourceDirectory>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.0</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
<configuration>
<scalaVersion>${scala.version}</scalaVersion>
</configuration>
</plugin>
<plugin>
<groupId>com.heroku.sdk</groupId>
<artifactId>heroku-maven-plugin</artifactId>
<version>0.3.0</version>
<configuration>
<appName>MyApp</appName>
</configuration>
</plugin>
</plugins>
</build>
</project>
Your appName is incorrect in the Heroku plugin configuration. Heroku apps must start with a letter and can only contain lowercase letters, numbers, and dashes. Therefore MyApp cannot be correct.
Check your apps page in the Heroku dashboard and find the name of your application then update the configuration accordingly. Here is an example of the plugin config with a randomly named Heroku app:
<plugin>
<groupId>com.heroku.sdk</groupId>
<artifactId>heroku-maven-plugin</artifactId>
<version>0.3.0</version>
<configuration>
<appName>hidden-badlands-1234</appName>
</configuration>
</plugin>
Related
I am getting below error while building Jar file. I am stuck with this issue from so long please advice.
Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.5.6:compile (default) on project Clinical_CDF: Execution default of goal net.alchim31.maven:scala-maven-plugin:4.5.6:compile failed: Compilation failed
below is my pom.xml, I am trying to clone my git branch in intellij, after that trying to create a code but before that I am not even able to resolve dependencies . below is the content/xml code from pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.optum.csgbdptenant.clinical.cdf</groupId>
<artifactId>Clinical_CDF</artifactId>
<version>4.0-SNAPSHOT</version>
<url>http://maven.apache.org</url>
<properties>
<java.version>1.8</java.version>
<scala.version>2.11.8</scala.version>
<scala.binary.version>2.11.8</scala.binary.version>
<spark.version>2.4.4.0-mapr-630</spark.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<repositories>
<repository>
<id>scala-tools.org</id>
<name>Scala-tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</repository>
<repository>
<id>mapr-releases</id>
<url>http://repository.mapr.com/maven/</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
<releases>
<enabled>true</enabled>
</releases>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>scala-tools.org</id>
<name>Scala-tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</pluginRepository>
</pluginRepositories>
<dependencies>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-nop</artifactId>
<version>1.7.12</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.4.4.1-mapr-630</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.4.4.1-mapr-630</version>
</dependency>
<dependency>
<groupId>net.liftweb</groupId>
<artifactId>lift-json_2.11</artifactId>
<version>3.4.1</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-csv</artifactId>
<version>1.4</version>
</dependency>
<!-- <dependency>-->
<!-- <groupId>org.apache.spark</groupId>-->
<!-- <artifactId>spark-hive_2.11</artifactId>-->
<!-- <version>2.4.4.0-mapr-630</version>-->
<!-- </dependency>-->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.7.0-mapr-1803-r1</version>
</dependency>
<!--<dependency>-->
<!--<groupId>org.scalatest</groupId>-->
<!--<artifactId>scalatest_2.11</artifactId>-->
<!--<version>3.0.1</version>-->
<!--<scope>test</scope>-->
<!--</dependency>-->
<dependency>
<groupId>org.scalactic</groupId>
<artifactId>scalactic_2.11</artifactId>
<version>3.0.5</version>
</dependency>
<dependency>
<groupId>org.scalatest</groupId>
<artifactId>scalatest_2.11</artifactId>
<version>3.0.5</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-compiler</artifactId>
<version>${scala.version}</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.github.nscala-time/nscala-time -->
<dependency>
<groupId>com.github.nscala-time</groupId>
<artifactId>nscala-time_2.11</artifactId>
<version>2.22.0</version>
</dependency>
<dependency>
<groupId>com.mapr.db</groupId>
<artifactId>maprdb-spark</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>2.1.0-mapr-1710</version>
<scope>provided</scope>
</dependency>
<!--<dependency>-->
<!--<groupId>org.apache.hbase</groupId>-->
<!--<artifactId>hbase</artifactId>-->
<!--<version>1.1.13.0-mapr-630</version>-->
<!--</dependency>-->
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch-spark-20_2.10</artifactId>
<version>6.2.3</version>
</dependency>
<!-- <dependency>-->
<!-- <groupId>org.apache.hive</groupId>-->
<!-- <artifactId>hive-hbase-handler</artifactId>-->
<!-- <version>2.3.6-mapr-1912</version>-->
<!-- </dependency>-->
<dependency>
<groupId>org.json4s</groupId>
<artifactId>json4s-native_2.11</artifactId>
<version>3.7.0-M2</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.hortonworks/shc-core -->
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-spark</artifactId>
<version>1.1.8-mapr-1904</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.4.1-mapr-1408</version>
</dependency>
<dependency>
<groupId>com.optum.csgbdptenant.Common</groupId>
<artifactId>Common</artifactId>
<version>6.4</version>
<scope>system</scope>
<systemPath>${project.basedir}/jars/Common-6.4.jar</systemPath>
</dependency>
<dependency>
<groupId>PBEEncryption_Optum</groupId>
<artifactId>PBEEncryption_Optum</artifactId>
<scope>system</scope>
<version>1.0</version>
<systemPath>${project.basedir}/jars/PBEEncryption_Optum.jar</systemPath>
</dependency>
<!-- <dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-jdbc</artifactId>
<version>2.0.0</version>
</dependency> -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<sourceDirectory>scripts/src/main/scala</sourceDirectory>
<resources>
<resource>
<directory>scripts/src/main/resources</directory>
</resource>
</resources>
<testSourceDirectory>scripts/src/test/scala</testSourceDirectory>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>3.1.2</version>
<configuration>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<classpathPrefix>lib/</classpathPrefix>
<mainClass>Acquire</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>4.5.6</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<includes>
<include>**/*TESTS*.*</include>
</includes>
</configuration>
<!--<configuration>-->
<!--<skipTests>false</skipTests>-->
<!--</configuration>-->
</plugin>
<plugin>
<groupId>org.scalatest</groupId>
<artifactId>scalatest-maven-plugin</artifactId>
<version>1.0</version>
<configuration>
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
<junitxml>.</junitxml>
<skipTests>false</skipTests>
<testFailureIgnore>false</testFailureIgnore>
<filereports>WDF TestSuite.txt</filereports>
</configuration>
<executions>
<execution>
<id>test</id>
<goals>
<goal>test</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
adding the log also
[INFO]
[INFO] ----------< com.optum.csgbdptenant.clinical.cdf:Clinical_CDF >----------
[INFO] Building Clinical_CDF 4.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) # Clinical_CDF ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory C:\Users\sbhatt56\IdeaProjects\CASE_DETAIL_FACT\scripts\src\main\resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) # Clinical_CDF ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- scala-maven-plugin:4.5.6:compile (default) # Clinical_CDF ---
[WARNING] Expected all dependencies to require Scala version: 2.11.8
[WARNING] net.liftweb:lift-json_2.11:3.4.1 requires scala version: 2.11.12
[WARNING] Multiple versions of scala libraries detected!
[INFO] Using incremental compilation using Mixed compile order
[INFO] Compiler bridge file: C:\Users\sbhatt56\.sbt\1.0\zinc\org.scala-sbt\org.scala-sbt-compiler-bridge_2.11-1.5.8-bin_2.11.8__52.0-1.5.8_20211211T222914.jar
[INFO] compiling 14 Scala sources to C:\Users\sbhatt56\IdeaProjects\CASE_DETAIL_FACT\target\classes ...
[ERROR] C:/Users/sbhatt56/IdeaProjects/CASE_DETAIL_FACT/scripts/src/main/scala/member_policy.scala:5: expected class or object definition
[ERROR] one error found
Code testing
object RDDWithSession {
def main(args: Array[String]): Unit = {
//val sparkSession=SparkSession.builder()
val sparkSession = SparkSession.builder()
.appName("RRDSparkSession")
.config("hive.metastore.uris","thrift://apsrp07754.uhc.com:11848")
.enableHiveSupport()
.master("local")
.getOrCreate()
var recordcounts = ""
var notifymsg = ""
val updateddate = sparkSession.sql(s"select distinct update_dt from clinical.mbr_cse_pol_ref SORT by date(update_dt) DESC LIMIT 3")
notifymsg = s"Updated Records : $updateddate"
}
Thanks for the help !!
I am working on a build file utilizing dependencies from scala and spark packages, etc, on Java 1.8 and Scala 2.10.5.
The build is throwing up on the jackson-bind package. I have added the entire pom and the exception I see:
Exception
central (https://repo.maven.apache.org/maven2, releases=true, snapshots=false)
[ERROR] : ClientProtocolException:
Invalid redirect URI: http://blog.goodstuff.im/repo-releases/com/fasterxml/jackson/core/jackson-databind/[2.7.0,)/jackson-databind-[2.7.0,).jar:
Illegal character in path at index 83: http://blog.goodstuff.im/repo-releases/com/fasterxml/jackson/core/jackson-databind/[2.7.0,)/jackson-databind-[2.7.0,).jar
Tried the exclusion tags, it is still failing with an error:
ClientProtocolException: Invalid redirect URI:
http://blog.goodstuff.im/repo-releases/com/fasterxml/jackson/core/jackson-databind/[2.7.0,)/jackson-databind-[2.7.0,).jar:
Illegal character in path at index 83:
http://blog.goodstuff.im/repo-releases/com/fasterxml/jackson/core/jackson-databind/[2.7.0,)/jackson-databind-[2.7.0,).jar
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
My pom.xml file:
<?xml version="1.0" encoding="utf-8" ?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.mycompany</groupId>
<artifactId>c-platform</artifactId>
<version>3.0.0-SNAPSHOT</version>
<name>${project.version} - ${project.artifactId}</name>
<packaging>pom</packaging>
<parent> <groupId>com.mycompany</groupId>
<artifactId>c-parent</artifactId>
<version>3.0.0-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>
<repositories>
<repository>
<id>Nexus</id>
<url>…-snapshots/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
<repository>
<id> Nexus shapshot</id>
<url>../nexus/content/groups/public</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
<repository>
<id>Cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
<repository>
<id>scala-tools.org</id>
<name>Scala-tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</repository>
<repository>
<id>maven2-repository.dev.java.net</id>
<name>Java.net Repository for Maven</name>
<url>http://download.java.net/maven/2/</url>
<layout>default</layout>
</repository>
</repositories>
<!-- list all dependencies - begin -->
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>
<dependency>
<groupId>com.maxmind.geoip2</groupId>
<artifactId>geoip2</artifactId>
<version>2.7.0</version>
</dependency>
<dependency>
<groupId>com.databricks</groupId>
<artifactId>spark-csv_2.10</artifactId>
<version>${spark.csv.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.10</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_2.10</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.module</groupId>
<artifactId>jackson-module-scala_2.11</artifactId>
<version>2.7.3</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.7.3</version>
<exclusions>
<exclusion>
<artifactId>jackson-annotations</artifactId>
<groupId>com.fasterxml.jackson.core</groupId>
</exclusion>
<exclusion>
<artifactId>jackson-core</artifactId>
<groupId>com.fasterxml.jackson.core</groupId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
<!-- list all dependencies - emd -->
<build>
<sourceDirectory>src/main/scala</sourceDirectory>
<testSourceDirectory>src/test/scala</testSourceDirectory>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.1.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<relocations>
<relocation>
<pattern>com.fasterxml.jackson.**</pattern>
<shadedPattern>com.shaded.fasterxml.jackson#1</shadedPattern>
</relocation>
</relocations>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<version>2.10</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
</goals>
<configuration>
<args>
<arg>-dependencyfile</arg>
<arg>${project.build.directory}/.scala_dependencies</arg>
</args>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.7.0</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<showDeprecation>true</showDeprecation>
<showWarnings>true</showWarnings>
</configuration>
</plugin>
</plugins>
</build>
</project>
It seems that jackson-databind 2.7.3 brings as transitive dependencies jackson-annotations and jackson-core 2.7.0 which might cause some dependency conflict in your build.
I couldn't reproduce exactly, your errors, since I don't have your entire pom. But I would suggest you fix the dependency conflict by:
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.7.3</version>
<exclusions>
<exclusion>
<artifactId>jackson-annotations</artifactId>
<groupId>com.fasterxml.jackson.core</groupId>
</exclusion>
<exclusion>
<artifactId>jackson-core</artifactId>
<groupId>com.fasterxml.jackson.core</groupId>
</exclusion>
</exclusions>
</dependency>
Now you jackon-bind wil miss "jackson-core" and jackson-annotation, but you could add them to the pom with the proper version (2.7.3) and apply the exclusion trick.
I am using Scala 2.11.8 and used maven to build the jar file in IntelliJ
I ran my program in mobaxterm:
/opt/spark-1.6.1-bin-hadoop2.6/bin/spark-submit --class CDR.SQL cdr-maven-1.0-SNAPSHOT.jar
This is the error message:
Exception in thread "main" java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaUniverse$JavaMirror;
at CDR.SQL$.(SQL.scala:24)
at CDR.SQL$.(SQL.scala)
at CDR.SQL.main(SQL.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
This is my pom.xml file:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>cdr-maven</groupId>
<artifactId>cdr-maven</artifactId>
<version>1.0-SNAPSHOT</version>
<repositories>
<repository>
<id>apache-repo</id>
<name>Apache Repository</name>
<url>https://repository.apache.org/content/repositories/releases</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
</repositories>
<build>
<plugins>
<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<version>2.15.2</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.11.8</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>1.4.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.11</artifactId>
<version>1.4.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>1.4.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_2.11</artifactId>
<version>1.4.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>1.4.1</version>
<scope>provided</scope>
</dependency>
</dependencies>
</project>
How do I resolve this?
Thanks for the help!
Have you downloaded Spark distribution or build it by yourself?
Spark 1.6 was built by default with Scala 2.10. Please change your Scala version to 2.10 or update Spark to 2.0, which is built by default with Scala 2.11
Check the version.
<build>
<plugins>
<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<version>2.15.2</version>
<executions>
<execution>
Looks like you are using 2.15.x version.
After using this command to run my spark application on mobaxterm:
/opt/spark-2.0.0-bin-hadoop2.6/bin/spark-submit --class CDR.SQL cdr-maven-1.0-SNAPSHOT.jar
This error occurs:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/DataFrame
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:713)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.DataFrame
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 10 more
Built with maven (pom.xml):
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>cdr-maven</groupId>
<artifactId>cdr-maven</artifactId>
<version>1.0-SNAPSHOT</version>
<repositories>
<repository>
<id>apache-repo</id>
<name>Apache Repository</name>
<url>https://repository.apache.org/content/repositories/releases</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
</repositories>
<build>
<plugins>
<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.7</source>
<target>1.7</target>
</configuration>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.11.8</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>1.4.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.11</artifactId>
<version>1.4.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>1.4.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_2.11</artifactId>
<version>1.4.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_2.11</artifactId>
<version>1.4.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>1.4.1</version>
<scope>provided</scope>
</dependency>
</dependencies>
</project>
I got a feeling this is because my .jar is missing the DataFrame dependency but I'm not sure how to solve it.
Thanks for the help.
You are using Spark 1.4 and deploying it on Spark 2.0 cluster, where the class DataFrame was removed. Currently, there is only type DataFrame = Dataset[Row]. Please change dependency version to 2.0 or change cluster version to < 2.0
Goto "p=Project Setting" --> "Modules" "Dependencies" tab. Change the scope of org.apache.spark:spark_core_2.11 to compile.
Disclaimer: I am new to Scala and trying to create a sample Scala Maven project using simple scala archtype in IntelliJ IDEA. IntelliJ version is 14.1.2
Below is my pom file, I did change the Scala version to 2.11.6 from 2.7 which the archetype generates by default.
<!-- language: lang-xml -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>learn.rxscala</groupId>
<artifactId>rxscala-reactive-course-play</artifactId>
<version>1.0.0-SNAPSHOT</version>
<inceptionYear>2008</inceptionYear>
<properties>
<scala.version>2.11.6</scala.version>
<slf4j.version>1.7.5</slf4j.version>
<reactivex.version>0.23.0</reactivex.version>
<rx.scala.compat.version>2.11</rx.scala.compat.version>
<scala.async.version>0.9.2</scala.async.version>
</properties>
<repositories>
<repository>
<id>scala-tools.org</id>
<name>Scala-Tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>scala-tools.org</id>
<name>Scala-Tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</pluginRepository>
</pluginRepositories>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.10</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.specs</groupId>
<artifactId>specs</artifactId>
<version>1.2.5</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.reactivex</groupId>
<artifactId>rxscala_${rx.scala.compat.version}</artifactId>
<version>${reactivex.version}</version>
</dependency>
<dependency>
<groupId>org.scala-lang.modules</groupId>
<artifactId>scala-async_${rx.scala.compat.version}</artifactId>
<version>${scala.async.version}</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>${slf4j.version}</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>${slf4j.version}</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.17</version>
</dependency>
</dependencies>
<build>
<sourceDirectory>src/main/scala</sourceDirectory>
<testSourceDirectory>src/test/scala</testSourceDirectory>
<plugins>
<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
<configuration>
<scalaVersion>${scala.version}</scalaVersion>
<args>
<arg>-target:jvm-1.5</arg>
</args>
</configuration>
</plugin>
</plugins>
</build>
<reporting>
<plugins>
<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<configuration>
<scalaVersion>${scala.version}</scalaVersion>
</configuration>
</plugin>
</plugins>
</reporting>
</project>
On running Maven test, I keep getting following error -
[WARNING] error: error while loading JUnit4, Scala signature JUnit4 has wrong version
[WARNING] expected: 5.0
[WARNING] found: 4.1 in JUnit4.class
I am not sure how to fix this problem?
Just make sure you use the up-to-date scala-archetype-simple, because IDEA requires new scala-archetype-simple to work with. But by default IDEA does not provide the correct scala-srchetype-simple to choose, you need to type in the right one by yourself.
It is like.
groupId:net.alchim31.maven
artifactId:scala-archetype-simple
version:1.6
packaging:maven-archetype
I had this issue as well and while trying to correct the ver number manually I found that simple using terminal or CMD you can just build a new project.
Using the mvn commands ensures that your project is built with the correct ver.
mvn -B archetype:generate -DarchetypeGroupId=org.apache.maven.archetypes -DgroupId=com.mycompany.app -DartifactId=my-app
Reference: http://maven.apache.org/guides/getting-started/index.html