Porting a Maven Java project that uses JavaCC to Scala - scala

I am trying to port a Maven based Java project that uses JavaCC to Scala.
I am happy to leave the JavaCC generated classes in Java at least for now.
However the JavaCC generated class depends on the classes being converted.
The project builds fine from the original Java sources, but if I replace them with the Scala sources Maven tries to compile the JavaCC generated class first and does not see and of the symbols provided by the new code.
The problem then seems to be how to get Maven to generate the Java code first then compile the Scala code and the generated code together that the compiler can find all dependencies. The solution seems to be to add some magical incantation to pom.xml. What should that be though?
My current pom.xml is:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>bobj</groupId>
<artifactId>bobj</artifactId>
<version>0.9</version>
<packaging>jar</packaging>
<name>BOBJ algebraic specification and verification system</name>
<url>http://github.com/fh-wedel/BOBJ</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<profiles>
<profile>
<id>scala-2.10</id>
<properties>
<scalaVersion>2.10.4</scalaVersion>
<scalaBinaryVersion>2.10</scalaBinaryVersion>
</properties>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scalaVersion}</version>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-swing</artifactId>
<version>${scalaVersion}</version>
</dependency>
</dependencies>
</profile>
</profiles>
<build>
<sourceDirectory>src/main/java</sourceDirectory>
<plugins>
<plugin>
<artifactId>maven-jar-plugin</artifactId>
<version>2.4</version>
<configuration>
<archive>
<manifest>
<mainClass>bobj.BOBJ</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>javacc-maven-plugin</artifactId>
<version>2.6</version>
<executions>
<execution>
<id>javacc</id>
<goals>
<goal>javacc</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.6</source>
<target>1.6</target>
</configuration>
</plugin>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.1.6</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
<configuration>
<args>
<!-- work-around for https://issues.scala-lang.org/browse/SI-8358 -->
<arg>-nobootcp</arg>
</args>
</configuration>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<version>1.9.1</version>
<executions>
<execution>
<id>add-source</id>
<phase>generate-sources</phase>
<goals>
<goal>add-source</goal>
</goals>
<configuration>
<sources>
<source>src/main/scala</source>
<source>src/main/javacc</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.8.2</version>
<scope>test</scope>
</dependency>
</dependencies>
</project>
I did not write this. I started with the pom.xml that came with the project and and added the parts for scala and build-helper-maven-plugin
which I copied from other sources. My Maven-fu is minimal so I would not be surprised if I am doing something deeply misguided.

It turns out I forgot set up the scala-maven-plugin declaration for a mixed scala java project
ie the scala-maven-plugin needs
<executions>
<execution>
<id>scala-process-resources</id>
<phase>process-resources</phase>
<goals>
<goal>add-source</goal>
<goal>compile</goal>
</goals>
</execution>
<execution>
<id>scala-process-test-resources</id>
<phase>process-test-resources</phase>
<goals>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
instead of the usual
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>

Related

cannot resolve symbol apache in spark scala maven

I am creating Spark application with scala, and it is Maven Project.
If Possible may Someone can share POM file. My application is only having SPARKSQL.
Do i need to set HADOOP_HOME to the directory containing winutils.exe
as i have not added in the config part of the code.
My POM file looks like:-
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>net.martinprobson.spark</groupId>
<artifactId>spark_example</artifactId>
<version>1.0-SNAPSHOT</version>
<name>${project.artifactId}</name>
<description>Spark Batch And Streaming Application</description>
<inceptionYear>2019</inceptionYear>
<properties>
<scala.version>2.11</scala.version>
<scala.full.version>2.11.8</scala.full.version>
<spark.version>2.4.4</spark.version>
<java.version>1.8</java.version>
<jackson.version>2.6.5</jackson.version>
<scala.maven.plugin.version>3.2.2</scala.maven.plugin.version>
<maven.surefire.plugin.version>2.13</maven.surefire.plugin.version>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
<dependencies>
<!--Scala dependencies-->
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.full.version}</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.2</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.4.4</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.version}</artifactId>
<version>2.4.4</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-hive -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_${scala.version}</artifactId>
<version>2.4.4</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-yarn -->
<dependency>
<groupId>com.typesafe</groupId>
<artifactId>config</artifactId>
<version>1.3.0</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-deploy-plugin</artifactId>
<version>2.7</version>
</plugin>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.1.6</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
<configuration>
<scalaVersion>${scala.version}</scalaVersion>
</configuration>
</plugin>
<!-- disable surefire -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.12.4</version>
<configuration>
<skipTests>true</skipTests>
</configuration>
</plugin>
<!-- enable scala test -->
<plugin>
<groupId>org.scalatest</groupId>
<artifactId>scalatest_2.10</artifactId>
<version>2.2.6</version>
<configuration>
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
<junitxml>.</junitxml>
<filereports>TestSuite.txt</filereports>
</configuration>
<executions>
<execution>
<id>test</id>
<goals>
<goal>test</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<executions>
<execution>
<id>jar-with-dependencies</id>
<phase>package</phase>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
<configuration>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
My Scala code looks like
package Batch
import org.apache.spark.sql.{DataFrame, SaveMode, SparkSession}
import org.apache.spark.sql.functions._
Object BatchJob {
def main(args: Array[String]) {
val spark = SparkSession.builder
.master("local")
.appName("Fraud Detector")
.config("spark.driver.memory", "2g")
.enableHiveSupport
.getOrCreate()
import spark.implicits._
val financesDF = spark.read.json("Data/finances-small.json")
}
}
But getting Error as
**Cannot Resolve Symbol Apache
cannot Resolve Symbol Savemode
cannot Resolve Symbol SparkSession**
Is any problem with POM........ Highly Appreciate suggestion.
Kind Regards
you can simply replace your build tag with below. It worked for me.
<build>
<sourceDirectory>src/main/scala</sourceDirectory>
<testSourceDirectory>src/test/scala</testSourceDirectory>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.3.2</version>
<executions>
<execution>
<id>scala-compile-first</id>
<phase>process-resources</phase>
<goals>
<goal>add-source</goal>
<goal>compile</goal>
</goals>
</execution>
<execution>
<id>scala-test-compile</id>
<phase>process-test-resources</phase>
<goals>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.3.2</version>
<executions>
<execution>
<phase>compile</phase>
<goals>
<goal>compile</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>

builiding maven built with spark in scala ide giving execute exception

I am using Scala IDE for creating a maven project with spark
1.I have created a maven project with skipped the archetype and added the following pom file.
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.nivi</groupId>
<artifactId>testSpark</artifactId>
<version>0.0.1-SNAPSHOT</version>
<repositories>
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
<repository>
<id>general</id>
<url>https://repo1.maven.org/maven2</url>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.10.6</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.6.0-cdh5.8.2</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.10</artifactId>
<version>1.6.0-cdh5.8.2</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_2.10</artifactId>
<version>1.6.0-cdh5.8.2</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<version>2.15.2</version>
<scope>provided</scope>
</dependency>
</dependencies>
<build>
<plugins>
<!-- mixed scala/java compile -->
<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<version>2.15.2</version>
<executions>
<execution>
<id>compile</id>
<goals>
<goal>compile</goal>
</goals>
<phase>compile</phase>
</execution>
<execution>
<id>test-compile</id>
<goals>
<goal>testCompile</goal>
</goals>
<phase>test-compile</phase>
</execution>
<execution>
<phase>process-resources</phase>
<goals>
<goal>compile</goal>
</goals>
</execution>
</executions>
<configuration>
<scalaVersion>2.10.6</scalaVersion>
</configuration>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.4</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<!-- for fatjar -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.4</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>assemble-all</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>2.4</version>
<configuration>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<mainClass>fully.qualified.MainClass</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
</plugins>
<pluginManagement>
<plugins>
<!--This plugin's configuration is used to store Eclipse m2e settings
only. It has no influence on the Maven build itself. -->
<plugin>
<groupId>org.eclipse.m2e</groupId>
<artifactId>lifecycle-mapping</artifactId>
<version>1.0.0</version>
<configuration>
<lifecycleMappingMetadata>
<pluginExecutions>
<pluginExecution>
<pluginExecutionFilter>
<groupId>org.scala-tools</groupId>
<artifactId>
maven-scala-plugin
</artifactId>
<versionRange>
[2.15.2,)
</versionRange>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</pluginExecutionFilter>
<action>
<execute></execute>
</action>
</pluginExecution>
</pluginExecutions>
</lifecycleMappingMetadata>
</configuration>
</plugin>
</plugins>
</pluginManagement>
</build>
</project>
I have added scala nature in Configure.
I have added src/main/scala folder in the Properties-->Build path-->Source
I have set java compiler 1.8 and scala compiler 2.10.6 with jvm 1.8 in my
project properties.
But i am getting the following error on maven clean install :
enter image description here
Go to this location
c:\users\Nivedita.m2\repository\org\apache\hadoop\hadoop-common\2.6.0-cdh5.8.2
and try deleting hadoop-common-2.6.0-cdh5.8.2.jar file and do maven build again.
Can you try with an recent version of scala plugin ? (it's the successor of the one you used, I'm the author of both)
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.3.2</version>
</plugin>
(see http://davidb.github.io/scala-maven-plugin/example_java.html)
if you use a jdk 9+ to compile (regardless of jvm 1.8 in the config) you can have this kind of issue. So check that the jdk is 1.8 .

Maven shade-plugin not copying classes to .jar file

I have downloaded Gatling Maven Example and trying add mvn shade plugin in it as below, the jar get created but it doesn't contains any classes, so it fails during execution
E:\projects\gatling-maven>java -jar target\gatling-maven-plugin-demo-2.2.3.jar
Error: Could not find or load main class Engine
Here is pom.xml I have added
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>io.gatling</groupId>
<artifactId>gatling-maven-plugin-demo</artifactId>
<version>2.2.3</version>
<properties>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<gatling.version>${project.version}</gatling.version>
<gatling-plugin.version>2.2.1</gatling-plugin.version>
<scala-maven-plugin.version>3.2.2</scala-maven-plugin.version>
</properties>
<dependencies>
<dependency>
<groupId>io.gatling.highcharts</groupId>
<artifactId>gatling-charts-highcharts</artifactId>
<version>${gatling.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>${scala-maven-plugin.version}</version>
</plugin>
<plugin>
<groupId>io.gatling</groupId>
<artifactId>gatling-maven-plugin</artifactId>
<version>${gatling-plugin.version}</version>
<executions>
<execution>
<goals>
<goal>execute</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>Engine</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
and package structure is as
You have to move all your resources and scala files to src\main\resources and src\main\scala. Shade plugin will not include your test resources and scala files. I have also tried shadedTestjar and it also does not work. The other option could be you use either
Maven Dependency plugin and move all dependency manually - Error prone and ugly
Use Assembly plugin - Not suitable
I have tried moving your resources and scala files in src/main and it has worked. Following is working pom content,
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>io.gatling</groupId>
<artifactId>gatling-maven-plugin-demo</artifactId>
<version>2.2.3</version>
<properties>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<gatling.version>${project.version}</gatling.version>
<gatling-plugin.version>2.2.1</gatling-plugin.version>
<scala-maven-plugin.version>3.2.2</scala-maven-plugin.version>
</properties>
<dependencies>
<dependency>
<groupId>io.gatling.highcharts</groupId>
<artifactId>gatling-charts-highcharts</artifactId>
<version>${gatling.version}</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>${scala-maven-plugin.version}</version>
</plugin>
<plugin>
<groupId>io.gatling</groupId>
<artifactId>gatling-maven-plugin</artifactId>
<version>${gatling-plugin.version}</version>
<executions>
<execution>
<goals>
<goal>execute</goal>
</goals>
<configuration>
<configFolder>src/main/resources</configFolder>
<dataFolder>src/main/resources/data</dataFolder>
<resultsFolder>target/gatling/results</resultsFolder>
<requestBodiesFolder>src/main/resources/request-bodies</requestBodiesFolder>
<simulationsFolder>src/main/scala</simulationsFolder>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>Engine</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
Hope it solves your problem.
Try after changing your following plugin configuration,
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>${scala-maven-plugin.version}</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
and
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>Engine</mainClass>
</transformer>
</transformers>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>
There are still file not found exception but I believe they are trivial to solve.
First learn about the Maven standard directory structure convention. Your main project source is now put under src/test/ instead of the correct src/main/, so they are treated as to be used in unit test. Therefore it will not be packaged.

Identify the class of object, Scala

I'm new in a Scala and need help to read some piece of the code. I'm looking now for the next code: ALS.scala from Apache Spark and try to understand, how it works and which classes/objects are involved in this source code.
Currently I have difficulty with the line 166 in the code, because I can not understand the nature of the SchemaUtils object.
I copied already the source code to my local repository and got an recognition error for SchemaUtils object. In my pom.xml I defined the Spark version 1.6.1, but I suppose this object is not anymore available in this version (probably older one). Consequently is not recognized by Scala. The error message is:
not found: value SchemaUtils
How I can fix this bug?
Here is my pom.xml:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>test.spark</groupId>
<artifactId>als</artifactId>
<version>0.0.1-SNAPSHOT</version>
<pluginRepositories>
<pluginRepository>
<id>scala-tools.org</id>
<name>Scala-tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</pluginRepository>
</pluginRepositories>
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>1.6.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.11</artifactId>
<version>1.6.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>1.6.1</version>
</dependency>
<!---->
</dependencies>
<build>
<plugins>
<!-- mixed scala/java compile -->
<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<executions>
<execution>
<id>compile</id>
<goals>
<goal>compile</goal>
</goals>
<phase>compile</phase>
</execution>
<execution>
<id>test-compile</id>
<goals>
<goal>testCompile</goal>
</goals>
<phase>test-compile</phase>
</execution>
<execution>
<phase>process-resources</phase>
<goals>
<goal>compile</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.7</source>
<target>1.7</target>
</configuration>
</plugin>
<!-- for fatjar -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.4</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>assemble-all</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<mainClass>fully.qualified.MainClass</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
</plugins>
<pluginManagement>
<plugins>
<!--This plugin's configuration is used to store Eclipse m2e settings
only. It has no influence on the Maven build itself. -->
<plugin>
<groupId>org.eclipse.m2e</groupId>
<artifactId>lifecycle-mapping</artifactId>
<version>1.0.0</version>
<configuration>
<lifecycleMappingMetadata>
<pluginExecutions>
<pluginExecution>
<pluginExecutionFilter>
<groupId>org.scala-tools</groupId>
<artifactId>
maven-scala-plugin
</artifactId>
<versionRange>
[2.15.2,)
</versionRange>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</pluginExecutionFilter>
<action>
<execute></execute>
</action>
</pluginExecution>
</pluginExecutions>
</lifecycleMappingMetadata>
</configuration>
</plugin>
</plugins>
</pluginManagement>
</build>
</project>
From the Spark docs
For the Scala API, Spark 1.6.1 uses Scala 2.10. You will need to use a compatible Scala version (2.10.x).
You can't use 2.11 with Spark (yet), so change your <artifactId>spark-core_2.11</artifactId> and all related dependencies with the Scala version encoded to read _2.10 and try that.

Maven filtered file in jar is out of date

I have a strange problem with maven. I am using the filtering option to put some version information into a properties file. I then include that in the jar file, so that "Help/About" can tell me something useful. The problem I am having is that the version in the jar file is the previous version. So, if, for example I run a build at 0930, and another at 0940, then the version of the properties file in the jar generated at 0940 will have a build time of 0930. I am using the buildNumber plugin as well, but this issue is present whether or not I enable it.
What is even stranger is that when I run my build from within eclipse and run the program, the "old" file shows up on help/about, but then when I "Refresh" (F5) the eclipse project and re-run the program, I get the correct version. So could it be that maven is taking the eclipse version somehow? And why do you need to "refresh" in eclipse to get it to have the most up to date version.
Anyway, my pom.xml is
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<scm>
<url>scm:git:https://github.com/gregryork/DayOneViewer</url>
<developerConnection>scm:git:https://github.com/gregryork/DayOneViewer</developerConnection>
<tag>master</tag>
</scm>
<groupId>uk.co.gregreynolds</groupId>
<artifactId>dayone</artifactId>
<version>0.0.2-SNAPSHOT</version>
<packaging>jar</packaging>
<name>dayone</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<version.template.file>src/main/resources/uk/co/gregreynolds/dayone/Version.properties.template</version.template.file>
<version.file>src/main/resources/uk/co/gregreynolds/dayone/Version.properties</version.file>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>buildnumber-maven-plugin</artifactId>
<version>1.0</version>
<executions>
<execution>
<phase>generate-resources</phase>
<goals>
<goal>create</goal>
</goals>
</execution>
</executions>
<configuration>
<doCheck>false</doCheck>
<doUpdate>false</doUpdate>
<revisionOnScmFailure>true</revisionOnScmFailure>
<format>{0,date,yyyy-MM-dd_HH-mm}_{1}</format>
<items>
<item>timestamp</item>
<item>${user.name}</item>
</items>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.0</version>
<configuration>
<source>1.7</source>
<target>1.7</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>uk.co.gregreynolds.dayone.DayOneViewer</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>com.google.code.maven-replacer-plugin</groupId>
<artifactId>maven-replacer-plugin</artifactId>
<version>1.4.0</version>
<executions>
<execution>
<phase>process-resources</phase>
<goals>
<goal>replace</goal>
</goals>
</execution>
</executions>
<configuration>
<file>${version.template.file}</file>
<outputFile>${version.file}</outputFile>
<replacements>
<replacement>
<token>#buildnumber#</token>
<value>${buildNumber}</value>
</replacement>
<replacement>
<token>#buildtime#</token>
<value>${maven.build.timestamp}</value>
</replacement>
<replacement>
<token>#pomversion#</token>
<value>${project.version}</value>
</replacement>
</replacements>
</configuration>
</plugin>
</plugins>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-site-plugin</artifactId>
<version>3.3</version>
<dependencies>
<dependency><!-- add support for ssh/scp -->
<groupId>org.apache.maven.wagon</groupId>
<artifactId>wagon-ssh</artifactId>
<version>1.0</version>
</dependency>
</dependencies>
</plugin>
<!--This plugin's configuration is used to store Eclipse m2e settings
only. It has no influence on the Maven build itself. -->
<plugin>
<groupId>org.eclipse.m2e</groupId>
<artifactId>lifecycle-mapping</artifactId>
<version>1.0.0</version>
<configuration>
<lifecycleMappingMetadata>
<pluginExecutions>
<pluginExecution>
<pluginExecutionFilter>
<groupId>
com.google.code.maven-replacer-plugin
</groupId>
<artifactId>
maven-replacer-plugin
</artifactId>
<versionRange>
[1.4.0,)
</versionRange>
<goals>
<goal>replace</goal>
</goals>
</pluginExecutionFilter>
<action>
<ignore></ignore>
</action>
</pluginExecution>
</pluginExecutions>
</lifecycleMappingMetadata>
</configuration>
</plugin>
</plugins>
</pluginManagement>
</build>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.googlecode.plist</groupId>
<artifactId>dd-plist</artifactId>
<version>1.3</version>
</dependency>
<dependency>
<groupId>org.swinglabs</groupId>
<artifactId>swingx</artifactId>
<version>1.6.1</version>
</dependency>
</dependencies>
<distributionManagement>
<site>
<id>langurmonkey.no-ip.org</id>
<url>scp://langurmonkey.no-ip.org/var/www/dayone/</url>
</site>
</distributionManagement>
</project>
The first thing is to activate the filtering of your resources like this:
<build>
<resources>
<resource>
<directory>src/main/resources/</directory>
<filtering>true</filtering>
</resource>
</resources>
...
</build>
This will activate the filtering for all files which are in the folder (and subfolders) of src/main/resources. Sometimes you need the filtering only for a limited number of files. In such sutuations just put those files into a subfolder and change the above configuration appropriately.
Next is to define a property file (in src/main/resources folder) which contains all information you need like this:
version=${project.version}
buildNumber=${buildNumber}
buildTime=${maven.build.timestamp}
Furthermore do the configuration for the buildnumber-maven-plugin like this:
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>buildnumber-maven-plugin</artifactId>
<version>1.2</version>
<executions>
<execution>
<phase>validate</phase>
<goals>
<goal>create</goal>
</goals>
</execution>
</executions>
<configuration>
<doCheck>false</doCheck>
<doUpdate>false</doUpdate>
<revisionOnScmFailure>UNKNOWN</revisionOnScmFailure>
<format>{0,date,yyyy-MM-dd_HH-mm}_{1}</format>
<items>
<item>timestamp</item>
<item>${user.name}</item>
</items>
</configuration>
</plugin>
After that you can simple remove the usuage of the maven-replacer-plugin which you don't need in such simple scenarios.