How do I use spark-testing-base with Maven? - scala

Question

I would like to find out how to test same spark code. Googling around I have found the spark-tetsting-base. Well, now I would like to try it out but I am not able to run it with Maven.
First, I have scala code which is packable with mvn package and has the following project structure:
pom.xml
src/main/scala/com/test/spark/mycode.scala
src/test/scala/com/test/spark/test.scala
The problem is when I run mvn test it does not run the test under src/main/com/test/spark/test.scala. The test is actually the first example from the spark-testing-base wiki.
The output of maven is:
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) # mycode ---
[INFO] Nothing to compile - all classes are up to date
And my pom.xml looks like this:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.test.spark</groupId>
<artifactId>mycode</artifactId>
<version>0.0.1</version>
<name>${project.artifactId}</name>
<description>Simple wtest</description>
<inceptionYear>2017</inceptionYear>
<!-- change from 1.6 to 1.7 depending on Java version -->
<properties>
<maven.compiler.source>1.6</maven.compiler.source>
<maven.compiler.target>1.6</maven.compiler.target>
<encoding>UTF-8</encoding>
<scala.version>2.11.5</scala.version>
<scala.compat.version>2.11</scala.compat.version>
<spark.version>1.6.1</spark.version>
</properties>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>
<!-- Spark dependency -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.compat.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<!-- Spark sql dependency -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.compat.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<!-- Spark hive dependency -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_${scala.compat.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<!-- spark-testing-base dependency -->
<dependency>
<groupId>com.holdenkarau</groupId>
<artifactId>spark-testing-base_2.11</artifactId>
<version>${spark.version}_0.6.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.scalactic</groupId>
<artifactId>scalactic_2.11</artifactId>
<version>3.0.1</version>
</dependency>
<dependency>
<groupId>org.scalatest</groupId>
<artifactId>scalatest_2.11</artifactId>
<version>3.2.0-SNAP5</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<sourceDirectory>src/main/scala</sourceDirectory>
<testSourceDirectory>src/test/scala</testSourceDirectory>
<!-- Create JAR with all dependencies -->
<plugins>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.0.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
</plugin>
<plugin>
<!-- see http://davidb.github.com/scala-maven-plugin -->
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.2</version>
<configuration>
<scalaVersion>${scala.version}</scalaVersion>
<scalaCompatVersion>${scala.compat.version}</scalaCompatVersion>
</configuration>
<executions>
<execution>
<phase>compile</phase>
<goals>
<goal>compile</goal>
</goals>
</execution>
</executions>
</plugin>
<!-- for testing scala code -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.20</version>
<configuration>
<argLine>-Xmx2048m -XX:MaxPermSize=2048m</argLine>
</configuration>
</plugin>
</plugins>
</build>
</project>
Any help appreciated. Thanks!
Edit 1:
I added this as test.scala in order to fail the test:
class test extends FunSuite with SharedSparkContext {
test("test initializing spark context") {
val list = List(1, 2, 3, 4)
val rdd = sc.parallelize(list)
val list2 = List(1, 2, 3, 4, 5)
assert(rdd.count === list2.length)
}
}
But I still get:
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) # mycode ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-surefire-plugin:2.20:test (default-test) # mycode ---
[INFO]------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]------------------------------------------------------------------------
although the comparison in assert is false as if the test doesn't run. :(
Edit 2:
I changed the pom to with this plugin, still no change:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.20</version>
<configuration>
<includes>
<include>**/*Test.scala</include>
</includes>
<argLine>-Xmx2048m -XX:MaxPermSize=2048m</argLine>
</configuration>
</plugin>
Edit 3:
Added <goal>testCompile</goal> to net.alchim31.maven in order to compile the test class. Altough the test still doesn't run. What is missing to finally get the test to run??
Edit 4:
Thanks for your comments. I changed the pom to the following:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.test.spark</groupId>
<artifactId>testJavaAndScala</artifactId>
<version>1.0-SNAPSHOT</version>
<name>Test for Java + Scala compilation</name>
<description>Test for Java + Scala compilation</description>
<inceptionYear>2017</inceptionYear>
<properties>
<scala.version>2.11.5</scala.version>
<scala.compat.version>2.11</scala.compat.version>
<spark.version>1.6.1</spark.version>
</properties>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>
<!-- Spark dependency -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.compat.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<!-- Spark sql dependency -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.compat.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<!-- Spark hive dependency -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_${scala.compat.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<!-- Spark-testing-base -->
<dependency>
<groupId>com.holdenkarau</groupId>
<artifactId>spark-testing-base_${scala.compat.version}</artifactId>
<version>${spark.version}_0.6.0</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.1</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.0.2</version>
</plugin>
</plugins>
</pluginManagement>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<executions>
<execution>
<id>scala-compile-first</id>
<phase>process-resources</phase>
<goals>
<goal>add-source</goal>
<goal>compile</goal>
</goals>
</execution>
<execution>
<id>scala-test-compile</id>
<phase>process-test-resources</phase>
<goals>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<executions>
<execution>
<phase>compile</phase>
<goals>
<goal>compile</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
And this is the mvn testoutput:
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Test for Java + Scala compilation 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) # testJavaAndScala ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /home/src/main/resources
[INFO]
[INFO] --- scala-maven-plugin:3.2.1:add-source (scala-compile-first) # testJavaAndScala ---
[INFO] Add Source directory: /home/src/main/scala
[INFO] Add Test Source directory: /home/src/test/scala
[INFO]
[INFO] --- scala-maven-plugin:3.2.1:compile (scala-compile-first) # testJavaAndScala ---
[INFO] /home/src/main/scala:-1: info: compiling
[INFO] Compiling 1 source files to /home/target/classes at 1497938635103
[INFO] prepare-compile in 0 s
[INFO] compile in 3 s
[INFO]
[INFO] --- maven-compiler-plugin:2.0.2:compile (default-compile) # testJavaAndScala ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-compiler-plugin:2.0.2:compile (default) # testJavaAndScala ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) # testJavaAndScala ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 0 resource
[INFO]
[INFO] --- scala-maven-plugin:3.2.1:testCompile (scala-test-compile) # testJavaAndScala ---
[INFO] /home/src/test/scala:-1: info: compiling
[INFO] Compiling 1 source files to /home/target/test-classes at 1497938639022
[INFO] prepare-compile in 0 s
[INFO] compile in 5 s
[INFO]
[INFO] --- maven-compiler-plugin:2.0.2:testCompile (default-testCompile) # testJavaAndScala ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) # testJavaAndScala ---
[INFO] Surefire report directory: /home/target/surefire-reports
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Results :
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 14.854 s
[INFO] Finished at: 2017-06-20T08:04:05+02:00
[INFO] Final Memory: 28M/360M
[INFO] ------------------------------------------------------------------------
I can find my test class in the target directory but it seems it is not executed. Why is that? As I said it is just the minimal exmaple and it does not rely on my actual main code.

Answer 1

Surefire wont find those scalatest tests on it's own. You need to use the scalatest-maven-plugin. Instructions are available on their website
To get started, modify your pom like this.
<!-- disable surefire -- >
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.7</version>
<configuration>
<skipTests>true</skipTests>
</configuration>
</plugin>
<!-- enable scalatest -- >
<plugin>
<groupId>org.scalatest</groupId>
<artifactId>scalatest-maven-plugin</artifactId>
<version>1.0</version>
<configuration>
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
<junitxml>.</junitxml>
<filereports>WDF TestSuite.txt</filereports>
</configuration>
<executions>
<execution>
<id>test</id>
<goals>
<goal>test</goal>
</goals>
</execution>
</executions>
</plugin>

Answer 2

The test cannot be run due to the folder structure.
Your folder structure
src/main/scala/com/test/spark/mycode.scala
src/main/test/com/test/spark/test.scala
Folder structure should be like
src/main/scala/com/test/spark/mycode.scala
src/test/scala/com/test/spark/test.scala
Here is the standard maven layout.
Hope this helps!
Edited
you are missing test execution plugin in your pom file please see scala maven plugin
please add the following in your net.alchim31.maven plugin part
<execution>
<id>scala-test-compile</id>
<phase>process-test-resources</phase>
<goals>
<goal>testCompile</goal>
</goals>
</execution>

Answer 3

Even though this is a very late reply, but I wrote it hoping that someone might get benefitted from it in case their scala unit test are not getting discovered. One of the major and silent culprit is the absolute path to the test classes. If there is a space in the name of any directory on the path, scalatest won't pick it up. Rename such directories for successful running of unit tests. Also in your case, you might want to add the sacalatest maven plugin after disabling the surefire plugin.