Build Jar with dependencies including multiple main classes using Gradle in IntelliJ - scala

Question

We are currently using Maven to build a Scala Application. Now we are trying to convert this project into Gradle. I have 3/4 main classes in this project and I want to build a jar with dependencies that include all the main classes and execute this jar with spark-submit by calling any of the classes.
I'm new to Gradle and facing issues with Gradle. Could some one help me.
Contents of pom.xml :
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org-groupId</groupId>
<artifactId>project-name</artifactId>
<version>3.0-SNAPSHOT</version>
<properties>
<scala.version>2.11.12</scala.version>
<scala.compat.version>2.11</scala.compat.version>
<scala.test.version>3.0.4</scala.test.version>
<project.type>application</project.type>
</properties>
<dependencies>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.8.2</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-all</artifactId>
<version>4.1.42.Final</version>
</dependency>
<dependency>
<groupId>org.scalatest</groupId>
<artifactId>scalatest_${scala.compat.version}</artifactId>
<version>3.0.4</version>
<scope>test</scope>
</dependency>
</dependencies>
<repositories>
<repository>
<id>maven-repo</id>
<name>Maven central repository</name>
<url>https://repo1.maven.org/maven2</url>
</repository>
</repositories>
<build>
<sourceDirectory>src/main/scala</sourceDirectory>
<testSourceDirectory>src/test/scala</testSourceDirectory>
<plugins>
<plugin>
<groupId>org.scalatest</groupId>
<artifactId>scalatest-maven-plugin</artifactId>
<version>1.0</version>
<configuration>
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
<junitxml>.</junitxml>
</configuration>
<executions>
<execution>
<id>test</id>
<goals>
<goal>test</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.0.0</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>copy-dependencies</goal>
</goals>
<configuration>
<outputDirectory>${project.build.directory}/lib</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>4.3.0</version>
<executions>
<execution>
<id>scala-compile-first</id>
<phase>process-resources</phase>
<goals>
<goal>add-source</goal>
<goal>compile</goal>
</goals>
</execution>
<execution>
<id>scala-test-compile</id>
<phase>process-test-resources</phase>
<goals>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.1</version>
<configuration>
<release>11</release>
</configuration>
<executions>
<execution>
<phase>compile</phase>
<goals>
<goal>compile</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>2.6</version>
<executions>
<execution>
<goals>
<goal>test-jar</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.scalastyle</groupId>
<artifactId>scalastyle-maven-plugin</artifactId>
<version>1.0.0</version>
<configuration>
<verbose>false</verbose>
<failOnViolation>true</failOnViolation>
<includeTestSourceDirectory>true</includeTestSourceDirectory>
<failOnWarning>false</failOnWarning>
<sourceDirectory>src/main/scala</sourceDirectory>
<testSourceDirectory>src/test/scala</testSourceDirectory>
<configLocation>${project.basedir}/src/main/resources/plugin/scalastyle_config.xml</configLocation>
<outputFile>${project.build.directory}/scalastyle-output.xml</outputFile>
<outputEncoding>UTF-8</outputEncoding>
</configuration>
<executions>
<execution>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
Contents of build.gradle:
apply plugin: 'java'
apply plugin: 'scala'
apply plugin: 'idea'
apply plugin: 'eclipse'
apply plugin: 'maven'
apply plugin: 'maven-publish'
repositories {
mavenLocal()
mavenCentral()
maven {
url = uri('https://repo1.maven.org/maven2')
}
}
dependencies {
// Some internal dependencies
implementation 'com.google.code.gson:gson:2.8.2'
implementation 'io.netty:netty-all:4.1.42.Final'
testImplementation "org.scalatest:scalatest_2.11:$scalaTestVersion"
}
tasks.withType(JavaCompile) {
options.encoding = 'UTF-8'
}
Contents of gradle.properties :
# Description
# Versions
version = 4.0
scalaVersion=2.11.12
scalaMajorVersion=2.11
scalaTestVersion=3.0.4
sparkVersion=2.4.6
sourceCompatibility = '1.8'
I have tried ./gradlew clean assemble and ./gradlew clean build.
Both of them building a regular jar but not dependencies jar.
Also, I'm trying to execute this in IntelliJ since Jar is not building, but with IntelliJ getting the below error.
21:36:59: Executing task 'mainclass1.main()'...
FAILURE: Build failed with an exception.
* Where:
Initialization script '/private/var/folders/sq/npjk1mkn7lgfm57mf9g_3rrh0000gn/T/mainclass1_main__.gradle' line: 4
* What went wrong:
A problem occurred configuring root project 'project-name'.
> Could not create task ':mainclass1.main()'.
> Unnecessarily replacing a task that does not exist is not supported. Use create() or register() directly instead. You attempted to replace a task named 'mainclass1.main()', but there is no existing task with that name.
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.4.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 567ms
Cause: invalid type code: FA
21:37:00: Task execution finished 'mainclass1.main()'.
While converting from Maven pom.xml to Gradle, I didn't included and build plugins in Gradle. What are the plugins I can use to build a the jar in Gradle.

Answer 1

It looks like you want a fat jar. This will contain all the dependencies that allow you to run the application. ​
Gradle does not have a straightforward out-of-the-box solution because its aim is to keep things simple.
You can do this with by setting the following on your build.gradle file:
jar {
​from {
​configurations.runtimeClasspath.collect { it.isDirectory() ? it : zipTree(it) }
​}
}
Alternatively, you can use this plugin, by setting the following on the build.gradle:
plugins {
id 'com.github.johnrengelman.shadow' version '5.2.0' // check version compatibiltiy
}
and the running the task gradlew shadowJar.
On another note, did I understand you have 3/4 main classes? I don't know SPARK, so maybe it allows for several main classes, but outside of SPARK, your manifest file must set which is the main class. There can be only one.
This will allows you to run java -jar your-jar.jar. The following SO question might be relevant: Multiple runnable classes inside JAR, how to run them?
To set your main class on the Manifest using gradle:
jar {
​manifest.attributes["Main-Class"] = "<your-main-class-full-path>"
}