Build sbt project in uber jar instead of individual submodule jars - scala

Been recently trying to change the way my Scala project does the assembly so that only a single jar is generated out of it instead of one per module.
main-project
| - inner-module-one [inner_module_one]
| - inner-module-two [inner_module_two]
What I've currently done is the following for the main module (the one I want its uber jar containing other modules content).
project
.in(file("."))
.settings(
name := "main-project",
assemblyName,
settings
)
.aggregate(
inner_module_one,
inner_module_two
)
Having the other two modules declared as follows.
lazy val inner_module_one = project
.in(file("inner-module-one"))
.settings(
name := "inner-module-one",
assemblyName,
settings,
dependencies and such (...)
)
.dependsOn(
inner_module_two
)
The jar file generated for the main-project is really, really small (no more than 5mbs in size) and only contains Scala related stuff, no project classes or such. However, other modules jars are complete and contains everything they require.
I've already tried adding the following setting to the main-project.
aggregate in assembly := false
But still no luck so far. Jars for submodules aren't generated but the main-project jar still doesn't contain the contents of the submodules.
Any clue where the issue could be?
EDIT
Tried what #LuisMiguelMejíaSuárez suggested and seems to be wanting to build, however, some errors arise that were already solved within their respective modules. In a given module there are some conflicts which are solved due to some overrides, but now they are appearing again.
[error] deduplicate: different file contents found in the following:
Having the dependsOn instead of aggregate affects the way dependencies are added, overridden and such?

Let's divide your question into two sections, how to include the submodules into the main-project, and you want to know how to not package assembly the 2 submodules.
Let's start from the easier, the second. sbt-assembly, is a plugin for sbt. Just like any other plugin, you cann disable it using the disablePlugins command. Therefore, if you define:
lazy val inner_module_one = project.disablePlugins(AssemblyPlugin)
.in(file("inner-module-one"))
.settings(
name := "inner-module-one"
)
.dependsOn(
inner_module_two
)
then inner_module_one won't build a jar.
To solve the first question, we need to understand the difference between aggregate and dependsOn. For that I'll quote from Jacek Laskowski great explanation:
aggregate causes the tasks to be executed in the aggregating module and all aggregated one while dependsOn sets a CLASSPATH dependency so the libraries are visible to the aggregateing module (depending on the configuration that's compile aka default in the example).
Therefore, when you aggregate inner_module_one, and inner_module_two, you just cause then to assembly as well. In order to get those to be part of main-project all you need to do is declare:
project
.in(file("."))
.settings(
name := "main-project",
assemblyName,
settings
)
.dependsOn(
inner_module_one,
inner_module_two
)
In order to test it, I created the following structue:
main-project
|-inner_module_one
|-main
|-scala
|One.scala
|-inner_module_two
|-main
|-scala
|Two.scala
In order to check whether the class One was in a jar I used the command:
jar tfv inner-module-one/target/scala-2.12/inner-module-one-assembly-0.1.0-SNAPSHOT.jar | grep One
Then, when running sbt assembly with the original aggregate, the command above provided empty results on the output jar. And when running sbt assembly with dependsOn, the command above provided one result:
480 Fri Oct 09 01:48:14 IDT 2020 One.class
If this causes conflicts, you can read about it at sbt-assembly: deduplication found error

Related

The meaning of _root_ in sbt

I have two separate Scala projects. In both projects, the build.sbt has the line:
lazy val V = _root_.scalafix.sbt.BuildInfo
The two projects contain several projects so both build.sbt files are multi-project builds. In my research, I came across the idea of a root project and I think _root_ in the above value denotes the root project.
My problem is that in one of the projects, sbt cannot resolve the symbol scalafix in lazy val V = _root_.scalafix.sbt.BuildInfo. When I search scalafix.sbt.BuildInfo in this project, I can find the related jar but sbt somehow misses it. The two projects in so far as I can tell are almost identical but one resolves scalafix while the other does not.
For both projects, the jar seems to be located at the below directory in my Mac:
/Users/soft/.ivy2/cache/scala_2.12/sbt_1.0/ch.epfl.scala/sbt-scalafix/jars/sbt-scalafix-0.6.0-M19.jar!/scalafix/sbt
What is _root_ and how can I help sbt resolve scalafix?
I think root in the above value denotes the root project.
No, this is not true.
All sbt files are scala code and _root_ has the same meaning as _root_ in scala. Thus the answer is the same as in What is the root package in Scala.
The two projects in so far as I can tell are almost identical but one resolves scalafix while the other does not.
The problem is most probably the missing sclafix plugin.
Check that the plugin is enabled in project/plugins.sbt with line
addSbtPlugin("ch.epfl.scala" % "sbt-scalafix" % "0.9.1")

How to use SBT's externalPom() command

I have a Maven POM file that the deployment engineers need to deploy the system in the enterprise. I have developers using SBT for a Scala project. They use SBT targets that just aren't supported in Maven. We'd like to use the Maven POM file to define the dependencies, slurp in those dependencies in SBT, and define SBT development targets there.
According to the SBT documentation, the externalPom() command is the way to do that. But even with the simplest POM file (two developers have tried this with two different simple POM files that defined different dependencies), the externalPom() command seems to half work. The SBT targets clearly recognize the dependency defined in the POM, but can't resolve it. This error arises:
Cannot add dependency 'commons-collections#commons-collections;3.2.2'
to configuration 'default' of module
default#maven-sbt$sources_javadoc;0.1-SNAPSHOT because this
configuration doesn't exist!
When the externalPom() command is commented out and the equivalent dependency added directly in the build.sbt file everything goes swimmingly. The dependency comes directly from Maven Central in both cases; one from copying the dependency from the Maven tab and one from copying the dependency from the SBT tab. Once again, two developers are seeing exactly the same thing, from two different dependencies. The thing that's common is that both developers have reduced the build.sbt file down to a single statement. In the "slurp from POM" case, that statement is externalPom(). In the "plain old SBT" case, that statement is the dependency copied from Maven Central. The POM file is a dependency list with a single dependency (as simple as we can make it and still test externalPom()).
We suspect that we need something else in the build.sbt to make the externalPom() command work but we don't know what it is. Any help with that would be greatly appreciated.
I did some experimentation with this, and was able to replicate your error in my experiments.
I'm still a bit of a Scala / SBT newbie, but I created a build.sbt file that looks like so:
val Default = config("default")
lazy val root = (project in file(".")).
configs(Default).
settings(
externalPom()
)
This did compile for me!
One non-obvious catch: I had to make sure to include the scala-library in my POM file as a dependency

sbt : Is there a better way of structuring large build.sbt files

Problem description: large sbt file
Our current build.sbt spanns 250+ lines.
We have two problems:
readability
current approach grouping of data and comments:
// Plugins ///////////////////////////////////////////////////
enablePlugins(DockerPlugin)
// basic configuration : projects ///////////////////////////
name := """projectName"""
lazy val projectName =
(project in file(".")).....
logic reuse
We have some configuration logic we would like to share between different projects.
Question
Is there a way to include other *.sbt files?
Or do you have a suggestion how to solve this problem without resorting to write a sbt plugin?
Is there a way to include other *.sbt files?
Yes, you can simply put parts of your build.sbt file into other *.sbt files within your project root. SBT picks up all *.sbt files and merges them together, as there was only a single large file.
One thing you can do is to factor out parts of your build info into scala files in the project directory.
E.g. in our build we have a file Dependencies.scala in the project directory, which contains all dependencies of the various projects in one place:
object Dependencies {
val akka_actor = "com.typesafe.akka" %% "akka-actor" % "2.3.13"
// ...
}
This can then be imported from the build.sbt:
import Dependencies._
lazy val foo = Project(...) dependsOn (akka_actor, ...)
You can also put tasks and commands into objects in the project directory.
Update: One thing I often do when looking for inspiration about how to organize a build is to look at the build of complex, high-profile scala projects such as akka. As you can see, they have moved a lot of logic into scala files in the project directory. The build itself is defined in AkkaBuild.scala.

What is the issue with this sbt file?

When I import a SBT project into intelliJ, the build.sbt file is showing lot of errors as shown in the following screenshot. Wondering what might be the issue
IDEA Version 13.1.4
I also see the following
The following source roots are outside of the corresponding base directories:
C:\Users\p0c\Downloads\recfun\src\main\resources
C:\Users\p0c\Downloads\recfun\src\test\java
C:\Users\p0c\Downloads\recfun\src\test\scala
C:\Users\p0c\Downloads\recfun\src\test\resources
These source roots cannot be included in the IDEA project model. Please consider using shared SBT projects instead of shared source roots.
I think the question perhaps does not provide all the required information to answer conclusively, but I'll give it a spin anyways -
Since sbt runs correctly when invoked from the shell, we know the sbt file is fine. I use Idea for my Scala and sbt projects and I am sure the Idea sbt support works very well, but! Idea is far more restrictive than sbt when it comes to project structure. It is really easy to create a valid sbt project structure that Idea can't support very well.
Given that the source roots error indicates that the recfun/src folder is not in the project folder, it seems clear that this error is not emitted during the processing of recfun/build.sbt. The screenshot shows you have at least three different sbt files, progfun-recfun, submission and scala-recfun. Since Idea will also create projects like submission-build, and you have an assignment-build project there, something is probably broken in the project structure, not from the sbt viewpoint - there you're fine, you can build - but from the Idea viewpoint, which is more restrictive.
My suggestion to resolve this would be to change your project structure as follows. First, have a top level project with a build.sbt. Then create a sub-project for each src folder you want. Do not put a src folder in your top level project. Each sub-project needs a build.sbt as well.
Second, make sure the sub-projects build correctly with sbt when run from the shell. Arrange the sub-project build.sbt files with the proper dependencies, using this syntax:
lazy val a = ProjectRef(file("../a"), "a")
lazy val b = ProjectRef(file("../b"), "b")
lazy val root = Project(id = "c", base = file(".")) dependsOn (a, b)
(This example has three sister projects a, b and c, where c depends on a and b. The three projects are placed in directories with the same name, within the root directory. The code snippet is from the build file for c.)
Third, arrange your top level build.sbt to aggregate the sub-projects, using this syntax in the top level build.sbt:
lazy val a = ProjectRef(file("a"), "a")
lazy val b = ProjectRef(file("b"), "b")
lazy val c = ProjectRef(file("c"), "c")
lazy val root = (project in file(".")).
aggregate(a, b, c)
Building this top level project will build each of the sub-projects a, b and c, and the dependencies established in the sub-project build files will ensure they are built in the right order.
Fourth, import the top level project into Idea, and all should be well.
You can get fancy with the file structure if you want, because the project references use relative paths, but it's usually nice to at least start simple.
I had a lot of frustration with sbt and Idea at the start, I hope this helps :)

how do I get sbt to use a local maven proxy repository (Nexus)?

I've got an sbt (Scala) project that currently pulls artifacts from the web. We'd like to move towards a corporate-standardized Nexus repository that would cache artifacts. From the Nexus documentation, I understand how to do that for Maven projects. But sbt obviously uses a different approach. (I understand Ivy is involved somehow, but I've never used it and don't understand how it works.)
How do I tell sbt and/or the underlying Ivy to use the corporate Nexus repository system for all dependencies? I'd like the answer to use some sort of project-level configuration file, so that new clones of our source repository will automatically use the proxy. (I.e., mucking about with per-user config files in a dot-directory is not viable.)
Thanks!
Step 1: Follow the instructions at Detailed Topics: Proxy Repositories, which I have summarised and added to below:
(If you are using Artifactory, you can skip this step.) Create an entirely separate Maven proxy repository (or group) on your corporate Maven repository, to proxy ivy-style repositories such as these two important ones:
http://repo.typesafe.com/typesafe/ivy-releases/
http://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/
This is needed because some repository managers cannot handle Ivy-style and Maven-style repositories being mixed together.
Create a file repositories, listing both your main corporate repository and any extra one that you created in step 1, in the format shown below:
[repositories]
my-maven-proxy-releases: http://repo.example.com/maven-releases/
my-ivy-proxy-releases: http://repo.example.com/ivy-releases/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
Either save that file in the .sbt directory inside your home directory, or specify it on the sbt command line:
sbt -Dsbt.repository.config=<path-to-your-repo-file>
Good news for those using older versions of sbt: Even though, in the sbt 0.12.0 launcher jar at least, the boot properties files for older sbt versions don't contain the required line (the one that mentions repository.config), it will still work for those versions of sbt if you edit those files to add the required line, and repackage them into the sbt 0.12.0 launcher jar! This is because the feature is implemented in the launcher, not in sbt itself. And the sbt 0.12.0 launcher is claimed to be able to launch all versions of sbt, right back to 0.7!
Step 2: To make sure external repositories are not being used, remove the default repositories from your resolvers. This can be done in one of two ways:
Add the command line option -Dsbt.override.build.repos=true mentioned on the Detailed Topics page above. This will cause the repositories you specified in the file to override any repositories specified in any of your sbt files. This might only work in sbt 0.12 and above, though - I haven't tried it yet.
Use fullResolvers := Seq( resolver(s) for your corporate maven repositories ) in your build files, instead of resolvers ++= or resolvers := or whatever you used to use.
OK, with some help from Mark Harrah on the sbt mailing list, I have an answer that works.
My build class now looks like the following (plus some other repos):
import sbt._
//By extending DefaultWebProject, we get Jetty support
class OurApplication(info: ProjectInfo) extends DefaultWebProject(info) {
// This skips adding the default repositories and only uses the ones you added
// explicitly. --Mark Harrah
override def repositories = Set("OurNexus" at "http://our.nexus.server:9001/nexus/content/groups/public/")
override def ivyRepositories = Seq(Resolver.defaultLocal(None)) ++ repositories
/* Squeryl */
val squeryl = "org.squeryl" % "squeryl_2.8.0.RC3" % "0.9.4beta5"
/* DATE4J */
val date4j = "hirondelle.date4j" % "date4j" % "1.0" from "http://www.date4j.net/date4j.jar"
// etc
}
Now, if I delete the Squeryl tree from my machine's .ivy2/cache directory, sbt tries to grab it from the Nexus tree with the appropriate URL. Problem solved!
All you need is to define a property file sbt.boot.properties which will allow you to:
redefine the ivy cache location (I need that because it would be otherwise part of our roaming Windows profile, which is severely limited in disk space in our shop. See Issue 74)
define any other Maven repo you want
C:\HOMEWARE\apps\sbt-0.74\sbt.boot.properties
[scala]
version: 2.7.7
# classifiers: sources, javadoc
[app]
org: org.scala-tools.sbt
name: sbt
version: read(sbt.version)
class: sbt.xMain
components: xsbti
cross-versioned: true
classifiers: sources, javadoc
[repositories]
local
my-nexus: http://my.nexus/nexus/content/repositories/scala-tools/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]
maven-local
# sbt-db: http://databinder.net/repo/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]
# maven-central
# scala-tools-releases
# scala-tools-snapshots
[boot]
directory: project/boot
properties: project/build.properties
prompt-create: Project does not exist, create new project?
prompt-fill: true
quick-option: true
[log]
level: debug
[app-properties]
project.name: quick=set(test), new=prompt(Name)[p], fill=prompt(Name)
project.organization: new=prompt(Organization)[org.vonc]
project.version: quick=set(1.0), new=prompt(Version)[1.0], fill=prompt(Version)[1.0]
build.scala.versions: quick=set(2.8.0.RC2), new=prompt(Scala version)[2.8.0.RC2], fill=prompt(Scala version)[2.8.0.RC2]
sbt.version: quick=set(0.7.4), new=prompt(sbt version)[0.7.4], fill=prompt(sbt version)[0.7.4]
project.scratch: quick=set(true)
project.initialize: quick=set(true), new=set(true)
[ivy]
cache-directory: C:\HOMEWARE\projects\.ivy2\cache
Note: this sbt.boot.properties file is inspired from:
the one mentioned in the "Generalized Launcher" page of the sbt project.
the one found within sbt-0.74 itself!
I have commented any external Maven repository definition, and added a reference to my own Nexus Maven repo.
The launcher may be configured in one of the following ways in increasing order of precedence:
Replace the /sbt/sbt.boot.properties file in the jar.
Put a configuration file named sbt.boot.properties on the classpath. Put it in the classpath root without the /sbt prefix.
Specify the location of an alternate configuration on the command line. This can be done by:
either specifying the location as the system property sbt.boot.properties
or as the first argument to the launcher prefixed by '#'.
The system property has lower precedence.
Resolution of a relative path is:
first attempted against the current working directory,
then against the user's home directory,
and then against the directory containing the launcher jar.
An error is generated if none of these attempts succeed.
Define a sbt.bat wrapper (in order to be sure to specify your sbt.boot.properties) like:
C:\HOMEWARE>more C:\HOMEWARE\bin\sbt.BAT
#echo off
set t=%~dp0
set adp0=%t:C:\="%"
set SBT_DIR=%adp0%..\apps\sbt-0.74
dir C:\%SBT_DIR%\sbt-launch-0.7.4.jar
# if needed, add your proxy settings
set PROXY_OPTIONS=-Dhttp.proxyHost=my.proxy -Dhttp.proxyPort=80xx -Dhttp.proxyUser=auser -Dhttp.proxyPassword=yyyy
set JAVA_OPTIONS=-XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256m -Xmx512M -cp C:\HOMEWARE\apps\sbt-0.74\sbt-launch-0.7.4
set SBT_BOOT_PROPERTIES=-Dsbt.boot.properties="sbt.boot.properties"
cmd /C C:\HOMEWARE\apps\jdk4eclipse\bin\java.exe %PROXY_OPTIONS% %JAVA_OPTIONS% %SBT_BOOT_PROPERTIES% -jar C:\HOMEWARE\apps\sbt-0.74\sbt-launch-0.7.4.jar %*
And your sbt will download artifacts only from:
your Nexus
your local Maven repo.
Just tested at home with an old Nexus opensource 1.6 I had running, java 1.6, sbt07.4
C:\Prog\Java\jdk1.6.0_18\jre\bin\java -Xmx512M -Dsbt.boot.properties=sbt.boot.properties - jar "c:\Prog\Scala\sbt\sbt-launch-0.7.4.jar"
That gives:
[success] Build completed successfully.
C:\Prog\Scala\tests\pp>sbt
Getting Scala 2.8.0 ...
downloading http://localhost:8081/nexus/content/repositories/scala/org/scala-lang/scala-compiler/2.8.0/scala-compiler-2.
8.0.jar ...
[SUCCESSFUL ] org.scala-lang#scala-compiler;2.8.0!scala-compiler.jar (311ms)
downloading http://localhost:8081/nexus/content/repositories/scala/org/scala-lang/scala-library/2.8.0/scala-library-2.8.
0.jar ...
[SUCCESSFUL ] org.scala-lang#scala-library;2.8.0!scala-library.jar (185ms)
:: retrieving :: org.scala-tools.sbt#boot-scala
confs: [default]
2 artifacts copied, 0 already retrieved (14484kB/167ms)
[info] Building project test 0.1 against Scala 2.8.0
[info] using sbt.DefaultProject with sbt 0.7.4 and Scala 2.7.7
If I try a funny value in the sbt.boot.properties file:
C:\Prog\Scala\tests\pp>sbt
Getting Scala 2.9.7 ...
:: problems summary ::
:::: WARNINGS
module not found: org.scala-lang#scala-compiler;2.9.7
==== nexus: tried
http://localhost:8081/nexus/content/repositories/scala/org/scala-lang/scala-compiler/2.9.7/scala-compiler-2.9.7.pom
-- artifact org.scala-lang#scala-compiler;2.9.7!scala-compiler.jar:
http://localhost:8081/nexus/content/repositories/scala/org/scala-lang/scala-compiler/2.9.7/scala-compiler-2.9.7.jar
So it does limit itself to the two repo I defined:
[repositories]
nexus: http://localhost:8081/nexus/content/repositories/scala
nexus2: http://localhost:8081/nexus/content/repositories/scala, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]
(I commented everything else: local, maven-local, ...)
If I comment all repositories and put a funny value (2.7.9) for the scala version in the sbt.boot.properties, I do get (like the OP did)
C:\Prog\Scala\tests\pp>sbt
Error during sbt execution: No repositories defined.
If I put 2.7.7 (while still having all repo commented), yes, it won't generate an error:
C:\Prog\Scala\tests\pp>sbt
[info] Building project test 0.1 against Scala 2.8.0
[info] using sbt.DefaultProject with sbt 0.7.4 and Scala 2.7.7
But that's only because it already had downloaded scala2.8.0 during my previous tries.
If I remove that library from my project/boot directory, then it will throw an Exception:
[info] using sbt.DefaultProject with sbt 0.7.4 and Scala 2.7.7
> C:\Prog\Scala\tests\pp>sbt
Error during sbt execution: No repositories defined.
at xsbt.boot.Pre$.error(Pre.scala:18)
at xsbt.boot.Update.addResolvers(Update.scala:197)
...
at xsbt.boot.Boot$.main(Boot.scala:15)
at xsbt.boot.Boot.main(Boot.scala)
Error loading project: Error during sbt execution: No repositories defined.
edit the config file in sbt_home/conf "sbtconfig.txt"
add two line
-Dsbt.override.build.repos=true
-Dsbt.repository.config="C:/Program Files (x86)/sbt/conf/repo.properties"
the repo.properties content is
[repositories]
local
public: http://222.vvfox.com/public <-fix this ,write your local nexus group url
Well this has bugged me for a while so I found a guy that has written an SBT plugin for maven out on github called maven-sbt so all you have to do is include it in your plugins project and make your project mixin with maven.MavenDependencies and all your operations like update and publish-local work with your local maven. The nice thing about that is if you are like me, your org is all maven. So, all you libs are in you local maven repo but if for some reason you build with sbt first, then you start getting a bunch or jars in ivy too. What a waste of space, and time since you will still need to get them for your maven builds.
That said, I wish this were built into sbt so I would not need to add it to every project. Maybe as a processor at least. He mentioned in one thing I read that he would like to add it to 0.9 but I have not been able to find it.
I got this error because I had a blank file in ~/.sbt/repositories. Both adding repositories to the file and removing the file solved the problem.