I have a multi module scala application that I use to do ML training. I have a core module that contains some generic configuration which I would like to add in the other modules that depends on the core module.
All my configuration files are located in the resources folder and my project structure looks like this:
core
src
main
resources
application.conf
mod1
src
main
resources
application.conf
mod2
src
main
resources
application.conf
So I would like to have in my mod1 and mod2 module's application.conf file, this as the first line:
include core/application.conf
So that I can override some of the settings from there. How do I do this? For example., here is the build.sbt sample on how I define my module mod1:
// Define Sub Modules and its settings
lazy val mod1 = (project in file(MODULE_NAME_CLEANSE)).dependsOn(core% "compile->compile;test->test", config)
.settings(
commonSettings,
enablingCoverageSettings,
dockerSettings(),
name := MODULE_1,
description := "Clean the incoming data for training purposes"
)
.enablePlugins(JavaAppPackaging, DockerPlugin)
Assuming you are using Lighbend's config library.
Once built/deployed, your 2 modules will each have in their classpath the application.conf file from core module and the one from the said module. Both will be loaded but you cannot guarantee the order.
The recommended way is to name it reference.conf file in your core module instead of application.conf because that way you are sure that reference.conf is loaded with lower priority than applicayion.conf.
See https://github.com/lightbend/config#standard-behavior :
The convenience method ConfigFactory.load() loads the following (first-listed are higher priority):
system properties
application.conf (all resources on classpath with this name)
application.json (all resources on classpath with this name)
application.properties (all resources on classpath with this name)
reference.conf (all resources on classpath with this name)
The idea is that libraries and frameworks should ship with a reference.conf in their jar. Applications should provide an application.conf
Related
I have trouble understanding the mechanism of merging resource files in Typesafe Config.
According to the Typesafe config documentation:
The convenience method ConfigFactory.load() loads the following
(first-listed are higher priority):
system properties
application.conf (all resources on classpath with this name)
application.json (all resources on classpath with this name)
application.properties (all resources on classpath with this name)
reference.conf (all resources on classpath with this name)
Suppose I have two modules: a and b, and module a depends on module b.
lazy val a = (project in file("a"))
...
.dependsOn(b % "compile->compile;test->test")
If both modules have a resource file reference.conf, how they will get merged?
Is it appending? Is so, how does it resolve conflicts?
I believe they are overriding in reverse classpath order (could not find the docs to confirm this).
classpath: a.jar;b.jar;
->
b reference.conf will be loaded first and then overridden by a reference.conf
This makes it consistent with how .properties files are read, the first in the classpath will be used.
I think it is a good idea to be explicit about your overrides however, either programmatically or by using reference.conf and application.conf as intended.
Problem description: large sbt file
Our current build.sbt spanns 250+ lines.
We have two problems:
readability
current approach grouping of data and comments:
// Plugins ///////////////////////////////////////////////////
enablePlugins(DockerPlugin)
// basic configuration : projects ///////////////////////////
name := """projectName"""
lazy val projectName =
(project in file(".")).....
logic reuse
We have some configuration logic we would like to share between different projects.
Question
Is there a way to include other *.sbt files?
Or do you have a suggestion how to solve this problem without resorting to write a sbt plugin?
Is there a way to include other *.sbt files?
Yes, you can simply put parts of your build.sbt file into other *.sbt files within your project root. SBT picks up all *.sbt files and merges them together, as there was only a single large file.
One thing you can do is to factor out parts of your build info into scala files in the project directory.
E.g. in our build we have a file Dependencies.scala in the project directory, which contains all dependencies of the various projects in one place:
object Dependencies {
val akka_actor = "com.typesafe.akka" %% "akka-actor" % "2.3.13"
// ...
}
This can then be imported from the build.sbt:
import Dependencies._
lazy val foo = Project(...) dependsOn (akka_actor, ...)
You can also put tasks and commands into objects in the project directory.
Update: One thing I often do when looking for inspiration about how to organize a build is to look at the build of complex, high-profile scala projects such as akka. As you can see, they have moved a lot of logic into scala files in the project directory. The build itself is defined in AkkaBuild.scala.
I am using scala typesafe config (version 1.2.1) in one of my projects to read the application.conf file, my project has dependency on multiple other projects and I create a jar with dependencies to run the dependency projects.
Problem - those projects also use typesafe and has application.conf files in top level directory and my maven jar with dependencies pick up only one application.conf in same classpath and drops rest of them (I tried using maven shade plugin to merge these conf files and I don't want to go that route). I am trying to place these application.conf files in packages, so they don't get overriden in jar with dependencies, but typesafe doesn't seem to recognize the files under a package name in resources directory, it can only find the conf file if placed under top level resources directory.
Is there a better solution to achieve this by using typesafe ? or I am open to using a totally different config library for scala if there is any.
Given a file structure like this:
$tree src/main/resources/
src/main/resources
└── path
└── to
└── file
└── xxx.conf
You can load the config with the following command:
val config = ConfigFactory.load("path/to/file/xxx.conf")
I'm using the sbt-native-packager plugin to generate a start script for my application, which is very convenient as this plugin generates the correct classpath specification with all my library dependencies. I am not distributing this applictaion, therefore I'm not packaging the entire thing into one tarball. I just use the lib directory generated by sbt-native-packager that contains all the jar-files on which my project depends, both third-party libraries as well as the jar-file that contains my own class and resource files.
In my project's src/main/resources directory I have files that I want to be able to edit without having to use sbt-native-packager to regenerate the entire installation, for example configuration files. This is difficult because those files are zipped up in the jar file with all my classes.
Question: how can I tell sbt-native-packager not to put my resource files into a jar-file, while still generating the start-script with the correct classpath for those resource files to be located and read by my application as they are now from within the jar file? If this means leaving all my class files out of a jar file that is fine, as long as the files from src/main/resources remain as files that I can change without re-invoking sbt stage and as long as the start-script works.
While it is possible to filter these resources I would suggest to put them into a different directory and add them to the classpath.
Modifying the start script generated by sbt-native-packager is a bit cumbersome as the class com.typesafe.sbt.packager.archetypes.JavaAppBashScript that is generating the classpath is prefixing all paths with $lib_dir/. The cleanest approach would probably be to provide your own implementation and use that to generate the bashScriptDefines.
A simpler but hacky way would be to just add the following lines to your build.sbt:
packageArchetype.java_server
// add your config files to the classpath for running inside sbt
unmanagedClasspath in Compile += Attributed.blank(sourceDirectory.value/"main"/"config")
// map all files in src/main/config to config in the packaged app
mappings in Universal ++= {
val configDir = sourceDirectory.value/"main"/"config"
for {
file <- (configDir ** AllPassFilter).get
relative <- file.relativeTo(configDir.getParentFile)
mapping = file -> relative.getPath
} yield mapping
}
scriptClasspath ~= (cp => "../config" +: cp)
This will prepend $lib_dir/../config to your start script's classpath. If your app has to run on Windows you will have to provide similar settings for the batScriptDefines.
The project I'm working on at work is a webapp on the Lift Framework. We're using xsbt web plugin as well. There's a "core" project, which contains the vast majority of the functionality; my current goal is to create two "distribution" projects that add a different set of classpath resources to the "core" project. The problem is that I either 1) can't get the "distribution" projects to run, or 2) Get them to run, but the required resource doesn't seem to be there.
What I've tried
Here's an abridged version of my project/Build.scala:
lazy val core = Project("Core", file("core"))
.settings( /*some dependencies, resolvers, webSettings */ )
lazy val app1 = Project("App1", file("app1"))
.aggregate(core)
.settings( /*the same settings as core */ )
lazy val app2 = Project("App2", file("app2"))
.aggregate(core)
.settings( /*the same settings as core*/ )
Then in the directory structure for both app1 and app2, I have a file at src/main/resources/aFileINeed. The core application is using the class.getResource approach to load the file from the classpath.
Problems
If I try to run one of the distribution projects, using container:start, it does not detect the required file in the classpath. Also, it claims that src/main/webapp is not an existing directory (that folder is included in the core project, as it is required by the xsbt web plugin).
How can I get these projects to "merge" their resources? I expected that using aggregate or dependsOn in the Build.scala project definition would handle that for me, but it apparently doesn't.
This is what I'm doing. I create a global 'resources' directory in the root of the project, and then the following variable:
lazy val globalResources = file("resources")
And then in every project's settings:
unmanagedResourceDirectories in Runtime += globalResources
For projects that use the xsbt-web-plugin or some other library that imports it you'll have to use the stronger
unmanagedResourceDirectories in Compile += globalResources
Beware that using this solution your projects will potentially have muliple resource directories and AFAIK if you define the same file twice compile:copy-resources is going to be angry at you.