I am using scala typesafe config (version 1.2.1) in one of my projects to read the application.conf file, my project has dependency on multiple other projects and I create a jar with dependencies to run the dependency projects.
Problem - those projects also use typesafe and has application.conf files in top level directory and my maven jar with dependencies pick up only one application.conf in same classpath and drops rest of them (I tried using maven shade plugin to merge these conf files and I don't want to go that route). I am trying to place these application.conf files in packages, so they don't get overriden in jar with dependencies, but typesafe doesn't seem to recognize the files under a package name in resources directory, it can only find the conf file if placed under top level resources directory.
Is there a better solution to achieve this by using typesafe ? or I am open to using a totally different config library for scala if there is any.
Given a file structure like this:
$tree src/main/resources/
src/main/resources
└── path
└── to
└── file
└── xxx.conf
You can load the config with the following command:
val config = ConfigFactory.load("path/to/file/xxx.conf")
Related
I have a multi module scala application that I use to do ML training. I have a core module that contains some generic configuration which I would like to add in the other modules that depends on the core module.
All my configuration files are located in the resources folder and my project structure looks like this:
core
src
main
resources
application.conf
mod1
src
main
resources
application.conf
mod2
src
main
resources
application.conf
So I would like to have in my mod1 and mod2 module's application.conf file, this as the first line:
include core/application.conf
So that I can override some of the settings from there. How do I do this? For example., here is the build.sbt sample on how I define my module mod1:
// Define Sub Modules and its settings
lazy val mod1 = (project in file(MODULE_NAME_CLEANSE)).dependsOn(core% "compile->compile;test->test", config)
.settings(
commonSettings,
enablingCoverageSettings,
dockerSettings(),
name := MODULE_1,
description := "Clean the incoming data for training purposes"
)
.enablePlugins(JavaAppPackaging, DockerPlugin)
Assuming you are using Lighbend's config library.
Once built/deployed, your 2 modules will each have in their classpath the application.conf file from core module and the one from the said module. Both will be loaded but you cannot guarantee the order.
The recommended way is to name it reference.conf file in your core module instead of application.conf because that way you are sure that reference.conf is loaded with lower priority than applicayion.conf.
See https://github.com/lightbend/config#standard-behavior :
The convenience method ConfigFactory.load() loads the following (first-listed are higher priority):
system properties
application.conf (all resources on classpath with this name)
application.json (all resources on classpath with this name)
application.properties (all resources on classpath with this name)
reference.conf (all resources on classpath with this name)
The idea is that libraries and frameworks should ship with a reference.conf in their jar. Applications should provide an application.conf
I created a new SBT project HelloScala, then I created a package called week6 and then a Scala worksheet under it. Eventually, I got the following directory structure:
~/HelloScala/.idea
~/HelloScala/project
~/HelloScala/src/main/scala
~/HelloScala/src/main/scala-2.12/week6/hello.sc
However, when I put package week6 in the worksheet hello.sc, IntelliJ warns that Project names doesn't correspond to directories structure. My questions are:
What should be the correct directory structure IntelliJ/SBT expects? Is it specified in some .xml file?
Notice that IntelliJ created two scala subdirectories, scala and scala-2.12. Is it correct? Or IntelliJ somehow failed to recognize the scala which had already existed but download and install another scala?
Thank you!
I don not use IntelliJ that often, but usually Scala packages should go into src/main/scala.
The structure of a Scala project is nothing too different from a Java one. Minimally you will need:
myproject
├── build.sbt
└── src
└── main
└── scala
└── mypackage
which is a project with a single package, a SBT script and nothing else.
For a more complex example, a Scala project could look like:
project
├── build.sbt
├── project
├── target
└── src
├── main
│ ├── java
│ ├── resource
│ └── scala
└── test
├── java
├── resource
└── scala
in which the project folder is used by SBT for various purposes, the target folder contains the compiled .class files and .jar packages.
To answer your questions:
The above structure is used implicitly by the SBT plugin. Maybe it is customizable, but in most cases it is suggested against doing this.
Your Scala source code should go into src/main/scala. I have no idea where does the other directory come from.
#sgu It seems I misunderstood your question and unfortunately I cannot leave a comment. The file "hello.sc" you are trying to deal with is treated by IntelliJ as a "Scala worksheet". I guess it is a REPL behind IntelliJ which evaluates as you edit. However it is not treated the same as a "Scala source file", so adding package xxx gives you the warning. If you want to create a package, the source files should be in ".scala" extension.
You should put all your source code under "main/scala" directory. "Scala-2.12" is the directory, that would be created with sbt under "project/target" after you compile/build it.
Have you tried clicking the Make project option on top of the scala sheet in Intellij Idea? I use that option when I need to import oher packages into one scala worksheet.
I am trying to use an LSH implementation of Scala(https://github.com/marufaytekin/lsh-spark) in my Spark project.I cloned the repository with some changes to the sbt file (added Organisation)
To use this implementation , I compiled it using sbt compile and moved the jar file to the "lib" folder of my project and updated the sbt configuration file of my project , which looks like this ,
Now when I try to compile my project using sbt compile , It fails to load the external jar file ,showing the error message "unresolved dependency: com.lendap.spark.lsh.LSH#lsh-scala_2.10;0.0.1-SNAPSHOT: not found".
Am i following the right steps for adding an external jar file ?
How do i solve the dependency issue
As an alternative, you can build the lsh-spark project and add the jar in your spark application.
To add the external jars, addJar option can be used while executing spark application. Refer Running spark application on yarn
This issue isn't related to spark but to sbt configuration.
Make sure you followed the correct folder structure imposed by sbt and added your jar in the lib folder, as explained here - lib folder should be at the same level as build.sbt (cf. this post).
You might also want to check out this SO post.
I'm using the sbt-native-packager plugin to generate a start script for my application, which is very convenient as this plugin generates the correct classpath specification with all my library dependencies. I am not distributing this applictaion, therefore I'm not packaging the entire thing into one tarball. I just use the lib directory generated by sbt-native-packager that contains all the jar-files on which my project depends, both third-party libraries as well as the jar-file that contains my own class and resource files.
In my project's src/main/resources directory I have files that I want to be able to edit without having to use sbt-native-packager to regenerate the entire installation, for example configuration files. This is difficult because those files are zipped up in the jar file with all my classes.
Question: how can I tell sbt-native-packager not to put my resource files into a jar-file, while still generating the start-script with the correct classpath for those resource files to be located and read by my application as they are now from within the jar file? If this means leaving all my class files out of a jar file that is fine, as long as the files from src/main/resources remain as files that I can change without re-invoking sbt stage and as long as the start-script works.
While it is possible to filter these resources I would suggest to put them into a different directory and add them to the classpath.
Modifying the start script generated by sbt-native-packager is a bit cumbersome as the class com.typesafe.sbt.packager.archetypes.JavaAppBashScript that is generating the classpath is prefixing all paths with $lib_dir/. The cleanest approach would probably be to provide your own implementation and use that to generate the bashScriptDefines.
A simpler but hacky way would be to just add the following lines to your build.sbt:
packageArchetype.java_server
// add your config files to the classpath for running inside sbt
unmanagedClasspath in Compile += Attributed.blank(sourceDirectory.value/"main"/"config")
// map all files in src/main/config to config in the packaged app
mappings in Universal ++= {
val configDir = sourceDirectory.value/"main"/"config"
for {
file <- (configDir ** AllPassFilter).get
relative <- file.relativeTo(configDir.getParentFile)
mapping = file -> relative.getPath
} yield mapping
}
scriptClasspath ~= (cp => "../config" +: cp)
This will prepend $lib_dir/../config to your start script's classpath. If your app has to run on Windows you will have to provide similar settings for the batScriptDefines.
I'm making webapp using scalatra framework via sbt & xsbt-web-plugin.
I want to package all resources(templates, css, js) into a single jar.
In sbt with sbt-assembly plugin, assembly command makes single jar which includes all of project's dependencies.
$ java -jar myproject.jar
and I open it in browser
Could not load resource: [/WEB-INF/views/index.scaml]; are you sure it's within [null]?
I unzipped jar to confirm that it does not include src/main/webapp/*.
How can I config sbt for including src/main/webapp/* and building executable jar?
Resources are meant to be put under the resources folders. There are two such folders:
src/main/resources for resources available at runtime
src/test/resources for resources available only during testing
sbt will package those automatically for you when you run package-war or test. The project does not need to have the assembly plugin for sbt to include resources.
In your case, you should put the WEB-INF directory in src/main/resources/WEB-INF/.