Cannot load resource files using Scala / Mill - scala

I cannot load resources in Scala, using Mill (0.10.5) as the build tool.
The minimal example is:
.
├── app
│   └── src
│   └── main
│   ├── resources
│   │   └── hello_world.txt
│   └── scala
│   └── Main.scala
├── build.sc
└── out
with build.sc:
import mill._, scalalib._, mill.modules.Jvm
object app extends ScalaModule {
def scalaVersion = "2.13.5"
}
and Main.scala:
import scala.io.Source
object Main {
def main(args: Array[String]): Unit = {
val helloWorldText : Iterator[String] = Source.fromResource("/hello_world.txt").getLines
helloWorldText.foreach(println)
}
}
While mill -i app.run compiles, the program throws an exception:
Exception in thread "main" java.io.FileNotFoundException: resource 'hello_world.txt' was not found in the classpath from the given classloader
What is the right approach to access resource files using Mill?

Mill ScalaModule does not use the Maven/SBT directory layout by default. Instead, it expects the following directory structure:
.
├── app
│ ├── src
│ | └── Main.scala
│ └── resources
| └── hello_world.txt
├── build.sc
└── out
This is also part of the Getting Started section in the documentation.
What you are looking for is a SBT-compatible project layout, which is discussed in the documentation under section SBT-compatible Modules. In short, you need to extend from SbtModule instead of ScalaModule.
Alternatively, you can also override the sources and resources targets, to customize to your needs.

Related

Scala share resources folder to open a file

This is my folder structure and I am trying to load the "grammar.txt" file from resources folder but I get not found error.
val source = Source.fromResource("grammar.txt")
Folder structure:
➜ cfg-tools tree -L 4
.
├── build.sbt
├── src
│   ├── main
│   │   └── scala
│   │   ├── Builer.scala
│   │   ├── Driver.scala
│   │   ├── tokens.scala
│   │   └── Tools.scala
│   ├── resources
│   │   └── grammar.txt
build.sbt
name := "cfg-tools"
version := "0.1"
scalaVersion := "3.0.2"
Compile / unmanagedResourceDirectories += sourceDirectory.value / "resources"
You don't need the custom SBT configuration: just use the standard place for resources which is src/main/resources (note that it's in main subfolder compared to your current structure).

Importing json resources inside .pex (Python Executable (format by Twitter))

I'm using a Twitter engineered build tool pants to manage many projects inside my monorepo. It outputs .pex files when I complete a build, this is a binary that packages the bare minimum dependencies I need for each project and makes them a "binary" (actually an archive that's decompressed at runtime), my issue is a utility that my code has used for a long time fails to detect some .json files(now that I'm using pants) I have stored under my environments library. all my other code seems to run fine. I'm pretty sure it has to do with my config, perhaps I'm not storing the resources properly so my code can find it, though when I use unzip my_app.pex the resources I desire are in the package and located in the proper location(dir). Here is the method my utility uses to load the json resources:
if test_env:
file_name = "test_env.json"
elif os.environ["ENVIRONMENT_TYPE"] == "PROD":
file_name = "prod_env.json"
else:
file_name = "dev_env.json"
try:
json_file = importlib.resources.read_text("my_apps.environments", file_name)
except FileNotFoundError:
logger.error(f"my_apps.environments->{file_name} was not found")
exit()
config = json.loads(json_file)
here is the the BUILD file I use for these resource currently:
python_library(
dependencies=[
":dev_env",
":prod_env",
":test_env"
]
)
resources(
name="dev_env",
sources=["dev_env.json"]
)
resources(
name="prod_env",
sources=["prod_env.json"]
)
resources(
name="test_env",
sources=["test_env.json"]
)
and here is the BUILD file for the utility that calls these resources of which the python code above is what you saw:
python_library(
name="environment_handler",
sources=["environment_handler.py"],
dependencies=[
"my_apps/environments:dev_env",
"my_apps/environments:prod_env",
"my_apps/environments:test_env"
]
)
I always get an FileNotFoundError exception and I'm confused because the files are available to the runtime, what's causing these files to not be accessible? and is there a different format I need to set up the JSON resources as?
Also for context here is the decompressed .pex file(actually just the source-code dir):
├── apps
│   ├── __init__.py
│   └── services
│   ├── charts
│   │   ├── crud
│   │   │   ├── __init__.py
│   │   │   └── patch.py
│   │   ├── __init__.py
│   │   └── main.py
│   └── __init__.py
├── environments
│   ├── dev_env.json
│   ├── prod_env.json
│   └── test_env.json
├── __init__.py
├── models
│   ├── charts
│   │   ├── base.py
│   │   └── __init__.py
│   └── __init__.py
└── utils
├── api_queries
│   ├── common
│   │   ├── connections.py
│   │   └── __init__.py
│   └── __init__.py
├── calculations
│   ├── common
│   │   ├── __init__.py
│   │   └── merged_user_management.py
│   └── __init__.py
├── environment_handler.py
├── __init__.py
├── json_response_toolset.py
└── security_toolset.py
I figured it out: I changed the way I access the files within the library and it works perfectly before and after the build to .pex format. I used:
import pkgutil
#json_file = importlib.resources.read_text("my_apps.environments", file_name)
json_file = pkgutil.get_data("my_apps.environments", file_name).decode("utf-8")

sbt 0.13 multi project build, import packages across projects

I have a play project that I'd like to build with a common library that handles things like clients, services, utilities, etc...
My dir structure is like so:
.
├── build.sbt
├── myapp
│   ├── app
│   │   ├── controllers
│   │   │   └── HomeController.scala
│   │   └── views
│   │   ├── index.scala.html
│   │   └── main.scala.html
│   ├── conf
│   │   ├── application.conf
│   │   ├── logback.xml
│   │   ├── messages
│   │   └── routes
│   ├── public
│   │   ├── images
│   │   │   └── favicon.png
│   │   ├── javascripts
│   │   │   └── main.js
│   │   └── stylesheets
│   │   └── main.css
│   └── test
│   └── controllers
│   └── HomeControllerSpec.scala
├── myapp-common
│   └── src
│   └── main
│   └── TesterObject.scala
└── project
├── build.properties
├── plugins.sbt
└── scaffold.sbt
My build.sbt is configured as follows:
lazy val commonSettings = Seq(
version := "1.0-SNAPSHOT",
scalaVersion := "2.12.3"
)
lazy val root = (project in file("."))
.settings(commonSettings)
.aggregate(myapp, `myapp-common`)
.dependsOn(myapp, `myapp-common`)
lazy val `myapp-common` = project
.settings(commonSettings)
lazy val myapp = project
.enablePlugins(PlayScala)
.settings(commonSettings)
.settings(libraryDependencies += guice)
.settings(libraryDependencies += "org.scalatestplus.play" %% "scalatestplus-play" % "3.1.2" % Test)
.dependsOn(`myapp-common`)
While I am able to successfully run the play project with sbt myapp/run, it fails when I try to import a package defined in myapp-common.
For instance, if I have myapp-common/src/main/TesterObject.scala:
package tester
Object TesterObject {
val testMe = 3
}
If I try to import tester._ and then use TesterObject.testMe in myapp/app/controllers/HomeController.scala, the project fails during compilation since it can't find the package.
Can anyone point me in the right direction here? The sbt guide on multi-projects is a bit tough for me to parse for this particular problem. It seems like my dependencies are appropriately set up.

cucumber-scala and Play Framework integration testing

I've managed to get cucumber-scala up and running on a Play/Scala. Now I want to run the entire play application so that I can use something like Selenium to test my application.
My Current attempts have lead me to
val app = new FakeApplication()
val port = 3333
lazy val browser: TestBrowser = TestBrowser.of(webDriverClass, Some("http://localhost:" + port))
lazy val server = TestServer(port, app)
Of course this FakeApplication() is not configured in any way... Am I approaching this incorrectly? This application is also multi-module and Ideally I would like to have the feature tests run per module (see output from tree below)
├── README.md
├── build.sbt
├── conf
│   ├── application.conf
│   └── routes
├── logs
│   └── application.log
├── modules
│   ├── module1
│   │   ├── app
│   │   ├── conf
│   │   ├── target
│   │   └── test
│   └── module2
│   ├── app
│   ├── conf
│   └── target
└── project
   ├── build.properties
   ├── plugins.sbt
   ├── project
   │   └── target
   └── target
   ├── config-classes
   ├── resolution-cache
   ├── scala-2.10
   └── streams
I am aware that Play has a selenium integration which can be used to drive my tests. However I have a business requirement for feature files, as they are used as a reporting mechanism. I am not absolutely tied to Cucumber so if anyone is aware of a way of driving browser based tests using Feature files that would also be acceptable to me?
Thanks,
Ben
Update:
I was running through IntelliJ, which causes a server to run with no routes or anything provided. I assume this is because it runs with a default blank application.
However When running through sbt test I get the following output:
Caused by: com.google.inject.ProvisionException: Unable to provision, see the following errors:
1) Error in custom provider, Configuration error: Configuration error[Router not found: admin.Routes]
while locating play.api.inject.guice.FakeRouterProvider
while locating play.api.routing.Router
for parameter 0 at play.api.http.JavaCompatibleHttpRequestHandler.<init>(HttpRequestHandler.scala:200)
while locating play.api.http.JavaCompatibleHttpRequestHandler
while locating play.api.http.HttpRequestHandler
for parameter 4 at play.api.DefaultApplication.<init>(Application.scala:221)
at play.api.DefaultApplication.class(Application.scala:221)
while locating play.api.DefaultApplication
while locating play.api.Application
If you're using the default configuration settings for IntegrationTest in sbt, and ScalaTestPlusPlay, here are the directories to use:
src/it/scala – scala IT code
src/it/java – java IT code
src/it/resources – resources, including configuration
Place your application.conf and routes files in the resources directory and FakeApplication will pick it up. You can also set the paths in your sbt script to some other place.
If all of your development/integration testing machines are Unix-like and you use git for VC, you can use a relative symlink1 to your real routes file.
For Windows or other VCS's, you'll likely have to copy the routes file and do double maintenance.
1 symlink using a relative path to the original, e.g., ../../../conf/routes.

SBT: Exclude resource subdirectory

I've been using Gradle for most of my Scala projects, but I want to evaluate the suitability of SBT as a replacement. One of the things I've done in Gradle is to exclude a certain resource directory from the final build (for example, using CoffeeScript to write JavaScript files that will be included as final resources).
In Gradle, I'd do this by:
sourceSets {
main {
resources {
exclude 'com/example/export/dev' // exclude development resources
}
}
}
And this would exclude the resource package com.example.export.dev package from the final build.
How would I do the same in SBT? I've tried
unmanagedResourceDirectories in Compile -= (resourceDirectory in Compile).value / "com/example/export/dev"
but that doesn't do a thing (I understand why, but that doesn't really help). And the documentation on the SBT web site only talks about excluding file patterns (at Classpaths, sources, and resources).
As a more descriptive image, say we have the following resource directory structure:
com
\---example
\---export
\---dev
\---something
In the final output, I want:
com
\---example
\---export
\---something
The way to think in SBT is a bit different and I know it can be hard at first.
In your example, you need to modify the task that generate the resource files (or the task that selects the folders to look for resource files).
Here is an example of how I can select only the resource files that start with character 'a'.
(unmanagedResources in Compile) := (unmanagedResources in Compile).value.filter(_.getName.startsWith("a"))
Similarly if you want to modify the entire directory of the resource files you can do that like this:
(unmanagedResourceDirectories in Compile) := (unmanagedResourceDirectories in Compile).value.filter(_.getName.startsWith("a"))
Obviously my filters here are just and example, you can have any complex pattern that Scala supports.
The nice thing about SBT is that it's interactive. So you can check the result of your task by simply typing these at the REPL of your project:
> show compile:unmanagedResources
> show compile: unmanagedResourceDirectories
To check all the dependencies to the task do this from the REPL:
> inspect tree compile:unmanagedResources
Assumption:
SBT knows where to find all resources using the standard maven build directory layout. The above solution assumes that all resources are under the /resources directory. You can then access them from your Scala code using getClass.getResource("/folderInsideResources/file.txt").
Here is a sample directory layout for a mixed Java/Scala project with resources:
.
├── main
│   ├── java
│   │   └── com
│   │   └── a
│   │   └── b
│   │   └── Hello.java
│   ├── resources
│   │   ├── a.tx
│   │   └── b.tx
│   └── scala
│   └── com
│   └── a
│   └── b
│   └── ScalaHello.scala
└── test
├── resources
└── scala
└── com
└── a
└── b
└── ScalaHello.scala
To access the resource file just use:
getClass.getResource("/a.txt")
getClass.getResource("/b.txt")
From https://github.com/sbt/sbt-jshint/issues/14:
excludeFilter in unmanagedResources := {
val public = ((resourceDirectory in Compile).value / "com" / "example" / "export" / "dev").getCanonicalPath
new SimpleFileFilter(_.getCanonicalPath startsWith public)
}