When attempting to execute a query via Slick (v3.0.3), I am receiving a com.typesafe.config.ConfigException$Missing exception (wrapped in a ExceptionInInitializerError), exclaiming:
No configuration setting found for key 'slick'
Apparently Slick requires a config value for slick.dumpPaths to be present when debug logging is enabled. Ordinarily, a default value will be provided by the reference.conf file that comes stock in Slick's jar-file, but for some reason that file (or that particular key) is not getting picked up, in this case.
In addition, adding an application.conf (which includes the requested config value, slick.dumpPaths) to my application's resource directory (src/main/resources/, by default) and/or to the test resource directory does not help the problem -- the exception still occurs.
It turns out this was (apparently) happening because I was attempting to run the Slick query via SBT's Tests.Setup hook. My hook, in build.sbt, looks something like this:
testOptions in Test += Tests.Setup(loader =>
loader.loadClass("TestSetup").newInstance)
My guess is that SBT has not properly instantiated the classpath at the time this TestSetup class gets instantiated (and when my Slick query tries to execute). Perhaps someone that knows more about SBT's internals can edit this answer to provide more insight, though.
Related
I am trying to run my first Scio pipeline on Dataflow .
The code in question can be found here. However I do not think that is too important.
My first experiment was to read some local CSV files and write another local CSV file, using the DirecRunner. That worked as expected.
Now, I am trying to read the files from GCS, write the output to BigQuery and run the pipeline using the DataflowRunner. I already made all the necessary changes (or that is what I believe). But I am unable to make it run.
I already gcloud auth application-default login and when I do
sbt run --runner=DataflowRunner --project=project-id --input-path=gs://path/to/data --output-table=dataset.table
I can see the Jb is submitted in Dataflow. However, after one hour the jobs fails with the following error message.
Workflow failed. Causes: The Dataflow job appears to be stuck because no worker activity has been seen in the last 1h.
(Note, the job did nothing in all that time, and since this is an experiment the data is simple too small to take more than a couple of minutes).
Checking the StackDriver I can find the follow error:
java.lang.ClassNotFoundException: scala.collection.Seq
Related to some jackson thing:
java.util.ServiceConfigurationError: com.fasterxml.jackson.databind.Module: Provider com.fasterxml.jackson.module.scala.DefaultScalaModule could not be instantiated
And that is what is killing each executor just at the start. I really do not understand why I can not find the Scala standard library.
I also tried to first create a template and runt it latter with:
sbt run --runner=DataflowRunner --project=project-id --input-path=gs://path/to/data --output-table=dataset.table --stagingLocation=gs://path/to/staging --templateLocation=gs://path/to/templates/template-1
But, after running the template, I get the same error.
Also, I noticed that in the staging folder there are a lot of jars, but the scala-library.jar is not in there.
I am missing something obvious?
It's a known issue with sbt 1.3.0 which introduced some breaking change w.r.t. class loaders. Try 1.2.8?
Also the Jackson issue is probably related to Java 11 or above. Stay with Java 8 for now.
Fix by setting the sbt classLoaderLayeringStrategy:
run / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.Flat
sbt uses a new classloader for the application that is run with run. This causes other classes already loaded by the JVM (Predef for instance) to be reused, reducing startup time. See in-process classloaders for details.
This doesn't play well with the Beam DataflowRunner because it explicitly does not stage classes from parent classloaders, see PipelineResources.java#L51:
Attempts to detect all the resources the class loader has access to. This does not recurse to class loader parents stopping it from pulling in resources from the system class loader.
So the fix is to force all classes used by your application to be loaded in the same classloader so that DataflowRunner stages everything.
Hope that helps
So the problem is really simple and I hope solution will be as well.
So basically I have two configuration files application.conf and dev.conf. I'm passing my config files from command line like that sbt -Dconfig.file=dev.conf.
The problem is when I use ConfigFactory.load from main object(the one which extends App) it loads config I passed via command line(in this case dev.conf), but when I load the config from different object it loads default application.conf.
Can I load somehow config passed from arguments from any object?
When you run your application with the runMain SBT task, then by default SBT won't create a separate JVM for your code. This has several consequences around the application lifecycle, and of course with regard to system properties as well.
In general, your approach should work, as long as your build configuration does not enable forking. However, I think the better approach would be to actually rely on forking and specify the system property explicitly. This is guaranteed to work. To do this, you need to set the fork setting in the run task to true, and then add a JVM command line option:
Compile / run / fork := true,
Compile / run / javaOptions += "-Dconfig.file=dev.conf",
Don't forget to restart SBT after that. You won't need to pass the config.file property to SBT with this approach; rather, it is controlled by the javaOptions setting, as in the example above.
Using scala playframework 2.5,
I build the app into a jar using sbt plugin PlayScala,
And then build and pushes a docker image out of it using sbt plugin DockerPlugin
Residing in the source code repository conf/development.conf (same where application.conf is).
The last line in application.conf says include development which means that in case development.conf exists, the entries inside of it will override some of the entries in application.conf in such way that provides all default values necessary for making the application runnable locally right out of the box after the source was cloned from source control with zero extra configuration. This technique allows every new developer to slip right in a working application without wasting time on configuration.
The only missing piece to make that architectural design complete is finding a way to exclude development.conf from the final runtime of the app - otherwise this overrides leak into production runtime and obviously the application fails to run.
That can be achieved in various different ways.
One way could be to some how inject logic into the build task (provided as part of the sbt pluging PlayScala I assume) to exclude the file from the jar artifact.
Other way could be injecting logic into the docker image creation process. this logic could manually delete development.conf from the existing jar prior to executing it (assuming that's possible)
If you ever implemented one of the ideas offered,
or maybe some different architectural approach that gives the same "works out of the box" feature, please be kind enough to share :)
I usually have the inverse logic:
I use the application.conf file (that Play uses by default) with all the things needed to run locally. I then have a production.conf file that starts by including the application.conf, and then overrides the necessary stuff.
for deploying to production (or staging) I specify the production/staging.conf file to be used
This is how I solved it eventually.
conf/application.conf is production ready configuration, it contains placeholders for environment variables whom values will be injected in runtime by k8s given the service's deployment.yaml file.
right next to it, conf/development.conf - its first line is include application.conf and the rest of it are overrides which will make the application run out of the box right after git clone by a simple sbt run
What makes the above work, is the addition of the following to build.sbt :
PlayKeys.devSettings := Seq(
"config.resource" -> "development.conf"
)
Works like a charm :)
This can be done via the mappings config key of sbt-native-packager:
mappings in Universal ~= (_.filterNot(_._1.name == "development.conf"))
See here.
I'm trying to set up a play framework 2.6 application in production, but I'm struggling a bit with it.
I've tried both sbt dist and sbt stage, but when I run the server startup file with -Dplay.http.secret.key (generated as per https://www.playframework.com/documentation/2.6.x/ApplicationSecret), nothing really happens. I simply get a prompt with >, and if I check the logs, there's nothing happening there.
It doesn't seem the server is up at all. I tried http://localhost:9000, as well as https://localhost:9443, but I get the usual "Site can't be reached".
I tried to set up the property file myself with -Dconfig.resource and -Dconfig.file. I also made sure I had execution permission for my startup file.
None of it worked. What am I missing? Any extra configuration I need to specify to run it in production, either in build.sbt or application.properties?
I increased Play logging level to debug, and still see absolutely nothing.
Did you quote the secret ? The secret usually contains chars that have a special meaning in a shell, so you need to quote :
./app -Dplay.http.secret.key='M%>secret^^%%'
You're also free to put the secret in the application.conf.
I am developing an application in Scala Play 2.5.4. I want to test my database interactions, and am attempting to do so by the method recommended in this page.
The trouble is, I am unable to import the object play.api.db.Databases. I suspect that I may need to add something to my build.sbt file, but since this is part of the Play API, I'm not so sure that this is the case.
There are some things available, but not what is shown in the API doc
Yes, you need to add in your build.sbt file this:
libraryDependencies += jdbc
After that, reload activator and update your dependencies (activator update/sbt update).
Note that after this you will need also to add the jdbc driver of the database you intend to use. See more info in https://www.playframework.com/documentation/2.5.x/ScalaDatabase
Edit
As stated in the comments, this may cause problems with Slick.
Unfortunately the classes you need to use are provided by that module, so if this causes you problems, you can try two things:
Extract the class/methods you need and put them in your project: you can get the Databases class code in https://github.com/playframework/playframework/blob/2.5.x/framework/src/play-jdbc/src/main/scala/play/api/db/Databases.scala
Try to disable the DBModule. I'm not quite sure about the syntax, so try each one of those:
play.modules.enabled -= "play.api.db.DBModule"
play.modules.disabled += "play.api.db.DBModule"
For people who do not have dependency to Slick but have the same problem. If you have added dependencies to jdbc and your DB driver (for example Postgresql) and problem still exists, reindexing of all dependencies via sbt is required. In my case closing IntelliJ Idea and subsequent reopening project back causes dependencies reindexing and problem was gone.