scala- error loading a properties file packed inside jar - scala

This is a newbie question, I read a lot but I am a bit confused.
I pass a properties file from inside a Jar, configuration is read, all is fine.
I wanted to add a try catch.I tried this but it does not work because the loading does not produce an exception if the properties file is not present. Therefore 3 questions:
Is it correct to load files like this?
Does it make sense to put a try/catch since the config is inside the jar?
If so, any suggestions on how to?
var appProps : Config = ConfigFactory.load()
try {
appProps = ConfigFactory.load("application.properties")
} catch {
case e: Exception => {
log.error("application.properties file not found")
sc.stop()
System.exit(1)
}
}

Your code looks ok in general.
ConfigFactory.load("resource.config") will handle a missing resource like an empty resource. As a result you get an empty Config. So, a try-catch-block does not really make sense.
You would usually specify a fallback configuration like this:
val appProps = ConfigFactory.load(
"application.properties"
).withFallBack(
ConfigFactory.load()
)
EDIT:
The sentence
As a result you get an empty Config
Is somewhat incomplete. ConfigFactory.load(resourceBaseName: String) applies defaultReference() and defaultOverrides(). So your resulting Config object probably contains some generic environment data and is not empty.
As far as I can see, your best option to check if the resource is there and emit an error message if not, is to look up the resource yourself:
val configFile = "application.properties"
val config = Option(getClass.getClassLoader.getResource(configFile)).fold {
println(s"Config not found!")
ConfigFactory.load()
} { resource =>
ConfigFactory.load(configFile)
}

Related

Pureconfig - is it possible to include in conf file another conf file?

Is it possible to include in *conf file another conf file?
Current implementation:
// db-writer.conf
writer: {
name="DatabaseWriter",
model="model1",
table-name="model1",
append=false,
create-table-file="sql/create_table_model1.sql",
source-file="abcd.csv"
}
Desired solution:
// model1.conf + others model2.conf, model3.conf..
table: {
name="model1",
table-name="model1",
create-table-file="../sql/create_table_model1.sql"
}
//db-writer.conf
import model1.conf <=== some import?
writer: {
name="DatabaseWriter",
model="model1", <=== some reference like this?
append=false,
source-file="abcd.csv"
}
Reason why I would like to have it like this is :
to reduce duplicated definitions
to pre-define user conf file which are rare modified
I guess it is not possible - if not do you have any suggestion how to separate configs & reuse them?
I'm using scala 2.12 lang and pureconfig 0.14 (can be updated to any newer)
Pureconfig uses HOCON (though some of the interpretation of things like durations differ). HOCON include is supported.
So assuming that you have model1.conf in your resources (e.g. src/main/resources), all you need in db-writer.conf is
include "model1"
HOCON-style overrides and concatenation are also supported:
writer: ${table} {
name = "DatabaseWriter"
model = "model1"
append = false
source-file = "abcd"
}

Play Framework request attributes with typed key

I seem to have issues accessing the attributes of the request attributes map in Play. Following the explanation offered by Play (Link), I should get the correct data from the attributes, but the Option is returned as None.
My structure is as follows. One controller (later injected named as "sec") has the typed attribute for shared access to it:
val AuthenticatedAsAttr: TypedKey[AuthenticatedEmail] = TypedKey("AuthenticatedAs")
The type AuthenticatedEmail is defined in the companion object of this controller as a case class:
case class AuthenticatedEmail(email: String)
The filter passes the attribute to the next request:
val attrs = requestHeader.attrs + TypedEntry[AuthenticatedEmail](sec.AuthenticatedAsAttr, AuthenticatedEmail(email))
nextFilter(requestHeader.withAttrs(attrs))
When trying to then access this attribute in another controller, the returned Option is None:
val auth = request.attrs.get(sec.AuthenticatedAsAttr)
I confirmed via println that the value is definitely in request.attrs but run out of options to debug the issue successfully. A fraction of the println output below.
(Request attrs,{HandlerDef -> HandlerDef(sun.misc .... ,POST, ... Cookies -> Container<Cookies(Cookie ... , AuthenticatedAs -> AuthenticatedEmail(a#test.de), ... })
My Scala version is 2.12.6, Play Framework version 2.6.18. Any help is highly appreciated.
It turns out that the TypedKey must be within an object, not an inject-able controller. So moving it to an object like the following resolves the issue:
object Attrs {
val AuthenticatedAsAttr: TypedKey[AuthenticatedEmail] = TypedKey("AuthenticatedAs")
}
The reason is the implementation of TypedKey (Link), which does not contain an equals method and therefore reverts to comparing memory references.

Using start_bundle() in apache-beam job not working. Unpickleable storage.Client()

I'm getting this error
pickle.PicklingError: Pickling client objects is explicitly not
supported. Clients have non-trivial state that is local and
unpickleable.
When trying to use beam.ParDo to call a function that looks like this
class ExtractBlobs(beam.DoFn):
def start_bundle(self):
self.storageClient = storage.Client()
def process(self, element):
client = self.storageClient
bucket = client.get_bucket(element)
blobs = list(bucket.list_blobs(max_results=100))
return blobs
I thought the whole point of the start_bundle was to initialize self.someProperty and then use that self.someProperty in the 'process' method to get rid of the pickling problem (from sources below)
Could anyone point me into the right direction of how to solve this?
[+] What I've read:
https://github.com/GoogleCloudPlatform/google-cloud-python/issues/3191
How do I resolve a Pickling Error on class apache_beam.internal.clients.dataflow.dataflow_v1b3_messages.TypeValueValuesEnum?
UPDATED: The issue was actually a library issue. I had to have the correct apache-beam SDK version with the correct google-cloud versions:
gapic-google-cloud-pubsub-v1==0.15.4
gax-google-logging-v2==0.8.3
gax-google-pubsub-v1==0.8.3
google-api-core==1.1.2
google-api-python-client==1.6.7
google-apitools==0.5.10
google-auth==1.4.1
google-auth-httplib2==0.0.3
google-cloud-bigquery==1.1.0
google-cloud-core==0.28.1
google-cloud-datastore==1.6.0
google-cloud-pubsub==0.26.0
google-cloud-storage==1.10.0
google-gax==0.12.5
apache-beam==2.3.0
Was able to solve this by what seems a combination of things, first I don't serialize anything (ugly one liner in the yield) and second is using threading.local()
class ExtractBlobs(beam.DoFn):
def start_bundle(self):
self.threadLocal = threading.local()
self.threadLocal.client = storage.Client()
def process(self, element):
yield list(storage.Client().get_bucket(element).list_blobs(max_results=100))

How to log in settingKey in SBT?

I would like to do some logging in SBT. I tried to get streams when initializing settingKey. However the compiler complains that A setting cannot depend on a task.
The config snippet is here.
val appConfig = settingKey[Config]("The parsed application.conf in SBT")
appConfig := {
// ...
streams.value.log.error("Cannot find application.conf. Please check if -Dconfig.file/resource is setting correctly.")
// ...
}
Is there any method to do logging in settingKey? Thanks.
The proper way to log from an sbt setting is by using the Keys.sLog setting. This setting is set up by sbt and it holds the logger that it's used by other settings in the sbt universe.
val appConfig = settingKey[Config]("The parsed application.conf in SBT")
appConfig := {
// ...
Keys.sLog.value.log.error("Cannot find application.conf. Please check if -Dconfig.file/resource is setting correctly.")
// ...
}
A setting should only hold settable data or something directly computable from other settings . streams is a task, so you can make another task depend on it and the appConfig setting. For example:
val appConfig = settingKey[File]("application.conf file")
val parsedAppConfig = taskKey[Config]("The parsed application.conf in SBT")
parsedAppConfig := {
// ...
parse(appConfig.value)
// ...
streams.value.log.error("Cannot find application.conf. Please check if -Dconfig.file/resource is setting correctly.")
// ...
}
It's possible to go around streams and create a new ConsoleLogger() directly.
val appConfig = settingKey[Config]("The parsed application.conf in SBT")
appConfig := {
// ...
val logger: Logger = ConsoleLogger()
logger.error("Cannot find application.conf. Please check if -Dconfig.file/resource is setting correctly.")
// ...
}
Caveat: if streams.value.log gets set to something other than a ConsoleLogger, your log statement will be oblivious and still log to the console. That may be acceptable for your purposes though.

Playframework Scala Async controller for JSON request

I am trying to write an async PlayFramework controller that receives a POST request and creates a new object in the database:
def register = Action(BodyParsers.parse.json) { request =>
val businessInfoResult = request.body.validate[BusinessInfo]
businessInfoResult.fold(errors =>{
BadRequest(Json.obj("status"-> "Error", "message"->JsError.toJson(errors))) //Error on this line
}, businessInfo=> {
//save the object
Ok(Json.obj("status" ->"OK", "message" -> ("Place '"+ businessInfo.businessName +"' saved.") )) //Error on this line
})
}
However, it keeps throwing the error below:
reference to Json is ambiguous; it is imported twice in the same scope by import play.libs.Json and import play.mvc.BodyParser.Json AsyncController.scala
The errors are thrown at line 108 and 105 which correspond to lines commented with //Error on this line above (lines with BadRequest(..) and Ok(..))
How do I fix this issue? I can using new JsValue(Map(..)) but was wondering if there's any other way.
Thank you so much for your help.
Rather than Json, you probably want to call play.libs.Json. The problem here is that, considering the imports in your file, you have two objects / classes called Json and the compiler can't choose which one it should use. Calling play.libs.Json, you'll give the compiler enough information.
You could use one or more aliases in your imports:
import play.libs.Json
import play.mvc.BodyParser.{Json => JsonParser}
JsonParser is just an example. You can use anything you like as long as it's unique within the file.
Instead of writing Json (for play.mvc.BodyParser.Json) you could now use the alias JsonParser.
But are you sure you even need to import play.mvc.BodyParser.Json? Because you don't seem to use it.