Is it possible to include in *conf file another conf file?
Current implementation:
// db-writer.conf
writer: {
name="DatabaseWriter",
model="model1",
table-name="model1",
append=false,
create-table-file="sql/create_table_model1.sql",
source-file="abcd.csv"
}
Desired solution:
// model1.conf + others model2.conf, model3.conf..
table: {
name="model1",
table-name="model1",
create-table-file="../sql/create_table_model1.sql"
}
//db-writer.conf
import model1.conf <=== some import?
writer: {
name="DatabaseWriter",
model="model1", <=== some reference like this?
append=false,
source-file="abcd.csv"
}
Reason why I would like to have it like this is :
to reduce duplicated definitions
to pre-define user conf file which are rare modified
I guess it is not possible - if not do you have any suggestion how to separate configs & reuse them?
I'm using scala 2.12 lang and pureconfig 0.14 (can be updated to any newer)
Pureconfig uses HOCON (though some of the interpretation of things like durations differ). HOCON include is supported.
So assuming that you have model1.conf in your resources (e.g. src/main/resources), all you need in db-writer.conf is
include "model1"
HOCON-style overrides and concatenation are also supported:
writer: ${table} {
name = "DatabaseWriter"
model = "model1"
append = false
source-file = "abcd"
}
Related
I am using a HOCON configuration file which also has substitution variables. But in the case of substitution variables, the key is not overridden by another value into the same file.
For example, consider the following HOCON config:
{
"x":5
"x":6
"y":{"a":1}
"y":{"a":11}
"z":${y.a}
"z":${y.a}
}
Now when I load this from ConfigFactor.parseURL, the resulted config is:
{"x":6,"y":{"a":11},"z":${y.a},"z":${y.a}}
Here y has to be resolved, but this does not happen with z.
Questions:
What is the reason for this output?
How could be enabled to resolve "z" as well?
You're just parsing the config file without resolving it. You have to call resolve() method.
Check following example
val options: ConfigRenderOptions = ConfigRenderOptions
.defaults()
.setComments(false)
.setOriginComments(false)
.setFormatted(false)
.setJson(true)
val parsed = ConfigFactory.parseString("""
|{
| "x":5
| "x":6
| "y":{"a":1}
| "y":{"a":11}
| "z":${y.a}
| "z":${y.a}
|}
|""".stripMargin)
println(parsed.root().render(options))
println(parsed.resolve().root().render(options))
Prints
{"x":6,"y":{"a":11},"z":${y.a},"z":${y.a}}
{"x":6,"y":{"a":11},"z":11}
Please note that parse/resolve methods are used for advanced/customised configuration loading.
If you are just loading application.conf and reference.conf files, I suggest to stick to load* methods only. Or use ConfigFactory.load(ConfigFactory.parse...) way of resolving the parsed config.
This is a newbie question, I read a lot but I am a bit confused.
I pass a properties file from inside a Jar, configuration is read, all is fine.
I wanted to add a try catch.I tried this but it does not work because the loading does not produce an exception if the properties file is not present. Therefore 3 questions:
Is it correct to load files like this?
Does it make sense to put a try/catch since the config is inside the jar?
If so, any suggestions on how to?
var appProps : Config = ConfigFactory.load()
try {
appProps = ConfigFactory.load("application.properties")
} catch {
case e: Exception => {
log.error("application.properties file not found")
sc.stop()
System.exit(1)
}
}
Your code looks ok in general.
ConfigFactory.load("resource.config") will handle a missing resource like an empty resource. As a result you get an empty Config. So, a try-catch-block does not really make sense.
You would usually specify a fallback configuration like this:
val appProps = ConfigFactory.load(
"application.properties"
).withFallBack(
ConfigFactory.load()
)
EDIT:
The sentence
As a result you get an empty Config
Is somewhat incomplete. ConfigFactory.load(resourceBaseName: String) applies defaultReference() and defaultOverrides(). So your resulting Config object probably contains some generic environment data and is not empty.
As far as I can see, your best option to check if the resource is there and emit an error message if not, is to look up the resource yourself:
val configFile = "application.properties"
val config = Option(getClass.getClassLoader.getResource(configFile)).fold {
println(s"Config not found!")
ConfigFactory.load()
} { resource =>
ConfigFactory.load(configFile)
}
So people have been having problems compressing the output of Scalding Jobs including myself. After googling I get the odd hiff of an answer in a some obscure forum somewhere but nothing suitable for peoples copy and paste needs.
I would like an output like Tsv, but writes compressed output.
Anyway after much faffification I managed to write a TsvCompressed output which seems to do the job (you still need to set the hadoop job system configuration properties, i.e. set compress to true, and set the codec to something sensible or it defaults to crappy deflate)
import com.twitter.scalding._
import cascading.tuple.Fields
import cascading.scheme.local
import cascading.scheme.hadoop.{TextLine, TextDelimited}
import cascading.scheme.Scheme
import org.apache.hadoop.mapred.{OutputCollector, RecordReader, JobConf}
case class TsvCompressed(p: String) extends FixedPathSource(p) with DelimitedSchemeCompressed
trait DelimitedSchemeCompressed extends Source {
val types: Array[Class[_]] = null
override def localScheme = new local.TextDelimited(Fields.ALL, false, false, "\t", types)
override def hdfsScheme = {
val temp = new TextDelimited(Fields.ALL, false, false, "\t", types)
temp.setSinkCompression(TextLine.Compress.ENABLE)
temp.asInstanceOf[Scheme[JobConf,RecordReader[_,_],OutputCollector[_,_],_,_]]
}
}
I have also small project showing how to achieve compressed output from Tsv. WordCount-Compressed.
Scalding was setting null to the Cascading TextDelimeted parameter which disables compression.
One question I have about current Scala couchdb drivers is whether they can work with "partial" schemas". I'll try to explain what I mean: the libraries I've see seem to all want to do a complete conversion from JSON docs in the database to a Scala object, handle the Scala object, and convert it back to JSON. This is is fine if your application knows everything about that type of object --- especially if it is the sole piece of software interacting with that database. However, what if I want to write a little application that only knows about part of the JSON object: for example, what if I'm only interested in a 'mybook' component embedded like this:
{
_id: "0ea56a7ec317138700743cdb740f555a",
_rev: "2-3e15c3acfc3936abf10ea4f84a0aeced",
type: "user",
profiles: {
mybook: {
key: "AGW45HWH",
secret: "g4juh43ui9hg929gk4"
},
.. 6 or 7 other profiles
},
.. lots of other stuff
}
I really don't want to convert the whole JSON AST to a Scala object. On the other hand, in couchdb, you must save back the entire JSON doc, so this needs to be preserved somehow. I think what I really what is something like this:
class MyBook {
private val userJson: JObject = ... // full JSON retrieved from the database
lazy val _id: String = ... // parsed from the JSON
lazy val _rev: String = ... // parsed from the JSON
lazy val key: String = ... // parsed from the JSON
lazy val secret: String = ... // (ditto)
def withSecret(secret: String): MyBook = ... // new object with altered userJson
def save(db: CouchDB) = ... // save userJson back to couchdb
}
Advantages:
computationally cheaper to extract only needed fields
don't have to sync with database evolution except for 'mybook' part
more suitable for development with partial schemas
safer, because there is less change as inadvertently deleting fields if we didn't keep up with the database schema
Disadavantages:
domain objects in Scala are not pristinely independent of couch/JSON
more memory use per object
Is this possible with any of the current Scala drivers? With either of scouchdb or the new Sohva library, it seems not.
As long as you have a good JSON library and a good HTTP client library, implementing a schemaless CouchDB client library is really easy.
Here is an example in Java: code, tests.
My couchDB library uses spray-json for (de)serialization, which is very flexible and would enable you to ignore parts of a document but still save it. Let's look at a simplified example:
Say we have a document like this
{
dontcare: {
...
},
important: "foo"
}
Then you could declare a class to hold information from this document and define how the conversion is done:
case class Dummy(js:JsValue)
case class PartialDoc(dontcare: Dummy, important: String)
implicit object DummyFormat extends JsonFormat[Dummy] {
override def read(js:JsValue):Dummy = Dummy(js)
override def write(d:Dummy):JsValue = d.js
}
implicit val productFormat = jsonFormat2(PartialDoc)
This will ignore anything in dontcare but still safe it as a raw JSON AST. Of course this example is not as complex as the one in your question, but it should give you an idea how to solve your problem.
Is there any intrinsic support for namespacing in coffeescript?
Adequate namespacing seems like something coffeescript could really help with although I don't seem to be able to find anything to suggest that there is support for this.
I prefer using this pattern for "namespacing". It isn't really a namespace but an object tree, but it does the job:
Somewhere in the startup of the app, you define the namespaces globally (replace window with exports or global based on your environment.
window.App =
Models: {}
Collections: {}
Views: {}
Then, when you want to declare classes, you can do so:
class App.Models.MyModel
# The class is namespaced in App.Models
And when you want to reference it:
myModel = new App.Models.MyModel()
If you don't like the global way of defining namespaces, you can do so before your class:
window.App.Models ?= {} # Create the "namespace" if Models does not already exist.
class App.Models.MyModel
A way to make it simple to reference the class both in it's own "namespace" (the closed function) and the global namespace is to assign it immediately. Example:
# Define namespace unless it already exists
window.Test or= {}
# Create a class in the namespace and locally
window.Test.MyClass = class MyClass
constructor: (#a) ->
# Alerts 3
alert new Test.MyClass(1).a + new MyClass(2).a
As you see, now you can refer to it as MyClass within the file, but if you need it outside it's available as Test.MyClass. If you only want it in the Test namespace you can simplify it even more:
window.Test or= {}
# Create only in the namespace
class window.Test.MyClass
constructor: (#a) ->
Here's my personal implementation :
https://github.com/MaksJS/Namespace-in-CoffeeScript
How to use in the browser :
namespace Foo:SubPackage1:SubPackage2:
class Bar extends Baz
#[...]
How to use in CommonJS environment :
require './path/to/this/file' # once
namespace Foo:SubPackage1:SubPackage2:
class Bar extends Baz
#[...]
From the section about namespacing on the wiki: https://github.com/jashkenas/coffee-script/wiki/FAQ
# Code:
#
namespace = (target, name, block) ->
[target, name, block] = [(if typeof exports isnt 'undefined' then exports else window), arguments...] if arguments.length < 3
top = target
target = target[item] or= {} for item in name.split '.'
block target, top
# Usage:
#
namespace 'Hello.World', (exports) ->
# `exports` is where you attach namespace members
exports.hi = -> console.log 'Hi World!'
namespace 'Say.Hello', (exports, top) ->
# `top` is a reference to the main namespace
exports.fn = -> top.Hello.World.hi()
Say.Hello.fn() # prints 'Hi World!'
You must really check out CoffeeToaster:
https://github.com/serpentem/coffee-toaster
It comes with a packaging system that when enabled will use your folder's hierarchy as namespaces declarations to your classes if you want so, then you can extends classes from multiple files, do imports and son, such as like:
#<< another/package/myclass
class SomeClass extends another.package.MyClass
The build configuration is extremely minimalist and simple, made to be obvious:
# => SRC FOLDER
toast 'src_folder'
# => VENDORS (optional)
# vendors: ['vendors/x.js', 'vendors/y.js', ... ]
# => OPTIONS (optional, default values listed)
# bare: false
# packaging: true
# expose: ''
# minify: false
# => HTTPFOLDER (optional), RELEASE / DEBUG (required)
httpfolder: 'js'
release: 'www/js/app.js'
debug: 'www/js/app-debug.js'
There's also a debug option that compile files individually for ease the debugging processes and another useful features.
Hope it helps.
Note that it is possible to write:
class MyObject.MyClass
constructor: () ->
initializeStuff()
myfunction: () ->
doStuff()
if you declared an object/ns MyObject.
And while we're at it, here's my implementation of a jquery-ns-function:
(function($) {
$.namespace = function(namespace, initVal) {
var nsParts = namespace.split("."),
nsPart = nsParts.shift(),
parent = window[nsPart] = window[nsPart] || {},
myGlobal = parent;
while(nsPart = nsParts.shift()) {
parent = parent[nsPart] = parent[nsPart] || {};
}
return myGlobal;
}
})(jQuery);
I Strongly suggest using requirejs.org or similar battle tested module loaders.
Especially if you want to load stuff asynchronously.
Rolling your own namespacing/module scheme is really hard if you disregard
the simply, easy and naive approaches
As I'm also busy to learn the best way of structuring the files and use coffeescript in combination with backbone and cake, I have created a small project on github to keep it as a reference for myself, maybe it will help you too around cake and some basic things. All .js (with cake compiled files) are in www folder so that you can open them in your browser and all source files (except for cake configuration) are in src folder. In this example, all .coffee files are compiled and combined in one output .js file which is then included in html.
Based on some of the answers here on StackOverflow I have created small util.coffee file (in src folder) that exposes "namespaces" to the rest of the code.