Suppose I have a scala case class with the ability to serialize into json (using json4s or some other library):
case class Weather(zip : String, temp : Double, isRaining : Boolean)
If I'm using a HOCON config file:
allWeather {
BeverlyHills {
zip : 90210
temp : 75.0
isRaining : false
}
Cambridge {
zip : 10013
temp : 32.0
isRainging : true
}
}
Is there any way to use typesafe config to automatically instantiate a Weather object?
I'm looking for something of the form
val config : Config = ConfigFactory.parseFile(new java.io.File("weather.conf"))
val bevHills : Weather = config.getObject("allWeather.BeverlyHills").as[Weather]
The solution could leverage the fact that the value referenced by "allWeather.BeverlyHills" is a json "blob".
I could obviously write my own parser:
def configToWeather(config : Config) =
Weather(config.getString("zip"),
config.getDouble("temp"),
config.getBoolean("isRaining"))
val bevHills = configToWeather(config.getConfig("allWeather.BeverlyHills"))
But that seems inelegant since any change to the Weather definition would also require a change to configToWeather.
Thank you in advance for your review and response.
typesafe config library has API to instantiate object from config that uses java bean convention. But as I understand case class does not follow those rules.
There are several scala libraries that wrap typesafe config and provide scala specific functionality that you are looking for.
For example using pureconfig reading config could look like
val weather:Try[Weather] = loadConfig[Weather]
where Weather is a case class for values in config
Expanding on Nazarii's answer, the following worked for me:
import scala.beans.BeanProperty
//The #BeanProperty and var are both necessary
case class Weather(#BeanProperty var zip : String,
#BeanProperty var temp : Double,
#BeanProperty var isRaining : Boolean) {
//needed by configfactory to conform to java bean standard
def this() = this("", 0.0, false)
}
import com.typesafe.config.ConfigFactory
val config = ConfigFactory.parseFile(new java.io.File("allWeather.conf"))
import com.typesafe.config.ConfigBeanFactory
val bevHills =
ConfigBeanFactory.create(config.getConfig("allWeather.BeverlyHills"), classOf[Weather])
Follow up: based on the comments below it may be the case that only Java Collections, and not Scala Collections, are viable options for the parameters of the case class (e.g. Seq[T] will not work).
A simple solution without external libraries, inspired from playframework Configuration.scala
trait ConfigLoader[A] { self =>
def load(config: Config, path: String = ""): A
def map[B](f: A => B): ConfigLoader[B] = (config, path) => f(self.load(config, path))
}
object ConfigLoader {
def apply[A](f: Config => String => A): ConfigLoader[A] = f(_)(_)
implicit val stringLoader: ConfigLoader[String] = ConfigLoader(_.getString)
implicit val booleanLoader: ConfigLoader[Boolean] = ConfigLoader(_.getBoolean)
implicit val doubleLoader: ConfigLoader[Double] = ConfigLoader(_.getDouble)
}
object Implicits {
implicit class ConfigOps(private val config: Config) extends AnyVal {
def apply[A](path: String)(implicit loader: ConfigLoader[A]): A = loader.load(config, path)
}
implicit def configLoader[A](f: Config => A): ConfigLoader[A] = ConfigLoader(_.getConfig).map(f)
}
Usage:
import Implicits._
case class Weather(zip: String, temp: Double, isRaining: Boolean)
object Weather {
implicit val loader: ConfigLoader[Weather] = (c: Config) => Weather(
c("zip"), c("temp"), c("isRaining")
)
}
val config: Config = ???
val bevHills: Weather = config("allWeather.BeverlyHills")
Run the code in Scastie
Another option is to use circe.config with the code below. See https://github.com/circe/circe-config
import io.circe.generic.auto._
import io.circe.config.syntax._
def configToWeather(conf: Config): Weather = {
conf.as[Weather]("allWeather.BeverlyHills") match {
case Right(c) => c
case _ => throw new Exception("invalid configuration")
}
}
Another tried-and-tested solution is to use com.fasterxml.jackson.databind.ObjectMapper. You don't need to tag #BeanProperty to any of your case class parameters but you will have to define a no-arg constructor.
case class Weather(zip : String, temp : Double, isRaining : Boolean) {
def this() = this(null, 0, false)
}
val mapper = new ObjectMapper().registerModule(DefaultScalaModule)
val bevHills = mapper.convertValue(config.getObject("allWeather.BeverlyHills").unwrapped, classOf[Weather])
Using config loader
implicit val configLoader: ConfigLoader[Weather] = (rootConfig: Config, path: String) => {
val config = rootConfig.getConfig(path)
Weather(
config.getString("zip"),
config.getDouble("temp"),
config.getBoolean("isRaining")
)
}
Related
I want to use JCommander to parse args.
I wrote some code:
import com.beust.jcommander.{JCommander, Parameter}
import scala.collection.mutable.ArrayBuffer
object Config {
#Parameter(names = Array("--categories"), required = true)
var categories = new ArrayBuffer[String]
}
object Main {
def main(args: Array[String]): Unit = {
val cfg = Config
JCommander
.newBuilder()
.addObject(cfg)
.build()
.parse(args.toArray: _*)
println(cfg.categories)
}
}
Howewer it fails with
com.beust.jcommander.ParameterException: Could not invoke null
Reason: Can not set static scala.collection.mutable.ArrayBuffer field InterestRulesConfig$.categories to java.lang.String
What am i doing wrong?
JCommander uses knowledge about types in Java to map values to parameters. But Java doesn't have a type scala.collection.mutable.ArrayBuffer. It has a type java.util.List. If you want to use JCommander you have to stick to Java's build-in types.
If you want to use Scala's types use one of Scala's libraries that handle in in more idiomatic manner: scopt or decline.
Working example
import java.util
import com.beust.jcommander.{JCommander, Parameter}
import scala.jdk.CollectionConverters._
object Config {
#Parameter(names = Array("--categories"), required = true)
var categories: java.util.List[Integer] = new util.ArrayList[Integer]()
}
object Hello {
def main(args: Array[String]): Unit = {
val cfg = Config
JCommander
.newBuilder()
.addObject(cfg)
.build()
.parse(args.toArray: _*)
println(cfg.categories)
println(cfg.categories.getClass())
val a = cfg.categories.asScala
for (x <- a) {
println(x.toInt)
println(x.toInt.getClass())
}
}
}
I would like to access the following configuration
customers {
"cust1" {
clientId: "id1"
attrs {
att1: "str1"
att2: "str2"
att3: "str3"
att4: "str4"
}
}
"cust2" {
clientId: "id2"
attrs: {
att1: "faldfjalfj"
att2: "reqwrewrqrq"
}
}
"cust3" {
clientId: "id3"
attrs {
att2: "xvcbzxbv"
}
}
}
as a Map[String, CustomerConfig] where CustomerConfig is
package models
import play.api.ConfigLoader
import com.typesafe.config.Config // I added this import in order to make the documentation snippet compile to the best of my knowledge.
case class CustomerConfig(clientId: String, attrs: Map[String, String])
object CustomerConfig {
implicit val configLoader: ConfigLoader[CustomerConfig] = new ConfigLoader[CustomerConfig] {
def load(rootConfig: Config, path: String): CustomerConfig = {
val config = rootConfig.getConfig(path)
CustomerConfig(
clientId = config.getString("clientId"),
attrs = config.get[Map[String, String]]("attrs").map { case (attr, attrVal) =>
(attr, attrVal)
})
}
}
}
For reference, this is how I am currently attempting to reference it:
val resellerEnvMap = conf.get[Map[String, CustomerConfig]]("customers").map {
case (customer, customerConfig) =>
customer -> customerConfig.attrs.map {
case (attr, attrVal) =>
attr -> new Obj(attrVal, customerConfig.clientId)
}
}
based on custom config loader documentation.
The problem is that config.get[A] does not exist (neither does config.getMap[K, V]) per the what I believe to be the API docs. I would like a way to populate that map from the configuration file. My end goal is to populate anything in the functional vicinity of Map[String, Map[String, (String, String)]] where the first String is the customer name, the 2nd is attribute name, the 3rd is the attribute value, and lastly, the 4th is the client ID.
Play 2.6 uses com.typesafe:config 1.3.2 while the link you posted seems to be from version 2? Here's one way to do it:
object CustomerConfig {
implicit val configLoader: ConfigLoader[CustomerConfig] = new ConfigLoader[CustomerConfig] {
def load(rootConfig: Config, path: String): CustomerConfig = {
val config = rootConfig.getConfig(path)
import scala.collection.JavaConverters._
val attrConfig = config.getConfig("attrs")
CustomerConfig(
clientId = config.getString("clientId"),
attrs = attrConfig.entrySet().asScala.map { entry =>
(entry.getKey, attrConfig.getString(entry.getKey))
}.toMap
)
}
}
}
I am experiencing a reproducible error while producing Avro messages with reactive kafka and avro4s. Once the identityMapCapacity of the client (CachedSchemaRegistryClient) is reached, serialization fails with
java.lang.IllegalStateException: Too many schema objects created for <myTopic>-value
This is unexpected, since all messages should have the same schema - they are serializations of the same case class.
val avroProducerSettings: ProducerSettings[String, GenericRecord] =
ProducerSettings(system, Serdes.String().serializer(),
avroSerde.serializer())
.withBootstrapServers(settings.bootstrapServer)
val avroProdFlow: Flow[ProducerMessage.Message[String, GenericRecord, String],
ProducerMessage.Result[String, GenericRecord, String],
NotUsed] = Producer.flow(avroProducerSettings)
val avroQueue: SourceQueueWithComplete[Message[String, GenericRecord, String]] =
Source.queue(bufferSize, overflowStrategy)
.via(avroProdFlow)
.map(logResult)
.to(Sink.ignore)
.run()
...
queue.offer(msg)
The serializer is a KafkaAvroSerializer, instantiated with a new CachedSchemaRegistryClient(settings.schemaRegistry, 1000)
Generating the GenericRecord:
def toAvro[A](a: A)(implicit recordFormat: RecordFormat[A]): GenericRecord =
recordFormat.to(a)
val makeEdgeMessage: (Edge, String) => Message[String, GenericRecord, String] = { (edge, topic) =>
val edgeAvro: GenericRecord = toAvro(edge)
val record = new ProducerRecord[String, GenericRecord](topic, edge.id, edgeAvro)
ProducerMessage.Message(record, edge.id)
}
The schema is created deep in the code (io.confluent.kafka.serializers.AbstractKafkaAvroSerDe#getSchema, invoked by io.confluent.kafka.serializers.AbstractKafkaAvroSerializer#serializeImpl) where I have no influence on it, so I have no idea how to fix the leak. Looks to me like the two confluent projects do not work well together.
The issues I have found here, here and here do not seem to address my use case.
The two workarounds for me are currently:
not use schema registry - not a long-term solution obviously
create custom SchemaRegistryClient not relying on object identity - doable but I would like to avoid creating more issues than by reimplementing
Is there a way to generate or cache a consistent schema depending on message/record type and use it with my setup?
edit 2017.11.20
The issue in my case was that each instance of GenericRecord carrying my message has been serialized by a different instance of RecordFormat, containing a different instance of the Schema. The implicit resolution here generated a new instance each time.
def toAvro[A](a: A)(implicit recordFormat: RecordFormat[A]): GenericRecord = recordFormat.to(a)
The solution was to pin the RecordFormat instance to a val and reuse it explicitly. Many thanks to https://github.com/heliocentrist for explaining the details.
original response:
After waiting for a while (also no answer for the github issue) I had to implement my own SchemaRegistryClient. Over 90% is copied from the original CachedSchemaRegistryClient, just translated into scala. Using a scala mutable.Map fixed the memory leak. I have not performed any comprehensive tests, so use at your own risk.
import java.util
import io.confluent.kafka.schemaregistry.client.rest.entities.{ Config, SchemaString }
import io.confluent.kafka.schemaregistry.client.rest.entities.requests.ConfigUpdateRequest
import io.confluent.kafka.schemaregistry.client.rest.{ RestService, entities }
import io.confluent.kafka.schemaregistry.client.{ SchemaMetadata, SchemaRegistryClient }
import org.apache.avro.Schema
import scala.collection.mutable
class CachingSchemaRegistryClient(val restService: RestService, val identityMapCapacity: Int)
extends SchemaRegistryClient {
val schemaCache: mutable.Map[String, mutable.Map[Schema, Integer]] = mutable.Map()
val idCache: mutable.Map[String, mutable.Map[Integer, Schema]] =
mutable.Map(null.asInstanceOf[String] -> mutable.Map())
val versionCache: mutable.Map[String, mutable.Map[Schema, Integer]] = mutable.Map()
def this(baseUrl: String, identityMapCapacity: Int) {
this(new RestService(baseUrl), identityMapCapacity)
}
def this(baseUrls: util.List[String], identityMapCapacity: Int) {
this(new RestService(baseUrls), identityMapCapacity)
}
def registerAndGetId(subject: String, schema: Schema): Int =
restService.registerSchema(schema.toString, subject)
def getSchemaByIdFromRegistry(id: Int): Schema = {
val restSchema: SchemaString = restService.getId(id)
(new Schema.Parser).parse(restSchema.getSchemaString)
}
def getVersionFromRegistry(subject: String, schema: Schema): Int = {
val response: entities.Schema = restService.lookUpSubjectVersion(schema.toString, subject)
response.getVersion.intValue
}
override def getVersion(subject: String, schema: Schema): Int = synchronized {
val schemaVersionMap: mutable.Map[Schema, Integer] =
versionCache.getOrElseUpdate(subject, mutable.Map())
val version: Integer = schemaVersionMap.getOrElse(
schema, {
if (schemaVersionMap.size >= identityMapCapacity) {
throw new IllegalStateException(s"Too many schema objects created for $subject!")
}
val version = new Integer(getVersionFromRegistry(subject, schema))
schemaVersionMap.put(schema, version)
version
}
)
version.intValue()
}
override def getAllSubjects: util.List[String] = restService.getAllSubjects()
override def getByID(id: Int): Schema = synchronized { getBySubjectAndID(null, id) }
override def getBySubjectAndID(subject: String, id: Int): Schema = synchronized {
val idSchemaMap: mutable.Map[Integer, Schema] = idCache.getOrElseUpdate(subject, mutable.Map())
idSchemaMap.getOrElseUpdate(id, getSchemaByIdFromRegistry(id))
}
override def getSchemaMetadata(subject: String, version: Int): SchemaMetadata = {
val response = restService.getVersion(subject, version)
val id = response.getId.intValue
val schema = response.getSchema
new SchemaMetadata(id, version, schema)
}
override def getLatestSchemaMetadata(subject: String): SchemaMetadata = synchronized {
val response = restService.getLatestVersion(subject)
val id = response.getId.intValue
val version = response.getVersion.intValue
val schema = response.getSchema
new SchemaMetadata(id, version, schema)
}
override def updateCompatibility(subject: String, compatibility: String): String = {
val response: ConfigUpdateRequest = restService.updateCompatibility(compatibility, subject)
response.getCompatibilityLevel
}
override def getCompatibility(subject: String): String = {
val response: Config = restService.getConfig(subject)
response.getCompatibilityLevel
}
override def testCompatibility(subject: String, schema: Schema): Boolean =
restService.testCompatibility(schema.toString(), subject, "latest")
override def register(subject: String, schema: Schema): Int = synchronized {
val schemaIdMap: mutable.Map[Schema, Integer] =
schemaCache.getOrElseUpdate(subject, mutable.Map())
val id = schemaIdMap.getOrElse(
schema, {
if (schemaIdMap.size >= identityMapCapacity)
throw new IllegalStateException(s"Too many schema objects created for $subject!")
val id: Integer = new Integer(registerAndGetId(subject, schema))
schemaIdMap.put(schema, id)
idCache(null).put(id, schema)
id
}
)
id.intValue()
}
}
I'd like to build a generic method for transforming Scala Case Classes to Mongo Documents.
A promising Document constructor is
fromSeq(ts: Seq[(String, BsonValue)]): Document
I can turn a case class into a Map[String -> Any], but then I've lost the type information I need to use the implicit conversions to BsonValues. Maybe TypeTags can help with this?
Here's what I've tried:
import org.mongodb.scala.bson.BsonTransformer
import org.mongodb.scala.bson.collection.immutable.Document
import org.mongodb.scala.bson.BsonValue
case class Person(age: Int, name: String)
//transform scala values into BsonValues
def transform[T](v: T)(implicit transformer: BsonTransformer[T]): BsonValue = transformer(v)
// turn any case class into a Map[String, Any]
def caseClassToMap(cc: Product) = {
val values = cc.productIterator
cc.getClass.getDeclaredFields.map( _.getName -> values.next).toMap
}
// transform a Person into a Document
def personToDocument(person: Person): Document = {
val map = caseClassToMap(person)
val bsonValues = map.toSeq.map { case (key, value) =>
(key, transform(value))
}
Document.fromSeq(bsonValues)
}
<console>:24: error: No bson implicit transformer found for type Any. Implement or import an implicit BsonTransformer for this type.
(key, transform(value))
def personToDocument(person: Person): Document = {
Document("age" -> person.age, "name" -> person.name)
}
Below code works without manual conversion of an object.
import reactivemongo.api.bson.{BSON, BSONDocument, Macros}
case class Person(name:String = "SomeName", age:Int = 20)
implicit val personHandler = Macros.handler[Person]
val bsonPerson = BSON.writeDocument[Person](Person())
println(s"${BSONDocument.pretty(bsonPerson.getOrElse(BSONDocument.empty))}")
You can use Salat https://github.com/salat/salat. A nice example can be found here - https://gist.github.com/bhameyie/8276017. This is the piece of code that will help you -
import salat._
val dBObject = grater[Artist].asDBObject(artist)
artistsCollection.save(dBObject, WriteConcern.Safe)
I was able to serialize a case class to a BsonDocument using the org.bson.BsonDocumentWriter. The below code runs using scala 2.12 and mongo-scala-driver_2.12 version 2.6.0
My quest for this solution was aided by this answer (where they are trying to serialize in the opposite direction): Serialize to object using scala mongo driver?
import org.mongodb.scala.bson.codecs.Macros
import org.mongodb.scala.bson.codecs.DEFAULT_CODEC_REGISTRY
import org.bson.codecs.configuration.CodecRegistries.{fromRegistries, fromProviders}
import org.bson.codecs.EncoderContext
import org.bson.BsonDocumentWriter
import org.mongodb.scala.bson.BsonDocument
import org.bson.codecs.configuration.CodecRegistry
import org.bson.codecs.Codec
case class Animal(name : String, species: String, genus: String, weight: Int)
object TempApp {
def main(args: Array[String]) {
val jaguar = Animal("Jenny", "Jaguar", "Panthera", 190)
val codecProvider = Macros.createCodecProvider[Animal]()
val codecRegistry: CodecRegistry = fromRegistries(fromProviders(codecProvider), DEFAULT_CODEC_REGISTRY)
val codec = Macros.createCodec[Animal](codecRegistry)
val encoderContext = EncoderContext.builder.isEncodingCollectibleDocument(true).build()
var doc = BsonDocument()
val writr = new BsonDocumentWriter(doc) // need to call new since Java lib w/o companion object
codec.encode(writr, jaguar, encoderContext)
print(doc)
}
};
I have this Scala/Play application and I have to fetch a bunch of templates via AJAX. I'm doing something like this now:
def home = Action {
Ok(views.html.home())
}
def about = Action {
Ok(views.html.about())
}
def contact = Action {
Ok(views.html.contact())
}
//etc
But this is just creating an action for every template. Can I do something like this instead:
def loadTemplate(templateName) = Action {
//Load template from "views" with name being value of parameter templateName
}
Is this possible on Play Framework? If so then how?
Play Framework 2.2.1 / Scala 2.10.3 / Java 8 64bit
UPDATE: My original question might have been misunderstood. I don't want to compile a template, I want to fetch already compiled one in a more dynamic way.
UPDATE2: I think I found something very close, if not exactly what I need on this answer, but it's in Java and I need it in Scala.
Using scala reflection:
object Application extends Controller {
import reflect.runtime.universe._
val currentMirror = runtimeMirror(Play.current.classloader)
val packageName = "views.html."
def index(name: String) = Action {
val templateName = packageName + name
val moduleMirror = currentMirror.reflectModule(currentMirror.staticModule(templateName))
val methodSymbol = moduleMirror.symbol.typeSignature.declaration(newTermName("apply")).asMethod
val instanceMirror = currentMirror.reflect(moduleMirror.instance)
val methodMirror = instanceMirror.reflectMethod(methodSymbol)
Ok(methodMirror.apply().asInstanceOf[Html])
}
}
Based on #Nilanjan answer and for play-2.4 scala-2.11.7
def page(page: String) = Action.async { implicit request =>
Future.successful(Ok(loadTemplate(page)))
}
import reflect.runtime.universe._
import play.api._
import play.twirl.api.Html
val currentMirror = runtimeMirror(Play.current.classloader)
val packageName = "views.html."
private def loadTemplate(name: String) = {
val templateName = packageName + name
val moduleMirror = currentMirror.reflectModule(currentMirror.staticModule(templateName))
val methodSymbol = moduleMirror.symbol.info.member(TermName("apply")).asMethod
val instanceMirror = currentMirror.reflect(moduleMirror.instance)
val methodMirror = instanceMirror.reflectMethod(methodSymbol)
methodMirror.apply().asInstanceOf[Html]
}
Here we go again! #Nilanjan answer adopted to Scala 2.12 and Play 2.6 (as moving to DI play.api.Play.current, TermName and declaration have been deprecated). For app, I had to use injector because #Inject()(app: Application) was causing circular dependency
class HomeController #Inject()(injector: Injector, cc: ControllerComponents) extends AbstractController(cc) {
def index(name: String = "index") = Action { implicit request: Request[AnyContent] =>
import reflect.runtime.universe._
val app = injector.instanceOf[Application]
val currentMirror = runtimeMirror(app.classloader)
val packageName = "views.html."
val templateName = packageName + name
val moduleMirror = currentMirror.reflectModule(currentMirror.staticModule(templateName))
val methodSymbol = moduleMirror.symbol.typeSignature.decl(TermName("apply")).asMethod
val instanceMirror = currentMirror.reflect(moduleMirror.instance)
val methodMirror = instanceMirror.reflectMethod(methodSymbol)
Ok(methodMirror.apply("some", "content").asInstanceOf[Html])
}
}
It's not quite good idea to allow search view by any text (for security reasons) as such can be passed in param, instead, you can resolve this quite easy with match statement - it will allow you restrict request to allowed views only and will handle wrong requests as well, probably Scala geeks can demonstrate nicer code, but this will work for you out of the box:
def loadTemplate(templateName: String) = Action {
templateName match {
case "about" => Ok(about())
case "home" => Ok(home())
case "contact" => Ok(contact())
case _ => NotFound(notFoundView())
}
}
route:
GET /load-template/:templateName controllers.Application.loadTemplate(templateName)
Additional benefit is that you can grab additional data from request and pass it to the resolved view depending on templateName param
In your Template file
general_template.scala.html
#(templateName : String = "none")
#templateName match {
case "about" => { //html elements}
case "home" => {//html elements}
case "contact" => {//html elements}
case _ => { }
}
and in your controller
def loadTemplate(templateName) = Action {
Ok(views.html.general_template(templateName))
}