I have a HOCON config like this:
[
{
name = 1
url = "http://example.com"
},
{
name = 2
url = "http://example2.com"
},
{
name = 3
url = {
A = "http://example3.com"
B = "http://example4.com"
}
}
]
I want to parse it with pureconfig.
How can I represent that the URL can be either a string or a map of multiple urls, each having a key?
I have tried this:
import pureconfig.ConfigSource
import pureconfig.generic.auto.exportReader
case class Site(name: Int, url: Either[String, Map[String, String]])
case class Config(sites: List[Site])
ConfigSource.default.loadOrThrow[Config]
But it resulted in "Expected type OBJECT. Found STRING instead."
I know pureconfig supports Option. I have found no mention of supporting Either, does it mean it can be replaced with something else?
As you can see Either in not on list of types supported out of the box.
However Either falls under sealed family, so:
# ConfigSource.string("""{ type: left, value: "test" }""").load[Either[String, String]]
res15: ConfigReader.Result[Either[String, String]] = Right(Left("test"))
# ConfigSource.string("""{ type: right, value: "test" }""").load[Either[String, String]]
res16: ConfigReader.Result[Either[String, String]] = Right(Right("test"))
works. If you have a sealed hierarchy, what pureconfig will do is require an object which has a field type - this field will be used to dispatch parsing to a specific subtype. All the other fields will be passed as fields to parse into that subtype.
If that doesn't work for you, you might try to implement the codec yourself:
// just an example
implicit def eitherReader[A: ConfigReader, B: ConfigReader] =
new ConfigReader[Either[A, B]] {
def from(cur: ConfigCursor) =
// try left, if fail try right
ConfigReader[A].from(cur).map(Left(_)) orElse ConfigReader[B].from(cur).map(Right(_))
}
which now will not require discrimination value:
# ConfigSource.string("""{ test: "test" }""").load[Map[String, Either[String, String]]]
res26: ConfigReader.Result[Map[String, Either[String, String]]] = Right(Map("test" -> Left("test")))
This is not provided by default because you would have to answer a few things yourself:
how do you decide if you should go with Left or Right decoding?
does Left fallback Right or Right fallback Left make sense?
how about Either[X, X]?
If you have an idea what is expected behavior you can implement your own codec and use it in derivation.
There are might be several ways of doing it, but I don't like using Either as a config representation. Thus, I would suggest to use ADT approach with sealed trait:
sealed trait NameUrl {
val name: Int
}
case class Name(
name: Int,
url: String
) extends NameUrl
case class NameUrlObj(
name: Int,
url: Map[String, String]
) extends NameUrl
Sorry for my naming here. This would be a representation of of your config.
We need to modify a bit our config to parse easily the config with you ADT. In order to support generic types you should add your spefici type name for each subtype.
I'm going to put here full example so that you can run it on your machine:
import com.typesafe.config.ConfigFactory
import pureconfig.generic.auto._
import pureconfig.ConfigSource
object TstObj extends App {
sealed trait NameUrl {
val name: Int
}
case class Name(
name: Int,
url: String
) extends NameUrl
case class NameUrlObj(
name: Int,
url: Map[String, String]
) extends NameUrl
val cfgStr = ConfigFactory.parseString(
"""
|abc: [
| {
| type: name,
| name = 1
| url = "http://example.com"
| },
| {
| type: name,
| name = 1
| url = "http://example.com"
| },
| {
| type: name-url-obj,
| name = 3
| url = {
| "A": "http://example3.com"
| "B": "http://example4.com"
| }
| }
|]
|""".stripMargin
)
case class RootA(abc: List[NameUrl])
println(ConfigSource.fromConfig(cfgStr).loadOrThrow[RootA])
}
You can read more here about Sealed Families here
Related
I'm trying to define a tapir endpoint, which will accept two potential different payloads (in the snippet below, two different ways of defining a Thing). I'm broadly following the instructions here: https://circe.github.io/circe/codecs/adt.html, and defining my endpoint:
endpoint
.post
.in(jsonBody[ThingSpec].description("Specification of the thing"))
.out(jsonBody[Thing].description("Thing!"))
ThingSpec is a sealed trait, which both the classes representing possible payloads extend:
import io.circe.{Decoder, Encoder, derivation}
import io.circe.derivation.{deriveDecoder, deriveEncoder}
import sttp.tapir.Schema
import sttp.tapir.Schema.annotations.description
import sttp.tapir.generic.Configuration
import cats.syntax.functor._
import io.circe.syntax.EncoderOps
sealed trait ThingSpec {
def kind: String
}
object ThingSpec {
implicit val config: Configuration = Configuration.default.withSnakeCaseMemberNames
implicit val thingConfigDecoder
: Decoder[ThingSpec] = Decoder[ThingOneSpec].widen or Decoder[ThingTwoSpec].widen
implicit val thingConfigEncoder: Encoder[ThingSpec] = {
case one # ThingOneSpec(_, _) => one.asJson
case two # ThingTwoSpec(_, _) => two.asJson
}
implicit val thingConfigSchema: Schema[ThingSpec] =
Schema.oneOfUsingField[ThingSpec, String](_.kind, _.toString)(
"one" -> ThingOneSpec.thingConfigSchema,
"two" -> ThingTwoSpec.thingConfigSchema
)
}
case class ThingOneSpec(
name: String,
age: Long
) extends ThingSpec {
def kind: String = "one"
}
object ThingOneSpec {
implicit val config: Configuration = Configuration.default.withSnakeCaseMemberNames
implicit val thingConfigEncoder: Encoder[ThingOneSpec] = deriveEncoder(
derivation.renaming.snakeCase
)
implicit val thingConfigDecoder: Decoder[ThingOneSpec] = deriveDecoder(
derivation.renaming.snakeCase
)
implicit val thingConfigSchema: Schema[ThingOneSpec] = Schema.derived
}
case class ThingTwoSpec(
height: Long,
weight: Long,
) extends ThingSpec {
def kind: String = "two"
}
object ThingTwoSpec {
implicit val config: Configuration = Configuration.default.withSnakeCaseMemberNames
implicit val thingConfigEncoder: Encoder[ThingTwoSpec] = deriveEncoder(
derivation.renaming.snakeCase
)
implicit val thingConfigDecoder: Decoder[ThingTwoSpec] = deriveDecoder(
derivation.renaming.snakeCase
)
implicit val thingConfigSchema: Schema[ThingTwoSpec] = Schema.derived
}
Which seems to be working OK - except for the redoc docs which are generated. The "request body section" of the redoc, which I believe is generated from
.in(jsonBody[ThingSpec].description("Specification of the thing"))
only includes details of the ThingOneSpec object, there is no mention of ThingTwoSpec. The "payload" example section includes both.
My main question is how to get the request body section of the docs to show both possible payloads.
However - I'm aware that I might not have done this in the best way (from a circe/tapir point of view). Ideally, I'd like not to include an explicit discriminator (kind) in the trait/classes, because I'd rather it not be exposed to the end user in the 'Payload' sections of the docs. Despite reading
https://tapir.softwaremill.com/en/v0.17.7/endpoint/customtypes.html
https://github.com/softwaremill/tapir/blob/master/examples/src/main/scala/sttp/tapir/examples/custom_types/SealedTraitWithDiscriminator.scala
https://github.com/softwaremill/tapir/issues/315
I cannot get this working without the explicit discriminator.
You can get rid of the discriminator by defining a one-of schema by hand:
implicit val thingConfigSchema: Schema[ThingSpec] =
Schema(
SchemaType.SCoproduct(List(ThingOneSpec.thingConfigSchema, ThingTwoSpec.thingConfigSchema), None) {
case one: ThingOneSpec => Some(SchemaWithValue(ThingOneSpec.thingConfigSchema, one))
case two: ThingTwoSpec => Some(SchemaWithValue(ThingTwoSpec.thingConfigSchema, two))
},
Some(Schema.SName(ThingSpec.getClass.getName))
)
(Yes, it is unnecessarily hard to write; I'll look if this can be possibly generated by a macro or otherwise.)
When rendered by redoc, I get a "one of" switch, so I think this is the desired outcome:
New to Scala and Play and running into a problem where I have the following:
case class Media(
name: String,
id: Id,
type: String,
properties: PodcastProperties
)
object Media {
implicit val format: OFormat[Media] = Json.format[Media]
case class PodcastProperties(
x: Int,
y: DateTime,
z: String
)
object PodcastProperties {
implicit val format: OFormat[PodcastProperties] = Json.format[PodcastProperties]
Say I want to define Media to accept different media types. Let's say I have a Json Media object, and it's type is "newspaper" and it's properties should be parse using "NewspaperProperties"
case class NewspaperProperties(
Title: String,
Publisher: String
)
object NewspaperProperties {
implicit val format: OFormat[NewspaperProperties] = Json.format[NewspaperProperties]
How can I define Media, so it can parse the "type" field, and then read the "properties" field correctly using the right Json parser?
You need to defined the media properties as sealed family.
import play.api.libs.json._
import java.time.OffsetDateTime
sealed trait MediaProperties
case class NewspaperProperties(
title: String, // Do not use 'Title' .. initial cap is not valid
publisher: String // ... not 'Publisher'
) extends MediaProperties
object NewspaperProperties {
implicit val format: OFormat[NewspaperProperties] = Json.format[NewspaperProperties]
}
case class PodcastProperties(
x: Int,
y: OffsetDateTime,
z: String
) extends MediaProperties
object PodcastProperties {
implicit val format: OFormat[PodcastProperties] =
Json.format[PodcastProperties]
}
Then a OFormat can be materialized for MediaProperties.
implicit val mediaPropertiesFormat: OFormat[MediaProperties] = Json.format
This managed discriminator in the JSON representation (by default _type field, naming can be configured).
val props1: MediaProperties = PodcastProperties(1, OffsetDateTime.now(), "z")
val obj1 = Json.toJson(props1)
// > JsValue = {"_type":"PodcastProperties","x":1,"y":"2020-11-23T22:53:35.301603+01:00","z":"z"}
obj1.validate[MediaProperties]
JsResult[MediaProperties] = JsSuccess(PodcastProperties(1,2020-11-23T23:02:24.752063+01:00,z),)
The implicit format for MediaProperties should probably be defined in the companion object MediaProperties.
Then the format for Media can be materialized automatically.
final class Id(val value: String) extends AnyVal
object Id {
implicit val format: Format[Id] = Json.valueFormat
}
case class Media(
name: String,
id: Id,
//type: String, -- Not needed for the JSON representation
properties: MediaProperties
)
object Media {
implicit val format: OFormat[Media] = Json.format[Media] // <--- HERE
}
I am trying to parse json with null values for some fields using Play library. There is a case class which represents the data:
case class Id(value: Int) extends AnyVal
case class Name(value: String) extends AnyVal
case class Number(value: Int) extends AnyVal
case class Data(id: Option[Id], name: Option[Name], number: Option[Number])
Here is how parsing currently works:
def parse(jsValue: JsValue): Try[Seq[Data]] = Try {
jsValue.as[JsArray].value
.flatMap { record =>
val id = Id((record \ "id").as[Int])
val name = Name((record \ "name").as[String])
val number = Number((record \ "number").as[Int])
Some(Data(Some(id), Some(name), Some(number)))
}
}
Parsing with specific data types doesn't handle null cases, so this implementation returns:
Failure(play.api.libs.json.JsResultException: JsResultException(errors:List((,List(JsonValidationError(List(error.expected.jsstring),WrappedArray()))))))
For the input data like this:
{
"id": 1248,
"default": false,
"name": null,
"number": 2
}
I would like to have something like this: Seq(Data(Some(Id(1248)), None, Some(Number(2))))
I am going to write the data into the database so I do not mind writing some null values for these fields.
How can I handle null values for fields in parsed json?
You can simply let the play-json library generate the Reads for your case classes instead of writing them manually:
import play.api.libs.json._
object Data {
implicit val reads: Reads[Data] = {
// move these into the corresponding companion objects if used elsewhere...
implicit val idReads = Json.reads[Id]
implicit val numberReads = Json.reads[Number]
implicit val nameReads = Json.reads[Name]
Json.reads[Data]
}
}
def parse(jsValue: JsValue): Try[Seq[Data]] = Json.fromJson[Seq[Data]](jsValue).toTry
That way, your code will work even in case you change the arguments of your case classes.
If you still want to code it manually, you can use the readNullable parser:
val name: Option[Name] = Name(record.as((__ \ "name").readNullable[String]))
Note, however, that using Try is somewhat frowned upon in FP and directly using JsResult would be more idiomatic.
If you are not locked to use play-json then let me show how it can be done easily with jsoniter-scala:
import com.github.plokhotnyuk.jsoniter_scala.core._
import com.github.plokhotnyuk.jsoniter_scala.macros._
implicit val codec: JsonValueCodec[Seq[Data]] = JsonCodecMaker.make(CodecMakerConfig)
val json: Array[Byte] = """[
{
"id": 1248,
"default": false,
"name": null,
"number": 2
}
]""".getBytes("UTF-8")
val data: Seq[Data] = readFromArray(json)
println(data)
That will produce the following output:
List(Data(Some(Id(1248)),None,Some(Number(2))))
Here you can see an example how to integrate it with the Play framework.
Background
I'm working on an API using both Play Framework and Slick. In an effort to avoid repetitive boiler plate, I want to define my public JSON models without their ID field and wrap them in a WithId container.
import play.api.libs.json._
import play.api.libs.functional.syntax._
case class WithId[T](id: Long, item: T)
case class Wiki(name: String, source: Option[String], text: String)
object WithId {
implicit def withIdRead[T : Reads] : Reads[WithId[T]] = (
(JsPath \ "id").read[Long] and
JsPath.read[T]
)((id, item) => WithId(id, item))
implicit def withIdWrite[T : Writes] : Writes[WithId[T]] = (
(JsPath \ "id").write[Long] and
JsPath.write[T]
).apply(unlift(WithId.unapply[T]))
}
Thanks to the magic of the Reads and Writes definition, I can easily handle JSON with or without the id.
scala> val rawIdJson = """{"id": 123, "name": "My First Wiki", "text": "This is my first wiki article"}"""
rawIdJson: String = {"id": 123, "name": "My First Wiki", "text": "This is my first wiki article"}
scala> val withId = Json.parse(rawIdJson).validate[WithId[Wiki]].get
withId: model.util.WithId[model.entity.Wiki] = WithId(123,Wiki(My First Wiki,None,This is my first wiki article))
scala> val withIdJson = Json.toJson(withId)
withIdJson: play.api.libs.json.JsValue = {"id":123,"name":"My First Wiki","text":"This is my first wiki article"}
scala> val rawJson = """{"name": "My First Wiki", "text": "This is my first wiki article"}"""
rawJson: String = {"name": "My First Wiki", "text": "This is my first wiki article"}
scala> val withoutId = Json.parse(rawJson).validate[Wiki].get
withoutId: model.entity.Wiki = Wiki(My First Wiki,None,This is my first wiki article)
scala> val withoutIdJson = Json.toJson(withoutId)
withoutIdJson: play.api.libs.json.JsValue = {"name":"My First Wiki","text":"This is my first wiki article"}
All well and good.
The problem I now have is that Slick will return rows from the database in the form of tuples or case classes, depending on the query that I'm using. Obviously I could write a lot of pretty straight forward helper methods to transform the tuple/case class into each public model:
object Wiki {
implicit val wikiFmt = Json.format[Wiki]
def fromRow(row: WikiRow) : WithId[Wiki] = WithId(row.id, Wiki(row.name, row.source, row.text))
def fromRow(tup: (Long, String, Option[String], String)) : WithId[Wiki] = WithId(tup._1, Wiki(tup._2, tup._3, tup._4))
}
... but that is a lot of boilerplate to maintain as the number of public models grow.
Problem
Is there a clean way to take a Tuple4[Long, String, Option[String], String] or a case class WikiRow(id: Long, name: String, source: Option[String], text: String) and convert it into WithId[Wiki] (and vice-versa)?
Once I introduce another public model like case class Template(name: String, description: String), can we generalize the solution from #1 to now handle converting a Tuple3[Long, String, Strong] into WithId[Template] (and vice-versa)?
What happens if we throw a field into the private model that isn't used in the public model? Eg, case class WikiRow(id: Long, name: String, source: Option[String], text: String, hidden: Boolean). The hidden field needs to dropped when going WikiRow => WithId[Wiki], and supplied from another source when going WithId[Wiki] => WikiRow.
As for questions 1 and 2: yes it is possible with shapeless, as has been suggested. Here is a solution to go from a tuple or row to WithId[T].
scala> :paste
// Entering paste mode (ctrl-D to finish)
case class WikiRow(id: Long, name: String, source: Option[String], text: String)
case class Wiki(name: String, source: Option[String], text: String)
case class WithId[T](id: Long, item: T)
def createWithId[T] = new WithIdCreator[T]
class WithIdCreator[Out] {
import shapeless._
import shapeless.ops.hlist.IsHCons
def apply[In, InGen <: HList, Tail <: HList](in: In)(
implicit
genIn: Generic.Aux[In,InGen],
hcons: IsHCons.Aux[InGen,Long,Tail],
genOut: Generic.Aux[Out,Tail]
): WithId[Out] = {
val rep = genIn.to(in)
val id = hcons.head(rep)
val tail = hcons.tail(rep)
WithId(id, genOut.from(tail))
}
}
// Exiting paste mode, now interpreting.
defined class WikiRow
defined class Wiki
defined class WithId
createWithId: [T]=> WithIdCreator[T]
defined class WithIdCreator
scala> createWithId[Wiki](WikiRow(3L, "foo", None, "barbaz"))
res1: WithId[Wiki] = WithId(3,Wiki(foo,None,barbaz))
scala> createWithId[Wiki]((3L, "foo", None: Option[String], "barbaz"))
res2: WithId[Wiki] = WithId(3,Wiki(foo,None,barbaz))
scala> case class Template(name: String, description: String)
defined class Template
scala> createWithId[Template]((3L, "foo", "barbaz"))
res3: WithId[Template] = WithId(3,Template(foo,barbaz))
A conversion in the other direction will be more or less analogous.
I see no reason why 3 wouldn't be possible either, but then you will have to rewrite the conversions again to handle dropping or manually supplying parameters.
You can learn more about this in the shapeless guide. All the necessary concepts are explained in there.
For my project, I would like to make a tree model; let's say it's about files and directories. But files can be in multiple directories at the same time, so more like the same way you add tags to email in gmail.
I want to build a model for competences (say java, scala, angular, etc) and put them in categories. In this case java and scala are languages, agila and scrum are ways of working, angular is a framework / toolkit and so forth. But then we want to group stuff flexibly, ie play, java and scala are in a 'backend' category and angular, jquery, etc are in a frontend category.
I figured I would have a table competences like so:
case class Competence (name: String, categories: Option[Category])
and the categories as follows:
case class Category ( name: String, parent: Option[Category] )
This will compile, but SORM will generate an error (from activator console):
scala> import models.DB
import models.DB
scala> import models.Category
import models.Category
scala> import models.Competence
import models.Competence
scala> val cat1 = new Category ( "A", None )
cat1: models.Category = Category(A,None)
scala> val sav1 = DB.save ( cat1 )
sorm.Instance$ValidationException: Entity 'models.Category' recurses at 'models.Category'
at sorm.Instance$Initialization$$anonfun$2.apply(Instance.scala:216)
at sorm.Instance$Initialization$$anonfun$2.apply(Instance.scala:216)
at scala.Option.map(Option.scala:146)
at sorm.Instance$Initialization.<init>(Instance.scala:216)
at sorm.Instance.<init>(Instance.scala:38)
at models.DB$.<init>(DB.scala:5)
at models.DB$.<clinit>(DB.scala)
... 42 elided
Although I want the beautiful simplicity of sorm, will I need to switch to Slick for my project to implement this? I had the idea that link tables would be implicitly generated by sorm. Or could I simply work around the problem by making a:
case class Taxonomy ( child: Category, parent: Category )
and then do parsing / formatting work on the JS side? It seems to make the simplicity of using sorm disappear somewhat.
To give some idea, what I want is to make a ajaxy page where a user can add new competences in a list on the left, and then link/unlink them to whatever category tag in the tree he likes.
I encountered the same question. I needed to define an interation between two operant, which can be chained(recursive). Like:
case class InteractionModel(
val leftOperantId: Int,
val operation: String ,
val rightOperantId: Int,
val next: InteractionModel)
My working around: change this case class into Json(String) and persist it as String, when retreiving it, convert it from Json. And since it's String, do not register it as sorm Entity.
import spray.json._
case class InteractionModel(
val leftOperantId: Int,
val operation: String ,
val rightOperantId: Int,
val next: InteractionModel) extends Jsonable {
def toJSON: String = {
val js = this.toJson(InteractionModel.MyJsonProtocol.ImJsonFormat)
js.compactPrint
}
}
//define protocol
object InteractionModel {
def fromJSON(in: String): InteractionModel = {
in.parseJson.convertTo[InteractionModel](InteractionModel.MyJsonProtocol.ImJsonFormat)
}
val none = new InteractionModel((-1), "", (-1), null) {
override def toJSON = "{}"
}
object MyJsonProtocol extends DefaultJsonProtocol {
implicit object ImJsonFormat extends RootJsonFormat[InteractionModel] {
def write(im: InteractionModel) = {
def recWrite(i: InteractionModel): JsObject = {
val next = i.next match {
case null => JsNull
case iNext => recWrite(i.next)
}
JsObject(
"leftOperantId" -> JsNumber(i.leftOperantId),
"operation" -> JsString(i.operation.toString),
"rightOperantId" -> JsNumber(i.rightOperantId),
"next" -> next)
}
recWrite(im)
}
def read(value: JsValue) = {
def recRead(v: JsValue): InteractionModel = {
v.asJsObject.getFields("leftOperantId", "operation", "rightOperantId", "next") match {
case Seq(JsNumber(left), JsString(operation), JsNumber(right), nextJs) =>
val next = nextJs match {
case JsNull => null
case js => recRead(js)
}
InteractionModel(left.toInt, operation, right.toInt, next)
case s => InteractionModel.none
}
}
recRead(value)
}
}
}
}