I'm trying to marshall a case class in json which is easy enough with json4s.
case class container(id: String, `type`: Option[String], things: List[???])
Json4s does an excellent job of omitting fields that are null. The issues is I'd like options at the top level of container to be preserved but everything in things (it's filled with nested options) to preserve these empty options and serve them as Nulls.
The follow question pointed me in the right direction Json4s ignoring None fields during seriallization (instead of using 'null')
But using implicit val f = DefaultFormats.preservingEmptyValues targets all of the case classes in scope. Is there anyway I can just target fields within things. I can't specifically target fields as there's quite a few and doing this could become unmanageable.
Thanks in advance
Related
Following is my use case
I am using Cats for validation of my config. My config file is in json.
I deserialize my config file to my case class Config using lift-json and then validate it using Cats. I am using this as a guide.
My motive for using Cats is to collect all errors iff present at time of validation.
My problem is the examples given in the guide, are of the type
case class Person(name: String, age: Int)
def validatePerson(name: String, age: Int): ValidationResult[Person] = {
(validateName(name),validate(age)).mapN(Person)
}
But in my case I already deserialized my config into my case class ( below is a sample ) and then I am passing it for validation
case class Config(source: List[String], dest: List[String], extra: List[String])
def vaildateConfig(config: Config): ValidationResult[Config] = {
(validateSource(config.source), validateDestination(config.dest))
.mapN { case _ => config }
}
The difference here is mapN { case _ => config }. As I already have a config if everything is valid I dont want to create the config anew from its members. This arises as I am passing config to validate function not it's members.
A person at my workplace told me this is not the correct way, as Cats Validated provides a way to construct an object if its members are valid. The object should not exist or should not be constructible if its members are invalid. Which makes complete sense to me.
So should I make any changes ? Is the above I'm doing acceptable ?
PS : The above Config is just an example, my real config can have other case classes as its members which themselves can depend on other case classes.
One of the central goals of the kind of programming promoted by libraries like Cats is to make invalid states unrepresentable. In a perfect world, according to this philosophy, it would be impossible to create an instance of Config with invalid member data (through the use of a library like Refined, where complex constraints can be expressed in and tracked by the type system, or simply by hiding unsafe constructors). In a slightly less perfect world, it might still be possible to construct invalid instances of Config, but discouraged, e.g. through the use of safe constructors (like your validatePerson method for Person).
It sounds like you're in an even less perfect world where you have instances of Config that may or may not contain invalid data, and you want to validate them to get "new" instances of Config that you know are valid. This is totally possible, and in some cases reasonable, and your validateConfig method is a perfectly legitimate way to solve this problem, if you're stuck in that imperfect world.
The downside, though, is that the compiler can't track the difference between the already-validated Config instances and the not-yet-validated ones. You'll have Config instances floating around in your program, and if you want to know whether they've already been validated or not, you'll have to trace through all the places they could have come from. In some contexts this might be just fine, but for large or complex programs it's not ideal.
To sum up: ideally you'd validate Config instances whenever they are created (possibly even making it impossible to create invalid ones), so that you don't have to remember whether any given Config is good or not—the type system can remember for you. If that's not possible, because of e.g. APIs or definitions you don't control, or if it just seems too burdensome for a simple use case, what you're doing with validateConfig is totally reasonable.
As a footnote, since you say above that you're interested in looking in more detail at Refined, what it provides for you in a situation like this is a way to avoid even more functions of the shape A => ValidationResult[A]. Right now your validateName method, for example, probably takes a String and returns a ValidationResult[String]. You can make exactly the same argument against this signature as I have against Config => ValidationResult[Config] above—once you're working with the result (by mapping a function over the Validated or whatever), you just have a string, and the type doesn't tell you that it's already been validated.
What Refined allows you to do is write a method like this:
def validateName(in: String): ValidationResult[Refined[String, SomeProperty]] = ...
…where SomeProperty might specify a minimum length, or the fact that the string matches a particular regular expression, etc. The important point is that you're not validating a String and returning a String that only you know something about—you're validating a String and returning a String that the compiler knows something about (via the Refined[A, Prop] wrapper).
Again, this may be (okay, probably is) overkill for your use case—you just might find it nice to know that you can push this principle (tracking validation in types) even further down through your program.
I have a 3-level nested case class model with a bunch of options that represents some data in a database. It's essentially:
case class User(settings: Option[Settings])
case class Settings(keys: Option[List[KeySet]])
case class KeySet(privateKey: String, publicKey: String)
I understand how to get deeply nested fields out of this using some for comprehension or chains of flatMap (Scala Option object inside another Option object) and I also understand how to update it using a lens library, but I want to figure out how to update the fields even if some stuff in the tree is None and automatically make Somes of those if they don't exist yet.
For example, how would I handle the case where I want to add to the keys List but the user hasn't yet set any settings? Is it possible to, in some sense, automatically create a Some(settings) field and also a Some(keys) field?
I have an idea of how to do it with a lot of pattern matching, but this seems wrong because of 1. rightward drift of code and 2. not using map or flatMap very much with the options.
Is this possible using a lens library on its own? I read here that it might not be possible: https://github.com/julien-truffaut/Monocle/issues/215 as in the case of Monocle it can't update an Option that is a None. Maybe I need to think about the problem another way?
Thanks
I'm not sure why you use Option[List[KeySet]]. Is there an important distinction between None and an empty List?
In any case, I find fold to be a handy tool when working with Options.
def updateUser(u :User, ks :KeySet) :User = {
u.copy(settings =
Some(u.settings.fold(Settings(Some(ks::Nil))) (stngs =>
stngs.copy(keys = Some(stngs.keys.fold(ks::Nil) (ks::_))))))
}
val pat = updateUser(User(None), KeySet("a","b"))
//pat: User = User(Some(Settings(Some(List(KeySet(a,b))))))
val upat = updateUser(pat, KeySet("c","d"))
//upat: User = User(Some(Settings(Some(List(KeySet(c,d), KeySet(a,b))))))
Following on form this excellent set of answers on how to define union types in Scala. I've been using the Miles Sabin definition of Union types, but one questions remains.
How do you work with these if the type isn't know until Runtime? For example:
trait inv[-A] {}
type Or[A,B] = {
type check[X] = (inv[A] with inv[B]) <:< inv[X]
}
case class Foo[A : (Int Or String)#check](a: A)
Foo(1) // Foo[Int] = Foo(1)
Foo("hi") // Foo[String] = Foo(hi)
Foo(2.0) // Error!
This example works since the parameter A is know at compile time, and calling Foo(1) is really calling Foo[Int](1). However, what do you do if parameter A isn't known until runtime? Maybe you're paring a file that contains the data for Foo's, in which case the type parameter of Foo isn't know until you read the data. There's no easy way to set parameter A in this case.
The best solutions I've been able to come up with are:
Pattern Match on the data you've read and then create different Foo's based that type. In my case this isn't feasible because my case-class actually contains dozens of union types, so there'd be hundreds of combinations of types to pattern match.
Cast the type you've just read to be (String or Int), so you have a single type to pass around, that passes the Type Class constraint when you create Foo with it. Then return Foo[_] instead. This puts the onus back on the Foo user to work out the type of each field (since they'll appear to be type Any), but at least it defers having to know the type until the field is actually used, in which case a pattern match seems more tractable.
The second solution looks like this:
def parseLine: Any // Parses data point, but can be either a String or
// Int, so returns Any.
def mkFoo: Foo[_] = {
val a = parseLine.asInstanceOf[Int with String]
Foo(a) // Passes type constraint now
}
In practice I've ended up using the second solution, but I'm wondering if there's something better I can do?
Another way to state the problem is: What does it mean to return a Union Type? Functions can only return a single type, and the trickery we use with Miles Sabin union types is only useful for the types you pass in, not for the types you return.
PS. For context, why this is a problem in my case is that I'm generating a set of case-classes from a Json schema file. Json naturally supports union types, so I would like to make my case classes reflect that too. This works great in one direction: users creating case-classes to be serialized out to Json. But gets sticky in the other direction: user's parsing Json files to have a set of populated case classes returned to them.
The "standard" Scala solution to this problem is to use an ordinary discriminated-union type (ie, to forego true union types altogether):
sealed trait Foo
case class IntFoo(x: Int) extends Foo
case class StringFoo(x: String) extends Foo
This reflects the fact that, as you observe, the particular type of the member is a runtime value; the JVM type-tag of the Foo instance provides this runtime value.
Miles Sabin's implementation of union types is very clever, but I'm not sure if it provides any practical benefit, because it only restricts the type of thing that can go into a Foo, but provides the user of a Foo with no computable version of that restriction, in the way a match provides you with a computable version of the sealed trait. In general, for a restriction to be useful, it needs two sides: a check that only the right things are put in, and an extractor (aka an eliminator) that allows the same right things to come out the other end.
Perhaps if you gave some explanation of why you're looking for a purer union type it would illuminate whether regular discriminated unions are sufficient or if you really need something more.
There's a reason every JSON parser for Scala requires well defined types into which the JSON will be converted, even if some fields have to be dropped: you cannot work with something you don't know the type of.
To given an example, say you have a, and maybe a is a String, maybe it's an Int, but you don't know what it is. Why computation could you possibly make with a, not knowing its type? Why would your code compute the sum of all a's, for instance, if you didn't know in advance it was a number?
Generally, the answer to that is to perform user-provided data manipulation at runtime over data with unknown characteristics, as the user itself sees that it's a number and decides they want to know what the sum of that field is. Fine, but you are going the wrong way about it if so.
There is a well defined way to represent JSON data in Scala (and, for that matter, any data that has the same characteristics as JSON. Which is using a hierarchy of classes. A json value may be a json object, array or one of a number of primitives. A json object contains a list of key/value pairs, whose keys are json strings and values are json values. And so on. This is easy to represent, and there are many library doing so already. In fact, there are so many that there's a project called Json4s which presents a unified API which can be used and is implemented by many of the aforementioned libraries.
Things like the records which Miles Sabin's Shapeless library provide are intended to be used when the input doesn't have a well defined schema, but the program knows what it needs from that input. And, yes, the program might know what to do with a if it is an Int or a String, but not every possible value.
The next Scala 3 (mid 2020) based on Dotty will implement the proposal for Union Type from last Sept. 2018
You see it in "a tour of Scala 3" (June 2019)
Union Types Provide ad-hoc combinations of types
Subsetting = Subtyping
No boxing overhead
case class UserName(name: String)
case class Password(hash: Hash)
def help(id: UserName | Password) = {
val user = id match {
case UserName(name) => lookupName(name)
case Password(hash) => lookupPassword(hash)
}
...
}
Union Types Work also with singleton types
Great for JS interop
type Command = "Click" | "Drag" | "KeyPressed"
def handleEvent(kind: Command) = kind match {
case "Click" => MouseClick()
case "Drag" => MoveTo()
case "KeyPressed" => KeyPressed()
}
As an overview, I am trying to dynamically create a constructor for a case class from a Cassandra Java Row using reflection to find the primary constructor for the case class, and then trying to extract the values from the Cassandra Row.
Specifically, I want to support an Option in a case class as being an optional field in the Row, such that
case class Person(name: String, age: Option[Int])
will successfully populate if the Row has a name and an age, or just the name (and fill in a None for age).
To this end, I followed this very helpful blog post that achieves a similar objective between Case Classes and Maps.
However, I seem to be stuck trying to consolidate the dynamic nature of reflectively extracting types from the Case Class and the compile-time nature of quasiquotes. As an example:
I have a type fieldType which could be a native type or an Option of a native type. If it is an Option, I want to pass returnType.typeArgs.head to my quasiquote construction, so that it can extract the parameterized type from the Row, and if it is not an Option, I will just pass returnType.
if (fieldType <:< typeOf[Option[_]])
q"r.getAs[${returnType.typeArgs.head}]($fieldName)"
else
q"r.as[$returnType]($fieldName)"
(assuming r is a Cassandra Row and as and getAs exist for this Row)
When I try to compile this, I get an error saying that it does not know how to deal with doing r.as[Option[String]]. This makes conceptual sense to me because there is no way the compiler would know which way the runtime comparison will resolve and so needs to check both cases.
So how might I go about making this type check? If I could maybe compare the types fieldType and typeOf[Option[_]] within the quasiquote, it might stop complaining, but I can't figure out how to compare types in a quasiquote, and I'm not sure it's even possible. If I could extract the parameterized type of the Option within the quasiquote, it might stop complaining, but I could not figure that out either.
Sorry, I am very new to Scala and this stuff is, at the moment, very confusing and esoteric to me. If you want to look more closely at what I am doing, I have a repo: https://github.com/thurstonsand/scala-cass/blob/master/src/main/scala/com/weather/scalacass/ScalaCass.scala
where the interesting part is ScalaCass.CaseClassRealizer, and I am testing it in CaseClassUnitTests.
I found help from #liff on the gitter scala/scala page.
Apparently, I was finding my fieldType incorrectly.
I was doing: val fieldType = tpe.decl(encodedName).typeSignature where I should have been doing val fieldType = field.infoIn(tpe). Will update once I know what this difference means.
Motivations
This question is about working with Lists of data in Scala, and about resorting to either tuples or class objects for holding data. Perhaps some of my assumptions are wrong, so there it goes.
My current approach
As I understand, tuples do not afford the possibility of elegantly addressing their elements beyond the provided ._1, ._2, etc. I can use them, but code will be a bit unpleasant wherever data is extracted far from the lines of code that had defined it.
Also, as I understand, a Scala Map can only use a single type declaration for its values, so it can't diversify the value type of its values except for the case of type inheritance. (to the later point, considering the use of a type hierarchy for Map values "diversity" - may seem to be very artificial unless a class hierarchy fits any "model" intuition to begin with).
So, when I need to have lists where each element contains two or more named data entities, e.g. as below one of type String and one of type List, each accessible through an intelligible name, I resort to:
case class Foo (name1: String, name2: List[String])
val foos: List[Foo] = ...
Then I can later access instances of the list using .name1 and .name2.
Shortcomings and problems I see here
When the list is very large, should I assume this is less performant or more memory consuming than using a tuple as the List's type? alternatively, is there a different elegant way of accomplishing struct semantics in Scala?
In terms of performance, I don't think there is going to be any distinction between a tuple and an instance of a cases class. In fact, a tuple is an instance of a case class.
Secondly, if you're looking for another, more readable way to get the data out of the tuple, I suggest you consider pattern matching:
val (name1, name2) = ("first", List("second", "third"))