Does anyone know how to parse DBObject to case class object using subset2 ? Super concise documentation doesn't help me :(
Consider following case class
case class MenuItem(id : Int, name: String, desc: Option[String], prices: Option[Array[String]], subitems: Option[Array[MenuItem]])
object MenuItem {
implicit val asBson = BsonWritable[MenuItem](item =>
{
val buf: DBObjectBuffer = DBO("id" -> item.id, "name" -> item.name)
item.desc match { case Some(value) => buf.append("desc" -> value) case None => }
item.prices match { case Some(value) => buf.append("prices" -> value) case None => }
item.subitems match { case Some(value) => buf.append("subitems" -> value) case None => }
buf()
}
)
}
and I wrote this parser
val menuItemParser: DocParser[MenuItem] = int("id") ~ str("name") ~ str("desc").opt ~ get[Array[String]]("prices").opt ~ get[Array[MenuItem]]("subitems").opt map {
case id ~ name ~ desc_opt ~ prices_opt ~ subitems => {
MenuItem(id, name, desc_opt, prices_opt, subitems)
}
}
It works if I remove last field subitems. But version shown above doesn't compile because MenuItem has field that references itself. It gives me following error
Cannot find Field for Array[com.borsch.model.MenuItem]
val menuItemParser: DocParser[MenuItem] = int("id") ~ str("name") ~ str("desc").opt ~ get[Array[String]]("prices").opt ~ get[Array[MenuItem]]("subitems").opt map {
^
It obviously doesn't compile because last get wants Field[MenuItem] implicit. But if I define it for MenuItem wouldn't it be pretty much copy-paste of DocParser[MenuItem] ?
How would you do it elegantly ?
I am an author of Subset (both 1.x and 2).
The README states that you need to have Field[T] per every T you would like to read (it's under "deserialization" section)
Just a side note. Frankly I don't find very logical to name a deserializer for MenuItem to be jodaDateTime.
Anyway Field[T] must translate from vanilla BSON types to your T. BSON cannot store MenuItem natively, see native BSON types here
But certainly the main problem is that you have a recursive data structure, so your "serializer" (BsonWritable) and "deserializer" (Field) must be recursive as well. Subset has implicit serializer/deserializer for List[T], but they require you to provide those for MenuItem : recursion.
To keep things short, I shall demonstrate you how you would write something like that for simpler "case class".
Suppose we have
case class Rec(id: Int, children: Option[List[Rec]])
Then the writer may look like
object Rec {
implicit object asBson extends BsonWritable[Rec] {
override def apply(rec: Rec) =
Some( DBO("id" -> rec.id, "children" -> rec.children)() )
}
Here, when you are writing rec.children into "DBObject", BsonWriteable[Rec] is being used and it requires "implicit" Field[Rec] in turn. So, this serializer is recursive.
As of the deserializer, the following will do
import DocParser._
implicit lazy val recField = Field({ case Doc(rec) => rec })
lazy val Doc: DocParser[Rec] =
get[Int]("id") ~ get[List[Rec]]("children").opt map {
case id ~ children => new Rec(id, children)
}
}
These are mutually recursive (remember to use lazy val!)
You would use them like so:
val dbo = DBO("y" -> Rec(123, Some(Rec(234, None) :: Nil))) ()
val Y = DocParser.get[Rec]("y")
dbo match {
case Y(doc) => doc
}
Related
There are many nice libraries for writing/reading Scala case classes to/from CSV files. I'm looking for something that goes beyond that, which can handle nested cases classes. For example, here a Match has two Players:
case class Player(name: String, ranking: Int)
case class Match(place: String, winner: Player, loser: Player)
val matches = List(
Match("London", Player("Jane",7), Player("Fred",23)),
Match("Rome", Player("Marco",19), Player("Giulia",3)),
Match("Paris", Player("Isabelle",2), Player("Julien",5))
)
I'd like to effortlessly (no boilerplate!) write/read matches to/from this CSV:
place,winner.name,winner.ranking,loser.name,loser.ranking
London,Jane,7,Fred,23
Rome,Marco,19,Giulia,3
Paris,Isabelle,2,Julien,5
Note the automated header line using the dot "." to form the column name for a nested field, e.g. winner.ranking. I'd be delighted if someone could demonstrate a simple way to do this (say, using reflection or Shapeless).
[Motivation. During data analysis it's convenient to have a flat CSV to play around with, for sorting, filtering, etc., even when case classes are nested. And it would be nice if you could load nested case classes back from such files.]
Since a case-class is a Product, getting the values of the various fields is relatively easy. Getting the names of the fields/columns does require using Java reflection.
The following function takes a list of case-class instances and returns a list of rows, each is a list of strings. It is using a recursion to get the values and headers of child case-class instances.
def toCsv(p: List[Product]): List[List[String]] = {
def header(c: Class[_], prefix: String = ""): List[String] = {
c.getDeclaredFields.toList.flatMap { field =>
val name = prefix + field.getName
if (classOf[Product].isAssignableFrom(field.getType)) header(field.getType, name + ".")
else List(name)
}
}
def flatten(p: Product): List[String] =
p.productIterator.flatMap {
case p: Product => flatten(p)
case v: Any => List(v.toString)
}.toList
header(classOf[Match]) :: p.map(flatten)
}
However, constructing case-classes from CSV is far more involved, requiring to use reflection for getting the types of the various fields, for creating the values from the CSV strings and for constructing the case-class instances.
For simplicity (not saying the code is simple, just so it won't be further complicated), I assume that the order of columns in the CSV is the same as if the file was produced by the toCsv(...) function above.
The following function starts by creating a list of "instructions how to process a single CSV row" (the instructions are also used to verify that the column headers in the CSV matches the the case-class properties). The instructions are then used to recursively produce one CSV row at a time.
def fromCsv[T <: Product](csv: List[List[String]])(implicit tag: ClassTag[T]): List[T] = {
trait Instruction {
val name: String
val header = true
}
case class BeginCaseClassField(name: String, clazz: Class[_]) extends Instruction {
override val header = false
}
case class EndCaseClassField(name: String) extends Instruction {
override val header = false
}
case class IntField(name: String) extends Instruction
case class StringField(name: String) extends Instruction
case class DoubleField(name: String) extends Instruction
def scan(c: Class[_], prefix: String = ""): List[Instruction] = {
c.getDeclaredFields.toList.flatMap { field =>
val name = prefix + field.getName
val fType = field.getType
if (fType == classOf[Int]) List(IntField(name))
else if (fType == classOf[Double]) List(DoubleField(name))
else if (fType == classOf[String]) List(StringField(name))
else if (classOf[Product].isAssignableFrom(fType)) BeginCaseClassField(name, fType) :: scan(fType, name + ".")
else throw new IllegalArgumentException(s"Unsupported field type: $fType")
} :+ EndCaseClassField(prefix)
}
def produce(instructions: List[Instruction], row: List[String], argAccumulator: List[Any]): (List[Instruction], List[String], List[Any]) = instructions match {
case IntField(_) :: tail => produce(tail, row.drop(1), argAccumulator :+ row.head.toString.toInt)
case StringField(_) :: tail => produce(tail, row.drop(1), argAccumulator :+ row.head.toString)
case DoubleField(_) :: tail => produce(tail, row.drop(1), argAccumulator :+ row.head.toString.toDouble)
case BeginCaseClassField(_, clazz) :: tail =>
val (instructionRemaining, rowRemaining, constructorArgs) = produce(tail, row, List.empty)
val newCaseClass = clazz.getConstructors.head.newInstance(constructorArgs.map(_.asInstanceOf[AnyRef]): _*)
produce(instructionRemaining, rowRemaining, argAccumulator :+ newCaseClass)
case EndCaseClassField(_) :: tail => (tail, row, argAccumulator)
case Nil if row.isEmpty => (Nil, Nil, argAccumulator)
case Nil => throw new IllegalArgumentException("Not all values from CSV row were used")
}
val instructions = BeginCaseClassField(".", tag.runtimeClass) :: scan(tag.runtimeClass)
assert(csv.head == instructions.filter(_.header).map(_.name), "CSV header doesn't match target case-class fields")
csv.drop(1).map(row => produce(instructions, row, List.empty)._3.head.asInstanceOf[T])
}
I've tested this using:
case class Player(name: String, ranking: Int, price: Double)
case class Match(place: String, winner: Player, loser: Player)
val matches = List(
Match("London", Player("Jane", 7, 12.5), Player("Fred", 23, 11.1)),
Match("Rome", Player("Marco", 19, 13.54), Player("Giulia", 3, 41.8)),
Match("Paris", Player("Isabelle", 2, 31.7), Player("Julien", 5, 16.8))
)
val csv = toCsv(matches)
val matchesFromCsv = fromCsv[Match](csv)
assert(matches == matchesFromCsv)
Obviously this should be optimized and hardened if you ever want to use this for production...
Some nested case classes and the field addresses is a Seq[Address]:
// ... means other fields
case class Street(name: String, ...)
case class Address(street: Street, ...)
case class Company(addresses: Seq[Address], ...)
case class Employee(company: Company, ...)
I have an employee:
val employee = Employee(Company(Seq(
Address(Street("aaa street")),
Address(Street("bbb street")),
Address(Street("bpp street")))))
It has 3 addresses.
And I want to capitalize the streets start with "b" only. My code is mess like following:
val modified = employee.copy(company = employee.company.copy(addresses =
employee.company.addresses.map { address =>
address.copy(street = address.street.copy(name = {
if (address.street.name.startsWith("b")) {
address.street.name.capitalize
} else {
address.street.name
}
}))
}))
The modified employee is then:
Employee(Company(List(
Address(Street(aaa street)),
Address(Street(Bbb street)),
Address(Street(Bpp street)))))
I'm looking for a way to improve it, and can't find one. Even tried Monocle, but can't apply it to this problem.
Is there any way to make it better?
PS: there are two key requirements:
use only immutable data
don't lose other existing fields
As Peter Neyens points out, Shapeless's SYB works really nicely here, but it will modify all Street values in the tree, which may not always be what you want. If you need more control over the path, Monocle can help:
import monocle.Traversal
import monocle.function.all._, monocle.macros._, monocle.std.list._
val employeeStreetNameLens: Traversal[Employee, String] =
GenLens[Employee](_.company).composeTraversal(
GenLens[Company](_.addresses)
.composeTraversal(each)
.composeLens(GenLens[Address](_.street))
.composeLens(GenLens[Street](_.name))
)
val capitalizer = employeeStreeNameLens.modify {
case s if s.startsWith("b") => s.capitalize
case s => s
}
As Julien Truffaut points out in an edit, you can make this even more concise (but less general) by creating a lens all the way to the first character of the street name:
import monocle.std.string._
val employeeStreetNameFirstLens: Traversal[Employee, Char] =
GenLens[Employee](_.company.addresses)
.composeTraversal(each)
.composeLens(GenLens[Address](_.street.name))
.composeOptional(headOption)
val capitalizer = employeeStreetNameFirstLens.modify {
case 'b' => 'B'
case s => s
}
There are symbolic operators that would make the definitions above a little more concise, but I prefer the non-symbolic versions.
And then (with the result reformatted for clarity):
scala> capitalizer(employee)
res3: Employee = Employee(
Company(
List(
Address(Street(aaa street)),
Address(Street(Bbb street)),
Address(Street(Bpp street))
)
)
)
Note that as in the Shapeless answer, you'll need to change your Employee definition to use List instead of Seq, or if you don't want to change your model, you could build that transformation into the Lens with an Iso[Seq[A], List[A]].
If you are open to replacing the addresses in Company from Seq to List, you can use "Scrap Your Boilerplate" from shapeless (example).
import shapeless._, poly._
case class Street(name: String)
case class Address(street: Street)
case class Company(addresses: List[Address])
case class Employee(company: Company)
val employee = Employee(Company(List(
Address(Street("aaa street")),
Address(Street("bbb street")),
Address(Street("bpp street")))))
You can create a polymorphic function which capitalizes the name of a Street if the name starts with a "b".
object capitalizeStreet extends ->(
(s: Street) => {
val name = if (s.name.startsWith("b")) s.name.capitalize else s.name
Street(name)
}
)
Which you can use as :
val afterCapitalize = everywhere(capitalizeStreet)(employee)
// Employee(Company(List(
// Address(Street(aaa street)),
// Address(Street(Bbb street)),
// Address(Street(Bpp street)))))
Take a look at quicklens
You could do it like this
import com.softwaremill.quicklens._
case class Street(name: String)
case class Address(street: Street)
case class Company(address: Seq[Address])
case class Employee(company: Company)
object Foo {
def foo(e: Employee) = {
modify(e)(_.company.address.each.street.name).using {
case name if name.startsWith("b") => name.capitalize
case name => name
}
}
}
I have a simple database consisting of 2 tables - movie and comment, where comments are related to movies, and then I have following piece of scala anorm code:
case class Comment(commentId: Long, comment: String)
case class Movie(movieId: Long, name: String, movieType: String)
object MovieDao {
val movieParser: RowParser[Movie] = {
long("movieId") ~
str("name") ~
str("movieType") map {
case movieId ~ name ~ movieType => Movie(movieId, name, movieType)
}
}
val commentParser: RowParser[Comment] = {
long("commentId") ~
str("comment") map {
case commentId ~ comment => Comment(commentId, comment)
}
}
def getAll(movieType: String) = DB.withConnection {
implicit connection =>
SQL(
"""
|SELECT
|movie.movieId,
|movie.name,
|movie.movieType,
|comment.commentId,
|comment.comment
|FROM movie
|LEFT JOIN comment USING(movieId)
|WHERE movieType = {movieType}
""".stripMargin)
.on("movieType" -> movieType)
.as(((movieParser ~ (commentParser ?)) map (flatten)) *)
.groupBy(_._1) map {(mc: (Movie, List[(Movie, Option[Comment])])) =>
mc match {
case (a, b) => (a, b filter { //filter rows with no comments
case (c, Some(d)) => true
case _ => false
} map(_._2))
}
} toList
}
}
My goal is to return List[(Movie, Option[List[Comment]])] from getAll method, so I can iterate over movies and check if there are any comments as simple as possible, e.i. match None or Some on comments List. I'm currently returning List[(Movie, Option[List[Option[Comment]])] and I'm only able to check size of comments List (thanks to using filter method), which I don't consider as the right way to do it in scala.
My second question is about parsing query itself, I think it's just to complicated the way I did it. Is there any simpler and nicer solution to parse 0..N relation using anorm?
Peter, it's possibly more style than anything dramatically different, but with a MovieComments case class, you could write something like:
case class MovieComments(movie: Movie, comments: List[Comment])
val movieCommentsP =
movieParser ~ (commentParser ?) map {
case movie ~ comment =>
MovieComments(movie,if (comment.isEmpty) List() else List(comment.get))
}
val movieSqlSelector = "m.movieId, m.name, m.movieType"
val commentSqlSelector = "c.commentId, c.comment"
def getAll(movieType: String) :List[MovieComments]= DB.withConnection {
implicit connection =>
(SQL(
s"""
|SELECT
|$movieSqlSelector,
|$commentSqlSelector
|FROM movie
|LEFT JOIN comment USING(movieId)
|WHERE movieType = {movieType}
""".stripMargin)
.on('movieType -> movieType)
.as(movieCommentsP *)
.groupBy(_.movie.movieId) map {
case (movieId,movieComments) =>
MovieComments(
movieComments.head.movie,
movieComments.flatMap(_.comments))
}
).toList
}
You may really need an Option[List[Comment]], but wouldn't a List[Comment] do? List() is the "no comment" case after all. (P.S. I find the use of sqlSelector variables helps with refactoring.)
As part of a macro, I want to manipulate the case definitions of a partial function.
To do so, I use a Transformer to manipulate the case definitions of the partial function and a Traverser to inspect the patterns of the case definitions:
def myMatchImpl[A: c.WeakTypeTag, B: c.WeakTypeTag](c: Context)
(expr: c.Expr[A])(patterns: c.Expr[PartialFunction[A, B]]): c.Expr[B] = {
import c.universe._
val transformer = new Transformer {
override def transformCaseDefs(trees: List[CaseDef]) = trees map {
case caseDef # CaseDef(pattern, guard , body) => {
// println(show(pattern))
val traverser = new Traverser {
override def traverse(tree: Tree) = tree match {
// match against a specific pattern
}
}
traverser.traverse(pattern)
}
}
}
val transformedPartialFunction = transformer.transform(patterns.tree)
c.Expr[B](q"$transformedPartialFunction($expr)")
}
Now let us assume, the interesting data I want to match against is represented by the class Data (which is part of the object Example):
case class Data(x: Int, y: String)
When now invoking the macro on the example below
abstract class Foo
case class Bar(data: Data) extends Foo
case class Baz(string: String, data: Data) extends Foo
def test(foo: Foo) = myMatch(foo){
case Bar(Data(x,y)) => y
case Baz(_, Data(x,y)) => y
}
the patterns of the case definitions of the partial function are transformed by the compiler as following (the Foo, Bar, and Baz classes are members of the object Example, too):
(data: Example.Data)Example.Bar((x: Int, y: String)Example.Data((x # _), (y # _)))
(string: String, data: Example.Data)Example.Baz(_, (x: Int, y: String)Example.Data((x # _), (y # _)))
This is the result of printing the patterns as hinted in the macro above (using show), the raw abstract syntax trees (printed using showRaw) look like this:
Apply(TypeTree().setOriginal(Select(This(newTypeName("Example")), Example.Bar)), List(Apply(TypeTree().setOriginal(Select(This(newTypeName("Example")), Example.Data)), List(Bind(newTermName("x"), Ident(nme.WILDCARD)), Bind(newTermName("y"), Ident(nme.WILDCARD))))))
Apply(TypeTree().setOriginal(Select(This(newTypeName("Example")), Example.Baz)), List(Ident(nme.WILDCARD), Apply(TypeTree().setOriginal(Select(This(newTypeName("Example")), Example.Data)), List(Bind(newTermName("x"), Ident(nme.WILDCARD)), Bind(newTermName("y"), Ident(nme.WILDCARD))))))
How do I write a pattern-quote which matches against these trees?
First of all, there is a special flavor of quasiquotes specifically for CaseDefs called cq:
override def transformCaseDefs(trees: List[CaseDef]) = trees map {
case caseDef # cq"$pattern if $guard => $body" => ...
}
Secondly, you should use pq to deconstruct patterns:
pattern match {
case pq"$name # $nested" => ...
case pq"$extractor($arg1, $arg2: _*)" => ...
...
}
If you are interested in internals of trees that are used for pattern matching they are created by patvarTransformer defined in TreeBuilder.scala
On the other hand if you're are working with UnApply trees (that are being produced after typechecking) I have bad news for you: quasiquotes currently don't support them. Follow SI-7789 to get notified when this is fixed.
After Den Shabalin pointed out, that quasiquotes can't be used in this particular setting, I managed to find a pattern which matches against the patterns of a partial function's case definitions.
The key problem is, that the constructor we want to match against (in our example Data) is stored in the TypeTree of the Apply node. Matching against a tree wrapped up in a TypeTree is a bit tricky, since the only extractor of this class (TypeTree()) isn't very helpful for this particular task. Instead we have to select the wrapped up tree using the original method:
override def transform(tree: Tree) = tree match {
case Apply(constructor # TypeTree(), args) => constructor.original match {
case Select(_, sym) if (sym == newTermName("Data")) => ...
}
}
In our use case the wrapped up tree is a Select node and we can now check if the symbol of this node is the one we are looking for.
I have a tree-like structure of abstract classes and case classes representing an Abstract Syntax Tree of a small language.
For the top abstract class i've implemented a method map:
abstract class AST {
...
def map(f: (AST => AST)): AST = {
val b1 = this match {
case s: STRUCTURAL => s.smap(f) // structural node for example IF(expr,truebranch,falsebranch)
case _ => this // leaf, // leaf, like ASSIGN(x,2)
}
f(b1)
}
...
The smap is defined like:
override def smap(f: AST => AST) = {
this.copy(trueb = trueb.map(f), falseb = falseb.map(f))
}
Now im writing different "transformations" to insert, remove and change nodes in the AST.
For example, remove adjacent NOP nodes from blocks:
def handle_list(l:List[AST]) = l match {
case (NOP::NOP::tl) => handle_list(tl)
case h::tl => h::handle_list(tl)
case Nil => Nil
}
ast.map {
case BLOCK(listofstatements) => handle_list(listofstatements)
}
If I write like this, I end up with MatchError and I can "fix it" by changing the above map to:
ast.map {
case BLOCK(listofstatements) => handle_list(listofstatements)
case a => a
}
Should I just live with all those case a => a or could I improve my map method(or other parts) in some way?
Make the argument to map a PartialFunction:
def map(f: PartialFunction[AST, AST]): AST = {
val idAST: PartialFunction[AST, AST] = {case a => a}
val g = f.orElse(idAST)
val b1 = this match {
case s: STRUCTURAL => s.smap(g)
case _ => this
}
g(b1)
}
If tree transformations are more than a minor aspect of your project, I highly recommend you use Kiama's Rewriter module to implement them. It implements Stratego-like strategy-driven transformations. It has a very rich set of strategies and strategy combinators that permit a complete separation of traversal logic (which for the vast majority of cases can be taken "off the shelf" from the supplied strategies and combinators) from (local) transformations (which are specific to your AST and you supply, of course).