Mapping a class as a column type within a class - scala

I understand how this is done when using types such as Long, Int, String etc.. But say I have a class that has fields within another class like so:
case class Foo(a:String, b:String)
case class Bar(foo:Option[Foo], c:String)
How would I set up a mapper for my custom type (the Foo in my Bar class)?
class Bars(tag:Tag) extends Table[Bar](tag, "BARS") {
def foo = column[Foo]("FOO") // <- won't work
def c = column[String]("C")
def * = (foo, c) <> (Bar.tupled, Bar.unapply)
}
(documentation link)
Update:
DB Driver: slick.driver.PostgresDriver
Slick 2
I'm guessing the raw SQL would look like this:
"BARS" (
"A" VARCHAR(254) NOT NULL,
"B" VARCHAR(254) NOT NULL,
"C" VARCHAR(254) NOT NULL
);
Should be able to call Bar like so:
val bar = Bar(Foo("1", "2"), "3")
barTable.insert(bar)
bar.foo.a // 1
bar.foo.b // 2
bar.c // 3

You can write a mapper between the case class and some type that can be stored in the database.
See an example from Slick here:http://slick.typesafe.com/doc/1.0.0/lifted-embedding.html, at the end of the page.
One easy way in your case might be to transform your case class into json and store as a string. (And if your DB supports json type directly, like PostgreSQL, you can specify JSON type in column mapper, that would give you an advantage when making queries related to the content of your case classes.)
import org.json4s._
import org.json4s.native.Serialization
import org.json4s.native.Serialization.{read, write}
//place this in scope of your table definition
implicit val FooTypeMapper = MappedTypeMapper.base[Foo, String](
{ f => write(f) }, // map Foo to String
{ s => read[Too](s) } // map String to Foo
)
class Bars(tag:Tag) extends Table[Bar](tag, "BARS") {
def foo = column[Foo]("FOO") // <- should work now
def c = column[String]("C")
def * = (foo, c) <> (Bar.tupled, Bar.unapply)
}
With PostgreSQL >=9.3, you can also write:
def foo = column[Foo]("FOO", O.DBType("json"))
So that DB treats your json properly.
UPDATE: there is a connection property that should be set if send a String for a JSON field. Something like this:
val prop = new java.util.Properties
prop.setProperty("stringtype", "unspecified")
val db = Database.forURL(<db-uri>,
driver="org.postgresql.Driver",
user=<username>,
password=<password>,
prop=prop)

You need to provide columns for A and B in the Bars table:
class Bars(tag:Tag) extends Table[Bar](tag, "BARS") {
def a = column[String]("A")
def b = column[String]("B")
def foo = (a, b) <> (Foo.tupled, Foo.unapply)
def c = column[String]("C")
def * = (foo, c) <> (Bar.tupled, Bar.unapply)
}

Related

Is there any way to specify type in scala dynamically

I'm new in Spark, Scala, so sorry for stupid question. So I have a number of tables:
table_a, table_b, ...
and number of corresponding types for these tables
case class classA(...), case class classB(...), ...
Then I need to write a methods that read data from these tables and create dataset:
def getDataFromSource: Dataset[classA] = {
val df: DataFrame = spark.sql("SELECT * FROM table_a")
df.as[classA]
}
The same for other tables and types. Is there any way to avoid routine code - I mean individual fucntion for each table and get by with one? For example:
def getDataFromSource[T: Encoder](table_name: String): Dataset[T] = {
val df: DataFrame = spark.sql(s"SELECT * FROM $table_name")
df.as[T]
}
Then create list of pairs (table_name, type_name):
val tableTypePairs = List(("table_a", classA), ("table_b", classB), ...)
Then to call it using foreach:
tableTypePairs.foreach(tupl => getDataFromSource[what should I put here?](tupl._1))
Thanks in advance!
Something like this should work
def getDataFromSource[T](table_name: String, encoder: Encoder[T]): Dataset[T] =
spark.sql(s"SELECT * FROM $table_name").as(encoder)
val tableTypePairs = List(
"table_a" -> implicitly[Encoder[classA]],
"table_b" -> implicitly[Encoder[classB]]
)
tableTypePairs.foreach {
case (table, enc) =>
getDataFromSource(table, enc)
}
Note that this is a case of discarding a value, which is a bit of a code smell. Since Encoder is invariant, tableTypePairs isn't going to have that useful of a type, and neither would something like
tableTypePairs.map {
case (table, enc) =>
getDataFromSource(table, enc)
}
One option is to pass the Class to the method, this way the generic type T will be inferred:
def getDataFromSource[T: Encoder](table_name: String, clazz: Class[T]): Dataset[T] = {
val df: DataFrame = spark.sql(s"SELECT * FROM $table_name")
df.as[T]
}
tableTypePairs.foreach { case (table name, clazz) => getDataFromSource(tableName, clazz) }
But then I'm not sure of how you'll be able to exploit this list of Dataset without .asInstanceOf.

How to dynamically create an Enum type in Scala?

I have a basic enum type Currency that will include all major currencies traded e.g. EUR, USD, JPY, etc. This code I can write or generate one time. However, I'd also like to have strong enum type for all currency pair combinations e.g. EURCHF, USDCHF, etc. Is there any provision in Scala that would allow me to build such a derived enum type dynamically? I could also do it with some script generator from outside ... but I wonder whether it would be possible.
object Ccy extends Enumeration {
type Type = Value
val USD = Value("USD")
val CHF = Value("CHF")
val EUR = Value("EUR")
val GBP = Value("GBP")
val JPY = Value("JPY")
}
object CcyPair extends Enumeration {
type Type = Value
// ??? Ccy.values.toSeq.combinations(2) ...
}
UPDATE using the accepted answer as reference this was my solution implementation:
import scala.language.dynamics
object CcyPair extends Enumeration with Dynamic {
type Type = Value
/*
* contains all currency combinations including the symmetric AB and BA
*/
private val byCcy: Map[(Ccy.Value, Ccy.Value), Value] =
Ccy.values.toSeq.combinations(2).map { case Seq(c1, c2) =>
Seq(
(c1, c2) -> Value(c1.toString + c2.toString),
(c2, c1) -> Value(c2.toString + c1.toString)
)
}.flatten.toMap
/**
* reverse lookup to find currencies by currency pair, needed to find
* the base and risk components.
*/
private val revByCcy = byCcy.toSeq.map { case (((ccyRisk, ccyBase), ccyPair)) =>
ccyPair -> (ccyRisk, ccyBase)
}.toMap
def apply(ccy1: Ccy.Value, ccy2: Ccy.Value): Value = {
assert(ccy1 != ccy2, "currencies should be different")
byCcy((ccy1, ccy2))
}
implicit class DecoratedCcyPair(ccyPair: CcyPair.Type) {
def base: Ccy.Type = {
revByCcy(ccyPair)._1
}
def risk: Ccy.Type = {
revByCcy(ccyPair)._2
}
def name: String = ccyPair.toString()
}
def selectDynamic(ccyPair: String): Value = withName(ccyPair)
}
and then I can do things like:
val ccyPair = CcyPair.EURUSD
// or
val ccyPair = CcyPair(Ccy.EUR, Ccy.USD)
// and then do
println(ccyPair.name)
// and extract their parts like:
// print the base currency of the pair i.e. EUR
println(CcyPair.EURUSD.base)
// print the risk currency of the pair i.e. USD
println(CcyPair.EURUSD.risk)
There is no magic in Scala's Enumeration. The call to the Value function inside simply does some modifications to Enumeration's internal mutable structures. So you just have to call Value for each pair of currencies. The following code will work:
object CcyPair1 extends Enumeration {
Ccy.values.toSeq.combinations(2).foreach {
case Seq(c1, c2) =>
Value(c1.toString + c2.toString)
}
}
It's not very comfortable to work with though. You can access the values only through withName or values functions.
scala> CcyPair1.withName("USDEUR")
res20: CcyPair1.Value = USDEUR
But it's possible to extend this definition, for example, to allow retrieving CcyPair.Value by a pair of Ccy.Values, or to allow access by object fields with Dynamic, or to provide other facilities you may need:
import scala.language.dynamics
object CcyPair2 extends Enumeration with Dynamic {
val byCcy: Map[(Ccy.Value, Ccy.Value), Value] =
Ccy.values.toSeq.combinations(2).map {
case Seq(c1, c2) =>
(c1, c2) -> Value(c1.toString + c2.toString)
}.toMap
def forCcy(ccy1: Ccy.Value, ccy2: Ccy.Value): Value = {
assert(ccy1 != ccy2, "currencies should be different")
if (ccy1 < ccy2) byCcy((ccy1, ccy2))
else byCcy((ccy2, ccy1))
}
def selectDynamic(pairName: String): Value =
withName(pairName)
}
This definition is a bit more useful:
scala> CcyPair2.forCcy(Ccy.USD, Ccy.EUR)
res2: CcyPair2.Value = USDEUR
scala> CcyPair2.forCcy(Ccy.EUR, Ccy.USD)
res3: CcyPair2.Value = USDEUR
scala> CcyPair2.USDCHF
res4: CcyPair2.Value = USDCHF

Allocation of Function Literals in Scala

I have a class that represents sales orders:
class SalesOrder(val f01:String, val f02:Int, ..., f50:Date)
The fXX fields are of various types. I am faced with the problem of creating an audit trail of my orders. Given two instances of the class, I have to determine which fields have changed. I have come up with the following:
class SalesOrder(val f01:String, val f02:Int, ..., val f50:Date){
def auditDifferences(that:SalesOrder): List[String] = {
def diff[A](fieldName:String, getField: SalesOrder => A) =
if(getField(this) != getField(that)) Some(fieldName) else None
val diffList = diff("f01", _.f01) :: diff("f02", _.f02) :: ...
:: diff("f50", _.f50) :: Nil
diffList.flatten
}
}
I was wondering what the compiler does with all the _.fXX functions: are they instanced just once (statically), and can be shared by all instances of my class, or will they be instanced every time I create an instance of my class?
My worry is that, since I will use a lot of SalesOrder instances, it may create a lot of garbage. Should I use a different approach?
One clean way of solving this problem would be to use the standard library's Ordering type class. For example:
class SalesOrder(val f01: String, val f02: Int, val f03: Char) {
def diff(that: SalesOrder) = SalesOrder.fieldOrderings.collect {
case (name, ord) if !ord.equiv(this, that) => name
}
}
object SalesOrder {
val fieldOrderings: List[(String, Ordering[SalesOrder])] = List(
"f01" -> Ordering.by(_.f01),
"f02" -> Ordering.by(_.f02),
"f03" -> Ordering.by(_.f03)
)
}
And then:
scala> val orderA = new SalesOrder("a", 1, 'a')
orderA: SalesOrder = SalesOrder#5827384f
scala> val orderB = new SalesOrder("b", 1, 'b')
orderB: SalesOrder = SalesOrder#3bf2e1c7
scala> orderA diff orderB
res0: List[String] = List(f01, f03)
You almost certainly don't need to worry about the perfomance of your original formulation, but this version is (arguably) nicer for unrelated reasons.
Yes, that creates 50 short lived functions. I don't think you should be worried unless you have manifest evidence that that causes a performance problem in your case.
But I would define a method that transforms SalesOrder into a Map[String, Any], then you would just have
trait SalesOrder {
def fields: Map[String, Any]
}
def diff(a: SalesOrder, b: SalesOrder): Iterable[String] = {
val af = a.fields
val bf = b.fields
af.collect { case (key, value) if bf(key) != value => key }
}
If the field names are indeed just incremental numbers, you could simplify
trait SalesOrder {
def fields: Iterable[Any]
}
def diff(a: SalesOrder, b: SalesOrder): Iterable[String] =
(a.fields zip b.fields).zipWithIndex.collect {
case ((av, bv), idx) if av != bv => f"f${idx + 1}%02d"
}

Scala Macros: Checking for a certain annotation

Thanks to the answers to my previous question, I was able to create a function macro such that it returns a Map that maps each field name to its value of a class, e.g.
...
trait Model
case class User (name: String, age: Int, posts: List[String]) extends Model {
val numPosts: Int = posts.length
...
def foo = "bar"
...
}
So this command
val myUser = User("Foo", 25, List("Lorem", "Ipsum"))
myUser.asMap
returns
Map("name" -> "Foo", "age" -> 25, "posts" -> List("Lorem", "Ipsum"), "numPosts" -> 2)
This is where Tuples for the Map are generated (see Travis Brown's answer):
...
val pairs = weakTypeOf[T].declarations.collect {
case m: MethodSymbol if m.isAccessor =>
val name = c.literal(m.name.decoded)
val value = c.Expr(Select(model, m.name))
reify(name.splice -> value.splice).tree
}
...
Now I want to ignore fields that have #transient annotation. How would I check if a method has a #transient annotation?
I'm thinking of modifying the snippet above as
val pairs = weakTypeOf[T].declarations.collect {
case m: MethodSymbol if m.isAccessor && !m.annotations.exists(???) =>
val name = c.literal(m.name.decoded)
val value = c.Expr(Select(model, m.name))
reify(name.splice -> value.splice).tree
}
but I can't find what I need to write in exists part. How would I get #transient as an Annotation so I could pass it there?
Thanks in advance!
The annotation will be on the val itself, not on the accessor. The easiest way to access the val is through the accessed method on MethodSymbol:
def isTransient(m: MethodSymbol) = m.accessed.annotations.exists(
_.tpe =:= typeOf[scala.transient]
)
Now you can just write the following in your collect:
case m: MethodSymbol if m.isAccessor && !isTransient(m) =>
Note that the version of isTransient I've given here has to be defined in your macro, since it needs the imports from c.universe, but you could factor it out by adding a Universe argument if you're doing this kind of thing in several macros.

Deep access of fields in Scala using runtime reflection

I have code that deeply walks a case class' constructor fields, which of course may themselves be complex (list of things, maps, options, and other case classes). The code I found to extract field values at runtime works great on the highest-level fields but explodes when I try to access deeper fields. Example below.
I real life my application introspects the fields at each level, so I know that 'stuff' is another case class (I have the Symbol/Type), and I know Dos' field Symbols/Types. But this is obtained at runtime so I think it's blowing up because it doesn't know [T]/Manifest[T]. Is there a way to get this at runtime via reflection? How might my code change? The examples I found seemed to all require various things[T], which I wouldn't have for 'dos', right?
case class Uno( name:String, age:Int, pets:List[String], stuff:Dos )
case class Dos( foo:String )
object Boom extends App {
val ru = scala.reflect.runtime.universe
val m = ru.runtimeMirror(getClass.getClassLoader)
val u = Uno("Marcus",19,List("fish","bird"),Dos("wow"))
println("NAME: "+unpack(u,"name")) // Works
println("PETS: "+unpack(u,"pets")) // Works
// ----- Goes Boom -------
val dos = unpack(u,"stuff")
println("Other: "+unpack(dos,"foo")) // Boom!
// -----------------------
// Get object value for named parameter of target
def unpack[T]( target:T, name:String )(implicit man:Manifest[T]) : Any = {
val im = m.reflect(target)
val fieldX = ru.typeOf[T].declaration(ru.newTermName(name)).asTerm.accessed.asTerm
im.reflectField(fieldX).get
}
}
You're exactly right, the type of your dos is Any.
FieldMirror.symbol.typeSignature is what you'd get from typeOf[Dos].
So consider returning a pair (Any, Type) from unpack to have something to pass to unpack(target, type, name). Somewhat like:
case class Uno(name: String, age: Int, pets: List[String], stuff: Dos)
case class Dos(foo: String)
object Boom extends App {
import scala.reflect.runtime.universe._
import scala.reflect.runtime.{ currentMirror => cm }
import scala.reflect.ClassTag
val u = Uno("Marcus", 19, List("fish", "bird"), Dos("wow"))
println("NAME: " + unpack(u, "name")) // Works
println("PETS: " + unpack(u, "pets")) // Works
// ----- Goes Boom -------
val (dos, dosT) = unpack(u, "stuff")
println("Other: " + unpack(dos, dosT, "foo")) // Boom! ...or fizzle
// -----------------------
def unpack[T: TypeTag](target: T, name: String): (Any, Type) = unpack(target, typeOf[T], name)
// Get object value for named parameter of target
def unpack[T](target: T, t: Type, name: String): (Any, Type) = {
val im = cm.reflect(target)(ClassTag(target.getClass))
val fieldX = t.declaration(newTermName(name)).asTerm.accessed.asTerm
val fm = im.reflectField(fieldX)
(fm.get, fm.symbol.typeSignature)
}
}