Customer Type Mapper for Slick SQL - scala

I've found this example from slick testing:
https://github.com/slick/slick/blob/master/slick-testkit/src/main/scala/com/typesafe/slick/testkit/tests/MapperTest.scala
sealed trait Bool
case object True extends Bool
case object False extends Bool
implicit val boolTypeMapper = MappedColumnType.base[Bool, String](
{ b =>
assertNotNull(b)
if(b == True) "y" else "n"
}, { i =>
assertNotNull(i)
if(i == "y") True else False
}
)
But I'm trying to create a TypeMapper for org.joda.time.DateTime to/from java.sql.Timestamp - but without much success. The Bool example is very particular and I'm having trouble adapting it. Joda Time is super common - so any help would be much appreciated.
To be clear, I'm using interpolated sql"""select colA,colB from tableA where id = ${id}""" and such. When doing a select the system works well by using jodaDate types in the implicit GetResult converter.
However, for inserts there doesn't seem to be a way to do an implicit conversion or it is ignoring the code provided below in Answer #1 - same error as before:
could not find implicit value for parameter pconv: scala.slick.jdbc.SetParameter[(Option[Int], String, String, Option[org.joda.time.DateTime])]
I'm not using the Lifted style Slick configuration with the annotated Table objects perhaps that is why it is not finding/using the TypeMapper

I use the following in my code, which might also work for you:
import java.sql.Timestamp
import org.joda.time.DateTime
import org.joda.time.DateTimeZone.UTC
import scala.slick.lifted.MappedTypeMapper.base
import scala.slick.lifted.TypeMapper
implicit val DateTimeMapper: TypeMapper[DateTime] =
base[DateTime, Timestamp](
d => new Timestamp(d millis),
t => new DateTime(t getTime, UTC))
Edit (after your edit =^.~= ): (a bit late but I hope it still helps)
Ah, OK, since you're not using lifted embedding, you'll have to define different implicit values (as indicated by the error message from the compiler). Something like the following should work (though I haven't tried myself):
implicit val SetDateTime: SetParameter[DateTime] = new SetParameter {
def apply(d: DateTime, p: PositionedParameters): Unit =
p setTimestamp (new Timestamp(d millis))
}
For the other way round (retrieving the results of the SELECT), it looks like you'd need to define a GetResult:
implicit val GetDateTime: GetResult[DateTime] = new GetResult {
def apply(r: PositionedResult) = new DateTime(r.nextTimestamp getTime, UTC))
}
So, basically this is just the same as with the lifted embedding, just encoded with different types.

Why not to digg into something that works great?
Look at
https://gist.github.com/dragisak/4756344
and
https://github.com/tototoshi/slick-joda-mapper
First you could copy-paste to your project, and the second is available from Maven central.

Related

How to normalise a Union Type (T | Option[T])?

I have the following case class:
case class Example[T](
obj: Option[T] | T = None,
)
This allows me to construct it like Example(myObject) instead of Example(Some(myObject)).
To work with obj I need to normalise it to Option[T]:
lazy val maybeIn = obj match
case o: Option[T] => o
case o: T => Some(o)
the type test for Option[T] cannot be checked at runtime
I tried with TypeTest but I got also warnings - or the solutions I found look really complicated - see https://stackoverflow.com/a/69608091/2750966
Is there a better way to achieve this pattern in Scala 3?
I don't know about Scala3. But you could simply do this:
case class Example[T](v: Option[T] = None)
object Example {
def apply[T](t: T): Example[T] = Example(Some(t))
}
One could also go for implicit conversion, regarding the specific use case of the OP:
import scala.language.implicitConversions
case class Optable[Out](value: Option[Out])
object Optable {
implicit def fromOpt[T](o: Option[T]): Optable[T] = Optable(o)
implicit def fromValue[T](v: T): Optable[T] = Optable(Some(v))
}
case class SomeOpts(i: Option[Int], s: Option[String])
object SomeOpts {
def apply(i: Optable[Int], s: Optable[String]): SomeOpts = SomeOpts(i.value, s.value)
}
println(SomeOpts(15, Some("foo")))
We have a specialized Option-like type for this purpose: OptArg (in Scala 2 but should be easily portable to 3)
import com.avsystem.commons._
def gimmeLotsOfParams(
intParam: OptArg[Int] = OptArg.Empty,
strParam: OptArg[String] = OptArg.Empty
): Unit = ???
gimmeLotsOfParams(42)
gimmeLotsOfParams(strParam = "foo")
It relies on an implicit conversion so you have to be a little careful with it, i.e. don't use it as a drop-in replacement for Option.
The implementation of OptArg is simple enough that if you don't want external dependencies then you can probably just copy it into your project or some kind of "commons" library.
EDIT: the following answer is incorrect. As of Scala 3.1, flow analysis is only able to check for nullability. More information is available on the Scala book.
I think that the already given answer is probably better suited for the use case you proposed (exposing an API can can take a simple value and normalize it to an Option).
However, the question in the title is still interesting and I think it makes sense to address it.
What you are observing is a consequence of type parameters being erased at runtime, i.e. they only exist during compilation, while matching happens at runtime, once those have been erased.
However, the Scala compiler is able to perform flow analysis for union types. Intuitively I'd say there's probably a way to make it work in pattern matching (as you did), but you can make it work for sure using an if and isInstanceOf (not as clean, I agree):
case class Example[T](
obj: Option[T] | T = None
) {
lazy val maybeIn =
if (obj.isInstanceOf[Option[_]]) {
obj
} else {
Some(obj)
}
}
You can play around with this code here on Scastie.
Here is the announcement from 2019 when flow analysis was added to the compiler.

Case classes - Copy multiple fields with transforms

I have a case class with 25 fields and need to convert it into another with 22, of which 19 of these are shared and 3 are simply renamed.
I have found a few examples of how to do this using shapeless (e.g. an answer here and some code examples from Miles Sabin here and here) but the last of those looks a bit out of date, and I can't figure out from the Github example how I can use shapeless to rename multiple fields, or do more manipulation on a field before adding it to the new object. Can anyone help me out?
Simplified code example;
import shapeless.LabelledGeneric
case class A(fieldA:Int, fieldB:String, fieldC:String)
case class B(fieldARenamed:Int, fieldB:String, fieldC:String, fieldCTransformed:String)
val aGen = LabelledGeneric[A]
val bGen = LabelledGeneric[B]
val freddie = new A(1,"Freddie","somestring")
val record = aGen.to(freddie)
val atmp = freddie.fieldA
record.Remove("fielda")
val freddieB = bGen.from(record +
(Symbol("fieldARenamed") ->> atmp) +
(Symbol("fieldCTransformed") ->> freddie.fieldC.toUpperCase)
) //Errors everywhere, even if I replace + with :: etc.
I have a feeling Align is going to come into the picture somewhere here, but understanding how to do this in the leanest possible fashion - e.g. without creating additional traits like Field, as in that third link above - would be interesting.
In The Shapeless Guide, there is also some usage of a single quote, (e.g. 'fieldC) notation, which I haven't been able to find much information on, so if that plays a role some explanation would also be really helpful. Fairly new to this depth of Scala sorcery, so apologies if the question seems obtuse or covers too many disparate topics.
EDIT: For the avoidance of doubt, I am not looking for answers which suggest that I just manually create a new case class by referencing fields from the first, as in;
val freddieB = B(fieldARenamed = freddie.fieldA, fieldB = freddie.fieldB, fieldC = freddie.fieldC, fieldCTransformed =freddie.fieldC.toUpperCase)
See below comment for various reasons why this is inappropriate.
One other option is to use automapper; in particular, Dynamic Mappings feature.
For your particular example it would look like the following:
import io.bfil.automapper._
case class A(fieldA:Int, fieldB:String, fieldC:String)
case class B(fieldARenamed:Int, fieldB:String, fieldC:String, fieldCTransformed:String)
val freddie = new A(1,"Freddie","somestring")
val freddieB = automap(freddie).dynamicallyTo[B](
fieldARenamed = freddie.fieldA,
fieldCTransformed = freddie.fieldC.toUpperCase
)
and I guess you can make it a function
def atob(a: A): B = {
automap(a).dynamicallyTo[B](
fieldARenamed = a.fieldA,
fieldCTransformed = a.fieldC.toUpperCase
)
}
From efficiency point of view, this lib uses macros, so the generated code is practically as good as one could've written by hand
The simplest solution is to construct an instance of the new case class using the values from the old one, applying functions to the values as necessary. The code will be very efficient, the purpose of the code will be very clear, it will take less time to write than any other solution, it will be more robust and maintainable than a solution that depends on third-party libraries, and it avoids a hidden dependency between the two classes.
Just FYI, here's one way to get your question code to work.
import shapeless._
import shapeless.labelled.FieldType
import shapeless.ops.hlist.{Align,Intersection}
import shapeless.syntax.singleton._
case class A(fieldA:Int, fieldB:String, fieldC:String)
case class B(fieldARenamed:Int, fieldB:String, fieldC:String, fieldCTransformed:String)
val fromGen = LabelledGeneric[A]
val toGen = LabelledGeneric[B]
val freddie = A(1, "Freddie", "somestring")
val putARename = Symbol("fieldARenamed") ->> freddie.fieldA
val putCTrans = Symbol("fieldCTransformed") ->> freddie.fieldC.toUpperCase
trait Field { type K; type V; type F = FieldType[K, V] }
object Field {
def apply[K0,V0](sample: FieldType[K0,V0]) =
new Field { type K = K0; type V = V0 }
}
val pFieldA = Field(putARename)
val pFieldCT = Field(putCTrans)
val inter = Intersection[pFieldA.F :: pFieldCT.F :: fromGen.Repr, toGen.Repr]
val align = Align[inter.Out, toGen.Repr]
toGen.from(align(inter(putARename :: putCTrans :: fromGen.to(freddie))))
//res0: B = B(1,Freddie,somestring,SOMESTRING)

Polymorphism with Spark / Scala, Datasets and case classes

We are using Spark 2.x with Scala for a system that has 13 different ETL operations. 7 of them are relatively simple and each driven by a single domain class, and differ primarily by this class and some nuances in how the load is handled.
A simplified version of the load class is as follows, for the purposes of this example say that there are 7 pizza toppings being loaded, here's Pepperoni:
object LoadPepperoni {
def apply(inputFile: Dataset[Row],
historicalData: Dataset[Pepperoni],
mergeFun: (Pepperoni, PepperoniRaw) => Pepperoni): Dataset[Pepperoni] = {
val sparkSession = SparkSession.builder().getOrCreate()
import sparkSession.implicits._
val rawData: Dataset[PepperoniRaw] = inputFile.rdd.map{ case row : Row =>
PepperoniRaw(
weight = row.getAs[String]("weight"),
cost = row.getAs[String]("cost")
)
}.toDS()
val validatedData: Dataset[PepperoniRaw] = ??? // validate the data
val dedupedRawData: Dataset[PepperoniRaw] = ??? // deduplicate the data
val dedupedData: Dataset[Pepperoni] = dedupedRawData.rdd.map{ case datum : PepperoniRaw =>
Pepperoni( value = ???, key1 = ???, key2 = ??? )
}.toDS()
val joinedData = dedupedData.joinWith(historicalData,
historicalData.col("key1") === dedupedData.col("key1") &&
historicalData.col("key2") === dedupedData.col("key2"),
"right_outer"
)
joinedData.map { case (hist, delta) =>
if( /* some condition */) {
hist.copy(value = /* some transformation */)
}
}.flatMap(list => list).toDS()
}
}
In other words the class performs a series of operations on the data, the operations are mostly the same and always in the same order, but can vary slightly per topping, as would the mapping from "raw" to "domain" and the merge function.
To do this for 7 toppings (i.e. Mushroom, Cheese, etc), I would rather not simply copy/paste the class and change all of the names, because the structure and logic is common to all loads. Instead I'd rather define a generic "Load" class with generic types, like this:
object Load {
def apply[R,D](inputFile: Dataset[Row],
historicalData: Dataset[D],
mergeFun: (D, R) => D): Dataset[D] = {
val sparkSession = SparkSession.builder().getOrCreate()
import sparkSession.implicits._
val rawData: Dataset[R] = inputFile.rdd.map{ case row : Row =>
...
And for each class-specific operation such as mapping from "raw" to "domain", or merging, have a trait or abstract class that implements the specifics. This would be a typical dependency injection / polymorphism pattern.
But I'm running into a few problems. As of Spark 2.x, encoders are only provided for native types and case classes, and there is no way to generically identify a class as a case class. So the inferred toDS() and other implicit functionality is not available when using generic types.
Also as mentioned in this related question of mine, the case class copy method is not available when using generics either.
I have looked into other design patterns common with Scala and Haskell such as type classes or ad-hoc polymorphism, but the obstacle is the Spark Dataset basically only working on case classes, which can't be abstractly defined.
It seems that this would be a common problem in Spark systems but I'm unable to find a solution. Any help appreciated.
The implicit conversion that enables .toDS is:
implicit def rddToDatasetHolder[T](rdd: RDD[T])(implicit arg0: Encoder[T]): DatasetHolder[T]
(from https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.SQLImplicits)
You are exactly correct in that there's no implicit value in scope for Encoder[T] now that you've made your apply method generic, so this conversion can't happen. But you can simply accept one as an implicit parameter!
object Load {
def apply[R,D](inputFile: Dataset[Row],
historicalData: Dataset[D],
mergeFun: (D, R) => D)(implicit enc: Encoder[D]): Dataset[D] = {
...
Then at the time you call the load, with a specific type, it should be able to find an Encoder for that type. Note that you will have to import sparkSession.implicits._ in the calling context as well.
Edit: a similar approach would be to enable the implicit newProductEncoder[T <: Product](implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Encoder[T] to work by bounding your type (apply[R, D <: Product]) and accepting an implicit JavaUniverse.TypeTag[D] as a parameter.

Optional boolean parameters in Scala

I've been lately working on the DSL-style library wrapper over Apache POI functionality and faced a challenge which I can't seem to good solution for.
One of the goals of the library is to provide user with ability to build a spreadsheet model as a collection of immutable objects, i.e.
val headerStyle = CellStyle(fillPattern = CellFill.Solid, fillForegroundColor = Color.AquaMarine, font = Font(bold = true))
val italicStyle = CellStyle(font = Font(italic = true))
with the following assumptions:
User can optionally specify any parameter (that means, that you can create CellStyle with no parameters as well as with the full list of explicitly specified parameters);
If the parameter hasn't been specified explicitly by the user it is considered undefined and the default environment value (default value for the format we're converting to) will be used;
The 2nd point is important, as I want to convert this data model into multiple formats and i.e. the default font in Excel doesn't have to be the same as default font in HTML browser (and if user doesn't define the font family explicitly I'd like him to see the data using those defaults).
To deal with the requirements I've used the variation of the null pattern described here: Pattern for optional-parameters in Scala using null and also suggested here Scala default parameters and null (below a simplified example).
object ModelObject {
def apply(modelParam : String = null) : ModelObject = ModelObject(
modelParam = Option(modelParam)
)
}
case class ModelObject private(modelParam : Option[String])
Since null is used only internally in the companion object and very localized I decided to accept the null-sacrifice for the sake of the simplicity of the solution. The pattern works well with all the reference classes.
However for Scala primitive types wrappers null cannot be specified. This is especially a huge problem with Boolean for which I effectively consider 3 states (true, false and undefined). Wanting to provide the interface, where user still be able to write bold = true I decided to reach to Java wrappers which accept nulls.
object ModelObject {
def apply(boolParam : java.lang.Boolean = null) : ModelObject = ModelObject(
boolParam = Option(boolParam).map(_.booleanValue)
)
}
case class ModelObject private(boolParam : Option[Boolean])
This however doesn't right and I've been wondering whether there is a better approach to the problem. I've been thinking about defining the union types (with additional object denoting undefined value): How to define "type disjunction" (union types)?, however since the undefined state shouldn't be used explicitly the parameter type exposed by IDE to the user, it is going to be very confusing (ideally I'd like it to be Boolean).
Is there any better approach to the problem?
Further information:
More DSL API examples: https://github.com/norbert-radyk/spoiwo/blob/master/examples/com/norbitltd/spoiwo/examples/quickguide/SpoiwoExamples.scala
Sample implementation of the full class: https://github.com/norbert-radyk/spoiwo/blob/master/src/main/scala/com/norbitltd/spoiwo/model/CellStyle.scala
You can use a variation of the pattern I described here: How to provide helper methods to build a Map
To sum it up, you can use some helper generic class to represent optional arguments (much like an Option).
abstract sealed class OptArg[+T] {
def toOption: Option[T]
}
object OptArg{
implicit def autoWrap[T]( value: T ): OptArg[T] = SomeArg(value)
implicit def toOption[T]( arg: OptArg[T] ): Option[T] = arg.toOption
}
case class SomeArg[+T]( value: T ) extends OptArg[T] {
def toOption = Some( value )
}
case object NoArg extends OptArg[Nothing] {
val toOption = None
}
You can simply use it like this:
scala>case class ModelObject(boolParam: OptArg[Boolean] = NoArg)
defined class ModelObject
scala> ModelObject(true)
res12: ModelObject = ModelObject(SomeArg(true))
scala> ModelObject()
res13: ModelObject = ModelObject(NoArg)
However as you can see the OptArg now leaks in the ModelObject class itself (boolParam is typed as OptArg[Boolean] instead of Option[Boolean].
Fixing this (if it is important to you) just requires to define a separate factory as you have done yourself:
scala> :paste
// Entering paste mode (ctrl-D to finish)
case class ModelObject private(boolParam: Option[Boolean])
object ModelObject {
def apply(boolParam: OptArg[Boolean] = NoArg): ModelObject = new ModelObject(
boolParam = boolParam.toOption
)
}
// Exiting paste mode, now interpreting.
defined class ModelObject
defined module ModelObject
scala> ModelObject(true)
res22: ModelObject = ModelObject(Some(true))
scala> ModelObject()
res23: ModelObject = ModelObject(None)
UPDATE The advantage of using this pattern, over simply defining several overloaded apply methods as shown by #drexin is that in the latter case the number of overloads grows very fast with the number of arguments(2^N). If ModelObject had 4 parameters, that would mean 16 overloads to write by hand!

Scala Macros, generating type parameter calls

I'm trying to generalize setting up Squeryl (Slick poses the same problems AFAIK). I want to avoid having to name every case class explicitly for a number of general methods.
table[Person]
table[Bookmark]
etc.
This also goes for generating indexes, and creating wrapper methods around the CRUD methods for every case class.
So ideally what I want to do is have a list of classes and make them into tables, add indexes and add a wrapper method:
val listOfClasses = List(classOf[Person], classOf[Bookmark])
listOfClasses.foreach(clazz => {
val tbl = table[clazz]
tbl.id is indexed
etc.
})
I thought Scala Macros would be the thing to apply here, since I don't think you can have values as type parameters. Also I need to generate methods for every type of the form:
def insert(model: Person): Person = persons.insert(model)
I've got my mits on an example on Macros but I don't know how to generate a generic datastructure.
I got this simple example to illustrate what I want:
def makeList_impl(c: Context)(clazz: c.Expr[Class[_]]): c.Expr[Unit] = {
import c.universe._
reify {
println(List[clazz.splice]()) // ERROR: error: type splice is not a member of c.Expr[Class[_]]
}
}
def makeList(clazz: Class[_]): Unit = macro makeList_impl
How do I do this? Or is Scala Macros the wrong tool?
Unfortunately, reify is not flexible enough for your use case, but there's good news. In macro paradise (and most likely in 2.11.0) we have a better tool to construct trees, called quasiquotes: http://docs.scala-lang.org/overviews/macros/quasiquotes.html.
scala> def makeList_impl(c: Context)(clazz: c.Expr[Class[_]]): c.Expr[Any] = {
| import c.universe._
| val ConstantType(Constant(tpe: Type)) = clazz.tree.tpe
| c.Expr[Any](q"List[$tpe]()")
| }
makeList_impl: (c: scala.reflect.macros.Context)(clazz: c.Expr[Class[_]])c.Expr[Any]
scala> def makeList(clazz: Class[_]): Any = macro makeList_impl
defined term macro makeList: (clazz: Class[_])Any
scala> makeList(classOf[Int])
res2: List[Int] = List()
scala> makeList(classOf[String])
res3: List[String] = List()
Quasiquotes are even available in 2.10.x with a minor tweak to the build process (http://docs.scala-lang.org/overviews/macros/paradise.html#macro_paradise_for_210x), so you might want to give them a try.
This will probably not fill all your needs here, but it may help a bit:
The signature of table method looks like this:
protected def table[T]()(implicit manifestT: Manifest[T]): Table[T]
As you can see, it takes implicit Manifest object. That object is passed automatically by the compiler and contains information about type T. This is actually what Squeryl uses to inspect database entity type.
You can just pass these manifests explicitly like this:
val listOfManifests = List(manifest[Person], manifest[Bookmark])
listOfManifests.foreach(manifest => {
val tbl = table()(manifest)
tbl.id is indexed
etc.
})
Unfortunately tbl in this code will have type similar to Table[_ <: CommonSupertypeOfAllGivenEntities] which means that all operations on it must be agnostic of concrete type of database entity.