When using tableQuery.ddl.create, it creates only the columns in the projection. However in our use case there are columns which are used ONLY for filtering and/or ordering, so they are not part of the projection, but they need to be created:
case class CacheEntry(source: String, destination: String, hops: Int)
class CacheTable(tag: Tag) extends Table[CacheEntry](tag, "cache") {
def source = column[String]("source")
def destination = column[String]("dest")
def hops = column[Int]("hops")
def timestamp = column[LocalDateTime]("ts", O.DBType("timestamp default now"))
def * = (source, destination, hops) <> ((CacheEntry.apply _).tupled, CacheEntry.unapply)
}
How can we convince Slick to create the timstamp column with TableQuery[CacheTable].ddl.create?
Are we approaching this in the wrong way? We definately do NOT want the ts to show up in the CacheEntry (we could live with it in this case, but we have more complicated cases where this is not desirable)
You could define something like a case class TimestampedEE](entry: E, timestamp: LocalDateTime), and change your CacheTable to a Table[Timestamped[CacheEntry]]. The def * projection is probably going to look ugly (if don't rely on some shapeless magic), but that's one way to do it.
Related
I have a database for objects called Campaigns containing three fields :
Id (int, not nullable)
Version (int, not nullable)
Stuff (Text, nullable)
Let's call CampaignsRow the corresponding slick entity class
When I select line from Campaigns, I don't always need to read stuff, which contains big chunks of text.
However, I'd very much like to work in the codebase with the class CampaignsRow instead of a tuple, and so to be able to sometimes just drop the Stuff column, while retaining the original type
Basically, I'm trying to write the following function :
//Force dropping the Stuff column from the current Query
def smallCampaign(campaigns: Query[Campaigns, CampaignsRow, Seq]): Query[Campaigns, CampaignsRow, Seq] = {
val smallCampaignQuery = campaigns.map {
row => CampaignsRow(row.id, row.version , None : Option[String])
}
smallCampaignQuery /* Fails because the type is now wrong, I have a Query[(Rep[Int], Rep[Int], Rep[Option[String]), (Int, Int, Option[String], Seq] */
}
Any idea how to do this ? I suspect this has to do with Shape in slick, but I can't find a resource to start understanding this class, and the slick source code is proving too complex for me to follow.
You're actually already doing almost what you want in def *, the default mapping. You can use the same tools in the map method. Your two tools are mapTo and <>.
As you've found, there is the mapTo method which you can only use if your case class exactly matches the shape of the tuple, so if you wanted a special case class just for this purpose:
case class CampaignLite(id: Int, version: Int)
val smallCampaignQuery = campaigns.map {
row => (row.id, row.version).mapTo[CampaignLite]
}
As you want to reuse your existing class, you can write your own convert functions instead of using the standard tupled and unapply and pass those to <>:
object CampaignRow {
def tupleLite(t: (Int, Int)) = CampaignRow(t._1, t._2, None)
def unapplyLite(c: CampaignRow) = Some((c.id, c.version))
}
val smallCampaignQuery = campaigns.map {
row => (row.id, row.version) <> (CampaignRow.tupleLite, CampaignRow.unapplyLite)
}
This gives you the most flexibility, as you can do whatever you like in your convert functions, but it's a bit more wordy.
As row is an instance of the Campaigns table you could always define it there alongside *, if you need to use it regularly.
class Campaigns ... {
...
def * = (id, version, stuff).mapTo[CampaignRow]
def liteMapping = (id, version) <> (CampaignRow.tupleLite, CampaignRow.unapplyLite)
}
val liteCampaigns = campaigns.map(_.liteMapping)
Reference: Essential Slick 3, section 5.2.1
If I understand your requirement correctly, you could consider making CampaignRow a case class that models your Campaigns table class by having Campaigns extend Table[CampaignRow] and providing the bidirectional mapping for the * projection:
case class CampaignRow(id: Int, version: Int, stuff: Option[String])
class Campaigns(tag: Tag) extends Table[CampaignRow](tag, "CAMPAIGNS") {
// ...
def * = (id, version, stuff) <> (CampaignRow.tupled, CampaignRow.unapply)
}
You should then be able to do something like below:
val campaigns = TableQuery[CampaignRow]
val smallCampaignQuery = campaigns.map( _.copy(stuff = None) )
For a relevant example, here's a Slick doc.
We are looking for a viable design pattern for building Scalacheck Gen (generators) that can produce both positive and negative test scenarios. This will allow us to run forAll tests to validate functionality (positive cases), and also verify that our case class validation works correctly by failing on all invalid combinations of data.
Making a simple, parameterized Gen that does this on a one-off basis is pretty easy. For example:
def idGen(valid: Boolean = true): Gen[String] = Gen.oneOf(ID.values.toList).map(s => if (valid) s else Gen.oneOf(simpleRandomCode(4), "").sample.get)
With the above, I can get a valid or invalid ID for testing purposes. The valid one, I use to make sure business logic succeeds. The invalid one, I use to make sure our validation logic rejects the case class.
Ok, so -- problem is, on a large scale, this becomes very unwieldly. Let's say I have a data container with, oh, 100 different elements. Generating a "good" one is easy. But now, I want to generate a "bad" one, and furthermore:
I want to generate a bad one for each data element, where a single data element is bad (so at minimum, at least 100 bad instances, testing that each invalid parameter is caught by validation logic).
I want to be able to override specific elements, for instance feeding in a bad ID or a bad "foobar." Whatever that is.
One pattern we can look to for inspiration is apply and copy, which allows us to easily compose new objects while specifying overridden values. For example:
val f = Foo("a", "b") // f: Foo = Foo(a,b)
val t = Foo.unapply(f) // t: Option[(String, String)] = Some((a,b))
Foo(t.get._1, "c") // res0: Foo = Foo(a,c)
Above we see the basic idea of creating a mutating object from the template of another object. This is more easily expressed in Scala as:
val f = someFoo copy(b = "c")
Using this as inspiration we can think about our objectives. A few things to think about:
First, we could define a map or a container of key/values for the data element and generated value. This could be used in place of a tuple to support named value mutation.
Given a container of key/value pairs, we could easily select one (or more) pairs at random and change a value. This supports the objective of generating a data set where one value is altered to create failure.
Given such a container, we can easily create a new object from the invalid collection of values (using either apply() or some other technique).
Alternatively, perhaps we can develop a pattern that uses a tuple and then just apply() it, kind of like the copy method, as long as we can still randomly alter one or more values.
We can probably explore developing a reusable pattern that does something like this:
def thingGen(invalidValueCount: Int): Gen[Thing] = ???
def someTest = forAll(thingGen) { v => invalidV = v.invalidate(1); validate(invalidV) must beFalse }
In the above code, we have a generator thingGen that returns (valid) Things. Then for all instances returned, we invoke a generic method invalidate(count: Int) which will randomly invalidate count values, returning an invalid object. We can then use that to ascertain whether our validation logic works correctly.
This would require defining an invalidate() function that, given a parameter (either by name, or by position) can then replace the identified parameter with a value that is known to be bad. This implies have an "anti-generator" for specific values, for instance, if an ID must be 3 characters, then it knows to create a string that is anything but 3 characters long.
Of course to invalidate a known, single parameter (to inject bad data into a test condition) we can simply use the copy method:
def thingGen(invalidValueCount: Int): Gen[Thing] = ???
def someTest = forAll(thingGen) { v => v2 = v copy(id = "xxx"); validate(v2) must beFalse }
That is the sum of my thinking to date. Am I barking up the wrong tree? Are there good patterns out there that handle this kind of testing? Any commentary or suggestions on how best to approach this problem of testing our validation logic?
We can combine a valid instance and an set of invalid fields (so that every field, if copied, would cause validation failure) to get an invalid object using shapeless library.
Shapeless allows you to represent your class as a list of key-value pairs that are still strongly typed and support some high-level operations, and converting back from this representation to your original class.
In example below I'll be providing an invalid instance for each single field provided
import shapeless._, record._
import shapeless.labelled.FieldType
import shapeless.ops.record.Updater
A detailed intro
Let's pretend we have a data class, and a valid instance of it (we only need one, so it can be hardcoded)
case class User(id: String, name: String, about: String, age: Int) {
def isValid = id.length == 3 && name.nonEmpty && age >= 0
}
val someValidUser = User("oo7", "Frank", "A good guy", 42)
assert(someValidUser.isValid)
We can then define a class to be used for invalid values:
case class BogusUserFields(name: String, id: String, age: Int)
val bogusData = BogusUserFields("", "1234", -5)
Instances of such classes can be provided using ScalaCheck. It's much easier to write a generator where all fields would cause failure. Order of fields doesn't matter, but their names and types do. Here we excluded about from User set of fields so we can do what you asked for (feeding only a subset of fields you want to test)
We then use LabelledGeneric[T] to convert User and BogusUserFields to their corresponding record value (and later we will convert User back)
val userLG = LabelledGeneric[User]
val bogusLG = LabelledGeneric[BogusUserFields]
val validUserRecord = userLG.to(someValidUser)
val bogusRecord = bogusLG.to(bogusData)
Records are lists of key-value pairs, so we can use head to get a single mapping, and the + operator supports adding / replacing field to another record. Let's pick every invalid field into our user one at a time. Also, here's the conversion back in action:
val invalidUser1 = userLG.from(validUserRecord + bogusRecord.head)// invalid name
val invalidUser2 = userLG.from(validUserRecord + bogusRecord.tail.head)// invalid ID
val invalidUser3 = userLG.from(validUserRecord + bogusRecord.tail.tail.head) // invalid age
assert(List(invalidUser1, invalidUser2, invalidUser3).forall(!_.isValid))
Since we basically are applying the same function (validUserRecord + _) to every key-value pair in our bogusRecord, we can also use map operator, except we use it with an unusual - polymorphic - function. We can also easily convert it to List, because every element will be of a same type now.
object polymerge extends Poly1 {
implicit def caseField[K, V](implicit upd: Updater[userLG.Repr, FieldType[K, V]]) =
at[FieldType[K, V]](upd(validUserRecord, _))
}
val allInvalidUsers = bogusRecord.map(polymerge).toList.map(userLG.from)
assert(allInvalidUsers == List(invalidUser1, invalidUser2, invalidUser3))
Generalizing and removing all the boilerplate
Now the whole point of this was that we can generalize it to work for any two arbitrary classes. The encoding of all relationships and operations is a bit cumbersome and it took me a while to get it right with all the implicit not found errors, so I'll skip the details.
class Picks[A, AR <: HList](defaults: A)(implicit lgA: LabelledGeneric.Aux[A, AR]) {
private val defaultsRec = lgA.to(defaults)
object mergeIntoTemplate extends Poly1 {
implicit def caseField[K, V](implicit upd: Updater[AR, FieldType[K, V]]) =
at[FieldType[K, V]](upd(defaultsRec, _))
}
def from[B, BR <: HList, MR <: HList, F <: Poly](options: B)
(implicit
optionsLG: LabelledGeneric.Aux[B, BR],
mapper: ops.hlist.Mapper.Aux[mergeIntoTemplate.type, BR, MR],
toList: ops.hlist.ToTraversable.Aux[MR, List, AR]
) = {
optionsLG.to(options).map(mergeIntoTemplate).toList.map(lgA.from)
}
}
So, here it is in action:
val cp = new Picks(someValidUser)
assert(cp.from(bogusData) == allInvalidUsers)
Unfortunately, you cannot write new Picks(someValidUser).from(bogusData) because implicit for mapper requires a stable identifier. On the other hand, cp instance can be reused with other types:
case class BogusName(name: String)
assert(cp.from(BogusName("")).head == someValidUser.copy(name = ""))
And now it works for all types! And bogus data is required to be any subset of class fields, so it will work even for class itself
case class Address(country: String, city: String, line_1: String, line_2: String) {
def isValid = Seq(country, city, line_1, line_2).forall(_.nonEmpty)
}
val acp = new Picks(Address("Test country", "Test city", "Test line 1", "Test line 2"))
val invalidAddresses = acp.from(Address("", "", "", ""))
assert(invalidAddresses.forall(!_.isValid))
You can see the code running at ScalaFiddle
I'm experimenting with scalikejdbc (trying to move from Slick), and I'm stuck on creating my schema from the entities (read: case classes).
// example Slick equivalent
case class X(id: Int, ...)
class XTable(tag: Tag) extends Table[X] (tag, "x") {
def id = column[Int]("id")
... //more columns
def * = (id, ...) <> (X.tupled, X.unapply)
}
val xTable = TableQuery[XTable]
db.run(xtable.schema.create) //creates in the DB a table named "x", with "id" and all other columns
It seemed like using SQLSyntaxSupport could be a step in the right direction, with something like
// scalikejdbc
case class X (id: Int, ...)
object X extends SQLSyntaxSupport[X] {
def apply (x: ResultName[X])(rs: WrappedResultSet): X = new X(id = rs.get(x.id, ...))
}
X.table.??? // what to do now?
but could not figure out the next step.
What I'm looking for is the opposite of the tool described under [Reverse-engineering]: http://scalikejdbc.org/documentation/reverse-engineering.html
Any help/ideas, in particular directions to a relevant part of the documentation, will be appreciated
You can use the the statements method to get the SQL code, like for
most other SQL-based Actions. Schema Actions are currently the only
Actions that can produce more than one statement.
schema.create.statements.foreach(println)
schema.drop.statements.foreach(println)
http://slick.typesafe.com/doc/3.0.0/schemas.html
I have a Slick 3.0 table definition similar to the following:
case class Simple(a: String, b: Int, c: Option[String])
trait Tables { this: JdbcDriver =>
import api._
class Simples(tag: Tag) extends Table[Simple](tag, "simples") {
def a = column[String]("a")
def b = column[Int]("b")
def c = column[Option[String]]("c")
def * = (a, b, c) <> (Simple.tupled, Simple.unapply)
}
lazy val simples = TableQuery[Simples]
}
object DB extends Tables with MyJdbcDriver
I would like to be able to do 2 things:
Get a list of the column names as Seq[String]
For an instance of Simple, generate a Seq[String] that would correspond to how the data would be inserted into the database using a raw query (e.g. Simple("hello", 1, None) becomes Seq("'hello'", "1", "NULL"))
What would be the best way to do this using the Slick table definition?
First of all it is not possible to trick Slick and change the order on the left side of the <> operator in the * method without changing the order of values in Simple, the row type of Simples, i.e. what Ben assumed is not possible. The ProvenShape return type of the * projection method ensures that there is a Shape available for translating between the Column-based type in * and the client-side type and if you write def * = (c, b, a) <> Simple.tupled, Simple.unapply) having Simple defined as case class Simple(a: String, b: Int, c: Option[String]), Slick will complain with an error "No matching Shape found. Slick does not know how to map the given types...". Ergo, you can iterate over all the elements of an instance of Simple with its productIterator.
Secondly, you already have the definition of the Simples table in your code and querying metatables to get the same information you already have is not sensible. You can get all you column names with a one-liner simples.baseTableRow.create_*.map(_.name). Note that the * projection of the table also defines the columns generated when you create the table schema. So the columns not mentioned in the projection are not created and the statement above is guaranteed to return exactly what you need and not to drop anything.
To recap briefly:
To get a list of the column names of the Simples table as Seq[String] use
simples.baseTableRow.create_*.map(_.name).toSeq
To generate a Seq[String] that would correspond to how the data
would be inserted into the database using a raw query for aSimple,
an instance of Simple use aSimple.productIterator.toSeq
To get the column names, try this:
db.run(for {
metaTables <- slick.jdbc.meta.MTable.getTables("simples")
columns <- metaTables.head.getColumns
} yield columns.map {_.name}) foreach println
This will print
Vector(a, b, c)
And for the case class values, you can use productIterator:
Simple("hello", 1, None).productIterator.toVector
is
Vector(hello, 1, None)
You still have to do the value mapping, and guarantee that the order of the columns in the table and the values in the case class are the same.
I have a simple class Feature, currently implemented as a case class
case class Feature(value :String)
There are multiple operations decorating a feature with different properties, for example there is a function that might count the number of appearances of the feature so then I might need a CountedFeature. Besides counting I might need also a WeightedFeature, an IndexedFeature and so on.
My intuition says it's suitable for traits so I defined the following traits
trait Counted {def count :Long}
trait Weighted {def weight :Double}
trait Indexed {def index :Int}
Two issues pop up with this:
1. Do I need to create a concrete class implementing each combination of traits (e.x. implement a CountedWeightedFeature, CountedIndexedfeature and so on) ,or is there some way to avoid it. If I will move to more decorations it will be impossible to mantain classes for all combinations.
2. I want to design a function that weights features based on their count. It's signature should look something like :
def computeWeightsByCount(features :List[T <: Feature with Counted]) :List[T with Weighted] = {...}
T here may be Indexed or not, so this function should have some way to take a class and instansiate a new class that has all the traits of the original class stacked inside plus an additional one.
Is there some elegant way to do this in Scala, or should I totaly rethink this design?
The design looks fine to me, except extending case classes is not recommended. A brief summary of the reasons why can be found here: https://stackoverflow.com/a/12705634/2186890
So you will might want to rewrite Feature as something like this:
trait Feature { def value: String }
Now you can define case classes for pattern matching etc. like this:
case class CountedFeature(value: String, count: Long) extends Feature with Counted
There is no easy way to avoid combinatorial explosion of case classes like this, but you are enabled to use types such as Feature with Counted wherever you like. Keep in mind that you can easily create objects that match type Feature with Counted on the fly. For instance:
val x: Feature with Counted = new Feature with Counted { val value = ""; val count = 0L }
Implementing computeWeightsByCount like you want is a little tricky, because there is no easy way to build a T with Weighted without knowing more about type T. But it can be done with implicit methods. Essentially, we need to have a defined path for generating a T with Weighted from a T for every Feature with Counted that you want to apply this method to. For instance, we start with this:
trait Feature { def value: String }
trait Counted { def count: Long }
trait Weighted { def weight: Double }
trait Indexed { def index: Int }
We want to define computeWeightsByCount like you did in your question, but also taking an implicit method that takes a T and a weight, and produces a T with Weighted:
def computeWeightsByCount[
T <: Feature with Counted](
features: List[T])(
implicit weighted: (T, Double) => T with Weighted
): List[T with Weighted] = {
def weight(fc: Feature with Counted): Double = 0.0d
features map { f => weighted(f, weight(f)) }
}
Now we need to define an implicit method to produce weighted features from the input features. Let's start with getting a Feature with Counted with Weighted from a Feature with Counted. We'll put it in companion object for Feature:
object Feature {
implicit def weight(fc: Feature with Counted, weight: Double): Feature with Counted with Weighted = {
case class FCW(value: String, count: Long, weight: Double) extends Feature with Counted with Weighted
FCW(fc.value, fc.count, weight)
}
}
We can use it like so:
case class FC(value: String, count: Long) extends Feature with Counted
val fcs: List[Feature with Counted] = List(FC("0", 0L), FC("1", 1L))
val fcws: List[Feature with Counted with Weighted] = computeWeightsByCount[Feature with Counted](fcs)
For any type that you want to compute weighted counts for, you need to define a similar such implicit method.
Admittedly, this is far from a beautiful solution. So yes, you are right, you may want to rethink the design. The advantage to this approach, however, is that any further extensions to the Feature "hierarchy" can be made without needing to make any changes to computeWeightsByCount. Whoever writes the new trait can provide the appropriate implicit methods as well.