I'm using slick 3.2.3 and I'm trying to build a query that returns a Seq[Entity1, Seq[Entity2]] for two entities that have a one to many relationship (for each Entity1 are associated multiple Entity2).
So I have my two entities
case class Entity1(name: String, id: Option[Long] = None)
case class Entity2(entity1Id: Long, name: String, id: Option[Long] = None
with the table definitions (that are generated by slick codegen task)
class entity1Table(_tableTag: Tag) extends profile.api.Table[Entity1](_tableTag, "ENTITY_1") {
...
}
lazy val groupTable = new TableQuery(tag => new groupTable(tag))
class entity2Table(_tableTag: Tag) extends profile.api.Table[Entity2](_tableTag, "ENTITY_2") {
...
}
lazy val entity2Table = new TableQuery(tag => new entity2Table(tag))
Reading this article I've created a query like this
val q = (for {
e1 <- entity1Table
e2 <- entity2Table if e2.entity1Id === e1.id
} yield (e1, e2)).groupBy(_._1) map {
case (entity1, tuples) => (entity1, tuples.map(_._2))
}
db.run(q.result)
but I get this error at compile time:
Error:(19, 35) No matching Shape found.
Slick does not know how to map the given types.
Possible causes: T in Table[T] does not match your * projection,
you use an unsupported type in a Query (e.g. scala List),
or you forgot to import a driver api into scope.
Required level: slick.lifted.FlatShapeLevel
Source type: (my.namespace.models.entity1Table, slick.lifted.Query[my.namespace.models.entity2Table,my.namespace.models.Entity2,[+A]Seq[A]])
Unpacked type: T
Packed type: G
} yield (e1, e2)).groupBy(_._1) map {
I suspect that it cannot map the entity1Table and entity2Table.
How can I fix the error?
As specified in the Slick doc, groupBy currently doesn't support executing a query with nested values of type Query:
The intermediate query (i.e. the query ended with groupBy() without an
aggregate function) contains nested values of type Query. These would
turn into nested collections when executing the query, which is not
supported at the moment. Therefore it is necessary to flatten the
nested queries immediately by aggregating their values (or individual
columns)
In other words, your Slick groupBy query must be coupled with an aggregate function equivalent to SQL's count(), sum(), etc. For example, the following query which is equivalent to select count(*) ... group by ... having count(*) > 1 would work:
val query = ( for {
e1 <- entity1Table
e2 <- entity2Table if e2.entity1Id === e1.id
} yield (e1, e2)
).
groupBy(_._1).map{
case (entity1, group) => (entity1, group.size)
}.
filter(_._2 > 1)
Related
I am having tuples of format as (DBIO[Seq[Person]], DBIO[Seq[Address]]) as one to one mapping. Person and Address is separate table in RDBMS. Profile definition is Profile(person: Person, address: Address). Now I want to convert the former into DBIO[Seq[Profile]]. Following is code snippet for how I have got (DBIO[Seq[Person]], DBIO[Seq[Address]])
for {
person <- personQuery if person.personId === personId
address <- addressQuery if address.addressId === profile.addressId
} yield (person.result, address.result)
Need help with this transformation to DBIO[Seq[Profile].
Assuming you can't use a join and you need to work with two actions (two DBIOs), what you can do is combine the two actions into a single one:
// Combine two actions into a single action
val pairs: DBIO[ ( Seq[Person], Seq[Address] ) ] =
(person.result).zip(address.result)
(zip is just one of many combinators you can use to manipulate DBIO).
From there you can use DBIO.map to convert the pair into the datastructure you want.
For example:
// Use Slick's DBIO.map to map the DBIO value into a sequence of profiles:
val profiles: DBIO[Seq[Profile]] = pairs.map { case (ppl, places) =>
// We now use a regular Scala `zip` on two sequences:
ppl.zip(places).map { case (person, place) => Profile(person, place) }
}
I am unfamiliar with whatever DBIO is. Assuming it is a case class of some type T :
val (DBIO(people), DBIO(addresses)) = for {
person <- personQuery if person.personId === personId
address <- addressQuery if address.addressId === profile.addressId
} yield (person.result, address.result)
val profiles = DBIO(people.zip(addresses).map{ case (person, address) => Profile(person, address)})
I'm exploring the different possibilities on how to implement a generic DAO using the latest Slick 3.1.1 to boost productivity and yes there is need for it because basing the service layer of my Play Web application on TableQuery alone leads to a lot of boilerplate code. One of the methods I'd like to feature in my generic DAO implementation is the findByExample, possible in JPA with the help of the Criteria API. In my case, I'm using the Slick Code Generator to generate the model classes from a sql script.
I need the following to be able to dynamically access the attribute names, taken from Scala. Get field names list from case class:
import scala.reflect.runtime.universe._
def classAccessors[T: TypeTag]: List[MethodSymbol] = typeOf[T].members.collect {
case m: MethodSymbol if m.isCaseAccessor => m
}.toList
A draft implementation for findByExample would be:
def findByExample[T, R](example: R) : Future[Seq[R]] = {
var qt = TableQuery[T].result
val accessors = classAccessors[R]
(0 until example.productArity).map { i =>
example.productElement(i) match {
case None => // ignore
case 0 => // ignore
// ... some more default values => // ignore
// handle a populated case
case Some(x) => {
val columnName = accessors(i)
qt = qt.filter(_.columnByName(columnName) == x)
}
}
}
qt.result
}
But this doesn't work because I need better Scala Kungfu. T is the entity table type and R is the row type that is generated as a case class and therefore a valid Scala Product type.
The first problem in that code is that would be too inefficient because instead of doing e.g.
qt.filter(_.firstName === "Juan" && _.streetName === "Rosedale Ave." && _.streetNumber === 5)
is doing:
// find all
var qt = TableQuery[T].result
// then filter by each column at the time
qt = qt.filter(_.firstName === "Juan")
qt = qt.filter(_.streetName === "Rosedale Ave.")
qt = qt.filter(_.streetNumber === 5)
Second I can't see how to dynamically access the column name in the filter method i.e.
qt.filter(_.firstName == "Juan")
I need to instead have
qt.filter(_.columnByName("firstName") == "Juan")
but apparently there is no such possibility while using the filter function?
Probably the best ways to implement filters and sorting by dynamically provided column names would be either plain SQL or extending the code generator to generate extension methods, something like this:
implicit class DynamicPersonQueries[C[_]](q: Query[PersonTable, PersonRow, C]){
def dynamicFilter( column: String, value: String ) = column {
case "firstName" => q.filter(_.firstName === value)
case "streetNumber" => q.filter(_.streetNumber === value.toInt)
...
}
}
You might have to fiddle with the types a bit to get it to compile (and ideally update this post afterwards :)).
You can then filter by all the provided values like this:
val examples: Map[String, String] = ...
val t = TableQuery[PersonTable]
val query = examples.foldLeft(t){case (t,(column, value)) => t.dynamicFilter(column, value)
query.result
Extending the code generator is explained here: http://slick.lightbend.com/doc/3.1.1/code-generation.html#customization
After further researching found the following blog post Repository Pattern / Generic DAO Implementation.
There they declare and implement a generic filter method that works for any Model Entity type and therefore it is in my view a valid functional replacement to the more JPA findByExample.
i.e.
T <: Table[E] with IdentifyableTable[PK]
E <: Entity[PK]
PK: BaseColumnType
def filter[C <: Rep[_]](expr: T => C)(implicit wt: CanBeQueryCondition[C]) : Query[T, E, Seq] = tableQuery.filter(expr)
I have a database that contain activities with a one-to-many registrations relation.
The goal is to get all activities, with a list of their registrations.
By creating a cartesian product of activities with registrations, all necessary data to get that data is out is there.
But I can't seem to find a nice way to get it into a scala collection properly;
let's of type: Seq[(Activity, Seq[Registration])]
case class Registration(
id: Option[Int],
user: Int,
activity: Int
)
case class Activity(
id: Option[Int],
what: String,
when: DateTime,
where: String,
description: String,
price: Double
)
Assuming the appropriate slick tables and tablequeries exist, I would write:
val acts_regs = (for {
a <- Activities
r <- Registrations if r.activityId === a.id
} yield (a, r))
.groupBy(_._1.id)
.map { case (actid, acts) => ??? }
}
But I cannot seem to make the appropriate mapping. What is the idiomatic way of doing this? I hope it's better than working with a raw cartesian product...
In Scala
In scala code it's easy enough, and would look something like this:
val activities = db withSession { implicit sess =>
(for {
a <- Activities leftJoin Registrations on (_.id === _.activityId)
} yield a).list
}
activities
.groupBy(_._1.id)
.map { case (id, set) => (set(0)._1, set.map(_._2)) }
But this seems rather inefficient due to the unnecessary instantiations of Activity which the table mapper will create for you.
Neither does it look really elegant...
Getting a count of registrations
The in scala method is even worse when only interested in a count of registrations like so:
val result: Seq[Activity, Int] = ???
In Slick
My best attempt in slick would look like this:
val activities = db withSession { implicit sess =>
(for {
a <- Activities leftJoin Registrations on (_.id === _.activityId)
} yield a)
.groupBy(_._1.id)
.map { case (id, results) => (results.map(_._1), results.length) }
}
But this results in an error that slick cannot map the given types in the "map"-line.
I would suggest:
val activities = db withSession { implicit sess =>
(for {
a <- Activities leftJoin Registrations on (_.id === _.activityId)
} yield a)
.groupBy(_._1)
.map { case (activity, results) => (activity, results.length) }
}
The problem with
val activities = db withSession { implicit sess =>
(for {
a <- Activities leftJoin Registrations on (_.id === _.activityId)
} yield a)
.groupBy(_._1.id)
.map { case (id, results) => (results.map(_._1), results.length) }
}
is that you can't produce nested results in group by. results.map(_._1) is a collection of items. SQL does implicit conversions from collections to single rows in some cases, but Slick being type-safe doesn't. What you would like to do in Slick is something like results.map(_._1).head, but that is currently not supported. The closest you could get is something like (results.map(_.id).max, results.map(_.what).max, ...), which is pretty tedious. So grouping by the whole activities row is probably the most feasible workaround right now.
A solution for getting all registrations per activity:
// list of all activities
val activities = Activities
// map of registrations belonging to those activities
val registrations = Registrations
.filter(_.activityId in activities.map(_.id))
.list
.groupBy(_.activityId)
.map { case (aid, group) => (aid, group.map(_._2)) }
.toMap
// combine them
activities
.list
.map { a => (a, registrations.getOrElse(a.id.get, List()))
Which gets the job done in 2 queries. It should be doable to abstract this type of "grouping" function into a scala function.
The code below compiles and works fine as shown.
However, if I try to yield Some("SomeConstant"), I get the runtime error shown below.
Why is this happening, and how can I return expressions (e.g. Some(...)) from my query?
def cannotUnpack(db: Database) {
db.withSession {
val data = (for {
rw1 <- TableOne
rw2 <- TableTwo if rw1.cl1 === rw2.cl1 && rw1.cl2 === rw2.cl2 && rw1.cl1 === "0"
now = new Timestamp(System.currentTimeMillis())
six = 6
} yield (uuid, rw1.cl3, "SomeConstant", six, now) ).list // Works
// } yield (uuid, rw1.cl3, Some("SomeConstant"), six, now) ).list // Runtime error
}
}
Runtime error:
Don't know how to unpack (String, scala.slick.lifted.Column[Option[String]], Some[String], scala.slick.lifted.Column[Int], scala.slick.lifted.Column[java.sql.Timestamp]) to T and pack to G
rw2 <- TableTwo if rw1.cl1 === rw2.cl1 && rw1.cl2 === rw2.cl2 && rw1.cl1 === "0"
^
Environment:
scala 2.10 on Ubuntu, Java 7
Slick 1.0.0, SQL Server, JTDS driver
The short answer: It works, if you write
Some("SomeConstant") : Option[String].
The long answer: If you supply a constant as part of your Slick query, Slick has to put the value into the SQL query and later read it back from the results. This preserves composability, i.e. allows you to use the Slick query as a component in another Slick query. In order to encode the value into the SQL query, the method you are calling (in case of a comprehension: map or flatMap) needs to find an implicit value of type TypeMapper[T] where T is the type of the value. Slick does define a TyperMapper[Option[String]] but the problem is that it does not apply in your case, because Some("SomeConstant") is of type Some[String] and there is no TypeMapper[Some[String]] defined in Slick (and TypeMapper[T] is invariant in T). By explicitly supplying :Option[String], you loosen the type information, so a matching TypeMapper can be found.
We will think about if we can add support for constants of type Some into Slick. I added a ticket (https://www.assembla.com/spaces/typesafe-slick/tickets/268) and will bring it up in our next team meeting.
Well, I wouldn't involve constants in select but use them only when managing results loaded from DB.
Try it like:
def cannotUnpack(db: Database) {
db.withSession {
val data = (for {
rw1 <- TableOne
rw2 <- TableTwo if rw1.cl1 === rw2.cl1 && rw1.cl2 === rw2.cl2 && rw1.cl1 === "0"
} yield (uuid, rw1.cl3)
}
}
after that get your data ready for what you need:
for (
(uuid, rw1_cl3) <- data.list
) yield (uuid, rw1_cl3, Some("constant"), 6, new Timestamp(System.currentTimeMillis()))
I usually use an output case class when preparing final data, for instance:
case class Export(uuid: String, rw1: String, constant: Option[String], six: String, now: Timestamp)
for (
(uuid, rw1_cl3) <- data.list
) yield Export(
uuid,
rw1_cl3,
Some("constant"),
6,
new Timestamp(System.currentTimeMillis()))
I am attempting to learn to use Slick to query MySQL. I have the following type of query working to get a single Visit object:
Q.query[(Int,Int), Visit]("""
select * from visit where vistor = ? and location_code = ?
""").firstOption(visitorId,locationCode)
What I would like to know is how can I change the above to query to get a List[Visit] for a collection of Locations...something like this:
val locationCodes = List("loc1","loc2","loc3"...)
Q.query[(Int,Int,List[String]), Visit]("""
select * from visit where vistor = ? and location_code in (?,?,?...)
""").list(visitorId,locationCodes)
Is this possible with Slick?
As the other answer suggests, this is cumbersome to do with static queries. The static query interface requires you to describe the bind parameters as a Product. (Int, Int, String*)
is not valid scala, and using (Int,Int,List[String]) needs some kludges as well. Furthermore, having to ensure that locationCodes.size is always equal to the number of (?, ?...) you have in your query is brittle.
In practice, this is not too much of a problem because you want to be using the query monad instead, which is the type-safe and recommended way to use Slick.
val visitorId: Int = // whatever
val locationCodes = List("loc1","loc2","loc3"...)
// your query, with bind params.
val q = for {
v <- Visits
if v.visitor is visitorId.bind
if v.location_code inSetBind locationCodes
} yield v
// have a look at the generated query.
println(q.selectStatement)
// run the query
q.list
This is assuming you have your tables set up like this:
case class Visitor(visitor: Int, ... location_code: String)
object Visitors extends Table[Visitor]("visitor") {
def visitor = column[Int]("visitor")
def location_code = column[String]("location_code")
// .. etc
def * = visitor ~ .. ~ location_code <> (Visitor, Visitor.unapply _)
}
Note that you can always wrap your query in a method.
def byIdAndLocations(visitorId: Int, locationCodes: List[String]) =
for {
v <- Visits
if v.visitor is visitorId.bind
if v.location_code inSetBind locationCodes
} yield v
}
byIdAndLocations(visitorId, List("loc1", "loc2", ..)) list
It doesn't work because the StaticQuery object (Q) expects to implicitly set the parameters in the query string, using the type parameters of the query method to create a sort of setter object (of type scala.slick.jdbc.SetParameter[T]).
The role of SetParameter[T] is to set a query parameter to a value of type T, where the required types are taken from the query[...] type parameters.
From what I see there's no such object defined for T = List[A] for a generic A, and it seems a sensible choice, since you can't actually write a sql query with a dynamic list of parameters for the IN (?, ?, ?,...) clause
I did an experiment by providing such an implicit value through the following code
import scala.slick.jdbc.{SetParameter, StaticQuery => Q}
def seqParam[A](implicit pconv: SetParameter[A]): SetParameter[Seq[A]] = SetParameter {
case (seq, pp) =>
for (a <- seq) {
pconv.apply(a, pp)
}
}
implicit val listSP: SetParameter[List[String]] = seqParam[String]
with this in scope, you should be able to execute your code
val locationCodes = List("loc1","loc2","loc3"...)
Q.query[(Int,Int,List[String]), Visit]("""
select * from visit where vistor = ? and location_code in (?,?,?...)
""").list(visitorId,locationCodes)
But you must always manually guarantee that the locationCodes size is the same as the number of ? in your IN clause
In the end I believe that a cleaner workaround could be created using macros, to generalize on the sequence type. But I'm not sure it would be a wise choice for the framework, given the aforementioned issues with the dynamic nature of the sequence size.
You can generate in clause automaticly like this:
def find(id: List[Long])(implicit options: QueryOptions) = {
val in = ("?," * id.size).dropRight(1)
Q.query[List[Long], FullCard](s"""
select
o.id, o.name
from
organization o
where
o.id in ($in)
limit
?
offset
?
""").list(id ::: options.limits)
}
And use implicit SetParameter as pagoda_5b says
def seqParam[A](implicit pconv: SetParameter[A]): SetParameter[Seq[A]] = SetParameter {
case (seq, pp) =>
for (a <- seq) {
pconv.apply(a, pp)
}
}
implicit def setLongList = seqParam[Long]
If you have a complex query and the for comprehension mentioned above is not an option, you can do something like the following in Slick 3. But you need to make sure you validate the data in your list query parameter yourself to prevent SQL injection:
val locationCodes = "'" + List("loc1","loc2","loc3").mkString("','") + "'"
sql"""
select * from visit where visitor = $visitor
and location_code in (#$locationCodes)
"""
The # in front of the variable reference disables the type validation and allows you to solve this without supplying a function for the implicit conversion of the list query parameter.