Scala+Slick - Get counts on GroupBy results - scala

I'm trying to get counts from a DB using a groupBy on my Scala+Slick code.
Here's my partial code :
object DBJobs extends Table[DBJob]("encoder_job") {
object Status extends Enumeration {
val local = Value("LOCAL")
val encoding = Value("ENCODING")
val done = Value("DONE")
val error = Value("ERROR")
}
implicit val StatusMapper = MappedTypeMapper.base[Status.Value, String] (
{x => x.toString},
{x => x match {case "LOCAL"=>Status(0);case "ENCODING"=>Status(1);case "DONE"=>Status(2);case "ERROR"=>Status(3)}}
)
def id = column[Long]("id", O.PrimaryKey, O.AutoInc)
def status = column[DBJobs.Status.Value]("status", O.NotNull)
def getStats()(implicit session:Session):mutable.Map[Status.Value, Int] = {
var map = mutable.Map[Column[Status.Value], Column[Int]]()
val q = (for { j <- DBJobs } yield (j)).groupBy(_.status).map{
case (s, results) =>
map = map += (s -> results.length)
}
map
}
}
My problem is how to put data in my Map as [DBJobs.Status, Int] instead of [Column[Status.Value], Column[Int]].
Here's the SQL equivalent :
SELECT COUNT( 1 ), status FROM encoder_job GROUP BY STATUS
Slick version: 1.0.1
Thanks

def getStats()(implicit session:Session):mutable.Map[Status.Value, Int] = {
Query(DBJobs).groupBy(_.status).map{
case (s, results) => (s -> results.length)
}
}
// usage
val results = getStats.run
Be aware that putting methods in the table object directly cannot be easily migrated to Slick 2.0. Put them separate, e.g. as method extensions. Also see https://groups.google.com/d/msg/scalaquery/xNtPT6sexXI/zlkgxv6lZ6YJ

Related

generate list of case class with int field without repeat

I want to generate a List of some class which contains several fields. One of them is Int type and it doesn’t have to repeat. Could you help me to write the code?
I tried next:
case class Person(name: String, age: Int)
implicit val genPerson: Gen[Person] =
for {
name <- arbitrary[String]
age <- Gen.posNum[Int]
} yield Person(name, age)
implicit val genListOfPerson: Gen[scala.List[Person]] = Gen.listOfN(3, genPerson)
The problem is that I got an instance of a person with equal age.
If you're requiring that no two Persons in the generated list have the same age, you can
implicit def IntsArb: Arbitrary[Int] = Arbitrary(Gen.choose[Int](0, Int.MaxValue))
implicit val StringArb: Arbitrary[String] = Arbitrary(Gen.listOfN(5, Gen.alphaChar).map(_.mkString))
implicit val PersonGen = Arbitrary(Gen.resultOf(Person.apply _))
implicit val PersonsGen: Arbitrary[List[Person]] =
Arbitrary(Gen.listOfN(3, PersonGen.arbitrary).map { persons =>
val grouped: Map[Int, List[Person]] = persons.groupBy(_.age)
grouped.values.map(_.head) // safe because groupBy
})
Note that this will return a List with no duplicate ages but there's no guarantee that the list will have size 3 (it is guaranteed that the list will be nonempty, with size at most 3).
If having a list of size 3 is important, at the risk of generation failing if the "dice are against you", you can have something like:
def uniqueAges(persons: List[Person], target: Int): Gen[List[Person]] = {
val grouped: Map[Int, List[Person]] = persons.groupBy(_.age)
val uniquelyAged = grouped.values.map(_.head)
val n = uniquelyAged.size
if (n == target) Gen.const(uniquelyAged)
else {
val existingAges = grouped.keySet
val genPerson = PersonGen.arbitrary.retryUntil { p => !existingAges(p.age) }
Gen.listOf(target - n, genPerson)
.flatMap(l => uniqueAges(l, target - n))
.map(_ ++ uniquelyAged)
}
}
implicit val PersonsGen: Arbitrary[List[Person]] =
Arbitrary(Gen.listOfN(3, PersonGen.arbitrary).flatMap(l => uniqueAges(l, 3)))
You can do it as follows:
implicit def IntsArb: Arbitrary[Int] = Arbitrary(Gen.choose[Int](0, Int.MaxValue))
implicit val StringArb: Arbitrary[String] = Arbitrary(Gen.listOfN(5, Gen.alphaChar).map(_.mkString))
implicit val PersonGen = Arbitrary(Gen.resultOf(Person.apply _))
implicit val PersonsGen: Arbitrary[List[Person]] = Arbitrary(Gen.listOfN(3, PersonGen.arbitrary))

Slick inSetBind or inSet doesn't work

I'm trying to limit the results of a Slick 3.1.0 query using inSetBind and always get an empty set back.
Here is the setup:
case class VTag(tagID: Option[Long], kind: String, name: String) {
def toUITag = UITag(tagID = tagID.get, kind = kind, name = name)
}
class VTags(tag: Tag) extends Table[VTag](tag, TagsDao.tableName) {
def tagID = column[Long]("ID", O.PrimaryKey, O.AutoInc)
def name = column[String]("name")
def kind = column[String]("kind")
def * = (tagID.?, name, kind) <>((VTag.apply _).tupled, VTag.unapply)
}
object TagsDao extends SurgeonsTableObject {
val tableVTags = TableQuery[VTags]
val db = GlobalDAO.db
override def tableName: String = "VTAGS"
override def tableSchema: H2Driver.DDL = tableVTags.schema
def getTagsByName(tagsToSearch: Set[String]): Future[Seq[VTag]] = {
val tagList: List[String] = tagsToSearch.toList
val q = for (c <- tableVTags if c.name.inSetBind(tagList)) yield c
db.run(q.result)
}
def all: Future[Seq[VTag]] = {
val q = for (c <- tableVTags.sortBy(_.tagID.asc)) yield c
db.run(q.result)
}
}
getTagsByName is the method that creates problem. All return all 100 tags I insert there. getTagsByName return always zero. I've tried using inSet / inSetBind, passing as input a list(tagList) or a set (tagsToSearch). Always empty result set. What's going on.
Please don't tell me I have to write SQL ...

in Slick 3.0, how to I get from a query to a case class?

I am trying to use Slick for database in a Scala application, and running into some issues (or my misunderstandings) of how to query (find) and convert the result to a case class.
I am not mapping the case class, but the actual values, with the intent of creating the case class on the fly. so, my table is:
object Tables {
class Names(tag: Tag) extends Table[Name](tag, "NAMES") {
def id = column[Long]("id", O.PrimaryKey, O.AutoInc)
def first = column[String]("first")
def middle = column[String]("last")
def last = column[String]("last")
def * = (id.?, first, middle.?, last) <> ((Name.apply _).tupled, Name.unapply)
}
object NamesQueries {
lazy val query = TableQuery[Names]
val findById = Compiled { k: Rep[Long] =>
query.filter(_.id === k)
}
}
}
and here is the query:
object NamesDAO {
def insertName(name: Name) {
NamesQueries.query += name.copy(id = None)
}
def findName(nameId: Long) = {
val q = NamesQueries.findById(nameId) // AppliedCompiledFunction[Long, Query[Tables.Names, Tables.Names.TableElementType, Seq],Seq[Tables.Names.TableElementType]]
val resultSeq = Database.forConfig("schoolme").run(q.result) // Future[Seq[Tables.Names.TableElementType]]
val result = resultSeq.map { r => // val result: Future[(Option[Long], String, Option[String], String) => Name]
val rr = r.map{ name => // val rr: Seq[(Option[Long], String, Option[String], String) => Name]
Name.apply _
}
rr.head
}
result
}
}
however, the findName method seems to return Future((Option[Long], String, Option[String], String) => Name) instead of a Future(Name). What am i doing wrong? Is it just a matter of just using asInstanceOf[Name]?
EDIT: expanded findName to smaller chunks with comments for each one, as sap1ens suggested.
well, i'll be damned.
following sap1ens comment above, I broke findName to multiple steps (and edited the question). but after that, i went back and gave my val an explicit type, and that worked. see here:
def findName(nameId: Long) = {
val q = NamesQueries.findById(nameId)
val resultSeq: Future[Seq[Name]] = Database.forConfig("schoolme").run(q.result)
val result = resultSeq.map { r =>
val rr = r.map{ name =>
name
}
rr.head
}
result
}
so, type inference was the (/my) culprit this time. remember, remember.

Slick 3.0 many-to-many query with the join as an iterable

I've created a many-to-many collection using Slick 3.0, but I'm struggling to retrieve data in the way I want.
There is a many-to-many relationship between Events and Interests. Here are my tables:
case class EventDao(title: String,
id: Option[Int] = None)
class EventsTable(tag: Tag)
extends Table[EventDao](tag, "events") {
def id = column[Int]("event_id", O.PrimaryKey, O.AutoInc)
def title = column[String]("title")
def * = (
title,
id.?) <> (EventDao.tupled, EventDao.unapply)
def interests = EventInterestQueries.query.filter(_.eventId === id)
.flatMap(_.interestFk)
}
object EventQueries {
lazy val query = TableQuery[EventsTable]
val findById = Compiled { k: Rep[Int] =>
query.filter(_.id === k)
}
}
Here's EventsInterests:
case class EventInterestDao(event: Int, interest: Int)
class EventsInterestsTable(tag: Tag)
extends Table[EventInterestDao](tag, "events_interests") {
def eventId = column[Int]("event_id")
def interestId = column[Int]("interest_id")
def * = (
eventId,
interestId) <> (EventInterestDao.tupled, EventInterestDao.unapply)
def eventFk = foreignKey("event_fk", eventId, EventQueries.query)(e => e.id)
def interestFk = foreignKey("interest_fk", interestId, InterestQueries.query)(i => i.id)
}
object EventInterestQueries {
lazy val query = TableQuery[EventsInterestsTable]
}
And finally Interests:
case class InterestDao(name: String,
id: Option[Int] = None)
class InterestsTable(tag: Tag)
extends Table[InterestDao](tag, "interests") {
def id = column[Int]("interest_id", O.PrimaryKey, O.AutoInc)
def name = column[String]("name")
def name_idx = index("idx_name", name, unique = true)
def * = (
name,
id.?) <> (InterestDao.tupled, InterestDao.unapply)
def events = EventInterestQueries.query.filter(_.interestId === id)
.flatMap(_.eventFk)
}
object InterestQueries {
lazy val query = TableQuery[InterestsTable]
val findById = Compiled { k: Rep[Int] =>
query.filter(_.id === k)
}
}
I can query and retrieve tuples of (event.name, interest) with the following:
val eventInterestQuery = for {
event <- EventQueries.query
interest <- event.interests
} yield (event.title, interest.name)
Await.result(db.run(eventInterestQuery.result).map(println), Duration.Inf)
So this is what I currently have.
What I want is to be able to populate a case class like:
case class EventDao(title: String,
interests: Seq[InterestDao],
id: Option[Int] = None)
The trouble is that if I update my case class like this, it messes up my def * projection in EventsTable. Also, I'll have to rename the EventsTable.interests filter to something like EventsTable.interestIds which is a bit ugly but I could live with if necessary.
Also, I can't find a way of writing a for query that yields (event.name, Seq(interest.name)). Anyway, that's just a stepping stone to me being able to yield a (EventDao, Seq(InterestDao)) tuple which is what I really want to return.
Does anyone know how I can achieve these things? I also want to be able to 'take' a certain number of Interests, so for some queries all would be returned, but for others only the first 3 would be.
So after reading this page and chatting on the mailing list, I finally got it working:
val eventInterestQuery = for {
event <- EventQueries.query
interest <- event.interests
} yield (event, interest)
Await.result(db.run(eventInterestQuery.result
// convert the interests to a sequence.
.map {
_.groupBy(_._1)
.map {
case (k,v) => (k, v.map(_._2))
}.toSeq
}
), Duration.Inf)
The only issue with groupBy is you lose order. You could fold the result. I've written this helper for my current project:
def foldOneToMany[A, B](in: Seq[(A, Option[B])], eq: (A, B) => Boolean)
(f: (A, B) => A): Seq[A] =
in.foldLeft(List.empty[A]) {
case (head :: tail, (_, Some(rel))) if eq(head, rel) =>
f(head, rel) :: tail
case (r, (el, Some(rel))) => f(el, rel) :: r
case (r, (el, None)) => el :: r
}.reverse
It could do with some love. Now it takes in a function A,B => Boolean to determine if B belongs to A and a function A,B => A that adds B to A.
Virtualeyes also has a point. In Postgres you could use array_agg function to use a little less bandwidth from the db.

Crashing the compiler with a "MatchError: AnyRef" when I call my scala macro

Edit: I've fixed the problem - I was incorrectly calling .map(f => f.typeSignature.asInstanceOf[TypeRef].args.head) on recursiveOpt, which meant that field.name was giving me the wrong field name in my copy method. I've removed the map and everything is working now.
I am writing a macro that will create a map of update methods for a case class, e.g.
case class Inner(innerStr: String)
case class Outer(outerStr: String, inner: Inner, innerOpt: Option[Inner])
should produce an update map for Outer that is something like
val innerMap = Map("innerStr" -> {(json: JsValue) => Try{(inner: Inner) => inner.copy(innerStr = json)}})
val outerMap = Map("outerStr" -> {(json: JsValue) => Try{(outer: Outer) => outer.copy(outerStr = json)}},
"inner.innerStr" -> {(json: JsValue) => Try{(outer: Outer) => innerMap.get("innerStr").get(json).flatMap(update => outer.copy(inner = update(outer.inner)))},
"innerOpt.innerStr" -> {(json: JsValue) => Try{(outer: Outer) => innerMap.get("innerStr").get(json).flatMap(update => outer.copy(inner = outer.inner.map(inner => update(inner))))})
which would then be called like
val oldOuter = Outer("str", Inner("str"), Some(Inner("str")))
val updatedOuter = outerMap.get("inner.innerStr").get(JsString("newStr")).get(oldOuter)
The idea is that given a json kv pair, I can retrieve the appropriate update method from the map using the key and then apply the update using the value, using implicit conversions to convert from the json value to the appropriate type.
My macro is working for the case of a flat case class, e.g. Inner(innerStr: String), and for a nested case class, e.g. Outer(outerStr: String, inner: Inner). However, the case of the nested option case class, Outer(outerStr: String, innerOpt: Option[Inner]), is crashing the compiler. I'm not sure if I'm doing something disastrously wrong, or if there's a bug in the compiler, or third option. This was done using the Scala 2.11.0-M7 REPL
Below is my code - I'm constructing a Map that accepts String input instead of JsValue input so that I don't need to import the play framework into my REPL. The blacklist filters out fields that should not be in the update map (e.g. one of the case classes we're applying this to has fields like "crypted_password" and "salt" that should never be updated via json sent in through a REST route). baseMethods constructs the key -> method tuples for the flat case, recursiveMethods constructs the key-method tuples for the nested case, and recursiveOptMethods constructs the key-value tuples for the nested option case; at the bottom of the macro these are all merged into a flat sequence and a placed in a Map.
I've tested the code in the recursiveOptMethods quasiquotes to ensure that I'm constructing a properly typed sequence of tuples and haven't found an error (also, this code is extremely similar to the recursiveMethods quasiquotes, which are functioning correctly), and I've tested the code that constructs the base, recursive, and recursiveOpt sequences of symbols and they seem to be doing their job.
Any help as to why I'm crashing the compiler would be greatly appreciated.
import scala.language.experimental.macros
def copyTestImpl[T: c.WeakTypeTag](c: scala.reflect.macros.Context)(blacklist: c.Expr[String]*): c.Expr[Map[String, (String) => scala.util.Try[(T) => T]]] = {
import c.universe._
val blacklistList: Seq[String] = blacklist.map(e => c.eval(c.Expr[String](c.resetAllAttrs(e.tree))))
def isCaseClass(tpe: Type): Boolean = tpe.typeSymbol.isClass && tpe.typeSymbol.asClass.isCaseClass
def isCaseClassOpt(tpe: Type): Boolean = tpe.typeSymbol.name.decoded == "Option" && isCaseClass(tpe.asInstanceOf[TypeRef].args.head)
def rec(tpe: Type): c.Expr[Map[String, (String) => scala.util.Try[(T) => T]]] = {
val typeName = tpe.typeSymbol.name.decoded
val fields = tpe.declarations.collectFirst {
case m: MethodSymbol if m.isPrimaryConstructor => m
}.get.paramss.head.filterNot(field => blacklistList.contains(typeName + "." + field.name.decoded))
val recursive = fields.filter(f => isCaseClass(f.typeSignature))
val recursiveOpt = fields.filter(f => isCaseClassOpt(f.typeSignature))
val base = fields.filterNot(f => isCaseClass(f.typeSignature) || isCaseClassOpt(f.typeSignature))
val recursiveMethods = recursive.map {
field => {
val fieldName = field.name
val fieldNameDecoded = fieldName.decoded
val map = rec(field.typeSignature)
q"""{
val innerMap = $map
innerMap.toSeq.map(tuple => ($fieldNameDecoded + "." + tuple._1) -> {
(str: String) => {
val innerUpdate = tuple._2(str)
innerUpdate.map(innerUpdate => (outer: $tpe) => outer.copy($fieldName = innerUpdate(outer.$fieldName)))
}
})}"""
}}
val recursiveOptMethods = recursiveOpt.map {
field => {
val fieldName = field.name
val fieldNameDecoded = fieldName.decoded
val map = rec(field.typeSignature.asInstanceOf[TypeRef].args.head)
q"""{
val innerMap = $map
innerMap.toSeq.map(tuple => ($fieldNameDecoded + "." + tuple._1) -> {
(str: String) => {
val innerUpdate = tuple._2(str)
innerUpdate.map(innerUpdate => (outer: $tpe) => outer.copy($fieldName = (outer.$fieldName).map(inner => innerUpdate(inner))))
}
})}"""
}}
val baseMethods = base.map {
field => {
val fieldName = field.name
val fieldNameDecoded = fieldName.decoded
val fieldType = field.typeSignature
val fieldTypeName = fieldType.toString
q"""{
$fieldNameDecoded -> {
(str: String) => scala.util.Try {
val x: $fieldType = str
(t: $tpe) => t.copy($fieldName = x)
}.recoverWith {
case e: Exception => scala.util.Failure(new IllegalArgumentException("Failed to parse " + str + " as " + $typeName + "." + $fieldNameDecoded + ": " + $fieldTypeName))
}
}}"""
}}
c.Expr[Map[String, (String) => scala.util.Try[(T) => T]]] {
q"""{ Map((List(..$recursiveMethods).flatten ++ List(..$recursiveOptMethods).flatten ++ List(..$baseMethods)):_*) }"""
}
}
rec(weakTypeOf[T])
}
def copyTest[T](blacklist: String*) = macro copyTestImpl[T]
And the top and bottom of my error from the 2.11.0-M7 REPL when calling copyTest[Outer]() (where Outer has an Option[Inner] field)
scala> copyTest[Outer]()
scala.MatchError: AnyRef
with Product
with Serializable {
val innerStr: String
private[this] val innerStr: String
def <init>(innerStr: String): Inner
def copy(innerStr: String): Inner
def copy$default$1: String #scala.annotation.unchecked.uncheckedVariance
override def productPrefix: String
def productArity: Int
def productElement(x$1: Int): Any
override def productIterator: Iterator[Any]
def canEqual(x$1: Any): Boolean
override def hashCode(): Int
override def toString(): String
override def equals(x$1: Any): Boolean
} (of class scala.reflect.internal.Types$ClassInfoType)
at scala.reflect.internal.Variances$class.inType$1(Variances.scala:181)
at scala.reflect.internal.Variances$$anonfun$inArgs$1$1.apply(Variances.scala:176)
at scala.reflect.internal.Variances$$anonfun$inArgs$1$1.apply(Variances.scala:176)
at scala.reflect.internal.util.Collections$class.map2(Collections.scala:55)
at scala.reflect.internal.SymbolTable.map2(SymbolTable.scala:14)
at scala.reflect.internal.Variances$class.inArgs$1(Variances.scala:176)
at scala.reflect.internal.Variances$class.inType$1(Variances.scala:189)
at scala.reflect.internal.Variances$$anonfun$inArgs$1$1.apply(Variances.scala:176)
at scala.reflect.internal.Variances$$anonfun$inArgs$1$1.apply(Variances.scala:176)
at scala.reflect.internal.util.Collections$class.map2(Collections.scala:55)
at scala.reflect.internal.SymbolTable.map2(SymbolTable.scala:14)
at scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3.run(Analyzer.scala:93)
at scala.tools.nsc.Global$Run.compileUnitsInternal(Global.scala:1603)
at scala.tools.nsc.Global$Run.compileUnits(Global.scala:1588)
at scala.tools.nsc.Global$Run.compileSources(Global.scala:1583)
at scala.tools.nsc.interpreter.IMain.compileSourcesKeepingRun(IMain.scala:387)
at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.compileAndSaveRun(IMain.scala:816)
at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.compile(IMain.scala:775)
at scala.tools.nsc.interpreter.IMain$Request.compile$lzycompute(IMain.scala:951)
at scala.tools.nsc.interpreter.IMain$Request.compile(IMain.scala:946)
at scala.tools.nsc.interpreter.IMain.compile(IMain.scala:530)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:518)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:516)
at scala.tools.nsc.interpreter.ILoop.reallyInterpret$1(ILoop.scala:748)
at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:793)
at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:660)
at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:427)
at scala.tools.nsc.interpreter.ILoop.loop(ILoop.scala:444)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:862)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:848)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:848)
at scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:95)
at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:848)
at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:81)
at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:94)
at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:103)
at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)
That entry seems to have slain the compiler. Shall I replay
your session? I can re-run each line except the last one.
I found the problem - originally I had val recursiveOpt = fields.filter(f => isCaseClassOpt(f.typeSignature)).map(f => f.typeSignature.asInstanceOf[TypeRef].args.head), which meant that when I called field.name on the recursiveOpt fields I was getting the wrong name back.