Slick 3.0.0 Select and Create or Update - scala

I'am in a situation where in I have to do a select first, use the value to issue a create. It is some versioning that I'm trying to implement. Here is the table definition:
class Table1(tag: Tag) extends Table[(Int, String, Int)](tag, "TABLE1") {
def id = column[Int]("ID")
def name = column[String]("NAME")
def version = column[Int]("VERSION")
def indexCol = index("_a", (id, version))
val tbl1Elems = TableQuery[Table1]
}
So when a request comes to create or update an entry in Table1, I have to do the following:
1. Select for the given id, if exists, get the version
2. Increment the version
3. Create a new entry
All that should happen in a single transaction. Here is what I have got so far:
// this entry should be first checked if the id exists and if yes get //the complete set of columns by applying a filter that returns the max //version
val table1 = Table1(2, "some name", 1)
for {
tbl1: Table1 <- tbl1MaxVersionFilter(table1.id)
maxVersion: Column[Int] = tbl1.version
result <- tbl1Elems += table1.copy(version = maxVersion + 1) // can't use this!!!
} yield result
I will later wrap that entire block in one transaction. But I',m wondering how to complete that will create a new version? How can I get the value maxVersion out of the Column so that I can increment 1 to it and use it?

I would go with a static query, with something like this
import scala.slick.jdbc.{StaticQuery=>Q}
def insertWithVersion(id: Int,name:String) =
( Q.u + "insert into table1 select " +?id + "," +?name + ", (
select coalese(max(version),1) from table1 where id=" +?id +")" ).execute
If you want to write it using slick way then take a look at the following
val tableOne = TableQuery[Table1]
def updateWithVersion(newId:Int,name:String):Unit = {
val version = tableOne.filter( _.id === newId).map( _.version).max.run.getOrElse(1)
tableOne += (newId,name,version)
}
The idea is select the max version in the same query and if there is no version use 1 and insert it. Also, as the whole logic is issued in a single statement, no extra transaction management is needed.
P.S. There might be some error is sql and code.

Related

How to update table query in Slick

How can I convert Query[MappedProjection[Example, (Option[String], Int, UUID, UUID)], Example, Seq] to Query[Examples, Example, Seq]?
Details
I am trying to drop a column from an existing table(Examples in this case) and move the data to another table (Examples2 in this case). I don't want to change all the existing code base, so I plan to join these two tables and map the results to Example.
import slick.lifted.Tag
import slick.driver.PostgresDriver.api._
import java.util.UUID
case class Example(
field1: Option[String] = None,
field2: Int,
someForeignId: UUID,
id: UUID,
)
object Example
class Examples(tag: Tag) extends Table[Example](tag, "entityNotes") {
def field1 = column[Option[String]]("field1")
def field2 = column[Int]("field2")
def someForeignId = column[UUID]("someForeignId")
def id = column[UUID]("id", O.PrimaryKey)
def someForeignKey = foreignKey(
"someForeignIdToExamples2",
someForeignId,
Examples2.query,
)(
_.id.?
)
def * =
(
field1.?,
field2,
someForeignId,
id,
) <> ((Example.apply _).tupled, Example.unapply)
}
object Examples{
val query = TableQuery[Examples]
}
Basically, all the functions in the codebase call Examples.query. If I update that query by joining two tables, the problem will be solved (of course with a performance shortcoming because of one extra join for each call).
To use the query with the existing code base, we need to keep the type the same. For example, we we can use filter as follows:
val query_ = TableQuery[Examples]
val query: Query[Examples, Example, Seq] = query_.filter(_.field2 > 5)
Everything will work without a problem since we keep the type of the query as it is supposed to be.
However, I cannot do that with a join if I want to use data from the second table.
val query_ = TableQuery[Examples]
val query = query
.join(Examples2.query_)
.on(_.someForeignId === _.id)
.map({
case (e, e2) =>
((
e2.value.?,
e1.field2,
e2.id
e.id,
) <> ((Example.apply _).tupled, Example.unapply))
})
This is where I got stuck. Its type is Query[MappedProjection[Example, (Option[String], Int, UUID, UUID)], Example, Seq].
Can anyone help? Btw, we don't have to use map. This is just what I got so far.

Scala, Slick: How to use insert using auto generated number and byte array

object FingerprintsModel extends FingerprintDAO {
// Fingerprint class definition
class FingerprintsTable(tag: Tag) extends Table[Fingerprint](tag, "fingerprints") {
def id = column[Long]("id", O.PrimaryKey, O.AutoInc)
def customerId = column[String]("customer_id", O.NotNull)
def template_one = column[Array[Byte]]("template_one", O.NotNull)
def template_two = column[Array[Byte]]("template_two", O.NotNull)
def created = column[DateTime]("created", O.NotNull)
def updated = column[Option[DateTime]]("updated")
def * = (id, customerId, template_one, template_two) <> (Fingerprint.tupled, Fingerprint.unapply _)
def fingerprint = foreignKey("CUSTOMER", customerId, CustomersModel.customers)(_.id)
}
and this is my insert statement:
FingerprintsModel.fingerprints.map(fi => (fi.customerId, fi.template_one, fi.template_two, fi.created))
.insert((id, fingerprint.template_one, fingerprint.template_two, new DateTime()))
Summary
There are two main modifications you need:
You will want a TableQuery[FingerprintsTable] to call insert (or += or ++=on); and
To get back the IDs inserted you need to use the returning method in Slick.
Worked example
It's hard to tell from the code you posted exactly what you have in mind. It would be helpful next time to simplify your example first.
I've assumed your model is something like this:
case class Fingerprint(
id: Long,
customerId: String,
template_one: Array[Byte]
)
I've left out one of the byte arrays and the created and updated fields as they don't seem relevant to the question. In other words, I've simplified.
The FingerprintTable seems ok. I'm ignoring the foreign key as that doesn't seem relevant. Oh, the O.NotNull are now deprecated (in Slick 3 at least). You can leave them off because your columns are not Option values.
What we need is the table query, which I'd add inside FingerprintsModel:
lazy val fingerprints = TableQuery[FingerprintsTable]
lazy val fingerprintsWithID = fingerprints returning fingerprints.map(_.id)
You could use fingerprints to insert data. But you've asked for the IDs back, so you want to use fingerprintsWithID.
Putting it all together (again, using Slick 3 here):
object FingerprintExample extends App {
import FingerprintsModel._
val testData = Seq(
Fingerprint(0L, "Alice", Array(0x01, 0x02)),
Fingerprint(0L, "Bob", Array(0x03, 0x04))
)
// A program that will create the schema, and insert the data, returning the IDs
val program = for {
_ <- fingerprints.schema.create
ids <- fingerprintsWithID ++= testData
} yield ids
// Run the program using an in-memory database
val db = Database.forConfig("h2mem1")
val future = db.run(program)
val result = Await.result(future, 10 seconds)
println(s"The result is: $result")
}
Produces:
The result is: List(1, 2)

Scala's Slick with multiple PK insertOrUpdate() throws exception ERROR: syntax error at end of input

I am using Scala' Slick and PostgreSQL.
And I am working well with tables with single PK.
Now I need to use a table with multiple PKs:
case class Report(f1: DateTime,
f2: String,
f3: Double)
class Reports(tag: Tag) extends Table[Report](tag, "Reports") {
def f1 = column[DateTime]("f1")
def f2 = column[String]("f2")
def f3 = column[Double]("f3")
def * = (f1, f2, f3) <> (Report.tupled, Report.unapply)
def pk = primaryKey("pk_report", (f1, f2))
}
val reports = TableQuery[Reports]
when I have empty table and use reports.insert(report) it works well.
But when I use reports.insertOrUpdate(report) I receive and exception:
Exception in thread "main" org.postgresql.util.PSQLException: ERROR: syntax error at end of input
Position: 76
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2102)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1835)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
at ....
What am I doing wrong? How to fix it?
Thanks in advance.
PS. I tried workaround - tried to implement "if exist update else insert" logic by:
val len = reports.withFilter(_.f1 === report.f1).withFilter(_.f2 === report.f2).length.run.toInt
if(len == 1) {
println("Update: " + report)
reports.update(report)
} else {
println("Insert: " + report)
reports.insert(report)
}
But I still get exception on update:
Exception in thread "main" org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "pk_report"
Detail: Key ("f1", f2)=(2014-01-31 04:00:00, addon_io.aha.connect) already exists.
Concerning your initial question, insertOrUpdate on a table with compound keys is broken in Slick (at least with PGSql), so the error is not on your side. See bug report e.g., here: https://github.com/slick/slick/issues/966
So you have to design a workaround, however the "upsert" operation is very prone to race conditions, and is very hard to design properly as PostgreSQL does not provide native feature to perform this. See e.g., http://www.depesz.com/2012/06/10/why-is-upsert-so-complicated/
Anyway, another way to perform the operation which is a bit less prone to race condition is to first update (which will do not do anything if the row does not exist), and then perform a "insert select" query, which will only insert if the row does not exist. This is the wày Slick will perform the insertOrUpdate operation on PostgreSQL with single PK. However, "insert select" cannot be done using Slick directly, you will have to fallback to direct SQL.
Second part where you have
val len = reports.withFilter(_.f1 === report.f1).withFilter(_.f2 === report.f2).length.run.toInt
if(len == 1) {
println("Update: " + report)
reports.update(report)
} else {
println("Insert: " + report)
reports.insert(report)
}
change reports.update(report) with
reports.filter(_.id === report.id).update(report)
actually you can just make one filter call (replacing your first withFilter )
I've successfully applied the technique described here
so my upsert method looks like this:
def upsert(model: String, module: String, timestamp: Long) = {
// see this article http://www.the-art-of-web.com/sql/upsert/
val insert = s"INSERT INTO $ModulesAffectedTableName (model, affected_module, timestamp) SELECT '$model','$module','$timestamp'"
val upsert = s"UPDATE $ModulesAffectedTableName SET timestamp=$timestamp WHERE model='$model' AND affected_module='$module'"
val finalStmnt = s"WITH upsert AS ($upsert RETURNING *) $insert WHERE NOT EXISTS (SELECT * FROM upsert)"
conn.run(sqlu"#$finalStmnt")
}
Hopefully this issue will be fixed in 3.2.0
Currently, I work around this issue by creating a dummy table for table creation:
class ReportsDummy(tag: Tag) extends Table[Report](tag, "Reports") {
def f1 = column[DateTime]("f1")
def f2 = column[String]("f2")
def f3 = column[Double]("f3")
def * = (f1, f2, f3) <> (Report.tupled, Report.unapply)
def pk = primaryKey("pk_report", (f1, f2))
}
and a "real" table for upsert
class Reports(tag: Tag) extends Table[Report](tag, "Reports") {
def f1 = column[DateTime]("f1", O.PrimaryKey)
def f2 = column[String]("f2", O.PrimaryKey) //two primary keys here, which would throw errors on table creation. Hence a dummy one for the task
def f3 = column[Double]("f3")
def * = (f1, f2, f3) <> (Report.tupled, Report.unapply)
}

Recursive tree-like table query with Slick

My table data forms a tree structure where one row can reference a parent row in the same table.
What I am trying to achieve, using Slick, is to write a query that will return a row and all it's children. Also, I would like to do the same, but write a query that will return a child and all it's ancestors.
In other words:
findDown(1) should return
List(Group(1, 0, "1"), Group(3, 1, "3 (Child of 1)"))
findUp(5) should return
List(Group(5, 2, "5 (Child of 2)"), Group(2, 0, "2"))
Here is a fully functional worksheet (except for the missing solutions ;-).
package com.exp.worksheets
import scala.slick.driver.H2Driver.simple._
object ParentChildTreeLookup {
implicit val session = Database.forURL("jdbc:h2:mem:test1;", driver = "org.h2.Driver").createSession()
session.withTransaction {
Groups.ddl.create
}
Groups.insertAll(
Group(1, 0, "1"),
Group(2, 0, "2"),
Group(3, 1, "3 (Child of 1)"),
Group(4, 3, "4 (Child of 3)"),
Group(5, 2, "5 (Child of 2)"),
Group(6, 2, "6 (Child of 2)"))
case class Group(
id: Long = -1,
id_parent: Long = -1,
label: String = "")
object Groups extends Table[Group]("GROUPS") {
def id = column[Long]("ID", O.PrimaryKey, O.AutoInc)
def id_parent = column[Long]("ID_PARENT")
def label = column[String]("LABEL")
def * = id ~ id_parent ~ label <> (Group, Group.unapply _)
def autoInc = id_parent ~ label returning id into {
case ((_, _), id) => id
}
def findDown(groupId: Long)(implicit session: Session) = { ??? }
def findUp(groupId: Long)(implicit session: Session) = { ??? }
}
}
A really bad, and static attempt at findDown may be something like:
private def groupsById = for {
group_id <- Parameters[Long]
g <- Groups; if g.id === group_id
} yield g
private def childrenByParentId = for {
parent_id <- Parameters[Long]
g <- Groups; if g.id_parent === parent_id
} yield g
def findDown(groupId: Long)(implicit session: Session) = { groupsById(groupId).list union childrenByParentId(groupId).list }
But, I'm looking for a way for Slick to recursively search the same table using the id and id_parent link. Any other good ways to solve the problem is really welcome. Keep in mind though, that it would be best to minimise the number of database round-trips.
You could try calling SQL from slick. The SQL call to go up the hierarchy would look something like this (This is for SQL Server):
WITH org_name AS
(
SELECT DISTINCT
parent.id AS parent_id,
parentname.label as parent_label,
child.id AS child_id,
childname.ConceptName as child_label
FROM
Group parent RIGHT OUTER JOIN
Group child ON child.parent_id = parent.id
),
jn AS
(
SELECT
parent_id,
parent_label,
child_id,
child_label
FROM
org_name
WHERE
parent_id = 5
UNION ALL
SELECT
C.parent_id,
C.parent_label,
C.child_id,
C.child_label
FROM
jn AS p JOIN
org_name AS C ON C.child_id = p.parent_id
)
SELECT DISTINCT
jn.parent_id,
jn.parent_label,
jn.child_id,
jn.child_label
FROM
jn
ORDER BY
1;
If you want to go down the hierarchy change the line:
org_name AS C ON C.child_id = p.parent_id
to
org_name AS C ON C.parent_id = p.child_id
In plain SQL this would be tricky. You would have multiple options:
Use a stored procedure to collect the correct records (recursively). Then convert those records into a tree using code
Select all the records and convert those into a tree using code
Use more advanced technique as described here (from Optimized SQL for tree structures) and here. Then convert those records into a tree using code
Depending on the way you want to do it in SQL you need to build a Slick query. The concept of Leaky Abstractions is very evident here.
So getting the tree structure requires two steps:
Get the correct (or all records)
Build (using regular code) a tree from those records
Since you are using Slick I don't think it's an option, but another database type might be a better fit for your data model. Check out NoSQL for the differences between the different types.

Why is this simple scala code taking so much memory and finally crashing? (Play framework 2.0)

I'm trying a simple scala code in play framework 2.0 to fill my db (other options exists, such as importing a SQL file directly within the database, but that's not the point) :
def filldb = Action {
import play.api.db.DB
import anorm._
var result: Boolean = false
val tuples: List[(Long, String)] = DB
.withConnection("playground") { implicit c =>
for (i <- 1 until 1000000) {
SQL("""
INSERT INTO article (
id,
title
) VALUES (
""" + i + """,
'Article no """ + i + """');"""
).executeUpdate()
if (i % 1000 == 0) println("i:" + i)
}
val sqlQuery = SQL("select id, title from article order by id;")
sqlQuery().map(row =>
row[Long]("id") -> row[String]("title")).toList
}
Ok("done")
}
This runs well for a while (200K iterations), slows down, eats up memory progressively (up to 1.8GB), and finally crashes from lack of memory.
Can someone explain me what causes this behaviour?
It's clear that it's possible to code it in different ways, but the point is to understand what is wrong, so that the error would not be done in another context...
To be complete, here are the details :
OS : mac 10.6.8
play : 2.0
database : mysql 5.5.12
table :
CREATE TABLE article (
id bigint(20) NOT NULL UNIQUE,
title varchar(255) NOT NULL,
PRIMARY KEY (id)
);
Tried this as weel, with no more success :
def filldb = Action {
import play.api.db.DB
import anorm._
var result: Boolean = false
val connection = DB.getConnection("playground")
for (i <- 1 until 1000000) {
SQL("""
INSERT INTO article (
id,
title
) VALUES (
""" + i + """,
'Article no """ + i + """');"""
).executeUpdate()(connection)
if (i % 1000 == 0) println("i:" + i)
}
val tuples: List[(Long, String)] = {
val sqlQuery = SQL("select id, title from article order by id;")
sqlQuery()(connection).map(row =>
row[Long]("id") -> row[String]("title")).toList
}
connection.close()
Ok("done")
}
Not better : stuck at 283k iterations...
My first guess is that you might still be using the default in memory database. Can you check and make sure your conf/application.conf is not using jdbc:h2:mem:play? If so, all of your entries would be filling up your memory.
Also, each statement that you make opens a statement object that doesn't get closed until the end of the withConnection block. Since you have a million of them sitting in memory, that can build up. See http://www.playframework.org/documentation/2.0/ScalaDatabase
You could try populating the DB outside of your query operation. I would experiment with doing 1000 batches of 1000 and seeing if that identifies your problem.
I think that your problem lies here:
val tuples: List[(Long, String)] = {
val sqlQuery = SQL("select id, title from article order by id;")
sqlQuery()(connection).map(row =>
row[Long]("id") -> row[String]("title")).toList
}
You are populating a map with all the rows form the database so you are fillng a Scala/Java data stucture with 1M of rows.
Do you really need one million of rows all at once? Or do you need them in paginated way (i.e the first 20 the second 20, etc,etc).
It's not a play problem, you will encounter the same problem even with java and a single plain jdbc test. Tell us the real usage of tuples and we can provide some suggestions.