Let's say I want to re-order some elements in a single-linked list of records...
The approach that came to mind is to build an array of (previous, current) id tuples and then unnest that in the FROM clause of an UPDATE statement.
I.e. something like this:
private fun updateCompositionElementOrdering(elementIdsInExpectedOrder: List<Int>) {
val PREVIOUS_ELEMENT = DSL.field("previous_element_id", Int::class.javaObjectType)
val CURRENT_ELEMENT = DSL.field("current_element_id", Int::class.javaObjectType)
val previousAndCurrent = elementIdsInExpectedOrder.mapIndexed { i, currentElement ->
val previousElementId = if (i == 0) null else elementIdsInExpectedOrder[i - 1]
DSL.row(previousElementId, currentElement)
}
ctx
.update(COMPOSITION_ELEMENT)
.set(COMPOSITION_ELEMENT.PREVIOUS_COMPOSITION_ELEMENT_ID, PREVIOUS_ELEMENT)
.from(
DSL.unnest(previousAndCurrent).`as`(DSL.name("p"), PREVIOUS_ELEMENT.unqualifiedName, CURRENT_ELEMENT.unqualifiedName)
)
.where(COMPOSITION_ELEMENT.ID.eq(CURRENT_ELEMENT))
.execute()
}
where private val ctx: DSLContext, of course.
But this results in:
jOOQ; bad SQL grammar [
update "public"."composition_element"
set
"previous_composition_element_id" = previous_element_id,
"modified_by_id" = ?
from
unnest(cast(? as any[])) as "values" (previous_element_id, current_element_id)
where "public"."composition_element"."id" = current_element_id
];
nested exception is org.postgresql.util.PSQLException: ERROR: syntax error at or near "any"
(this is with jOOQ 3.17.4 and Postgres 13.5)
jOOQ currently doesn't support unnesting arrays of rows. It probably should: https://github.com/jOOQ/jOOQ/issues/14505
... but this is a hairy thing also in PostgreSQL. For example:
select *
from unnest(array[row(1, 2), row(3, 4)]) as t (a, b)
Produces:
SQL Error [42601]: ERROR: a column definition list is required for functions returning "record"
You'd need to specify the column type:
select *
from unnest(array[row(1, 2), row(3, 4)]) as t (a integer, b integer)
There's a feature request for this, but it's not too popular, given how esoteric the feature is: https://github.com/jOOQ/jOOQ/issues/5926
Maybe, better just use DSL.values()?
DSL.values(previousAndCurrent)
.`as`(DSL.name("p"),
PREVIOUS_ELEMENT.unqualifiedName,
CURRENT_ELEMENT.unqualifiedName)
Related
I'm trying to query dynamoDB through withFilterExpression. I get an error as the argument is a composite key
Filter Expression can only contain non-primary key attributes: Primary key attribute: question_id
and also as it uses OR operator in the query and it cannot be passed to withKeyConditionExpression.
The query that was passed to withFilterExpression is similar to this question_id = 1 OR question_id = 2. The entire code is like follows
def getQuestionItems(conceptCode : String) = {
val qIds = List("1","2","3")
val hash_map : java.util.Map[String, Object] = new java.util.HashMap[String, Object]()
var queries = ArrayBuffer[String]()
hash_map.put(":c_id", conceptCode)
for ((qId, index) <- qIds.zipWithIndex) {
val placeholder = ":qId" + index
hash_map.put(placeholder, qId)
queries += "question_id = " + placeholder
}
val query = queries.mkString(" or ")
val querySpec = new QuerySpec()
.withKeyConditionExpression("concept_id = :c_id")
.withFilterExpression(query)
.withValueMap(hash_map)
questionsTable.query(querySpec)
}
Apart from withFilterExpression and withConditionExpression is there any other methods that I can use which is a part of QuerySpec ?
Let's raise things up a level. With a Query (as opposed to a GetItem or Scan) you provide a single PK value and optionally an SK condition. That's what a Query requires. You can't provide multiple PK values. If you want multiple PK values, you can do multiple Query calls. Or possibly you may consider a Scan across all PK values.
You can also consider having a GSI that presents the data in a format more suitable to efficient lookup.
Side note: With PartiQL you can actually specify multiple PK values, up to a limit. So if you really truly want this, that's a possibility. The downside is it raises things up to a new level of abstraction and can make inefficiencies hard to spot.
I have a database structure with a many-to-many relationship between Dreams and Tags.
Dreams and Tags are kept in separate tables, and there is a join table between them as usual in this kind of situation, with the class DreamTag representing the connection:
protected class DreamTagTable(tag: Tag) extends Table[DreamTag](tag, "dreamtags") {
def dreamId = column[Long]("dream_id")
def dream = foreignKey("dreams", dreamId, dreams)(_.id)
def tagId = column[Long]("tag_id")
def tag = foreignKey("tags", tagId, tags)(_.id)
// default projection
def * = (dreamId, tagId) <> ((DreamTag.apply _).tupled, DreamTag.unapply)
}
I have managed to perform the appropriate double JOIN to retrieve a Dream with its Tags, but I struggled to do it in a fully non-blocking manner.
Here is my code for performing the retrieval, as this may shed some light on things:
def createWithTags(form: DreamForm): Future[Seq[Int]] = db.run {
logger.info(s"Creating dream [$form]")
// action to put the dream
val dreamAction: DBIO[Dream] =
dreams.map(d => (d.title, d.content, d.date, d.userId, d.emotion))
.returning(dreams.map(_.id))
.into((fields, id) => Dream(id, fields._1, fields._2, fields._3, fields._4, fields._5))
.+=((form.title, form.content, form.date, form.userId, form.emotion))
// action to put any tags that don't already exist (create a single action)
val tagActions: DBIO[Seq[MyTag]] =
DBIO.sequence(form.tags.map(text => createTagIfNotExistsAction(text)))
// zip allows us to get the results of both actions in a tuple
val zipAction: DBIO[(Dream, Seq[MyTag])] = dreamAction.zip(tagActions)
// put the entries into the join table, if the zipAction succeeds
val dreamTagsAction = exec(zipAction.asTry) match {
case Success(value) => value match {
case (dream, tags) =>
DBIO.sequence(tags.map(tag => createDreamTagAction(dream, tag)))
}
case Failure(exception) => throw exception
}
dreamTagsAction
}
private def createTagIfNotExistsAction(text: String): DBIO[MyTag] = {
tags.filter(_.text === text).result.headOption.flatMap {
case Some(t: MyTag) => DBIO.successful(t)
case None =>
tags.map(t => (t.text))
.returning(tags.map(_.id))
.into((text, id) => MyTag(id, text)) += text
}
}
private def createDreamTagAction(dream: Dream, tag: MyTag): DBIO[Int] = {
dreamTags += DreamTag(dream.id, tag.id)
}
/**
* Helper method for executing an async action in a blocking way
*/
private def exec[T](action: DBIO[T]): T = Await.result(db.run(action), 2.seconds)
Scenario
Now I'm at the stage where I want to be able to update a Dream and the list of Tags, and I'm struggling.
Given a situation where the existing list of tags is ["one", "two", "three"] and is being updated to ["two", "three", "four"] I want to:
Delete the Tag for "one", if no other Dreams reference it.
Not touch the entries for "two" and "three", as the Tag and DreamTag entries already exist.
Create Tag "four" if it doesn't exist, and add an entry to the join table for it.
I think I need to do something like list1.diff(list2) and list2.diff(list1) but that would require getting performing a get first, which seems wrong.
Perhaps my thinking is wrong - Is it best to just clear all entries in the join table for this Dream and then create every item in the new list, or is there a nice way to diff the two lists (previous and existing) and perform the deletes/adds as appropriate?
Thanks for the help.
N.B. Yes, Tag is a super-annoying class name to have, as it clashes with slick.lifted.Tag!
Update - My Solution:
I went for option 2 as mentioned by Richard in his answer...
// action to put any tags that don't already exist (create a single action)
val tagActions: DBIO[Seq[MyTag]] =
DBIO.sequence(form.tags.map(text => createTagIfNotExistsAction(text)))
// zip allows us to get the results of both actions in a tuple
val zipAction: DBIO[(Int, Seq[MyTag])] = dreamAction.zip(tagActions)
// first clear away the existing dreamtags
val deleteExistingDreamTags = dreamTags
.filter(_.dreamId === dreamId)
.delete
// put the entries into the join table, if the zipAction succeeds
val dreamTagsAction = zipAction.flatMap {
case (_, tags) =>
DBIO.sequence(tags.map(tag => createDreamTagAction(dreamId, tag)))
}
deleteExistingDreamTags.andThen(dreamTagsAction)
I struggled to do it in a fully non-blocking manner.
I see you have an eval call which is blocking. I looks like this can be replaced with a flatMap:
case class Dream()
case class MyTag()
val zipAction: DBIO[(Dream, Seq[MyTag])] =
DBIO.successful( (Dream(), MyTag() :: MyTag() :: Nil) )
def createDreamTagAction(dream: Dream)(tag: MyTag): DBIO[Int] =
DBIO.successful(1)
val action: DBIO[Seq[Int]] =
zipAction.flatMap {
case (dream, tags) => DBIO.sequence(tags.map(createDreamTagAction(dream)))
}
Is it best to just clear all entries in the join table for this Dream and then create every item in the new list, or is there a nice way to diff the two lists (previous and existing) and perform the deletes/adds as appropriate?
Broadly, you have three options:
Look in the database to see what tags exist, compare them to what you want the state to be, and compute a set of insert and delete actions.
Delete all the tags and insert the state you want to reach.
Move the problem to SQL so you insert tags where they don't already exist in the table, and delete tags that don't exist in your desired state. You'd need to look at the capabilities of your database and likely need to use Plain SQL in Slick to get the effect. I'm not sure what the insert would be for adding tags (perhaps a MERGE or upsert of some kind), but deleting would be of the form: delete from tags where tag not in (1,2) if you wanted a final state of just tags 1 and 2.
The trades off:
For 1, you need to run 1 query to fetch existing tags, and then 1 query for the deletes, and at least 1 for the inserts. This will change the smallest number of rows, but will be the largest number of queries.
For 2, you'll be executing at least 2 queries: a delete and 1 (potentially) for a bulk insert. This will change the largest number of rows.
For 3, you'll be executing a constant 2 queries (if your database can carry out the logic for you). If this is even possible, the queries will be more complicated.
I'm currently facing an issue with my update query in my scala-slick3 project. I have a Report-Class, which contains multiple Products and each Product contains multiple Parts. I want to implement a function that marks every Part of every Product within this Report as assessed.
I thought about doing something like this:
def markProductPartsForReportAsAssessed(reportId: Int) = {
val query = for {
(products, parts) <- (report_product_query filter(_.reportId === reportId)
join (part_query filter(_.isAssessed === false))
on (_.productId === _.productId))
} yield parts.isAssessed
db.run(query.update(true))
}
Now, when I run this code slick throws this exception:
SlickException: A query for an UPDATE statement must resolve to a comprehension with a single table.
I already looked at similiar problems of which their solutions (like this or this) weren't really satisfying to me.
Why does slick throw this excpetion or why is it a problem to begin with? I was under the impression that my yield already takes care of not "updating multiple tables".
Thanks in advance!
I guess it's because the UPDATE query requires just one table. If you write SQL for the above query, it can be
UPDATE parts a SET isAccessed = 'true'
WHERE a.isAccessed = 'false' and
exists(select 'x' from products b
where a.productId = b.producId and b.reportId = reportId)
Therefore, you can put conditions related with 'Product' table in the filter as follows.
val reportId = "123" // some variable
val subQuery = (reportId:Rep[String], productId:Rep[String]) =>
report_product_query.filter(r => r.report_id === reportId && r.product_id === productId)
val query = part_query.filter(p => p.isAccesssed === false:Rep[Boolean] &&
subQuery(reportId, p.productId).exists).map(_.isAccessed)
db.run(query.update(true))
I have a database table InventoryLotUsage which has columns id, inventoryLotId, and date. I want to delete an InventoryLot, but before I can do that I need to update the InventoryLotUsage rows that have a foreign key inventoryLotId, based on date and some other conditions.
My question is how do I query for data using monadic joins, do some computations on it and use the result of those computations to run an update all in one transaction in Slick?
I was attempting to get a sequence of rows like this
for {
il <- InventoryLot.filter(_.id === id)
lotUsage <- InventoryLotUsage.filter(_.inventoryLotId === id).result
groupedUsage = lotUsage.groupBy(_.date)
...
}
my IDE suggests that lotUsage will be a Seq[InventoryLotUsageRows], but when compiling I get a type error because of the .result.
type mismatch;
found : slick.dbio.DBIOAction[FacilityProductRepository.this.InventoryLot,slick.dbio.NoStream,slick.dbio.Effect.Read with slick.dbio.Effect.Read with slick.dbio.Effect.Read]
required: slick.lifted.Query[?,?,?]
lotUsage <- InventoryLotUsage.filter(_.inventoryLotId === id).result
Without using .result its type is the InventoryLotUsage table. How can I wrangle the query into something usable for computation?
You need to compose DBIOActions to archive desired result. For example:
Load all data that you need
val loadStaffAction = (for {
il <- InventoryLot.filter(_.id === id)
lotUsage <- InventoryLotUsage.filter(_.inventoryLotId === id)
} yield (il, lotUsage)).result
Then you could use map/flatMap on loadStaffAction to create update statements based on computations. You can also use for-comprehensions here.
val updateAction = loadStaffAction.map { result =>
// creating update statements based on some computations and conditions
DBIO.seq(
InventoryLotUsage.filter(_.id === inventory1.id).map(_.name).update("new value"),
InventoryLotUsage.filter(_.id === inventory2.id).map(_.name).update("another value"),
);
}
After this you can run all queries in one transaction
db.run(updateAction.transactionally)
I'm trying to build a list page like the one in the "Computers" sample. My environment is Play 2.0 and PostrgreSQL 9.0
I have the following method in my User object:
def list(page: Int = 0, pageSize: Int = 10, orderBy: Int = 1, filter: String = "%"): Page[User] = {
val offset = pageSize * page
val mode = if (orderBy > 0) "ASC NULLS FIRST" else "DESC NULLS LAST"
Logger.debug("Users.list with params: page[%d] pageSize[%d] orderBy[%d] filter[%s] order[%s]".format(page, pageSize, orderBy, filter, mode))
DB.withConnection {
implicit connection =>
val users = SQL(
"""
select * from publisher
where name ilike {filter}
order by {orderBy} %s
limit {pageSize} offset {offset}
""".format(mode)
).on(
'pageSize -> pageSize,
'offset -> offset,
'filter -> filter,
'orderBy -> scala.math.abs(orderBy)
).as(User.simple *)
val totalRows = SQL(
"""
select count(*) from publisher
where name like {filter}
"""
).on(
'filter -> filter
).as(scalar[Long].single)
Page(users, page, offset, totalRows)
}
}
Doesn't matter which value of 'orderBy' I provide, the order is always based on id of the entities.
The query generated by Anorm is valid PostgreSQL and it works fine when running it against the database directly. But it seems like if Anorm parser was ignoring the order in which the results are returned, and instead returns a list ordered by 'id'.
I've even tried to simplify the query to a "select * from publisher order by 2 ASC/DESC", but nothing is fixed, the ordering is ignored by Anorm on return.
Any suggestion on how to solve this issue?
Thanks to Guillaume on the mailing list of Play I found a workaround.
All placeholders work except the one in order by. The worse part is that when you follow the logs, the driver generates the correct query and PostgreSQL is receiving it. I'm not sure what's the deal, very confusing, but if I remove that placeholder, it just works.
Depressing :(
I solved it like this:
val users = SQL(
"""
select * from publisher
where name ilike {filter}
order by %d %s
limit {pageSize} offset {offset}
""".format(scala.math.abs(orderBy), mode)
).on(
'pageSize -> pageSize,
'offset -> offset,
'filter -> filter
).as(User.simple *)
Now you'll be screaming "SQL INJECTION". Relax. Although it may be possible somehow, orderBy is an integer (which we turn into abs value for more safety). If you try to call the controller that provides orderBy with a string, Play returns a 404 error. So only integers are allowed. And if there is no column corresponding to the given integer, the order by is ignored. So, not ideal, but not so bad.