Slick Combining query conditions effeciently - postgresql

I have two table and i want to run this query everytime.
def query(host: String, id: String, key: String, map: Map[String, List[String]) =
{
val query = (for {
t1 <- TableQuery[TableA]
t2 <- TableQuery[TableB]
if t1.host === host &&
(t2.host === host) &&
(t1.id === t2.id)
} yield
t2.name)
.result
db.run(query)
}
When host is not my-host do execute the above function as it is.
But want to add an additional check only when t1.host == 'my-host', then check if the key is present in the map, if yes add the condition t1.class in map values.
I want something like:
t1 <- TableQuery[TableA]
t2 <- TableQuery[TableB]
if t1.host === host &&
t1.host === 'my-host'
(t2.host === host) &&
(t1.id === t2.id) &&
t1.value inSet map(key)
else if t1.host === host &&
(t2.host === host) &&
(t1.id === t2.id)

I'm not sure if you want to construct a different query for the two cases, or write a single SQL query that handles both cases.
If it's a single query, I suggest writing out the SQL you'd expect and then it should be possible to map that to a filter (for comprehension if)
But I'm guessing you want to construct a different query based on the case of host. In that case, the pattern I'd suggest is:
construct a base query (for the things you always want to do);
add on additional filters for the case you care about; and finally
map the query to the result you need (select the right columns).
(One thing to watch out for is understanding what's happening in SQL (in the database), and what's happening in Scala (in the client). For example, where you say "I want something like:" you have difficulties because inside that for-comprehension you're already in database land. That might be possible using conditional logic in SQL, but I don't think that's what you're looking for).
To give a simplified example you might construct a basic query like this:
val baseQuery = for {
t1 <- TableQuery[TableA]
t2 <- TableQuery[TableB]
if t1.host === host && t2.host === host && t1.id === t2.id
} yield (t1, t2)
Then use that to make a new query to handle the special cases:
val queryAllowingForMyHost =
baseQuery.filterIf(host == "my-host") {
case (t1, t2) => t2.host inSet Seq("x", "y", "z")
}
I've used query.filterIf there, but you could (if you prefer) write regular Scala:
val queryAllowingForMyHost =
if (host == "my-host") {
baseQuery.filter( /* extra conditions here */ )
} else {
baseQuery
}
Lastly, add any finishing touches to the query before running it:
val query = queryAllowingForMyHost.map { case (t1, t2) => t2.name }
That's the query you'd run.

Related

Making a Slick query that filters from an optional id?

I would like to query an object by id, but if a fooId is also provided then I'd like to include it in the query.
def getById(id: Long, fooIdOpt: Option[Long]): Future[Option[Bar]] = {
val query = for {
b <- bars if b.id === id && fooIdOpt.fold(true)(b.fooId === _)
} yield { // Compiler error here ^
b
}
db.run(query.result.headOption)
}
The issue here is that fooIdOpt.fold(true)(b.fooId === _) needs to return a Rep[Boolean], but I'm initializing the fold with a Boolean - a clear type violation of fold's method signature.
However I can't seem to find a way to pass a Rep[Boolean] that evaluates to true in as the fold initializer. Shooting in the dark, I've tried Rep(true), and LiftedLiteral[Boolean](true), but neither quite work.
I could abandon fold entirely and go with:
b <- bars if {
val rep1 = b.id === id
fooIdOpt.map(fooId => rep1 && b.fooId === fooId).getOrElse(rep1)
}
But that just seems so overly complicated, and a fooIdOpt match { case Some ... wouldn't look much better.
Is there a way to instantiate a Rep[Boolean] literal that always evaluates to true?
If not, is there an alternative to my attempts at fold, map, or match above that will allow me to build a query that optionally compares a fooId value?
As you've already figured out, the ifEmpty value type for fold needs to match that of b.fooId === _, which is Rep[Boolean]. One approach would be to apply bind to the ifEmpty value as shown below:
val query = for {
b <- bars if b.id === id && fooIdOpt.fold(true.bind)(b.fooId === _)
} yield b
One alternative that isn't quite as bad as your other options:
(fooIdOpt.toList.map(b.fooId === _) :+ (b.id === id)).reduceLeft(_ && _)

How can I run inter-dependent queries alongside a non-DB operation in the same transaction using slick

Given the data model (<- indicating a foreign key dependency)
TableA <- TableB <- TableC
^ v
-----------------
I need to execute an api DELETE operation that soft-deletes a row in TableC. This delete must also trigger a call to another service (requiring values from TableA and TableB) if there are no more undeleted TableC entries that reference that row's parent in TableB. If the external call fails, I want to rollback the soft-delete. I want to do all of this in an idiomatic fashion (I'm effectively brand new to scala/slick), and use transactions for the rollback
Based on what I've read, I need to be using for comprehension to assemble the queries, but I'm having issues getting the database operations to gel nicely with the external service call. My original plan was:
val select = for {
tableCRow <- tableBDao.TableQueryC.filter(_.id === idParam)
tableBRow <- tableBDao.TableQueryB if tableCRow.tableBForeignKey === tableBRow.id
tableARow <- TableADao.TableQueryA if tableCRow.tableAForeignKey === tableARow.id
count <- tableBDao.TableQueryC.filter(_.tableBForeignKey === tableBRow.id).map(_.id).countDefined
_ <- tableBDao.softDeleteRow(idParam)
_ <- if (count > 1) DBIO.successful(httpRequestService.deleteOtherResource(tableARow.someValue, tableBRow.someValue))
} yield ()
db.run(select.result)
But this had problems because I couldn't pass Slick's Rep[T] values to my httpRequestService method. I then tried to break it down into two portions - SELECT first, then DELETE, like so:
val select = for {
tableCRow <- tableBDao.TableQueryC.filter(_.id === idParam)
tableBRow <- tableBDao.TableQueryB if tableCRow.tableBForeignKey === tableBRow.id
tableARow <- TableADao.TableQueryA if tableCRow.tableAForeignKey === tableARow.id
count <- tableBDao.TableQueryC.filter(_.tableBForeignKey === tableBRow.id).map(_.id).countDefined
} yield (tableBRow.date.formatted("yyyy-MM-DD"), tableARow.externalServiceId, count)
val result: Future[Option[(String, Long, Integer)]] = db.run(select.result.headOption)
result.map {
case None => throw new IllegalArgumentException("exception message")
case Some(data) =>
val delete = for {
_ <- tableBDao.softDeleteRow(idParam)
_ <- if (data._3 > 1) DBIO.successful(httpRequestService.cancelSchedulerJob(data._2, data._1))
} yield numRows
db.run(delete.transactionally)
}
But, despite this actually passing IntelliJ IDEA checks, it won't compile as my count query returns a Rep[Int], which lacks a map function. Additionally, each of my table(A|B|C)Row maps raises an error because they're expecting slick.lifted.Query[Nothing,Nothing,Seq] and they're getting slick.lifted.Query[Nothing,T,Seq]. Finally, the db.run statement doesn't want to use headOption, and apparently returns Any which doesn't support map
halp
Solved this by finally understanding how slick puts things together in a for-comprehension. I had to pull the count out of the query and into a followup groupby->map function, which accumulated my list of things I wanted to count and THEN counted them as opposed to counting as part of the query. This fixed all the rest of the problems too, as the count query was throwing off the expected return types of everything else.
basically, the solution looked like (thing1 was for a join):
val select = (for {
thing1 <- query1
thing2 <- query2
thing3 <- query3
listToCount <- query4 selecting everything I wanted to count
} yield (thing2, thing3, listToCount))
.groupBy({
case (thing2, thing3, listToCount) =>
(thing2, thing3)
})
.map({
case ((thing2, thing3), list) =>
(thing2.deliveryDate, thing3.schedulerJobId, list.map(_._3).length)
})

Group by in many-to-many join with Quill

I am trying to achieve with Quill what the following PostgreSQL query does:
select books.*, array_agg(authors.name) from books
join authors_books on(books.id = authors_books.book_id)
join authors on(authors.id = authors_books.author_id)
group by books.id
For now I have this in my Quill version:
val books = quote(querySchema[Book]("books"))
val authorsBooks = quote(querySchema[AuthorBook]("authors_books"))
val authors = quote(querySchema[Author]("authors"))
val q: db.Quoted[db.Query[(db.Query[Book], Seq[String])]] = quote{
books
.join(authorsBooks).on(_.id == _.book_id)
.join(authors).on(_._2.author_id == _.id)
.groupBy(_._1._1.id)
.map {
case (bId, q) => {
(q.map(_._1._1), unquote(q.map(_._2.name).arrayAgg))
}
}
}
How can I get rid of the nested query in the result (db.Query[Book]) and get a Book instead?
I might be a little bit rusty with SQL but are you sure that your query is valid? Particularly I find suspicious that you do select books.* while group by books.id i.e. you directly return fields that you didn't group by. And attempt to translate that wrong query directly is what makes things go wrong
One way to fix it is to do group by by all fields. Assuming Book is declared as:
case class Book(id: Int, name: String)
you can do
val qq: db.Quoted[db.Query[((Index, String), Seq[String])]] = quote {
books
.join(authorsBooks).on(_.id == _.book_id)
.join(authors).on(_._2.author_id == _.id)
.groupBy(r => (r._1._1.id, r._1._1.name))
.map {
case (bId, q) => {
// (Book.tupled(bId), unquote(q.map(_._2.name).arrayAgg)) // doesn't work
(bId, unquote(q.map(_._2.name).arrayAgg))
}
}
}
val res = db.run(qq).map(r => (Book.tupled(r._1), r._2))
Unfortunately it seems that you can't apply Book.tupled inside quote because you get error
The monad composition can't be expressed using applicative joins
but you can easily do it after db.run call to get back your Book.
Another option is to do group by just Book.id and then join the Book table again to get all the fields back. This might be actually cleaner and faster if there are many fields inside Book

Why can't I use Options inside of a slick query

In order to save me having to create so many methods, I tried passing in Option's into my method and then checking if the Option is defined, if so, then apply the filter.
def getUsers(locationId: Option[Int], companyId: Int, salary: Option[Int]): List[User] = {
val query = for {
u <- users if u.companyId === companyId && (locationId.isDefined && u.locationId === locationId.get) && (salary.isDefined && u.salary >= salary.get)
}
query.list()
}
I am getting errors saying:
polymorphic expression cannot be instantiated to expected type;
IntelliJ errors are expected Boolean actual Column[Boolean].
Is this type of clause just not possible in a slick query or I'm just doing it wrong?
I can't tell you why but this compiles for me:
def getUsers(locationId: Option[Int], companyId: Int, salary: Option[Int]): List[User] = {
val query = for {
u <- users if u.companyId === companyId && locationId.isDefined && u.locationId === locationId.get && salary.isDefined && u.salary >= salary.get
} yield(u)
query.list()
}
Note that there are no parenthesis and that you have to yield something otherwise the return type for query would be Unit.
Sure, don't see any issue here, just use filter (or withFilter) and map over the options.
def getUsers(locationId: Option[Int], companyId: Int, salary: Option[Int]): List[User] = (for {
u <- users filter(u=>
if (u.companyId === companyId.bind) &&
(locationId.map(_.bind === u.locationId).getOrElse(true)) &&
(salary.map(_.bind <= u.salary).getOrElse(true))
)
} yield u).list()
Using filter allows you to drop down to Scala for the map or true fallback expressions. If you start with u < users if... then there's no way to use Scala conditionals. The bind calls just escape potential malicious input (i.e. if params are coming from outside the application).
Why it doesn't work
As cvot has noted in his comment, the reason this doesn't work is because:
Slick translates the None as SQL NULL including SQLs 3-valued-logic NULL propagation, so (None === a) is None regardless of the value of a ... basically if anything is None in the expression, the whole expression will be None, so the filter expression will be treated as false and the query result will be empty.
That said, there is a way to get the same behavior you want (filtering only if an optional value is provided).
A way to arrive at the desired behavior
The key thing to note is that for comprehensions get compiled down by Scala to a combination of map / flatMap / withFilter / filter calls. Slick, if I understand it correctly, works with the resulting structure when it compiles the Scala comprehension into a SQL query.
This lets us build up a query in parts:
val baseQuery = for {
u <- users if u.companyId === companyId
} yield u
val possiblyFilteredByLocation = if (locationId.isDefined) {
baseQuery.withFilter(u => u.locationId === locationId.get
} else baseQuery
val possiblyFilteredBySalaryAndOrLocation = if (salary.isDefined) {
possiblyFilteredByLocation.withFilter(u => u.salary >= salary.get)
} else possiblyFilteredByLocation
possiblyFilteredBySalaryAndOrLocation.list()
We can simplify this by using a var and fold:
var query = for {
u <- users if u.companyId === companyId
} yield u
query = locationId.fold(query)(id => query.withFilter(u => u.locationId === id))
query = salary.fold(query)(salary => query.withFilter(u => u.salary >= salary))
query.list()
If we do this frequently, we can generalize this pattern of filtering on an Option into something like this:
// Untested, probably does not compile
implicit class ConditionalFilter(query: Query) {
def ifPresent[T](value: Option[T], predicate: (Query, T) => Query) = {
value.fold(query)(predicate(query, _))
}
}
Then we can simplify our whole filter chain to:
query
.ifPresent[Int](locationId, (q, id) => q.withFilter(u => u.locationId === id))
.ifPresent[Int](salary, (q, s) => q.withFilter(u => u.salary >= s))
.list()
You can use the following solution (with Slick 3.3.x):
def getUsers(locationId: Option[Int], companyId: Int, minSalary: Option[Int]) =
users.
.filter(_.company === companyId)
.filterOpt(locationId)(_.locationId === _)
.filterOpt(minSalary)(_.salary >= _)
Because the Slick query gets translated into SQL, which has no notion of the isDefined and get methods of the Option class.
But you can fix this by calling the methods outside the query and passing the results (via the map function on the options).
The following code should fix it:
def getUsers(locationId: Option[Int], companyId: Int, salary: Option[Int]): List[User] = {
val locationAndSalary = for {
locId <- locationId;
sal <- salary
} yield (locId, sal)
locationAndSalary.map(locAndSal => {
val query = for {
u <- users if u.companyId === companyId && u.locationId === locAndSal._1 && u.salary >= locAndSal._2)
} yield u
query.list()
}).getOrElse(List[User]()) //If the locationID or salary is None, return empty list.
}
The locationAndSalary may seem strange, but we are using for comprehensions to give use a value only when both locationId and salary has a value and storing the result in a tuple, with the locationId in the first position and salary at the second. The following links explains it: Scala: for comprehensions with Options.
Edit: According to #Ende Neu answer the code compiles if you add the yield-statement, but I still think my solution is more the "Scala way".

Slick/Scala - how do I access fields of the mapped projection/projected table part of a join in a where query

I have a number of basic queries define, and am using query composition to add stuff such as ordering, paging, where clauses and so on...
But I have a problem accessing the fields of the joined 2nd table in the where clause...
Here's my table queries and my table. All tables are mapped to case classes.
val basicCars = TableQuery[CarTable]
val basicCarValues = TableQuery[CarValueTable]
val carsWithValues = for {
(c, v) <- basicCars leftJoin basicCarValues on (_.id === _.carId)
} yield (c, v.?)
Now I reuse/compose queries by doing stuff such as
carsWithValues.where(_._1.id === someId)
which works perfectly...
But if I want to access any value of the 2nd table... and I try
carsWithValues.where(_._2.latestPrice === somePrice)
It tells me that somePrice is not a member of MappedProjection......
error: value somePrice is not a member of scala.slick.lifted.MappedProjection[Option[com......datastore.slick.generated.Tables.CarValue],(Option[Long], Option[Long], Option[String],.....
I understand that this kind of can't work, cause _._2 is a MappedProjection and not just a CarValue sitting in the tuple..
But I can't figure out how to use any field of the table that is in the MappedProjection in a where clause?
The .? from the Slick code generator is implemented using a MappedProjection, which doesn't have the members anymore. If you postpone the call to .? it works:
val carsWithValues = for {
(c, v) <- basicCars leftJoin basicCarValues on (_.id === _.carId)
} yield (c, v)
carsWithValues.where(_._2.latestPrice === somePrice).map{ case (c,v) => (c,v.?) }