Now sure how to do this correctly, I'm trying to do this:
def byId(id: Column[Int], locationId: Column[Int]) = {
for {
m <- users if m.id === id && m.locationId == locationId
} yield m
}
val byIdCompiled = Compiled(byId _) // ???????????? how to pass second parameter?
def getById(id: Int, locationId: Int): Option[User] {
byIdCompiled(id, locationId).firstOption
}
How do I curry a function with 2 parameters when compililng my slick query?
The example provided in Slick docs uses single underscore to encode muiltiple parameters.
http://slick.typesafe.com/doc/2.0.0/queries.html
def userNameByIDRange(min: Column[Int], max: Column[Int]) =
for {
u <- users if u.id >= min && u.id < max
} yield u.first
val userNameByIDRangeCompiled = Compiled(userNameByIDRange _)
// The query will be compiled only once:
val names1 = userNameByIDRangeCompiled(2, 5).run
val names2 = userNameByIDRangeCompiled(1, 3).run
Related
I am trying to understand the "add" and "extract" methods of the FPTree class:
(https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/fpm/FPGrowth.scala).
What is the purpose of 'summaries' variable?
where is the Group list?
I assume it is the following, am I correct:
val numParts = if (numPartitions > 0) numPartitions else data.partitions.length
val partitioner = new HashPartitioner(numParts)
What will 'summaries contain for 3 transactions of {a,b,c} , {a,b} , {b,c} where all are frequent?
def add(t: Iterable[T], count: Long = 1L): FPTree[T] = {
require(count > 0)
var curr = root
curr.count += count
t.foreach { item =>
val summary = summaries.getOrElseUpdate(item, new Summary)
summary.count += count
val child = curr.children.getOrElseUpdate(item, {
val newNode = new Node(curr)
newNode.item = item
summary.nodes += newNode
newNode
})
child.count += count
curr = child
}
this
}
def extract(
minCount: Long,
validateSuffix: T => Boolean = _ => true): Iterator[(List[T], Long)] = {
summaries.iterator.flatMap { case (item, summary) =>
if (validateSuffix(item) && summary.count >= minCount) {
Iterator.single((item :: Nil, summary.count)) ++
project(item).extract(minCount).map { case (t, c) =>
(item :: t, c)
}
} else {
Iterator.empty
}
}
}
After a bit experiments, it is pretty straight forward:
1+2) The partition is indeed the Group representative.
It is also how the conditional transactions calculated:
private def genCondTransactions[Item: ClassTag](
transaction: Array[Item],
itemToRank: Map[Item, Int],
partitioner: Partitioner): mutable.Map[Int, Array[Int]] = {
val output = mutable.Map.empty[Int, Array[Int]]
// Filter the basket by frequent items pattern and sort their ranks.
val filtered = transaction.flatMap(itemToRank.get)
ju.Arrays.sort(filtered)
val n = filtered.length
var i = n - 1
while (i >= 0) {
val item = filtered(i)
val part = partitioner.getPartition(item)
if (!output.contains(part)) {
output(part) = filtered.slice(0, i + 1)
}
i -= 1
}
output
}
The summaries is just a helper to save the count of items in transaction
The extract/project will generate the FIS by using up/down recursion and dependent FP-Trees (project), while checking summaries if traversal that path is needed.
summaries of node 'a' will have {b:2,c:1} and children of node 'a' are 'b' and 'c'.
I'm new to scala, and, I'm trying to pass a map i.e. Map[String, Any]("from_type" -> "Admin", "from_id" -> 1) to my service for dynamic filtering. I'm trying to avoid writing my code like this filter(_.fromType === val && _.fromId === val2)
When trying this example Slick dynamically filter by a list of columns and values
I get a Type mismatch. Required Function1[K, NotInfered T] Found: Rep[Boolean]
Service code:
val query = TableQuery[UserTable]
def all(perPage: Int page: Int, listFilters: Map[String, Any]): Future[ResultPagination[User]] = {
val baseQuery = for {
items <- query.filter( listFilters ).take(perPage).drop(page).result // <----I want to filter here
total <- query.length.result
} yield ResultPagination[User](items, total)
db.run(baseQuery)
}
Table code:
def fromId: Rep[Int] = column[Int]("from_id")
def fromType: Rep[String] = column[String]("from_type")
def columnToRep(column: String): Rep[_] = {
column match {
case "from_type" = this.fromType
case "from_id" = this.fromId
}
}
Well, I would not recommend to use Map[String, Any] construction, because of using Any you are loosing type safety: for instance you can pass to the function by mistake Map("fromId" -> "1") and compile won't help identify issue.
I guess, what you want is to pass some kind of structure representing variative filter. And Query.filterOpt can help you in this case. You can take a look usage examples at: https://scala-slick.org/doc/3.3.2/queries.html#sorting-and-filtering
Please, see code example below:
// Your domain filter structure. None values will be ignored
// So `UserFilter()` - will match all.
case class UserFilter(fromId: Option[Int] = None, fromString: Option[String] = None)
def all(perPage: Int, page: Int, filter: UserFilter): Future[ResultPagination[User]] = {
val baseQuery = for {
items <- {
query
.filterOpt(filter.fromId)(_.fromId === _)
.filterOpt(filter.fromString)(_.fromType === _)
.take(perPage)
.drop(page)
.result
}
total <- query.length.result
} yield ResultPagination[User](items, total)
db.run(baseQuery)
}
And this will type safe.
Hope this helps!
I want to be able to create a query with Slick that let me filter left joins in a dynamic way
case class Player(
id: Long,
createdAt: DateTime,
lastModificationDate: DateTime,
name: String
)
class PlayerTable(tag: Tag) extends Table[Player](tag, "players") {
def id = column[Long]("id", O.PrimaryKey, O.AutoInc)
def createdAt = column[DateTime]("createdAt")
def lastModificationDate = column[DateTime]("lastModificationDate")
def name = column[String]("name")
override def * : ProvenShape[Player] = (
id,
createdAt,
lastModificationDate,
updatedAt,
name
) <> (Player.tupled, Player.unapply)
}
case class PlayerGame(
id: Long,
createdAt: DateTime,
lastModificationDate: DateTime,
playerId: Long,
level: Int,
status: String
)
class PlayerGameTable(tag: Tag) extends Table[PlayerGame](tag, "player_games") {
def id = column[Long]("id", O.PrimaryKey, O.AutoInc)
def createdAt = column[DateTime]("createdAt")
def lastModificationDate = column[DateTime]("lastModificationDate")
def playerId = column[Long]("playerId")
def level = column[Int]("level")
def status = column[String]("status")
override def * : ProvenShape[PlayerGame] = (
id,
createdAt,
lastModificationDate,
playerId,
level,
status
) <> (PlayerGame.tupled, PlayerGame.unapply)
}
I want to write a query like this with Slick, where the WHERE CLAUSE is dynamic. I wrote two examples
SELECT *
FROM players
LEFT JOIN player_games AS playerGamesOne ON players.id = playerGamesOne.playerId AND playerGamesOne.level = 1
LEFT JOIN player_games AS playerGamesTwo ON players.id = playerGamesTwo.playerId AND playerGamesTwo.level = 2
WHERE playerGamesOne.status LIKE 'gameOver'
OR playerGamesTWO.status LIKE 'gameOver'
SELECT *
FROM players
LEFT JOIN player_games AS playerGamesOne ON players.id = playerGamesOne.playerId AND playerGamesOne.level = 1
LEFT JOIN player_games AS playerGamesTwo ON players.id = playerGamesTwo.playerId AND playerGamesTwo.level = 2
WHERE playerGamesOne.status LIKE 'playing'
OR playerGamesTwo.status NOT LIKE 'gameOver'
I was trying something like this, but I get Rep[Option[PlayerGameTable]] as the parameter. Maybe there is a different way of doing something like this
val baseQuery = for {
((p, g1), g2) <- PlayerTable.playerQuery joinLeft
PlayerGameTable.playerGameQuery ON ((x, y) => x.id === y.playerId && y.level === 1) joinLeft
PlayerGameTable.playerGameQuery ON ((x, y) => x._1.id === y.playerId && y.level === 2)
} yield (p, g1, g2)
private def filterPlayerGames(gameStatus: String, playerGamesOneOpt: Option[PlayerGameTable], playerGamesTwoOpt: Option[PlayerGameTable]) = {
(gameStatus, playerGamesOneOpt, playerGamesOneOpt) match {
case (gameStatus: String, Some(playerGamesOne: PlayerGameTable), Some(playerGamesOne: PlayerGameTable)) if gameStatus == "gameOver" => playerGamesOne.status === "gameOver" || playerGamesTwo.status === "gameOver"
}
}
It is a complex question, if soemthing is not clear please let me know and I will try to clarify it
There are a couple of issues:
With multiple conditions, the underscore placeholder used within your ON clause would not work the way intended
_.level = something is an assignment, not a condition
Assuming PlayerTable.playerQuery is TableQuery[PlayerTable] and PlayerGameTable.playerGameQuery is TableQuery[PlayerGameTable], your baseQuery should look like this:
val baseQuery = for {
((p, g1), g2) <- PlayerTable.playerQuery joinLeft
PlayerGameTable.playerGameQuery on ((x, y) => x.id === y.playerId && y.level === 1) joinLeft
PlayerGameTable.playerGameQuery on ((x, y) => x._1.id === y.playerId && y.level === 2)
} yield (p, g1, g2)
It's not entirely clear to me how your filterPlayerGames method is going to handle dynamic conditions. Nor do I think any filtering wrapper method will be flexible enough to cover multiple conditions with arbitrary and/or/negation operators. I would suggest that you use the baseQuery for the necessary joins and build filtering queries on top of it, similar to something like below:
val query1 = baseQuery.filter{ case (_, g1, g2) =>
g1.filter(_.status === "gameOver").isDefined || g2.filter(_.status === "gameOver").isDefined
}
val query2 = baseQuery.filter{ case (_, g1, g2) =>
g1.filter(_.status === "playing").isDefined || g2.filter(_.status =!= "gameOver").isDefined
}
Note that with the left joins, g1 and g2 are of Option type, thus isDefined is applied for the or operation.
On a separate note, given that your filtering conditions are only on PlayerGameTable, it would probably be more efficient to perform filtering before the joins.
I have an update function in my UserDAO class that takes a few optional values:
def update(id: Int, name: Option[String], password: Option[String], age: Option[Int])
I know how to update all of the values:
val query = for {
u <- users if u.id === id
} yield (u.name, u.password, u.age)
db.run(query.update(name.get, password.get, age.get))
But want to do it conditionally update the different columns, depending on if their Option value is defined. Something like this perhaps:
val query = for {
u <- users if u.id === id
} yield (u.name if name.isDefined, u.password if password.isDefined, u.age if age.isDefined) // Pseudo code
db.run(query(update(...)) // Unpack arguments here
For slick 3, you can try like this,
val query = for {
u <- db.run(users.filter(_.id === id).result)
u1 = if(u.nonEmpty && u.head.name.isDefined){
u.head.copy(name = u.head.name) //add more if needed
}
else
{
u
}
res <- db.run(users.update(u1))
} yield res
for slick 2 no need of for-yield
val u = db.run(users.filter(_.id === id).result)
val u1 = if(u.nonEmpty && u.head.name.isDefined){
u.head.copy(name = u.head.name) //add more if needed
}
else
{
u
}
val res = db.run(users.update(u1))
Hope my answer was helpful.
In order to save me having to create so many methods, I tried passing in Option's into my method and then checking if the Option is defined, if so, then apply the filter.
def getUsers(locationId: Option[Int], companyId: Int, salary: Option[Int]): List[User] = {
val query = for {
u <- users if u.companyId === companyId && (locationId.isDefined && u.locationId === locationId.get) && (salary.isDefined && u.salary >= salary.get)
}
query.list()
}
I am getting errors saying:
polymorphic expression cannot be instantiated to expected type;
IntelliJ errors are expected Boolean actual Column[Boolean].
Is this type of clause just not possible in a slick query or I'm just doing it wrong?
I can't tell you why but this compiles for me:
def getUsers(locationId: Option[Int], companyId: Int, salary: Option[Int]): List[User] = {
val query = for {
u <- users if u.companyId === companyId && locationId.isDefined && u.locationId === locationId.get && salary.isDefined && u.salary >= salary.get
} yield(u)
query.list()
}
Note that there are no parenthesis and that you have to yield something otherwise the return type for query would be Unit.
Sure, don't see any issue here, just use filter (or withFilter) and map over the options.
def getUsers(locationId: Option[Int], companyId: Int, salary: Option[Int]): List[User] = (for {
u <- users filter(u=>
if (u.companyId === companyId.bind) &&
(locationId.map(_.bind === u.locationId).getOrElse(true)) &&
(salary.map(_.bind <= u.salary).getOrElse(true))
)
} yield u).list()
Using filter allows you to drop down to Scala for the map or true fallback expressions. If you start with u < users if... then there's no way to use Scala conditionals. The bind calls just escape potential malicious input (i.e. if params are coming from outside the application).
Why it doesn't work
As cvot has noted in his comment, the reason this doesn't work is because:
Slick translates the None as SQL NULL including SQLs 3-valued-logic NULL propagation, so (None === a) is None regardless of the value of a ... basically if anything is None in the expression, the whole expression will be None, so the filter expression will be treated as false and the query result will be empty.
That said, there is a way to get the same behavior you want (filtering only if an optional value is provided).
A way to arrive at the desired behavior
The key thing to note is that for comprehensions get compiled down by Scala to a combination of map / flatMap / withFilter / filter calls. Slick, if I understand it correctly, works with the resulting structure when it compiles the Scala comprehension into a SQL query.
This lets us build up a query in parts:
val baseQuery = for {
u <- users if u.companyId === companyId
} yield u
val possiblyFilteredByLocation = if (locationId.isDefined) {
baseQuery.withFilter(u => u.locationId === locationId.get
} else baseQuery
val possiblyFilteredBySalaryAndOrLocation = if (salary.isDefined) {
possiblyFilteredByLocation.withFilter(u => u.salary >= salary.get)
} else possiblyFilteredByLocation
possiblyFilteredBySalaryAndOrLocation.list()
We can simplify this by using a var and fold:
var query = for {
u <- users if u.companyId === companyId
} yield u
query = locationId.fold(query)(id => query.withFilter(u => u.locationId === id))
query = salary.fold(query)(salary => query.withFilter(u => u.salary >= salary))
query.list()
If we do this frequently, we can generalize this pattern of filtering on an Option into something like this:
// Untested, probably does not compile
implicit class ConditionalFilter(query: Query) {
def ifPresent[T](value: Option[T], predicate: (Query, T) => Query) = {
value.fold(query)(predicate(query, _))
}
}
Then we can simplify our whole filter chain to:
query
.ifPresent[Int](locationId, (q, id) => q.withFilter(u => u.locationId === id))
.ifPresent[Int](salary, (q, s) => q.withFilter(u => u.salary >= s))
.list()
You can use the following solution (with Slick 3.3.x):
def getUsers(locationId: Option[Int], companyId: Int, minSalary: Option[Int]) =
users.
.filter(_.company === companyId)
.filterOpt(locationId)(_.locationId === _)
.filterOpt(minSalary)(_.salary >= _)
Because the Slick query gets translated into SQL, which has no notion of the isDefined and get methods of the Option class.
But you can fix this by calling the methods outside the query and passing the results (via the map function on the options).
The following code should fix it:
def getUsers(locationId: Option[Int], companyId: Int, salary: Option[Int]): List[User] = {
val locationAndSalary = for {
locId <- locationId;
sal <- salary
} yield (locId, sal)
locationAndSalary.map(locAndSal => {
val query = for {
u <- users if u.companyId === companyId && u.locationId === locAndSal._1 && u.salary >= locAndSal._2)
} yield u
query.list()
}).getOrElse(List[User]()) //If the locationID or salary is None, return empty list.
}
The locationAndSalary may seem strange, but we are using for comprehensions to give use a value only when both locationId and salary has a value and storing the result in a tuple, with the locationId in the first position and salary at the second. The following links explains it: Scala: for comprehensions with Options.
Edit: According to #Ende Neu answer the code compiles if you add the yield-statement, but I still think my solution is more the "Scala way".