I have an update function in my UserDAO class that takes a few optional values:
def update(id: Int, name: Option[String], password: Option[String], age: Option[Int])
I know how to update all of the values:
val query = for {
u <- users if u.id === id
} yield (u.name, u.password, u.age)
db.run(query.update(name.get, password.get, age.get))
But want to do it conditionally update the different columns, depending on if their Option value is defined. Something like this perhaps:
val query = for {
u <- users if u.id === id
} yield (u.name if name.isDefined, u.password if password.isDefined, u.age if age.isDefined) // Pseudo code
db.run(query(update(...)) // Unpack arguments here
For slick 3, you can try like this,
val query = for {
u <- db.run(users.filter(_.id === id).result)
u1 = if(u.nonEmpty && u.head.name.isDefined){
u.head.copy(name = u.head.name) //add more if needed
}
else
{
u
}
res <- db.run(users.update(u1))
} yield res
for slick 2 no need of for-yield
val u = db.run(users.filter(_.id === id).result)
val u1 = if(u.nonEmpty && u.head.name.isDefined){
u.head.copy(name = u.head.name) //add more if needed
}
else
{
u
}
val res = db.run(users.update(u1))
Hope my answer was helpful.
Related
I want to be able to create a query with Slick that let me filter left joins in a dynamic way
case class Player(
id: Long,
createdAt: DateTime,
lastModificationDate: DateTime,
name: String
)
class PlayerTable(tag: Tag) extends Table[Player](tag, "players") {
def id = column[Long]("id", O.PrimaryKey, O.AutoInc)
def createdAt = column[DateTime]("createdAt")
def lastModificationDate = column[DateTime]("lastModificationDate")
def name = column[String]("name")
override def * : ProvenShape[Player] = (
id,
createdAt,
lastModificationDate,
updatedAt,
name
) <> (Player.tupled, Player.unapply)
}
case class PlayerGame(
id: Long,
createdAt: DateTime,
lastModificationDate: DateTime,
playerId: Long,
level: Int,
status: String
)
class PlayerGameTable(tag: Tag) extends Table[PlayerGame](tag, "player_games") {
def id = column[Long]("id", O.PrimaryKey, O.AutoInc)
def createdAt = column[DateTime]("createdAt")
def lastModificationDate = column[DateTime]("lastModificationDate")
def playerId = column[Long]("playerId")
def level = column[Int]("level")
def status = column[String]("status")
override def * : ProvenShape[PlayerGame] = (
id,
createdAt,
lastModificationDate,
playerId,
level,
status
) <> (PlayerGame.tupled, PlayerGame.unapply)
}
I want to write a query like this with Slick, where the WHERE CLAUSE is dynamic. I wrote two examples
SELECT *
FROM players
LEFT JOIN player_games AS playerGamesOne ON players.id = playerGamesOne.playerId AND playerGamesOne.level = 1
LEFT JOIN player_games AS playerGamesTwo ON players.id = playerGamesTwo.playerId AND playerGamesTwo.level = 2
WHERE playerGamesOne.status LIKE 'gameOver'
OR playerGamesTWO.status LIKE 'gameOver'
SELECT *
FROM players
LEFT JOIN player_games AS playerGamesOne ON players.id = playerGamesOne.playerId AND playerGamesOne.level = 1
LEFT JOIN player_games AS playerGamesTwo ON players.id = playerGamesTwo.playerId AND playerGamesTwo.level = 2
WHERE playerGamesOne.status LIKE 'playing'
OR playerGamesTwo.status NOT LIKE 'gameOver'
I was trying something like this, but I get Rep[Option[PlayerGameTable]] as the parameter. Maybe there is a different way of doing something like this
val baseQuery = for {
((p, g1), g2) <- PlayerTable.playerQuery joinLeft
PlayerGameTable.playerGameQuery ON ((x, y) => x.id === y.playerId && y.level === 1) joinLeft
PlayerGameTable.playerGameQuery ON ((x, y) => x._1.id === y.playerId && y.level === 2)
} yield (p, g1, g2)
private def filterPlayerGames(gameStatus: String, playerGamesOneOpt: Option[PlayerGameTable], playerGamesTwoOpt: Option[PlayerGameTable]) = {
(gameStatus, playerGamesOneOpt, playerGamesOneOpt) match {
case (gameStatus: String, Some(playerGamesOne: PlayerGameTable), Some(playerGamesOne: PlayerGameTable)) if gameStatus == "gameOver" => playerGamesOne.status === "gameOver" || playerGamesTwo.status === "gameOver"
}
}
It is a complex question, if soemthing is not clear please let me know and I will try to clarify it
There are a couple of issues:
With multiple conditions, the underscore placeholder used within your ON clause would not work the way intended
_.level = something is an assignment, not a condition
Assuming PlayerTable.playerQuery is TableQuery[PlayerTable] and PlayerGameTable.playerGameQuery is TableQuery[PlayerGameTable], your baseQuery should look like this:
val baseQuery = for {
((p, g1), g2) <- PlayerTable.playerQuery joinLeft
PlayerGameTable.playerGameQuery on ((x, y) => x.id === y.playerId && y.level === 1) joinLeft
PlayerGameTable.playerGameQuery on ((x, y) => x._1.id === y.playerId && y.level === 2)
} yield (p, g1, g2)
It's not entirely clear to me how your filterPlayerGames method is going to handle dynamic conditions. Nor do I think any filtering wrapper method will be flexible enough to cover multiple conditions with arbitrary and/or/negation operators. I would suggest that you use the baseQuery for the necessary joins and build filtering queries on top of it, similar to something like below:
val query1 = baseQuery.filter{ case (_, g1, g2) =>
g1.filter(_.status === "gameOver").isDefined || g2.filter(_.status === "gameOver").isDefined
}
val query2 = baseQuery.filter{ case (_, g1, g2) =>
g1.filter(_.status === "playing").isDefined || g2.filter(_.status =!= "gameOver").isDefined
}
Note that with the left joins, g1 and g2 are of Option type, thus isDefined is applied for the or operation.
On a separate note, given that your filtering conditions are only on PlayerGameTable, it would probably be more efficient to perform filtering before the joins.
I'm building a family tree from a database on Apache Spark, using a recursive search to find the ultimate parent (i.e. the person at the top of the family tree) for each person in the DB.
It is assumed that the first person returned when searching for their id is the correct parent
val peopleById = peopleRDD.keyBy(f => f.id)
def findUltimateParentId(personId: String) : String = {
if((personId == null) || (personId.length() == 0))
return "-1"
val personSeq = peopleById.lookup(personId)
val person = personSeq(0)
if(person.personId == "0 "|| person.id == person.parentId) {
return person.id
}
else {
return findUltimateParentId(person.parentId)
}
}
val ultimateParentIds = peopleRDD.foreach(f => f.findUltimateParentId(f.parentId))
It is giving the following error
"Caused by: org.apache.spark.SparkException: RDD transformations and actions can only be invoked by the driver, not inside of other transformations; for example, rdd1.map(x => rdd2.values.count() * x) is invalid because the values transformation and count action cannot be performed inside of the rdd1.map transformation. For more information, see SPARK-5063."
I understand from reading other similar questions that the problem is that I'm calling the findUltimateParentId from within the foreach loop, and if I call the method from the shell with a person's id, it returns the correct ultimate parent id
However, none of the other suggested solutions work for me, or at least I can't see how to implement them in my program, can anyone help?
If I understood you correctly - here's a solution that would work for any size of input (although performance might not be great) - it performs N iterations over the RDD where N is the "deepest family" (largest distance from ancestor to child) in the input:
// representation of input: each person has an ID and an optional parent ID
case class Person(id: Int, parentId: Option[Int])
// representation of result: each person is optionally attached its "ultimate" ancestor,
// or none if it had no parent id in the first place
case class WithAncestor(person: Person, ancestor: Option[Person]) {
def hasGrandparent: Boolean = ancestor.exists(_.parentId.isDefined)
}
object RecursiveParentLookup {
// requested method
def findUltimateParent(rdd: RDD[Person]): RDD[WithAncestor] = {
// all persons keyed by id
def byId = rdd.keyBy(_.id).cache()
// recursive function that "climbs" one generation at each iteration
def climbOneGeneration(persons: RDD[WithAncestor]): RDD[WithAncestor] = {
val cached = persons.cache()
// find which persons can climb further up family tree
val haveGrandparents = cached.filter(_.hasGrandparent)
if (haveGrandparents.isEmpty()) {
cached // we're done, return result
} else {
val done = cached.filter(!_.hasGrandparent) // these are done, we'll return them as-is
// for those who can - join with persons to find the grandparent and attach it instead of parent
val withGrandparents = haveGrandparents
.keyBy(_.ancestor.get.parentId.get) // grandparent id
.join(byId)
.values
.map({ case (withAncestor, grandparent) => WithAncestor(withAncestor.person, Some(grandparent)) })
// call this method recursively on the result
done ++ climbOneGeneration(withGrandparents)
}
}
// call recursive method - start by assuming each person is its own parent, if it has one:
climbOneGeneration(rdd.map(p => WithAncestor(p, p.parentId.map(i => p))))
}
}
Here's a test to better understand how this works:
/**
* Example input tree:
*
* 1 5
* | |
* ----- 2 ----- 6
* | |
* 3 4
*
*/
val person1 = Person(1, None)
val person2 = Person(2, Some(1))
val person3 = Person(3, Some(2))
val person4 = Person(4, Some(2))
val person5 = Person(5, None)
val person6 = Person(6, Some(5))
test("find ultimate parent") {
val input = sc.parallelize(Seq(person1, person2, person3, person4, person5, person6))
val result = RecursiveParentLookup.findUltimateParent(input).collect()
result should contain theSameElementsAs Seq(
WithAncestor(person1, None),
WithAncestor(person2, Some(person1)),
WithAncestor(person3, Some(person1)),
WithAncestor(person4, Some(person1)),
WithAncestor(person5, None),
WithAncestor(person6, Some(person5))
)
}
It should be easy to map your input into these Person objects, and to map the output WithAncestor objects into whatever it is you need. Note that this code assumes that if any person has parentId X - another person with that id actually exists in the input
fixed this by using SparkContext.broadcast:
val peopleById = peopleRDD.keyBy(f => f.id)
val broadcastedPeople = sc.broadcast(peopleById.collectAsMap())
def findUltimateParentId(personId: String) : String = {
if((personId == null) || (personId.length() == 0))
return "-1"
val personOption = broadcastedPeople.value.get(personId)
if(personOption.isEmpty) {
return "0";
}
val person = personOption.get
if(person.personId == 0 || person.orgId == person.personId) {
return person.id
}
else {
return findUltimateParentId(person.parentId)
}
}
val ultimateParentIds = peopleRDD.foreach(f => f.findUltimateParentId(f.parentId))
working great now!
Now sure how to do this correctly, I'm trying to do this:
def byId(id: Column[Int], locationId: Column[Int]) = {
for {
m <- users if m.id === id && m.locationId == locationId
} yield m
}
val byIdCompiled = Compiled(byId _) // ???????????? how to pass second parameter?
def getById(id: Int, locationId: Int): Option[User] {
byIdCompiled(id, locationId).firstOption
}
How do I curry a function with 2 parameters when compililng my slick query?
The example provided in Slick docs uses single underscore to encode muiltiple parameters.
http://slick.typesafe.com/doc/2.0.0/queries.html
def userNameByIDRange(min: Column[Int], max: Column[Int]) =
for {
u <- users if u.id >= min && u.id < max
} yield u.first
val userNameByIDRangeCompiled = Compiled(userNameByIDRange _)
// The query will be compiled only once:
val names1 = userNameByIDRangeCompiled(2, 5).run
val names2 = userNameByIDRangeCompiled(1, 3).run
I'm trying to use Slick to query a many-to-many relationship, but I'm running into a variety of errors, the most prominent being "Don't know how to unpack (User, Skill) to T and pack to G".
The structure of the tables is similar to the following:
case class User(val name: String, val picture: Option[URL], val id: Option[UUID])
object Users extends Table[User]("users") {
def name = column[String]("name")
def picture = column[Option[URL]]("picture")
def id = column[UUID]("id")
def * = name ~ picture ~ id.? <> (User, User.unapply _)
}
case class Skill(val name: String, val id: Option[UUID])
object Skills extends Table[Skill]("skill") {
def name = column[String]("name")
def id = column[UUID]("id")
def * = name ~ id.? <> (Skill, Skill.unapply _)
}
case class UserSkill(val userId: UUID, val skillId: UUID, val id: Option[UUID])
object UserSkills extends Table[UserSkill]("user_skill") {
def userId = column[UUID]("userId")
def skillId = column[UUID]("skillId")
def id = column[UUID]("id")
def * = userId ~ skillId ~ id.? <> (UserSkill, UserSkill.unapply _)
def user = foreignKey("userFK", userId, Users)(_.id)
def skill = foreignKey("skillFK", skillId, Skills)(_.id)
}
Ultimately, what I want to achieve is something of the form
SELECT u.*, group_concat(s.name) FROM user_skill us, users u, skills s WHERE us.skillId = s.id && us.userId = u.id GROUP BY u.id
but before I spend the time trying to get group_concat to work as well, I have been trying to produce the simpler query (which I believe is still valid...)
SELECT u.* FROM user_skill us, users u, skills s WHERE us.skillId = s.id && us.userId = u.id GROUP BY u.id
I've tried a variety of scala code to produce this query, but an example of what causes the shape error above is
(for {
us <- UserSkills
user <- us.user
skill <- us.skill
} yield (user, skill)).groupBy(_._1.id).map { case(_, xs) => xs.first }
Similarly, the following produces a packing error regarding "User" instead of "(User, Skill)"
(for {
us <- UserSkills
user <- us.user
skill <- us.skill
} yield (user, skill)).groupBy(_._1.id).map { case(_, xs) => xs.map(_._1).first }
If anyone has any suggestions, I would be very grateful: I've spent most of today and yesterday scouring google/google groups as well as the slick source, but I haven't a solution yet.
(Also, I'm using postgre so group_concat would actually be string_agg)
EDIT
So it seems like when groupBy is used, the mapped projection gets applied because something like
(for {
us <- UserSkills
u <- us.user
s <- us.skill
} yield (u,s)).map(_._1)
works fine because _._1 gives the type Users, which has a Shape since Users is a table. However, when we call xs.first (as we do when we call groupBy), we actually get back a mapped projection type (User, Skill), or if we apply map(_._1) first, we get the type User, which is not Users! As far as I can tell, there is no shape with User as the mixed type because the only shapes defined are for Shape[Column[T], T, Column[T]] and for a table T <: TableNode, Shape[T, NothingContainer#TableNothing, T] as defined in slick.lifted.Shape. Furthermore, if I do something like
(for {
us <- UserSkills
u <- us.user
s <- us.skill
} yield (u,s))
.groupBy(_._1.id)
.map { case (_, xs) => xs.map(_._1.id).first }
I get a strange error of the form "NoSuchElementException: key not found: #1515100893", where the numeric key value changes each time. This is not the query I want, but it is a strange issue none the less.
I've run up against similar situations as well. While I love working with Scala and Slick, I do believe there are times when it is easier to denormalize an object in the database itself and link the Slick Table to a view.
For example, I have an application that has a Tree object that is normalized into several database tables. Since I'm comfortable with SQL, I think it is a cleaner solution than writing a plain Scala Slick query. The Scala code:
case class DbGFolder(id: String,
eTag: String,
url: String,
iconUrl: String,
title: String,
owner: String,
parents: Option[String],
children: Option[String],
scions: Option[String],
created: LocalDateTime,
modified: LocalDateTime)
object DbGFolders extends Table[DbGFolder]("gfolder_view") {
def id = column[String]("id")
def eTag = column[String]("e_tag")
def url = column[String]("url")
def iconUrl = column[String]("icon_url")
def title = column[String]("title")
def owner = column[String]("file_owner")
def parents = column[String]("parent_str")
def children = column[String]("child_str")
def scions = column[String]("scion_str")
def created = column[LocalDateTime]("created")
def modified = column[LocalDateTime]("modified")
def * = id ~ eTag ~ url ~ iconUrl ~ title ~ owner ~ parents.? ~
children.? ~ scions.? ~ created ~ modified <> (DbGFolder, DbGFolder.unapply _)
def findAll(implicit s: Session): List[GFolder] = {
Query(DbGFolders).list().map {v =>
GFolder(id = v.id,
eTag = v.eTag,
url = v.url,
iconUrl = v.iconUrl,
title = v.title,
owner = v.owner,
parents = v.parents.map { parentStr =>
parentStr.split(",").toSet }.getOrElse(Set()),
children = v.children.map{ childStr =>
childStr.split(",").toSet }.getOrElse(Set()),
scions = v.scions.map { scionStr =>
scionStr.split(",").toSet }.getOrElse(Set()),
created = v.created,
modified = v.modified)
}
}
}
And the underlying (postgres) view:
CREATE VIEW scion_view AS
WITH RECURSIVE scions(id, scion) AS (
SELECT c.id, c.child
FROM children AS c
UNION ALL
SELECT s.id, c.child
FROM children AS c, scions AS s
WHERE c.id = s.scion)
SELECT * FROM scions ORDER BY id, scion;
CREATE VIEW gfolder_view AS
SELECT
f.id, f.e_tag, f.url, f.icon_url, f.title, m.name, f.file_owner,
p.parent_str, c.child_str, s.scion_str, f.created, f.modified
FROM
gfiles AS f
JOIN mimes AS m ON (f.mime_type = m.name)
LEFT JOIN (SELECT DISTINCT id, string_agg(parent, ',' ORDER BY parent) AS parent_str
FROM parents GROUP BY id) AS p ON (f.id = p.id)
LEFT JOIN (SELECT DISTINCT id, string_agg(child, ',' ORDER BY child) AS child_str
FROM children GROUP BY id) AS c ON (f.id = c.id)
LEFT JOIN (SELECT DISTINCT id, string_agg(scion, ',' ORDER BY scion) AS scion_str
FROM scion_view GROUP BY id) AS s ON (f.id = s.id)
WHERE
m.category = 'folder';
Try this. Hope it may yield what you expected. Find the Slick Code below the case classes.
click here for the reference regarding lifted embedding .
case class User(val name: String, val picture: Option[URL], val id: Option[UUID])
class Users(_tableTag: Tag) extends Table[User](_tableTag,"users") {
def name = column[String]("name")
def picture = column[Option[URL]]("picture")
def id = column[UUID]("id")
def * = name ~ picture ~ id.? <> (User, User.unapply _)
}
lazy val userTable = new TableQuery(tag => new Users(tag))
case class Skill(val name: String, val id: Option[UUID])
class Skills(_tableTag: Tag) extends Table[Skill](_tableTag,"skill") {
def name = column[String]("name")
def id = column[UUID]("id")
def * = name ~ id.? <> (Skill, Skill.unapply _)
}
lazy val skillTable = new TableQuery(tag => new Skills(tag))
case class UserSkill(val userId: UUID, val skillId: UUID, val id: Option[UUID])
class UserSkills(_tableTag: Tag) extends Table[UserSkill](_tableTag,"user_skill") {
def userId = column[UUID]("userId")
def skillId = column[UUID]("skillId")
def id = column[UUID]("id")
def * = userId ~ skillId ~ id.? <> (UserSkill, UserSkill.unapply _)
def user = foreignKey("userFK", userId, Users)(_.id)
def skill = foreignKey("skillFK", skillId, Skills)(_.id)
}
lazy val userSkillTable = new TableQuery(tag => new UserSkills(tag))
(for {((userSkill, user), skill) <- userSkillTable join userTable.filter on
(_.userId === _.id) join skillTable.filter on (_._1.skillId === _.id)
} yield (userSkill, user, skill)).groupBy(_.2.id)
Maybe a silly question. But I have not found an answer so far. So how do you represent the SQL's "LIKE" operator in SLICK?
Exactly as you normally would!
val query = for {
coffee <- Coffees if coffee.name like "%expresso%"
} yield (coffee.name, coffee.price)
Will generate SQL like
SELECT name, price FROM coffees WHERE NAME like '%expresso%';
This is how I got it to work:
// argMap is map of type [Str, Str]
val query = for {
coffee <- coffees if (
argMap.map{ case (k,v) =>
metric.column[String](k) like s"%${v}%"
}.reduce(_ && _)
)
} yield(coffee.name)
And then you can run this using your db:
val res = db.run(query.result)
Of course res is a future here that you need to use await to get the actual result.
Suppose you have a table named, logs with 3 fields -
id
message
datetime
You want to perform LIKE operation. So it will be:
def data(data: ReqData): Future[Seq[Syslog]] = {
sysLogTable
.filter(_.datetime >= data.datetimeFrom)
.filter(_.datetime <= data.datetimeUntil)
.filter(_.message like s"%${data.phrase}%")
.result
}
Note: for sysLogTable
val sysLogTable: TableQuery[SyslogsTable] = TableQuery[SyslogsTable]
class SyslogsTable(tag: Tag) extends Table[Syslog](tag, "logs") {
def id = column[Long]("id", O.PrimaryKey, O.AutoInc)
def message = column[String]("message")
def datetime = column[Timestamp]("date")
def * = (id.?, message, datetime) <> ((Syslog.apply _).tupled, Syslog.unapply)
}
Note: for Syslog case class
case class Syslog(
id: Option[Long],
message: String,
datetime: Timestamp
)