I am using slick 3 and I am trying to perform some integration tests with some inserts, some code that uses the db and then I want to rollback all the insert or deletion at the end of the test itself but I cannot find any documentation about it.
Is it possible?
How can I achieve it?
You need to use . transactionally around the DBIOAction
eg
val a = (for {
ns <- coffees.filter(_.name.startsWith("ESPRESSO")).map(_.name).result
_ <- DBIO.seq(ns.map(n => coffees.filter(_.name === n).delete): _*)
} yield ()).transactionally
val f: Future[Unit] = db.run(a)
For more see
http://slick.typesafe.com/doc/3.1.1/dbio.html#transactions-and-pinned-sessions
I can advice to drop and create table schema before and after test using BeforeAndAfter scala-test trait with next code:
def createTable(): Future[Unit] = {
db.run(DBIO.seq(
MTable.getTables.map(tables =>
if (!tables.exists(_.name.name == table.baseTableRow.tableName))
db.run(table.schema.create)
)
))
}
def dropTable(): Future[Unit] = db.run(table.schema.drop)
Related
So I have this chunk of code
dbs.foreach({
var map = scala.collection.mutable.Map[String, mutable.MutableList[String]]()
db =>
val resultList = getTables(hive, db)
map+=(db -> resultList)
})
What this does is loops through a list of dbs, does a show tables in db call for each db, then adds the db -> table to a map. How can this be done concurrently since there is about a 5 seconds wait time for the hive query to return?
update code --
def getAllTablesConcurrent(hive: JdbcHive, dbs: mutable.MutableList[String]): Map[String, mutable.MutableList[String]] = {
implicit val context:ExecutionContext = ExecutionContext.fromExecutor(Executors.newFixedThreadPool(10))
val futures = dbs.map {
db =>
Future(db, getTables(hive, db))
}
val map = Await.result( Future.sequence(futures), Duration(10, TimeUnit.SECONDS) ).toMap
map
}
Don't use vars and mutable state, especially if you want concurrency.
val result: Future[Map[String, Seq[String]] = Future
.traverse(dbs) { name =>
Future(name -> getTables(hive, name) )
}.map(_.toMap)
if you want more control (how much time do you want to wait, how many threads do you want to use, what happens if all your threads are busy, etc) you can use ThreadPollExecutor and Future
implicit val context:ExecutionContext = ExecutionContext.fromExecutor(Executors.newFixedThreadPool(10))
val dbs = List("db1", "db2", "db3")
val futures = dbs.map {
name => Future(name, getables(hive, name))
}
val result = Await.result( Future.sequence(futures), Duration(TIMEOUT, TimeUnit.MILLISECONDS) ).toMap
just remember not to create a new ExecutionContext every time you need it
You can use .par on any Scala collection to perform the next transformation in parallel (using default parallelism which depends on number of cores).
Also - easier and cleaner to map into an (immutable) map instead of updating a mutable one.
val result = dbs.par.map(db => db -> getTables(hive, db)).toMap
To have more control on the number of concurrent threads used, see https://docs.scala-lang.org/overviews/parallel-collections/configuration.html
I am using slick3.1.1
I want to insert a list of objects in DB (postgres)
I have written following code which works
for(user<-x.userList)
{
val users = userClass(user.name,user.id)
val userAction = DBAccess.userTable.insertOrUpdate(users)
val f = DBAccess.db.run(DBIO.seq(userAction))
Await.result(f, Duration.Inf)
}
However I am running multiple DB queries. So I was looking some way to call only a single db.run.
So I wrote something like below
val userQuery = TableQuery[dbuserTable]
for(user<-x.userList)
{
val users = userClass(user.name,user.id)
userQuery += users
}
val f = DBAccess.db.run(DBIO.seq(userQuery.result))
Await.result(f, Duration.Inf)
However this second piece does not write to DB. Can someone point out where I am going wrong?
I know this old but since I just fell on it I'll give an updated answer for this.
You can use ++= to insert a list.
val usersTable = TableQuery[dbuserTable]
val listToInsert = x.userList.map(userClass(_.name, _.id))
val action = usersTable ++= listToInsert
DBAccess.db.run(action)
This will only do one request that insert everything at once.
+= doesn't mutate your userQuery. Every iteration of your for loop creates an insert action and then promptly discards it.
Try accumulating the insert actions instead of discarding them (NB use of yield):
val usersTable = TableQuery[dbuserTable]
val inserts = for (user <- x.userList) yield {
val userRow = userClass(user.name, user.id)
usersTable += userRow
}
DBAccess.db.run(DBIO.seq(inserts: _*))
Usually you would run two or more statements in a transaction. But in all the examples I could find on using transactionally in Slick 3, there's a for comprehension to group these statements, when I usually use the for in a loop.
This works (deleting from two tables in a transaction):
val action = db.run((for {
_ <- table1.filter(_.id1 === id).delete
_ <- table2.filter(_.id2=== id).delete
} yield ()).transactionally)
val result = Await.result(action, Duration.Inf)
But is the for/yield needed? is there an alternative way just to run two or more statements in a transaction?
You can use transactionally on every DBIOAction, not just those that are a result of the for comprehension. For example you can use transactionally in combination with the DBIO.seq method, which takes a sequence of actions and runs them sequentially:
val firstAction = table1.filter(_.id === id1).delete
val secondAction = table2.filter(_.id === id2).delete
val combinedAction = DBIO.seq(
firstAction,
secondAction
).transactionally
For your case for/yield is not the only way to get what you need. But you will have to substitute it for the equivalent representation.
The for comprehension is syntactic sugar for a combination of flatMaps and a map. We need to use them because we are using monadic composition to aggregate all the actions in a BDIOAction.
So you could also write it as:
val action = db.run(
table1.filter(_.id1 === id).delete.map ( _ =>
table2.filter(_.id2=== id).delete
).transactionally
)
val result = Await.result(action, Duration.Inf)
The for comprehension is usually used because is more clean, easier to understand and very easy to scale.
Lets have a look at an example with 4 statements in a transactions to see how it looks:
This would be with a for comprehension:
val action = db.run((for {
_ <- table1.filter(_.id1 === id).delete
_ <- table2.filter(_.id2=== id).delete
_ <- table3.filter(_.id3=== id).delete
_ <- table4.filter(_.id4=== id).delete
} yield ()).transactionally)
val result = Await.result(action, Duration.Inf)
This would be with flatMap/maps:
val action = db.run(
table1.filter(_.id1 === id).delete.flatMap ( _ =>
table2.filter(_.id2 === id).delete.flatMap ( _ =>
table3.filter(_.id3 === id).delete.map ( _ =>
table4.filter(_.id4=== id).delete
)
)
).transactionally
)
val result = Await.result(action, Duration.Inf)
I am looking for a way to generate an UPDATE query over multiple columns that are only known at runtime.
For instance, given a List[(String, Int)], how would I go about generating a query in the form of UPDATE <table> SET k1=v1, k2=v2, kn=vn for all key/value pairs in the list?
I have found that, given a single key/value pair, a plain SQL query can be built as sqlu"UPDATE <table> SET #$key=$value (where the key is from a trusted source to avoid injection), but I've been unsuccessful in generalizing this to a list of updates without running a query for each.
Is this possible?
This is one way to do it. I create a table definition T here with table and column names (TableDesc) as implicit arguments. I would have thought that it should be possible to set them explicitly, but I couldn't find it. For the example a create to table query instances, aTable and bTable. Then I insert and select some values and in the end I update a value in the bTable.
import slick.driver.H2Driver.api._
import scala.concurrent.Await
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration.Duration
import scala.util.{Failure, Success}
val db = Database.forURL("jdbc:h2:mem:test1;DB_CLOSE_DELAY=-1", "sa", "", null, "org.h2.Driver")
case class TableDesc(tableName: String, intColumnName: String, stringColumnName: String)
class T(tag: Tag)(implicit tableDesc: TableDesc) extends Table[(String, Int)](tag, tableDesc.tableName) {
def stringColumn = column[String](tableDesc.intColumnName)
def intColumn = column[Int](tableDesc.stringColumnName)
def * = (stringColumn, intColumn)
}
val aTable = {
implicit val tableDesc = TableDesc("TABLE_A", "sa", "ia")
TableQuery[T]
}
val bTable = {
implicit val tableDesc = TableDesc("TABLE_B", "sb", "ib")
TableQuery[T]
}
val future = for {
_ <- db.run(aTable.schema.create)
_ <- db.run(aTable += ("Hi", 1))
resultA <- db.run(aTable.result)
_ <- db.run(bTable.schema.create)
_ <- db.run(bTable ++= Seq(("Test1", 1), ("Test2", 2)))
_ <- db.run(bTable.filter(_.stringColumn === "Test1").map(_.intColumn).update(3))
resultB <- db.run(bTable.result)
} yield (resultA, resultB)
Await.result(future, Duration.Inf)
future.onComplete {
case Success(a) => println(s"OK $a")
case Failure(f) => println(s"DOH $f")
}
Thread.sleep(500)
I've got the sleep statement in the end to assert that the Future.onComplete gets time to finish before the application ends. Is there any other way?
I have methods in my Play app that query database tables with over hundred columns. I can't define case class for each such query, because it would be just ridiculously big and would have to be changed with each alter of the table on the database.
I'm using this approach, where result of the query looks like this:
Map(columnName1 -> columnVal1, columnName2 -> columnVal2, ...)
Example of the code:
implicit val getListStringResult = GetResult[List[Any]] (
r => (1 to r.numColumns).map(_ => r.nextObject).toList
)
def getSomething(): Map[String, Any] = DB.withSession {
val columns = MTable.getTables(None, None, None, None).list.filter(_.name.name == "myTable").head.getColumns.list.map(_.column)
val result = sql"""SELECT * FROM myTable LIMIT 1""".as[List[Any]].firstOption.map(columns zip _ toMap).get
}
This is not a problem when query only runs on a single database and single table. I need to be able to use multiple tables and databases in my query like this:
def getSomething(): Map[String, Any] = DB.withSession {
//The line below is no longer valid because of multiple tables/databases
val columns = MTable.getTables(None, None, None, None).list.filter(_.name.name == "table1").head.getColumns.list.map(_.column)
val result = sql"""
SELECT *
FROM db1.table1
LEFT JOIN db2.table2 ON db2.table2.col1 = db1.table1.col1
LIMIT 1
""".as[List[Any]].firstOption.map(columns zip _ toMap).get
}
The same approach can no longer be used to retrieve column names. This problem doesn't exist when using something like PHP PDO or Java JDBCTemplate - these retrieve column names without any extra effort needed.
My question is: how do I achieve this with Slick?
import scala.slick.jdbc.{GetResult,PositionedResult}
object ResultMap extends GetResult[Map[String,Any]] {
def apply(pr: PositionedResult) = {
val rs = pr.rs // <- jdbc result set
val md = rs.getMetaData();
val res = (1 to pr.numColumns).map{ i=> md.getColumnName(i) -> rs.getObject(i) }.toMap
pr.nextRow // <- use Slick's advance method to avoid endless loop
res
}
}
val result = sql"select * from ...".as(ResultMap).firstOption
Another variant that produces map with not null columns (keys in lowercase):
private implicit val getMap = GetResult[Map[String, Any]](r => {
val metadata = r.rs.getMetaData
(1 to r.numColumns).flatMap(i => {
val columnName = metadata.getColumnName(i).toLowerCase
val columnValue = r.nextObjectOption
columnValue.map(columnName -> _)
}).toMap
})