explicitly close db connection in slick - scala

Here is code I am trying to optimise:
object UserRepo
{
val users = TableQuery[Users]
val dbName = "db"
lazy val queryAllUsers = for (user <- users) yield user
type UserRow = (Int, String, String, String)
def getAll() : Future[ Seq[UserRow] ] =
{
val db = Database.forConfig( dbName )
val f: Future[Seq[UserRow]] = db.run( queryAllUsers.result )
f.onComplete {
case Success(_) => { db.close() }
case Failure(_) => { db.close() }
}
f
}
}
I going to have number of query to the DB I am trying to get rid of string where I am creating DB connection. Is there any execution context I can use to close connection explicitly ?? so code will look more concise ?
Is there option to get used db connection within Future.onComplete scope??
Thanks

As for your comment ( explicitly close db connection in slick ) normally what you do is to create a connection on an application startup (or lazily on first use) and then closing it at the application end.
This obviously all depends what kind of application you are running:
if you are having DI container you would probably manage some of this in your DI mechanisms (like Modules in Guice)
if you are having web application, specifically e.g. Play - you would probably use play-slick that does this initialization / shutting down for you (kind of).
General way (no DI)
The easiest general way (assuming you are not using DI or play-slick) of doing this would be perhaps something like this:
object DbManager {
lazy val db = createDb
private def createDb = {
Database.forConfig("db")
}
def close {
db.close
}
}
Then your code would be:
object UserRepo
{
val users = TableQuery[Users]
lazy val queryAllUsers = for (user <- users) yield user
type UserRow = (Int, String, String, String)
def getAll() : Future[ Seq[UserRow] ] =
{
DbManager.db.run( queryAllUsers.result )
}
}
Above code doesn't do any cleaning up - this would need to be added to some kind of hook when application is closing (in case e.g. of web application) or you would need to manually call DbManager.close at some specified time (when you are closing the application).
Play slick
You would probably need to start from here: https://github.com/playframework/play-slick/tree/master/samples/basic (most basic sample showing play-slick configuration).
Updating your answer with this would be:
class UserRepo #Inject() (dbConfigProvider: DatabaseConfigProvider) extends HasDatabaseConfigProvider[JdbcProfile])
{
import driver.api._
val users = TableQuery[Users]
lazy val queryAllUsers = for (user <- users) yield user
type UserRow = (Int, String, String, String)
def getAll() : Future[ Seq[UserRow] ] =
{
db.run( queryAllUsers.result )
}
}
In this scenario you wouldn't call:
UserRepo.getAll
but you would rather need to inject it:
class MyClientCode #Inject() (userRepo: UserRepo) {
...
userRepo.getAll
...
}
You would need to obviously configure it in your configuration but this should be very straightforward to do with the sample provided above.
So in short your Play application will have database connection configuration and would do all initialization / cleaning up. Your external modules (like the one you described in your comment) would simply pull DatabaseConfigProvider as Guice managed dependency (as show above).

Related

How to implement a concurrent processing in akka?

I have a method in which there are multiple calls to db. As I have not implemented any concurrent processing, a 2nd db call has to wait until the 1st db call gets completed, 3rd has to wait until the 2nd gets completed and so on.
All db calls are independent of each other. I want to make this in such a way that all DB calls run concurrently.
I am new to Akka framework.
Can someone please help me with small sample or references would help. Application is developed in Scala Lang.
There are three primary ways that you could achieve concurrency for the given example needs.
Futures
For the particular use case that is asked about in the question I would recommend Futures before any akka construct.
Suppose we are given the database calls as functions:
type Data = ???
val dbcall1 : () => Data = ???
val dbcall2 : () => Data = ???
val dbcall3 : () => Data = ???
Concurrency can be easily applied, and then the results can be collected, using Futures:
val f1 = Future { dbcall1() }
val f2 = Future { dbcall2() }
val f3 = Future { dbcall3() }
for {
v1 <- f1
v2 <- f2
v3 <- f3
} {
println(s"All data collected: ${v1}, ${v2}, ${v3}")
}
Akka Streams
There is a similar stack answer which demonstrates how to use the akka-stream library to do concurrent db querying.
Akka Actors
It is also possible to write an Actor to do the querying:
object MakeQuery
class DBActor(dbCall : () => Data) extends Actor {
override def receive = {
case _ : MakeQuery => sender ! dbCall()
}
}
val dbcall1ActorRef = system.actorOf(Props(classOf[DBActor], dbcall1))
However, in this use case Actors are less helpful because you still need to collect all of the data together.
You can either use the same technique as the "Futures" section:
val f1 : Future[Data] = (dbcall1ActorRef ? MakeQuery).mapTo[Data]
for {
v1 <- f1
...
Or, you would have to wire the Actors together by hand through the constructor and handle all of the callback logic for waiting on the other Actor:
class WaitingDBActor(dbCall : () => Data, previousActor : ActorRef) {
override def receive = {
case _ : MakeQuery => previousActor forward MakeQuery
case previousData : Data => sender ! (dbCall(), previousData)
}
}
If you want to querying database, you should use something like slick which is a modern database query and access library for Scala.
quick example of slick:
case class User(id: Option[Int], first: String, last: String)
class Users(tag: Tag) extends Table[User](tag, "users") {
def id = column[Int]("id", O.PrimaryKey, O.AutoInc)
def first = column[String]("first")
def last = column[String]("last")
def * = (id.?, first, last) <> (User.tupled, User.unapply)
}
val users = TableQuery[Users]
then your need to create configuration for your db:
mydb = {
dataSourceClass = "org.postgresql.ds.PGSimpleDataSource"
properties = {
databaseName = "mydb"
user = "myuser"
password = "secret"
}
numThreads = 10
}
and in your code you load configuration:
val db = Database.forConfig("mydb")
then run your query with db.run method which gives you future as result, for example you can get all rows by calling method result
val allRows: Future[Seq[User]] = db.run(users.result)
this query run without blocking current thread.
If you have task which take long time to execute or calling to another service, you should use futures.
Example of that is simple HTTP call to external service. you can find example in here
If you have task which take long time to execute and for doing so, you have to keep mutable states, in this case the best option is using Akka Actors which encapsulate your state inside an actor which solve problem of concurrency and thread safety as simple as possible.Example of suck tasks are:
import akka.actor.Actor
import scala.concurrent.Future
case class RegisterEndpoint(endpoint: String)
case class NewUpdate(update: String)
class UpdateConsumer extends Actor {
val endpoints = scala.collection.mutable.Set.empty[String]
override def receive: Receive = {
case RegisterEndpoint(endpoint) =>
endpoints += endpoint
case NewUpdate(update) =>
endpoints.foreach { endpoint =>
deliverUpdate(endpoint, update)
}
}
def deliverUpdate(endpoint: String, update: String): Future[Unit] = {
Future.successful(Unit)
}
}
If you want to process huge amount of live data, or websocket connection, processing CSV file which is growing over time, ... or etc, the best option is Akka stream. For example reading data from kafka topic using Alpakka:Alpakka kafka connector

Assign dynamically injected database name in Play Slick

I have the following Play Slick DAO class. Note that the database configuration is a constant control0001. The DAO has a function readUser that reads a user based on its user id:
class UsersDAO #Inject()(#NamedDatabase("control0001")
protected val dbConfigProvider: DatabaseConfigProvider)
extends HasDatabaseConfigProvider[JdbcProfile] {
import driver.api._
def readUser (userid: String) = {
val users = TableQuery[UserDB]
val action = users.filter(_.userid === userid).result
val future = db.run(action.asTry)
future.map{
case Success(s) =>
if (s.length>0)
Some(s(0))
else
None
case Failure(e) => throw new Exception ("Failure in readUser: " + e.getMessage)
}
}
}
Instead of having a constant in #NamedDatabase("control0001"), I need the database to be variable. In the application, I have multiple databases (control0001, control002 and so on) configured in application.conf. Depending on a variable value, I need to determine the database to be used in the DAO. All the databases are similar and have the same tables (the data in each database differs).
The following Play class calls the DAO function, but first it needs to determine the database name to be injected:
class TestSlick #Inject()(dao: UsersDAO) extends Controller {
def test(someCode: Int, userId: String) = Action { request =>
val databaseName = if (someCode == 1) "control0001" else "control0002"
// Run the method in UsersDAO accessing the database set by databaseName
val future = dao.readUser(userId)
future.map { result =>
result match {
case Some(user) => Ok(user.firstName)
case _ => Ok("user not found")
}
}
}
}
How can this be achieved in Play Slick?
You can try to initialize slick db object overriding default config:
val db = Database.forURL("jdbc:mysql://localhost/" + databaseName, driver="org.h2.Driver")
more information in slick docs http://slick.lightbend.com/doc/3.0.0/database.html
Instead of trying to use Play's runtime dependency injection utilities in this case, use the SlickApi directly in your DAO and pass the datasource name to the dbConfig(DbName(name)) method. To obtain the SlickApi, mix in the SlickComponents trait:
class UsersDAO extends SlickComponents {
def readUser(userid: String, dbName: String) = {
val users = TableQuery[UserDB]
val action = users.filter(_.userid === userid).result
val dbConfig = slickApi.dbConfig(DbName(dbName))
val future = dbConfig.db.run(action.asTry)
...
}
}
Then in your controller:
def test(someCode: Int, userId: String) = Action { request =>
val databaseName = if (someCode == 1) "control0001" else "control0002"
val future = dao.readUser(userId, databaseName)
...
}

Database transactions in Play framework scala applications (anorm)

I am developing an application using Play framework and scala. I am using anorm for data-access layer. And I've got a problem I could not solve.
Brief: I want to be able to have methods in data-access objects (dao) to work inside transactions as well as being called alone.
Details:
I have data-access layer consist of class with methods that only executes particular SQL over database. Traditionally they looks like:
def list() = DB.withConnection { implicit cn =>
...
}
Now I want to have some methods to be executed in a transaction scope. Like traditional select-update service methods but still be able to run them alone. So, what I have in my mind is like this:
class Service {
def fooTransacted() = {
inTransaction {
val old = dao.select(id = 2)
val newObj = old.copy(isActive = true)
dao.update(newObj)
}
}
def fooSinle() = {
dao.select(id = 2)
}
}
I tried around several ways, but could not come up with any solution.
What about
class Dao {
def foo(id: Long)(implicit connection: Connection) = {
SQL("select * from foo where id={id}").on('id->id).as(...)
}
}
class Service{
def withConnection = {
DB.withConnection {implicit connection =>
Dao.foo(1)
Dao.foo(2)
}
}
def withTransaction = {
DB.withTransaction {implicit connection =>
Dao.foo(1)
Dao.foo(2)
}
}
The solution I've seen used elsewhere (principally in Squeryl), is roughly the following:
import java.sql.Connection
object Helper {
private val conn: ThreadLocal[Connection] = new ThreadLocal
def inTransaction[X](f: Connection => X) = {
conn.get() match {
case null =>
DB.withConnection { newConn =>
conn.set(newConn)
try f(newConn)
finally conn.set(null)
}
case c => f(c)
}
}
}
This way, the inTransaction method is re-entrant, so there's no harm in calling it redundantly inside dao.select.
If you prefer, you can expose conn via a public method, and change the signature of f to => X - you lose some compile-time safety, but the API is a little cleaner.
One pitfall with this approach is that connections are tied to threads, which may cause problems if you're using futures or actors, and a process can resume on a different thread (this is a tricky area anyway, but one you should be aware of).
You might want to look into Squeryl too - it may already do what you need.

Play2 and Scala, How should I configure my integration tests to run with proper DB

I'm trying to figure out how to write my db integration tests in my Play2 app.
In my conf file I have specified two databases, xxx_test for regular use and h2 db for testing:
db.xxx_test.driver=com.mysql.jdbc.Driver
db.xxx_test.url="jdbc:mysql://localhost/xxx_test?characterEncoding=UTF-8"
db.xxx_test.user="root"
db.xxx_test.password=""
db.h2.driver=org.h2.Driver
db.h2.url="jdbc:h2:mem:play"
db.h2.user=sa
db.h2.password=""
In my User object I have specified xxx_test to be used when I run the application.
def createUser(user: User): Option[User] = {
DB.withConnection("xxx_test") {
implicit connection =>
SQL("insert into users(first_name, last_name, email, email_validated, last_login, created, modified, active) values({first_name},{last_name},{email},{email_validated},{last_login}, {created}, {modified}, {active})").on(
'first_name -> user.firstName,
'last_name -> user.lastName,
'email -> user.email,
'email_validated -> user.emailValidated,
'last_login -> user.lastLogin,
'created -> user.created,
'modified -> user.modified,
'active -> true
).executeInsert().map(id => {
return Some(User(new Id[Long](id), user.firstName, user.lastName, user.email, user.emailValidated, user.lastLogin, user.created, user.modified, true))
}
)
}
None
}
In my test I create a new inMemoryDatabase and then use the User to create and get my object
for testing.
class DBEvolutionsTest extends Specification {
"The Database" should {
"persist data properly" in {
running(FakeApplication(additionalConfiguration = inMemoryDatabase())) {
User.create(User(Id[Long](1L), "jakob",
"aa",
"aaa",
true,
DateTime.now(),
DateTime.now(),
DateTime.now(),
true))
val newUser = User.findBy(Id[Long](1L))
newUser.get.firstName must beEqualTo("jakob")
}
}
}
}
This of course is not the correct way since the User object uses the xxx_test and not the
h2 db. The test will create a User in the real db and not the one in memory since I have specified the db in the User(DB.withConnection("xxx_test")) object.
I guess there is some smart way of doing this, I do not want to pass the db name around
inte the application like User.create(User(...), "xxx_test")
How have you solved this problem?
You might want to checkout how to do dependency injection in scala. A good solution is to abstract your database out of your User model and then pass it as a dependency.
A simple way to do it would be to change the configuration file for testing. Play lets you specify which config file is used on the command line.
This is not the most practical though.
Another solution is to use implicits, define your database connection as an implicit parameter of your function:
def createUser(user: User)(implicit dbName: String): Option[User]=
DB.withConnection(dbName) { ... }
You will still have to propagate the parameter upwards in all your calls, but you can hide it:
def importUsers(csvFile: File)(implicit dbName: String): Seq[User] = { conn =>
...
User.createUser(u)
...
}
and when you call it from the top:
implicit dbName = "test"
importUsers(...)
This is build in in scala, so it's pretty easy to setup and doesn't need a lot of boilerplate supporting it. Personally, I think that implicits make the code unclear and I prefer the solution presented in this presentation Dead-Simple Dependency Injection.
The gist of it is that you make your createUser and all other methods that depend on a database connection to return a function depending on the connection, and not just the result. Here is how it would work with your example.
1- you create a Connection trait that configures a connection. A simple form would be:
trait ConnectionConfig {
def dbName: String
}
2- your method depending on that config returns a function:
def createUser(user: User): ConnectionConfig => Option[User] = { conn =>
DB.withConnection(conn.dbName) { ... }
}
3- when you use createUser in another method, that method becomes dependent on the connection too, so you mark it as such by returning the dependency on ConnectionConfig with a function return type, for example:
def importUsers(csvFile: File): ConnectionConfig => Seq[User] = { conn =>
...
User.createUser(u)(conn)
...
}
This is a good habit to have, as it will be clear in your code which methods depends on a connection to the database, and you can easily swap connections. So, in your main app, you would create the real connection:
class RealConncetionConfig extends ConnectionConfig {
val dbName = "xxx_test"
}
but in your test file, you create a test DB config:
class DBEvolutionsTest extends Specification {
class TestDBConfig extends ConnectionConfig {
val dbName = "h2"
}
val testDB = new TestDBConfig()
"The Database" should {
"persist data properly" in {
running(FakeApplication(additionalConfiguration = inMemoryDatabase())) {
User.create(User(Id[Long](1L), "jakob",
"aa",
"aaa",
true,
DateTime.now(),
DateTime.now(),
DateTime.now(),
true))(testDB)
val newUser = User.findBy(Id[Long](1L))
newUser.get.firstName must beEqualTo("jakob")
}
}
}
}
This is the gist of it. Check out the presentation and slides I mentioned, there is a nice way to abstract all of that so that you can loose the (conn) argument that is making this code ugly.
As a side comment, if I were you, I would even abstract the type of DB. So, instead of having the SQL in the User model object, have it in a separate implementation, this way you can easily switch the type of database (use mongodb, dynamo...).
It would just be something like this, extending from the previous code:
trait ConnectionConfig {
def createUser(user: User): Option[User]
}
and in the User model object:
def createUser(user: User): ConnectionConfig => Option[User] = { conn =>
conn.createUser(user)
}
this way, when testing parts of your code depending on the User model, you can make a mock DB where createUser always works and returns the expected result (or always fails...), without even using the in memory database (you would still need tests for the real SQL connection, but you could test other parts of your app):
trait ConnectionConfig {
def createUser(user: User): Option[User] = Some(user)
}
The inMemoryDatabase method is defined like this:
def inMemoryDatabase(
name: String = "default",
options: Map[String, String] = Map.empty[String, String]): Map[String, String]
My guess is that you should pass the xxx_test as the name parameter.
You must define a name other than your default(xxx_test) name for the in memory database. I think the following snippet should work.
FakeApplication(additionalConfiguration = inMemoryDatabase("h2"))
Please see also: https://stackoverflow.com/a/11029324/2153190

How could I know if a database table is exists in ScalaQuery

I'm trying ScalaQuery, it is really amazing. I could defined the database table using Scala class, and query it easily.
But I would like to know, in the following code, how could I check if a table is exists, so I won't call 'Table.ddl.create' twice and get a exception when I run this program twice?
object Users extends Table[(Int, String, String)]("Users") {
def id = column[Int]("id")
def first = column[String]("first")
def last = column[String]("last")
def * = id ~ first ~ last
}
object Main
{
val database = Database.forURL("jdbc:sqlite:sample.db", driver = "org.sqlite.JDBC")
def main(args: Array[String]) {
database withSession {
// How could I know table Users is alrady in the DB?
if ( ??? ) {
Users.ddl.create
}
}
}
}
ScalaQuery version 0.9.4 includes a number of helpful SQL metadata wrapper classes in the org.scalaquery.meta package, such as MTable:
http://scalaquery.org/doc/api/scalaquery-0.9.4/#org.scalaquery.meta.MTable
In the test code for ScalaQuery, we can see examples of these classes being used. In particular, see org.scalaquery.test.MetaTest.
I wrote this little function to give me a map of all the known tables, keyed by table name.
import org.scalaquery.meta.{MTable}
def makeTableMap(dbsess: Session) : Map[String, MTable] = {
val tableList = MTable.getTables.list()(dbsess);
val tableMap = tableList.map{t => (t.name.name, t)}.toMap;
tableMap;
}
So now, before I create an SQL table, I can check "if (!tableMap.contains(tableName))".
This thread is a bit old, but maybe someone will find this useful. All my DAOs include this:
def create = db withSession {
if (!MTable.getTables.list.exists(_.name.name == MyTable.tableName))
MyTable.ddl.create
}
Here's a full solution that checks on application start using a PostGreSQL DB for PlayFramework
import globals.DBGlobal
import models.UsersTable
import org.scalaquery.meta.MTable
import org.scalaquery.session.Session
import play.api.GlobalSettings
import play.api.Application
object Global extends GlobalSettings {
override def onStart(app: Application) {
DBGlobal.db.withSession { session : Session =>
import org.scalaquery.session.Database.threadLocalSession
import org.scalaquery.ql.extended.PostgresDriver.Implicit._
if (!makeTableMap(session).contains("tableName")) {
UsersTable.ddl.create(session)
}
}
}
def makeTableMap(dbsess: Session): Map[String, MTable] = {
val tableList = MTable.getTables.list()(dbsess)
val tableMap = tableList.map {
t => (t.name.name, t)
}.toMap
tableMap
}
}
With java.sql.DatabaseMetaData (Interface). Depending on your Database, more or less functions might be implemented.
See also the related discussion here.I personally prefer hezamu's suggestion and extend it as follows to keep it DRY:
def createIfNotExists(tables: TableQuery[_ <: Table[_]]*)(implicit session: Session) {
tables foreach {table => if(MTable.getTables(table.baseTableRow.tableName).list.isEmpty) table.ddl.create}
}
Then you can just create your tables with the implicit session:
db withSession {
implicit session =>
createIfNotExists(table1, table2, ..., tablen)
}
You can define in your DAO impl the following method (taken from Slick MTable.getTables always fails with Unexpected exception[JdbcSQLException: Invalid value 7 for parameter columnIndex [90008-60]]) that gives you a true o false depending if there a defined table in your db:
def checkTable() : Boolean = {
val action = MTable.getTables
val future = db.run(action)
val retVal = future map {result =>
result map {x => x}
}
val x = Await.result(retVal, Duration.Inf)
if (x.length > 0) {
true
} else {
false
}
}
Or, you can check if some "GIVENTABLENAME" or something exists with println method:
def printTable() ={
val q = db.run(MTable.getTables)
println(Await.result(q, Duration.Inf).toList(0)) //prints first MTable element
println(Await.result(q, Duration.Inf).toList(1))//prints second MTable element
println(Await.result(q, Duration.Inf).toList.toString.contains("MTable(MQName(public.GIVENTABLENAME_pkey),INDEX,null,None,None,None)"))
}
Don't forget to add
import slick.jdbc.meta._
Then call the methods from anywhere with the usual #Inject(). Using
play 2.4 and play-slick 1.0.0.
Cheers,