I am developing an application using Play framework and scala. I am using anorm for data-access layer. And I've got a problem I could not solve.
Brief: I want to be able to have methods in data-access objects (dao) to work inside transactions as well as being called alone.
Details:
I have data-access layer consist of class with methods that only executes particular SQL over database. Traditionally they looks like:
def list() = DB.withConnection { implicit cn =>
...
}
Now I want to have some methods to be executed in a transaction scope. Like traditional select-update service methods but still be able to run them alone. So, what I have in my mind is like this:
class Service {
def fooTransacted() = {
inTransaction {
val old = dao.select(id = 2)
val newObj = old.copy(isActive = true)
dao.update(newObj)
}
}
def fooSinle() = {
dao.select(id = 2)
}
}
I tried around several ways, but could not come up with any solution.
What about
class Dao {
def foo(id: Long)(implicit connection: Connection) = {
SQL("select * from foo where id={id}").on('id->id).as(...)
}
}
class Service{
def withConnection = {
DB.withConnection {implicit connection =>
Dao.foo(1)
Dao.foo(2)
}
}
def withTransaction = {
DB.withTransaction {implicit connection =>
Dao.foo(1)
Dao.foo(2)
}
}
The solution I've seen used elsewhere (principally in Squeryl), is roughly the following:
import java.sql.Connection
object Helper {
private val conn: ThreadLocal[Connection] = new ThreadLocal
def inTransaction[X](f: Connection => X) = {
conn.get() match {
case null =>
DB.withConnection { newConn =>
conn.set(newConn)
try f(newConn)
finally conn.set(null)
}
case c => f(c)
}
}
}
This way, the inTransaction method is re-entrant, so there's no harm in calling it redundantly inside dao.select.
If you prefer, you can expose conn via a public method, and change the signature of f to => X - you lose some compile-time safety, but the API is a little cleaner.
One pitfall with this approach is that connections are tied to threads, which may cause problems if you're using futures or actors, and a process can resume on a different thread (this is a tricky area anyway, but one you should be aware of).
You might want to look into Squeryl too - it may already do what you need.
Related
I have a method in which there are multiple calls to db. As I have not implemented any concurrent processing, a 2nd db call has to wait until the 1st db call gets completed, 3rd has to wait until the 2nd gets completed and so on.
All db calls are independent of each other. I want to make this in such a way that all DB calls run concurrently.
I am new to Akka framework.
Can someone please help me with small sample or references would help. Application is developed in Scala Lang.
There are three primary ways that you could achieve concurrency for the given example needs.
Futures
For the particular use case that is asked about in the question I would recommend Futures before any akka construct.
Suppose we are given the database calls as functions:
type Data = ???
val dbcall1 : () => Data = ???
val dbcall2 : () => Data = ???
val dbcall3 : () => Data = ???
Concurrency can be easily applied, and then the results can be collected, using Futures:
val f1 = Future { dbcall1() }
val f2 = Future { dbcall2() }
val f3 = Future { dbcall3() }
for {
v1 <- f1
v2 <- f2
v3 <- f3
} {
println(s"All data collected: ${v1}, ${v2}, ${v3}")
}
Akka Streams
There is a similar stack answer which demonstrates how to use the akka-stream library to do concurrent db querying.
Akka Actors
It is also possible to write an Actor to do the querying:
object MakeQuery
class DBActor(dbCall : () => Data) extends Actor {
override def receive = {
case _ : MakeQuery => sender ! dbCall()
}
}
val dbcall1ActorRef = system.actorOf(Props(classOf[DBActor], dbcall1))
However, in this use case Actors are less helpful because you still need to collect all of the data together.
You can either use the same technique as the "Futures" section:
val f1 : Future[Data] = (dbcall1ActorRef ? MakeQuery).mapTo[Data]
for {
v1 <- f1
...
Or, you would have to wire the Actors together by hand through the constructor and handle all of the callback logic for waiting on the other Actor:
class WaitingDBActor(dbCall : () => Data, previousActor : ActorRef) {
override def receive = {
case _ : MakeQuery => previousActor forward MakeQuery
case previousData : Data => sender ! (dbCall(), previousData)
}
}
If you want to querying database, you should use something like slick which is a modern database query and access library for Scala.
quick example of slick:
case class User(id: Option[Int], first: String, last: String)
class Users(tag: Tag) extends Table[User](tag, "users") {
def id = column[Int]("id", O.PrimaryKey, O.AutoInc)
def first = column[String]("first")
def last = column[String]("last")
def * = (id.?, first, last) <> (User.tupled, User.unapply)
}
val users = TableQuery[Users]
then your need to create configuration for your db:
mydb = {
dataSourceClass = "org.postgresql.ds.PGSimpleDataSource"
properties = {
databaseName = "mydb"
user = "myuser"
password = "secret"
}
numThreads = 10
}
and in your code you load configuration:
val db = Database.forConfig("mydb")
then run your query with db.run method which gives you future as result, for example you can get all rows by calling method result
val allRows: Future[Seq[User]] = db.run(users.result)
this query run without blocking current thread.
If you have task which take long time to execute or calling to another service, you should use futures.
Example of that is simple HTTP call to external service. you can find example in here
If you have task which take long time to execute and for doing so, you have to keep mutable states, in this case the best option is using Akka Actors which encapsulate your state inside an actor which solve problem of concurrency and thread safety as simple as possible.Example of suck tasks are:
import akka.actor.Actor
import scala.concurrent.Future
case class RegisterEndpoint(endpoint: String)
case class NewUpdate(update: String)
class UpdateConsumer extends Actor {
val endpoints = scala.collection.mutable.Set.empty[String]
override def receive: Receive = {
case RegisterEndpoint(endpoint) =>
endpoints += endpoint
case NewUpdate(update) =>
endpoints.foreach { endpoint =>
deliverUpdate(endpoint, update)
}
}
def deliverUpdate(endpoint: String, update: String): Future[Unit] = {
Future.successful(Unit)
}
}
If you want to process huge amount of live data, or websocket connection, processing CSV file which is growing over time, ... or etc, the best option is Akka stream. For example reading data from kafka topic using Alpakka:Alpakka kafka connector
Here is code I am trying to optimise:
object UserRepo
{
val users = TableQuery[Users]
val dbName = "db"
lazy val queryAllUsers = for (user <- users) yield user
type UserRow = (Int, String, String, String)
def getAll() : Future[ Seq[UserRow] ] =
{
val db = Database.forConfig( dbName )
val f: Future[Seq[UserRow]] = db.run( queryAllUsers.result )
f.onComplete {
case Success(_) => { db.close() }
case Failure(_) => { db.close() }
}
f
}
}
I going to have number of query to the DB I am trying to get rid of string where I am creating DB connection. Is there any execution context I can use to close connection explicitly ?? so code will look more concise ?
Is there option to get used db connection within Future.onComplete scope??
Thanks
As for your comment ( explicitly close db connection in slick ) normally what you do is to create a connection on an application startup (or lazily on first use) and then closing it at the application end.
This obviously all depends what kind of application you are running:
if you are having DI container you would probably manage some of this in your DI mechanisms (like Modules in Guice)
if you are having web application, specifically e.g. Play - you would probably use play-slick that does this initialization / shutting down for you (kind of).
General way (no DI)
The easiest general way (assuming you are not using DI or play-slick) of doing this would be perhaps something like this:
object DbManager {
lazy val db = createDb
private def createDb = {
Database.forConfig("db")
}
def close {
db.close
}
}
Then your code would be:
object UserRepo
{
val users = TableQuery[Users]
lazy val queryAllUsers = for (user <- users) yield user
type UserRow = (Int, String, String, String)
def getAll() : Future[ Seq[UserRow] ] =
{
DbManager.db.run( queryAllUsers.result )
}
}
Above code doesn't do any cleaning up - this would need to be added to some kind of hook when application is closing (in case e.g. of web application) or you would need to manually call DbManager.close at some specified time (when you are closing the application).
Play slick
You would probably need to start from here: https://github.com/playframework/play-slick/tree/master/samples/basic (most basic sample showing play-slick configuration).
Updating your answer with this would be:
class UserRepo #Inject() (dbConfigProvider: DatabaseConfigProvider) extends HasDatabaseConfigProvider[JdbcProfile])
{
import driver.api._
val users = TableQuery[Users]
lazy val queryAllUsers = for (user <- users) yield user
type UserRow = (Int, String, String, String)
def getAll() : Future[ Seq[UserRow] ] =
{
db.run( queryAllUsers.result )
}
}
In this scenario you wouldn't call:
UserRepo.getAll
but you would rather need to inject it:
class MyClientCode #Inject() (userRepo: UserRepo) {
...
userRepo.getAll
...
}
You would need to obviously configure it in your configuration but this should be very straightforward to do with the sample provided above.
So in short your Play application will have database connection configuration and would do all initialization / cleaning up. Your external modules (like the one you described in your comment) would simply pull DatabaseConfigProvider as Guice managed dependency (as show above).
Edit2
Okay, so maybe I should parse out two desires here.
I had in mind that when it came time to get to the setSendTimeout(0) part, I would be using something like implicitly[Socket].
new ZContext(1) {
createSocket(ZMQ.PUB).setSendTimeout(0).//RATS!
}
I also had in mind a more generic approach to it, that would be (in pseudo code terms):
This is how you wrap a reference of T at a point in time without copying it, so that moving forward, you can tease out state of the reference after potential state changes from the value of whatever expression used it.
If it could be thought of as a chain of map map map s from T to where ever it ended up, then it is easy to append / apply a value onto it - just map again...
This is the motivating example.
override def createLogic(inheritedAttributes: Attributes): GraphStageLogic = new GraphStageLogic(shape) {
logger.info("Initializing ZMQ context.")
val context = new ZContext(1)
logger.info(s"Binding PUB socket to ${endpoint}")
val socket = {
val s = context.createSocket(ZMQ.PUB)
s.setSendTimeOut(0)
s.bind(endpoint)
s
}
Look at socket down there. For some reason that feels uglier than it needs to be to me, but is a consequence of the fact that setters don't return stuff like setSendTimeOut.
I would normally try to improve it as follows:
new ZContext(1) {
createSocket(ZMQ.PUB).setSendTimeout(0).//RATS!
}
Here a version of #Dima's answer. Again the setup:
trait Instance {
def createPort(): Port
}
trait Port {
def makeSpecial(): Unit
def bindTo(address: Any): Unit
}
trait Provider {
def getTheInstance(i: Int): Instance
}
Now the trick:
implicit class InstanceOps(i: Instance) {
def withCreatePort(fun: (Unit => Port) => Any): Port = {
val res = i.createPort()
fun(_ => res)
res
}
}
And if you add an implicit modifier to the argument of the function passed into withCreatePort, you "import" the implicit conversion:
trait ConnectTest extends Provider {
getTheInstance(2).withCreatePort { implicit p =>
().makeSpecial().bindTo("foo")
}
}
This is potentially more dangerous, because you have an implicit conversion from Unit to Port, although it is locally encapsulated. This is generic because Connect is generic.
This trick is perhaps too clever and difficult to understand by some outside standing person reading your code.
Yes, you can create two wrappers, one giving you withCreatePort, the other giving you variants of the port method that return this:
trait Instance {
def createPort(): Port
}
trait Port {
def makeSpecial(): Unit
def bindTo(address: Any): Unit
}
class PortOps(p: Port) {
def makeSpecial() : this.type = { p.makeSpecial() ; this }
def bindTo(address: Any): this.type = { p.bindTo(address); this }
}
implicit class InstanceOps(i: Instance) {
def withCreatePort[A](fun: PortOps => A): A = fun(new PortOps(i.createPort()))
}
Example:
trait Provider {
def getTheInstance(i: Int): Instance
}
trait Plain extends Provider {
val instance = getTheInstance(2)
val port = instance.createPort()
port.makeSpecial()
port.bindTo("foo")
}
trait Rich extends Provider {
getTheInstance(2).withCreatePort { p =>
p.makeSpecial().bindTo("foo")
}
}
The question is if the effort is worth it. You can also experiment with import:
trait Import extends Provider {
val instance = getTheInstance(2)
val port = instance.createPort()
locally {
import port._
makeSpecial(); bindTo("foo")
}
}
I am not sure where you are going with this Zipped thingy ... But what you described in the beginning of your question (assuming that port in the end of that snippet is a typo, and you really meant to return instance) can be done with something like this:
object Taps {
implicit class Tap[T](t: T) extends Anyval {
def tap(f: T => Unit) = { f(t); t }
}
}
Then you can write:
import Taps._
val instance = getTheInstance(2).tap {
_.createPort
.makeSpecial
.bindTo(...)
}
Is this what you are looking for?
In my particular case I have a menu that is rendered on all pages. The menu content is loaded from a database using slick and passed implicitly to the view. The whole thing looks like this:
Controller
class Application #Inject()(
implicit val menuContext: MenuContext
) extends Controller {
def index = Action.async {
val content: Future[Content] = getContent
content.map(c => Ok(views.html.index(c)))
}
}
MenuContext
class MenuContext {
val models: Future[List[SomeModel]] = getModelsFromDB
}
View
#(content: Content)(implicit menuContext: MenuContext)
...
#menuContext.models // how to access my actual model and not the Future?
...
How do I access List[SomeModel] in my view? Is there an Action.async equivalent in play for passing implicit parameters? Or is there maybe even a better solution for stuff that is required in (almost) all views?
Definitely not a good idea to make a template have to deal with a Future - so the question becomes the one in your comment - how to non-blockingly (?) get the content from your async content source, as well as your menu items from your other async content source?
A for-comprehension on the two Future instances does the trick:
def index = Action.async {
val fContent:Future[Content] = getContent
val fMenus:Future[List[SomeModel] = getModelsFromDB
for {
content <- fContent
menus <- fMenus
} yield(Ok(views.html.index(content)(menus))))
}
Note: You may be tempted to try and save a few lines and put the method calls (getContent, getModelsFromDB) straight into the for block.
Unfortunately, while it'll compile and work, the two tasks won't run in parallel, thereby making the exercise somewhat futile.
OK I'm adding another answer here, to specifically try to DRY up the injection of menus into your Actions.
The principal problem is that you need to inject the menus at just the right time, namely:
When you have the data ready (or at least a Future holding the data)
When you know which template is going to be rendered
When you know what status code you'll be returning
Because of these constraints, we can't use an ActionBuilder or ActionRefiner - they assume that your inner block of controller code will produce a finished Result.
So instead, we'll define a trait we can mix into our controllers:
trait MenuDecoration {
def withMenuSimple(body: Future[List[SomeModel] => Result]):Future[Result] = {
val fm = getModelsFromDB
val fb = body
for {
m <- fm
b <- fb
} yield(b(m))
}
}
This should look pretty familiar from my other answer, and it works the same way - it will start the execution of both async tasks, bringing them together once they are both done.
An Action that needs to decorate the template with a menu looks like this:
class BlahController extends Controller with MenuDecoration {
def index = Action.async {
withMenuSimple {
getContent.map { content => implicit menu =>
Ok(views.html.index(content))
}
}
}
}
Why withMenuSimple ? Because at some point you'll probably want to examine the Request - so we have this alternative:
trait MenuDecoration {
...
def withMenu(body: RequestHeader => Future[List[SomeModel] => Result])(implicit request:RequestHeader):Future[Result] = {
val fm = fMenus
val fb = body(request)
for {
m <- fm
b <- fb
} yield(b(m))
}
}
which you'd use like this:
def indexWithReq = Action.async { implicit request =>
withMenu { req =>
getContent.map { content => implicit menu =>
Ok(views.html.index(content))
}
}
}
I am trying to find the best way to handle jedis commands from scala. I am trying to implement a finally block, and prevent the java exceptions from bubbling up to my caller.
Does the following code make sense, and is it the best I can do performance wise, if I want to ensure that I handle exceptions when redis may be down temporarily? This trait would be extended by an object, and I'd call objectname.del(key). I feel like I'm combining too many concepts (Either, Option, Try, feels like there should be a cleaner way)
trait MyExperiment {
implicit class TryOps[T](val t: Try[T]) {
def eventually[Ignore](effect: => Ignore): Try[T] = {
val ignoring = (_: Any) => { effect; t }
t transform (ignoring, ignoring)
}
}
val jpool:JedisPool = initialize()
// init the pool at object creation
private def initialize(): JedisPool =
{
val poolConfig = new JedisPoolConfig()
poolConfig.setMaxIdle(10)
poolConfig.setMinIdle(2)
poolConfig.setTestWhileIdle(true)
poolConfig.setTestOnBorrow(true)
poolConfig.setTestOnReturn(true)
poolConfig.setNumTestsPerEvictionRun(10)
new JedisPool( poolConfig , "localhost" )
}
// get a resource from pool. This can throw an error if redis is
// down
def getFromPool: Either[Throwable,Jedis] =
Try(jpool.getResource) match {
case Failure(m) => Left(m)
case Success(m) => Right(m)
}
// return an object to pool
// i believe this may also throw an error if redis is down?
def returnToPool(cache:Jedis): Unit =
Try(jpool.returnResource(cache))
// execute a command -- "del" in this case, (wrapped by
// the two methods above)
def del(key: String) : Option[Long] = {
getFromPool match {
case Left(m) => None
case Right(m) => Try(m.del(key)) eventually returnToPool(m) match {
case Success(r) => Option(r)
case Failure(r) => None
}
}
}
}
Not an exact answer, but I moved on after doing some performance testing. Using the standard java-ish exception blocks ended up being much faster at high iterations (at 10,000 iterations, it was about 2.5x faster than the (bad) code above). That also cleaned up my code, although it's more verbose.
So the answer I arrived at is to use the Java-style exception blocks which provide for the finally construct. I believe it should be significantly faster, as long as exceptions are a very rare occurance.