Slick 3: How to implement repository pattern with transactions? - scala

In my play framework (2.5) app, I need to write unit tests for services.
I need to isolate data access logic to able to test service layer in isolation,
for this I want to create repository interfaces and MOCK them in my unit tests:
class UserService {
def signUpNewUser(username: String, memberName: String): Future[Unit] {
val userId = 1 // Set to 1 for demo
val user = User(userId, username)
val member = Member(memberName, userId)
// ---- I NEED TO EXECUTE THIS BLOCK WITHIN TRANSACTION ----
for {
userResult <- userRepository.save(user)
memberRepository.save(member)
} yield ()
// ---- END OF TRANSACTION ----
}
}
In the above example, userRepository.save(User) and memberRepository.save(member) operations should be performed within transaction.
I don't want to use slick directly in my service layer because it will complicate my tests.
Also, I don't want to use embedded database for my unit tests, elsewhere it would be a NOT unit test, I need complete isolation.
I do not want my repository interfaces to be depended on slick at all, but need something like this:
trait UserRepository {
findById(id: Long): Future[Option[User]]
save(user: User): Future[Unit]
}
how can I achieve this with slick?

OK - let's decompose your question into three parts.
How to execute block in transaction
Basically read this answer: How to use transaction in slick
As soon as you convert DBIO to Future you are done. No chances to compose multiple operation within single transaction. End of story.
How to avoid using Slick in tests
This is basically a design question - if you want to have a business layer on top of Repository / DAO / whatever - than let this service layer deal with transactions. You won't need to interact with Slick outside of this layer.
Avoiding your repository interfaces to depend on Slick
In most straightforward way - you need to depend on Slick DBIO to compose operations within transaction (and composing Repository methods within transaction is something that you cannot avoid in any serious application).
If you want to avoid depending on DBIO you would perhaps create you own monadic type, say TransactionBoundary[T] or TransactionContext[T].
Then you would have something like TransactionManager that would execute this TransactionContext[T].
IMHO not worth the effort, I'd simply use DBIO which has a brilliant name already (like Haskell's IO monad - DBIO informs you that you have a description of IO operations performed on your storage). But let's assume that you still want to avoid it.
You could do something like that perhaps:
package transaction {
object Transactions {
implicit class TransactionBoundary[T](private[transaction] val dbio: DBIO[T]) {
// ...
}
}
class TransactionManager {
def execute[T](boundary: TransactionBoundary[T]): Future[T] = db.run(boundary.dbio)
}
}
Your trait would look like this:
trait UserRepository {
findById(id: Long): TransactionBoundary[Option[User]]
save(user: User): TransactionBoundary[Unit]
}
and somewhere in your code you would do like this:
transactionManager.execute(
for {
userResult <- userRepository.save(user)
memberRepository.save(member)
} yield ()
)
By using implicit conversion you would have your results of methods in Repository be automatically converted to your TransactionBoundary.
But again - IMHO all of the above doesn't bring any actual advantage over using DBIO (except perhaps taste of esthetics). If you want to avoid using Slick related classes outside of certain layer, just make a type alias like this:
type TransactionBoundary[T] = DBIO[T]
and use it everywhere.

Related

Good practices of writing and unit testing Utility methods in scala

In particular situations, you need to have some utility methods that are required across different classes. To solve this situation, you create an Util object wherein you place all these methods
object AggregatorUtil {
def aggregateValues(list : List[BigDecimal]) = //some logic...
}
// Import everything in the Utilities object
import AggregatorUtil._
and then import whichever members of util are required in your class. However, the downside to this is that, as all your methods are inside the singleton object and it becomes tricky to mock the object and unit test methods of the class that use utility methods.
To solve this problem again, the only solution that came to mind was Extracting the functionality out to a trait and then mocking the trait.
Please let me know if there is any other approach for handling and testing of util methods and which one is a rather cleaner approach.
Thanks in advance !!!
Note: -I am using scalatest and mockito in my project.
If you need to mock, putting this all in a mocked-out trait is the way forward. If mocking is unnecessary though, avoid it. Mocking unnecessarily is... unnecessary. You'll just be wasting time and effort for something which provides no additional value.
Mocking is best used when you have complex functionality or functionality in other files which you want to treat as a black box and just assume it works as expected (you'd then typically unit test this stuff separately). If you can avoid it and use the functions' actual functionality though, you'll get a much more realistic view of what your application does and will spot new bugs/breaking changes quicker (if you have mocked out functionality and forget to update your mocks, you might not any spot new bugs you introduce).
A good example of when mocking is necessary is when you're mocking calls to a database in a MVC application (e.g. a Scala Play microservice). You obviously don't want to have to run an actual database when testing your code, so you'd typically mock out your connector layer and return dummy/mocked data from your connector functions.
An example of something you wouldn't mock is something like:
trait MyTrait {
def toInt(str: String): Int
}
val mockedTrait = mock[MyTrait]
when(mockedTrait.toInt(eq("3")).thenReturn(3)
It's a bit of a silly example, but I think it explains the point clearly - doing something like this would be ridiculous. Mocking isn't always the answer.
I mostly mock with Test-Implementation which I find more readable and you don't have to learn a mocking framework.
Here an example:
The Interface:
trait DataRepo {
def persist(data: DataObject): Future[DataObject]
def idents(): Future[List[String]]
def insertData(dataCont: DataObject): Future[Int]
...
}
The Mocked Interface:
object DataRepoMock extends DataRepo {
def persist(data: DataObject): Future[DataObject] = ??? // only implement when needed
def idents(): Future[List[String]] = Future.successful((0 to 10).map(_=>Random.nextInt(100)))
def insertData(dataCont: DataObject): Future[Int] = Future.successful(Random.nextInt(100))
...
}
You can also use all the Scala goodies, like Pattern Matching, to make your Mock react differently to input.
Here is an example that this is not just used by myself;):
EPFLx: scala-reactiveX see Lecture 2.5 Testing Actor Systems:
def fakeGetter(url:String, depth: Int):Props =
Props(new Getter(url, depth){
override def webClient: WebClient = FakeWebClient
})

Transactional method in Scala Play with Slick (similar to Spring #Transactional, maybe?)

I know scala, as a funcional language, is supposed to work differently from a common OO language, such as Java, but I'm sure there has to be a way to wrap a group of database changes in a single transaction, ensuring atomicity as well as every other ACID property.
As explained in the slick docs (http://slick.lightbend.com/doc/3.1.0/dbio.html), DBIOAction allows to group db operations in a transaction like this:
val a = (for {
ns <- coffees.filter(_.name.startsWith("ESPRESSO")).map(_.name).result
_ <- DBIO.seq(ns.map(n => coffees.filter(_.name === n).delete): _*)
} yield ()).transactionally
val f: Future[Unit] = db.run(a)
However, my use case (and most real world examples I can think of), I have a code structure with a Controller, which exposes the code for my REST endpoint, that controller calls multiple services and each service will delegate database operations to DAOs.
A rough example of my usual code structure:
class UserController #Inject(userService: UserService) {
def register(userData: UserData) = {
userService.save(userData).map(result => Ok(result))
}
}
class UserService #Inject(userDao: UserDao, addressDao: AddressDao) {
def save(userData: UserData) = {
for {
savedUser <- userDao.save(userData.toUser)
savedAddress <- addressDao.save(userData.addressData.toAddress)
} yield savedUser.copy(address = savedAddress)
}
}
class SlickUserDao {
def save(user: User) = {
db.run((UserSchema.users returning UserSchema.users)).insertOrUpdate(user)
}
}
This is a simple example, but most have more complex business logic in the service layer.
I don't want:
My DAOs to have business logic and decide which database operations to run.
Return DBAction from my DAOs and expose the persistency classes. That completely defeats the purpose of using DAOs in the first place and makes further refactorings much harder.
But I definitely want a transaction around my entire Controller, to ensure that if any code fails, all the changes done in the execution of that method will be rolled back.
How can I implement full controller transactionality with Slick in a Scala Play application? I can't seem to find any documentation on how to do that.
Also, how can I disable auto-commit in slick? I'm sure there is a way and I'm just missing something.
EDIT:
So reading a bit more about it, I feel now I understand better how slick uses connections to the database and sessions. This helped a lot: http://tastefulcode.com/2015/03/19/modern-database-access-scala-slick/.
What I'm doing is a case of composing in futures and, based on this article, there's no way to use the same connection and session for multiple operation of the kind.
Problem is: I really can't use any other kind of composition. I have considerable business logic that needs to be executed in between queries.
I guess I can change my code to allow me to use action composition, but as I mentioned before, that forces me to code my business logic with aspects like transactionality in mind. That shouldn't happen. It pollutes the business code and it makes writing tests a lot harder.
Any workaround this issue? Any git project out there that sorts this out that I missed? Or, more drastic, any other persistence framework that supports this? From what I've read, Anorm supports this nicely, but I may be misunderstanding it and don't want to change framework to find out it doesn't (like it happened with Slick).
There is no such thing as transactional annotations or the like in slick. Your second "do not want" is actually the way to go. It's totally reasonable to return DBIO[User] from your DAO which does not defeat their purpose at all. It's the way slick works.
class UserController #Inject(userService: UserService) {
def register(userData: UserData) = {
userService.save(userData).map(result => Ok(result))
}
}
class UserService #Inject(userDao: UserDao, addressDao: AddressDao) {
def save(userData: UserData): Future[User] = {
val action = (for {
savedUser <- userDao.save(userData.toUser)
savedAddress <- addressDao.save(userData.addressData.toAddress)
whatever <- DBIO.successful(nonDbStuff)
} yield (savedUser, savedAddress)).transactionally
db.run(action).map(result => result._1.copy(result._2))
}
}
class SlickUserDao {
def save(user: User): DBIO[User] = {
(UserSchema.users returning UserSchema.users).insertOrUpdate(user)
}
}
The signature of save in your service class is still the same.
No db related stuff in controllers.
You have full control of transactions.
I cannot find a case where the code above is harder to maintain / refactor compared to your original example.
There is also a quite exhaustive discussion that might be interesting for you. See Slick 3.0 withTransaction blocks are required to interact with libraries.

How do I abstract the domain layer from the persistence layer in Scala

UPDATE:
I've edited the title and added this text to better explain what I'm trying to achieve: I'm trying to create a new application from the ground up, but don't want the business layer to know about the persistence layer, in the same way one would not want the business layer to know about a REST API layer. Below is an example of a persistence layer that I would like to use. I'm looking for good advice on integrating with this i.e. I need help with the design/architecture to cleanly split the responsibilities between business logic and persistence logic. Maybe a concept along the line of marshalling and unmarshalling of persistence objects to domain objects.
From a SLICK (a.k.a. ScalaQuery) test example, this is how you create a many-to-many database relationship. This will create 3 tables: a, b and a_to_b, where a_to_b keeps links of rows in table a and b.
object A extends Table[(Int, String)]("a") {
def id = column[Int]("id", O.PrimaryKey)
def s = column[String]("s")
def * = id ~ s
def bs = AToB.filter(_.aId === id).flatMap(_.bFK)
}
object B extends Table[(Int, String)]("b") {
def id = column[Int]("id", O.PrimaryKey)
def s = column[String]("s")
def * = id ~ s
def as = AToB.filter(_.bId === id).flatMap(_.aFK)
}
object AToB extends Table[(Int, Int)]("a_to_b") {
def aId = column[Int]("a")
def bId = column[Int]("b")
def * = aId ~ bId
def aFK = foreignKey("a_fk", aId, A)(a => a.id)
def bFK = foreignKey("b_fk", bId, B)(b => b.id)
}
(A.ddl ++ B.ddl ++ AToB.ddl).create
A.insertAll(1 -> "a", 2 -> "b", 3 -> "c")
B.insertAll(1 -> "x", 2 -> "y", 3 -> "z")
AToB.insertAll(1 -> 1, 1 -> 2, 2 -> 2, 2 -> 3)
val q1 = for {
a <- A if a.id >= 2
b <- a.bs
} yield (a.s, b.s)
q1.foreach(x => println(" "+x))
assertEquals(Set(("b","y"), ("b","z")), q1.list.toSet)
As my next step, I would like to take this up one level (I still want to use SLICK but wrap it nicely), to working with objects. So in pseudo code it would be great to do something like:
objectOfTypeA.save()
objectOfTypeB.save()
linkAtoB.save(ojectOfTypeA, objectOfTypeB)
Or, something like that. I have my ideas on how I might approach this in Java, but I'm starting to realize that some of my object-oriented ideas from pure OO languages are starting to fail me. Can anyone please give me some pointers as to how approach this problem in Scala.
For example: Do I create simple objects that just wrap or extend the table objects, and then include these (composition) into another class that manages them?
Any ideas, guidance, example (please), that will help me better approach this problem as a designer and coder will be greatly appreciated.
The best idea would be to implement something like data mapper pattern. Which, in contrast to active record, will not violate SRP.
Since I am not a Scala developer, I will not show any code.
The idea is following:
create domain object instance
set conditions on the element (for example setId(42), if you are looking for element by ID)
create data mapper instance
execute fetch() method on the mapper by passing in domain object as parameter
The mapper would look up current parameters of provided domain object and, based on those parameters, retrieve information from storage (which might be SQL database, or JSON file or maybe a remote REST API). If information is retrieved, it assigns the values to the domain object.
Also, I must note, that data mappers are created for work with specific domain object's interface, but the information, which they pass from domain object to storage and back, can be mapped to multiple SQL tables or multiple REST resources.
This way you can easily replace the mapper, when you switch to different storage medium, or even unit-test the logic in domain objects without touching the real storage. Also, if you decide to add caching at some point, that would be just another mapper, which tried to fetch information from cache, and, if it fails, the mapper for persistent storage kicks in.
Domain object (or, in some cases, a collection of domain objects) would be completely unaware of whether it is stored or retrieved. That would be the responsibility of the data mappers.
If this is all in MVC context, then, to fully implement this, you would need another group of structures in the model layer. I call them "services" (please share, of you come up with better name). They are responsible for containing the interaction between data mappers and domain objects. This way you can prevent the business logic from leaking in the presentation layer (controllers, to be exact), and these services create a natural interface for interaction between business (also know as model) layer and the presentation layer.
P.S. Once again, sorry that I cannot provide any code examples, because I am a PHP developer and have no idea how to write code in Scala.
P.P.S. If you are using data mapper pattern, the best option is to write mappers manually and not use any 3rd party ORM, which claims to implement it. It would give you more control over codebase and avoid pointless technical debt [1] [2].
A good solution for simple persistence requirements is the ActiveRecord pattern: http://en.wikipedia.org/wiki/Active_record_pattern . This is implemented in Ruby and in Play! framework 1.2, and you can easily implement it in Scala in a stand-alone application
The only requirement is to have a singleton DB or a singleton service to get a reference to the DB you require. I personally would go for an implementation based on the following:
A generic trait ActiveRecord
A generic typeclass ActiveRecordHandler
Exploiting the power of implicits, you could obtain an amazing syntax:
trait ActiveRecordHandler[T]{
def save(t:T):T
def delete[A<:Serializable](primaryKey:A):Option[T]
def find(query:String):Traversable[T]
}
object ActiveRecordHandler {
// Note that an implicit val inside an object with the same name as the trait
// is one of the way to have the implicit in scope.
implicit val myClassHandler = new ActiveRecordHandler[MyClass] {
def save(myClass:MyClass) = myClass
def delete[A <: Serializable](primaryKey: A) = None
def find(query: String) = List(MyClass("hello"),MyClass("goodbye"))
}
}
trait ActiveRecord[RecordType] {
self:RecordType=>
def save(implicit activeRecordHandler:ActiveRecordHandler[RecordType]):RecordType = activeRecordHandler.save(this)
def delete[A<:Serializable](primaryKey:A)(implicit activeRecordHandler:ActiveRecordHandler[RecordType]):Option[RecordType] = activeRecordHandler.delete(primaryKey)
}
case class MyClass(name:String) extends ActiveRecord[MyClass]
object MyClass {
def main(args:Array[String]) = {
MyClass("10").save
}
}
With such a solution, you only need your class to extends ActiveRecord[T] and have an implicit ActiveRecordHandler[T] to handle this.
There is actually also an implementation: https://github.com/aselab/scala-activerecord which is based on similar idea, but instead of making the ActiveRecord having an abstract type, it declares a generic companion object.
A general but very important comment on the ActiveRecord pattern is that it helps meet simple requirements in terms of persistence, but cannot deal with more complex requirements: for example is when you want to persist multiple objects under the same transaction.
If your application requires more complex persistence logic, the best approach is to introduce a persistence service which exposes only a limited set of functions to the client classes, for example
def persist(objectsofTypeA:Traversable[A],objectsOfTypeB:Traversable[B])
Please also note that according to your application complexity, you might want to expose this logic in different fashions:
as a singleton object in the case your application is simple, and you do not want your persistence logic to be pluggable
through a singleton object which acts as a sort as a "application context", so that in your application at startup you can decide which persistence logic you want to use.
with some sort of lookup service pattern, if your application is distributed.

Play 2.0 models best practices

I am looking for best practices regarding models and ways to persist objects in database with play 2.0. I have studied the Play and typesafe samples for play 2.0 using scala.
What I understand is :
The model is defined in a case class
All the insert/update/delete/select are defined in the companion object of this case class
So if I need to update my Car object to define a new owner i will have to do:
val updatedCar = myCar.copy(owner=newOwner)
Car.update(updatedCar)
// or
Car.updateOwner(myCar.id.get, newOwner)
I am wondering why the update or delete statements are not in the case class itself:
case class Car(id: Pk[Long] = NotAssigned, owner: String) {
def updateOwner(newOwner: String) {
DB.withConnection { implicit connection =>
SQL(
"""
update car
set owner = {newOwner}
where id = {id}
"""
).on(
'id -> id,
'newOwner -> newOwner
).executeUpdate()
}
copy(owner = newOwner)
}
}
Doing so would permit to do:
val updatedCar = myCar.updateOwner(newOwner)
Which is what I used to do with Play 1.X using Java and JPA.
Maybe the reason is obvious and due to my small knowledge of Scala.
I think part of the reason is the favoring of immutability in functional languages like Scala.
In your example, you modify 'this.owner'. What's your equivalent operation look like for a delete, and what happens to "this"?
With a companion object, it seems a bit more clear that the passed object (or ID) is not modified, and the returned object or ID is the relevant result of the operation.
Then also, I think another part of the issue is that your example requires an instance first. When you delete an Object, what if you just want to delete by Id you got off a form, and don't want to first build a whole instance of the object you intend to delete?
I've been playing with play2.0 with mongo, and my companion objects look like:
object MyObject extends SalatDAO[MyObject,ObjectId] (collection = getCollection("objectcollection")) {
}
These companion objects inherit CRUD like operations from SalatDAO (MyObject.save(), MyObject.find(), etc). I'm not entirely clear on how it is implemented internally, but it works nicely.

Serialize Function1 to database

I know it's not directly possible to serialize a function/anonymous class to the database but what are the alternatives? Do you know any useful approach to this?
To present my situation: I want to award a user "badges" based on his scores. So I have different types of badges that can be easily defined by extending this class:
class BadgeType(id:Long, name:String, detector:Function1[List[UserScore],Boolean])
The detector member is a function that walks the list of scores and return true if the User qualifies for a badge of this type.
The problem is that each time I want to add/edit/modify a badge type I need to edit the source code, recompile the whole thing and re-deploy the server. It would be much more useful if I could persist all BadgeType instances to a database. But how to do that?
The only thing that comes to mind is to have the body of the function as a script (ex: Groovy) that is evaluated at runtime.
Another approach (that does not involve a database) might be to have each badge type into a jar that I can somehow hot-deploy at runtime, which I guess is how a plugin-system might work.
What do you think?
My very brief advice is that if you want this to be truly data-driven, you need to implement a rules DSL and an interpreter. The rules are what get saved to the database, and the interpreter takes a rule instance and evaluates it against some context.
But that's overkill most of the time. You're better off having a little snippet of actual Scala code that implements the rule for each badge, give them unique IDs, then store the IDs in the database.
e.g.:
trait BadgeEval extends Function1[User,Boolean] {
def badgeId: Int
}
object Badge1234 extends BadgeEval {
def badgeId = 1234
def apply(user: User) = {
user.isSufficientlyAwesome // && ...
}
}
You can either have a big whitelist of BadgeEval instances:
val weDontNeedNoStinkingBadges = Map(
1234 -> Badge1234,
5678 -> Badge5678,
// ...
}
def evaluator(id: Int): Option[BadgeEval] = weDontNeedNoStinkingBadges.get(id)
def doesUserGetBadge(user: User, id: Int) = evaluator(id).map(_(user)).getOrElse(false)
... or if you want to keep them decoupled, use reflection:
def badgeEvalClass(id: Int) = Class.forName("com.example.badge.Badge" + id + "$").asInstanceOf[Class[BadgeEval]]
... and if you're interested in runtime pluggability, try the service provider pattern.
You can try and use Scala Continuations - they can give you the ability to serialize the computation and run it at later time or even on another machine.
Some links:
Continuations
What are Scala continuations and why use them?
Swarm - Concurrency with Scala Continuations
Serialization relates to data rather than methods. You cannot serialize functionality because it is a class file which is designed to serialize that and object serialization serializes the fields of an object.
So like Alex says, you need a rule engine.
Try this one if you want something fairly simple, which is string based, so you can serialize the rules as strings in a database or file:
http://blog.maxant.co.uk/pebble/2011/11/12/1321129560000.html
Using a DSL has the same problems unless you interpret or compile the code at runtime.