I'm working on a Scala project using cats library, mainly. In there, we have calls like
for {
_ <- initSomeServiceAndLog("something from a far away service")
_ <- initSomeOtherServiceAndLog("something from another far away service")
a <- b()
c <- d(a)
} yield c
Imagine that b also logs something or might throw a business error (I know, we avoid to throw in Scala, but it's not the case right now). I'm looking for a solution to accumulate logs and print them all in the end, in a single message.
For a happy path, I saw that Writer Monad from Cats might be an acceptable solution.
But what if b method throws? The requirements are to logs everything - all previous logs and the error message, in a single message, with some kind of unique trace ID.
Any thoughts? Thanks in advance
Implementing functional logging (in a way that preserves logs even if error happened) using monad transformers like Writer (WriterT) or State (StateT) is hard. However, if we don't be anal about FP approach we could do the following:
use some IO monad
with it create something like in-memory storage for logs
however implement in in a functional way
Personally I would pick either cats.effect.concurrent.Ref or monix.eval.TaskLocal.
Example using Ref (and Task):
type Log = Ref[Task, Chain[String]]
type FunctionalLogger = String => Task[Unit]
val createLog: Task[Log] = Ref.of[Task, Chain[String]](Chain.empty)
def createAppender(log: Log): FunctionalLogger =
entry => log.update(chain => chain.append(entry))
def outputLog(log: Log): Task[Chain[String]] = log.get
with helpers like that I could:
def doOperations(logger: FunctionalLogger) = for {
_ <- operation1(logger) // logging is a side effect managed by IO monad
_ <- operation2(logger) // so it is referentially transparent
} yield result
createLog.flatMap { log =>
doOperations(createAppender(log))
.recoverWith(...)
.flatMap { result =>
outputLog(log)
...
}
}
However, making sure that output is called is a bit of a pain so we could use some form of Bracket or Resource to handle it:
val loggerResource: Resource[Task, FunctionalLogger] = Resource.make {
createLog // acquiring resource - IO operation that accesses something
} { log =>
outputLog(log) // releasing resource - works like finally in try-catchso it should
.flatMap(... /* log entries or sth */) // be called no matter if error occured
}.map(createAppender)
loggerResource.use { logger =>
doSomething(logger)
}
If you don't like passing this appender around explicitly you could use Kleisli to inject it:
type WithLogger[A] = Kleisli[Task, FunctionalLogger, A]
// def operation1: WithLogger[A]
// def operation2: WithLogger[B]
def doSomething: WithLogger[C] = for {
a <- operation1
b <- operation2
} yield c
loggerResource.use { logger =>
doSomething(logger)
}
TaskLocal would be used in a very similar way.
At the end of the day you would end up with:
type that says that it is logging
mutability managed through IO, so referential transparency would not be lost
certainty that even if IO fails, log will be preserved and the results sent
I believe some purist would not like this solution, but it has all the benefits of FP, so I would personally use it.
Related
This might be a really dumb question but I am trying to understand the logic behind using #flatMap and not just #map in this method definition in Finatra's HttpClient definition:
def executeJson[T: Manifest](request: Request, expectedStatus: Status = Status.Ok): Future[T] = {
execute(request) flatMap { httpResponse =>
if (httpResponse.status != expectedStatus) {
Future.exception(new HttpClientException(httpResponse.status, httpResponse.contentString))
} else {
Future(parseMessageBody[T](httpResponse, mapper.reader[T]))
.transformException { e =>
new HttpClientException(httpResponse.status, s"${e.getClass.getName} - ${e.getMessage}")
}
}
}
}
Why create a new Future when I can just use #map and instead have something like:
execute(request) map { httpResponse =>
if (httpResponse.status != expectedStatus) {
throw new HttpClientException(httpResponse.status, httpResponse.contentString)
} else {
try {
FinatraObjectMapper.parseResponseBody[T](httpResponse, mapper.reader[T])
} catch {
case e => throw new HttpClientException(httpResponse.status, s"${e.getClass.getName} - ${e.getMessage}")
}
}
}
Would this be purely a stylistic difference and using Future.exception is just better style in this case, whereas throwing almost looks like a side-effect (in reality it's not, as it doesn't exit the context of a Future) or is there something more behind it, such as order of execution and such?
Tl;dr:
What's the difference between throwing within a Future vs returning a Future.exception?
From a theoretical point of view, if we take away the exceptions part (they cannot be reasoned about using category theory anyway), then those two operations are completely identical as long as your construct of choice (in your case Twitter Future) forms a valid monad.
I don't want to go into length over these concepts, so I'm just going to present the laws directly (using Scala Future):
import scala.concurrent.ExecutionContext.Implicits.global
// Functor identity law
Future(42).map(x => x) == Future(42)
// Monad left-identity law
val f = (x: Int) => Future(x)
Future(42).flatMap(f) == f(42)
// combining those two, since every Monad is also a Functor, we get:
Future(42).map(x => x) == Future(42).flatMap(x => Future(x))
// and if we now generalise identity into any function:
Future(42).map(x => x + 20) == Future(42).flatMap(x => Future(x + 20))
So yes, as you already hinted, those two approaches are identical.
However, there are three comments that I have on this, given that we are including exceptions into the mix:
Be careful - when it comes to throwing exceptions, Scala Future (probably Twitter too) violates the left-identity law on purpose, in order to trade it off for some extra safety.
Example:
import scala.concurrent.ExecutionContext.Implicits.global
def sneakyFuture = {
throw new Exception("boom!")
Future(42)
}
val f1 = Future(42).flatMap(_ => sneakyFuture)
// Future(Failure(java.lang.Exception: boom!))
val f2 = sneakyFuture
// Exception in thread "main" java.lang.Exception: boom!
As #randbw mentioned, throwing exceptions is not idiomatic to FP and it violates principles such as purity of functions and referential transparency of values.
Scala and Twitter Future make it easy for you to just throw an exception - as long as it happens in a Future context, exception will not bubble up, but instead cause that Future to fail. However, that doesn't mean that literally throwing them around in your code should be permitted, because it ruins the structure of your programs (similarly to how GOTO statements do it, or break statements in loops, etc.).
Preferred practice is to always evaluate every code path into a value instead of throwing bombs around, which is why it's better to flatMap into a (failed) Future than to map into some code that throws a bomb.
Keep in mind referential transparency.
If you use map instead of flatMap and someone takes the code from the map and extracts it out into a function, then you're safer if this function returns a Future, otherwise someone might run it outside of Future context.
Example:
import scala.concurrent.ExecutionContext.Implicits.global
Future(42).map(x => {
// this should be done inside a Future
x + 1
})
This is fine. But after completely valid refactoring (which utilizes the rule of referential transparency), your codfe becomes this:
def f(x: Int) = {
// this should be done inside a Future
x + 1
}
Future(42).map(x => f(x))
And you will run into problems if someone calls f directly. It's much safer to wrap the code into a Future and flatMap on it.
Of course, you could argue that even when using flatMap someone could rip out the f from .flatMap(x => Future(f(x)), but it's not that likely. On the other hand, simply extracting the response processing logic into a separate function fits perfectly with the functional programming's idea of composing small functions into bigger ones, and it's likely to happen.
From my understanding of FP, exceptions are not thrown. This would be, as you said, a side-effect. Exceptions are instead values that are handled at some point in the execution of the program.
Cats (and i'm sure other libraries, too) employs this technique too (https://github.com/typelevel/cats/blob/master/core/src/main/scala/cats/ApplicativeError.scala).
Therefore, the flatMap call allows the exception to be contained within a satisfied Future here and handled at a later point in the program's execution where other exception value handling may also occur.
I'm new to parallel programming and ZIO, i'm trying to get data from an API, by parallel requests.
import sttp.client._
import zio.{Task, ZIO}
ZIO.foreach(files) { file =>
getData(file)
Task(file.getName)
}
def getData(file: File) = {
val data: String = readData(file)
val request = basicRequest.body(data).post(uri"$url")
.headers(content -> "text", char -> "utf-8")
.response(asString)
implicit val backend: SttpBackend[Identity, Nothing, NothingT] = HttpURLConnectionBackend()
request.send().body
resquest.Response match {
case Success(value) => {
val src = new PrintWriter(new File(filename))
src.write(value.toString)
src.close()
}
case Failure(exception) => log error
}
when i execute the program sequentially, it work as expected,
if i tried to run parallel, by changing ZIO.foreach to ZIO.foreachPar.
The program is terminating prematurely, i get that, i'm missing something basic here,
any help is appreciated to help me figure out the issue.
Generally speaking I wouldn't recommend mixing synchronous blocking code as you have with asynchronous non-blocking code which is the primary role of ZIO. There are some great talks out there on how to effectively use ZIO with the "world" so to speak.
There are two key points I would make, one ZIO lets you manage resources effectively by attaching allocation and finalization steps and two, "effects" we could say are "things which actually interact with the world" should be wrapped in the tightest scope possible*.
So lets go through this example a bit, first of all, I would not suggest using the default Identity backed backend with ZIO, I would recommend using the AsyncHttpClientZioBackend instead.
import sttp.client._
import zio.{Task, ZIO}
import zio.blocking.effectBlocking
import sttp.client.asynchttpclient.zio.AsyncHttpClientZioBackend
// Extract the common elements of the request
val baseRequest = basicRequest.post(uri"$url")
.headers(content -> "text", char -> "utf-8")
.response(asString)
// Produces a writer which is wrapped in a `Managed` allowing it to be properly
// closed after being used
def managedWriter(filename: String): Managed[IOException, PrintWriter] =
ZManaged.fromAutoCloseable(UIO(new PrintWriter(new File(filename))))
// This returns an effect which produces an `SttpBackend`, thus we flatMap over it
// to extract the backend.
val program = AsyncHttpClientZioBackend().flatMap { implicit backend =>
ZIO.foreachPar(files) { file =>
for {
// Wrap the synchronous reading of data in a `Task`, but which allows runs this effect on a "blocking" threadpool instead of blocking the main one.
data <- effectBlocking(readData(file))
// `send` will return a `Task` because it is using the implicit backend in scope
resp <- baseRequest.body(data).send()
// Build the managed writer, then "use" it to produce an effect, at the end of `use` it will automatically close the writer.
_ <- managedWriter("").use(w => Task(w.write(resp.body.toString)))
} yield ()
}
}
At this point you will just have the program which you will need to run using one of the unsafe methods or if you are using a zio.App through the main method.
* Not always possible or convenient, but it is useful because it prevents resource hogging by yielding tasks back to the runtime for scheduling.
When you use a purely functional IO library like ZIO, you must not call any side-effecting functions (like getData) except when calling factory methods like Task.effect or Task.apply.
ZIO.foreach(files) { file =>
Task {
getData(file)
file.getName
}
}
A colleague of mine stated the following, about using a Java ReentrantReadWriteLock in some Scala code:
Acquiring the lock here is risky. It's "reentrant", but that internally depends on the thread context. F may run different stages of the same computation in different threads. You can easily cause a deadlock.
F here refers to some effectful monad.
Basically what I'm trying to do is to acquire the same reentrant lock twice, within the same monad.
Could somebody clarify why this could be a problem?
The code is split into two files. The outermost one:
val lock: Resource[F, Unit] = for {
// some other resource
_ <- store.writeLock
} yield ()
lock.use { _ =>
for {
// stuff
_ <- EitherT(store.doSomething())
// other stuff
} yield ()
}
Then, in the store:
import java.util.concurrent.locks.{Lock, ReentrantReadWriteLock}
import cats.effect.{Resource, Sync}
private def lockAsResource[F[_]](lock: Lock)(implicit F: Sync[F]): Resource[F, Unit] =
Resource.make {
F.delay(lock.lock())
} { _ =>
F.delay(lock.unlock())
}
private val lock = new ReentrantReadWriteLock
val writeLock: Resource[F, Unit] = lockAsResource(lock.writeLock())
def doSomething(): F[Either[Throwable, Unit]] = writeLock.use { _ =>
// etc etc
}
The writeLock in the two pieces of code is the same, and it's a cats.effect.Resource[F, Unit] wrapping a ReentrantReadWriteLock's writeLock. There are some reasons why I was writing the code this way, so I wouldn't want to dig into that. I would just like to understand why (according to my colleague, at least), this could potentially break stuff.
Also, I'd like to know if there is some alternative in Scala that would allow something like this without the risk for deadlocks.
IIUC your question:
You expect that for each interaction with the Resource lock.lock and lock.unlock actions happen in the same thread.
1) There is no guarantee at all since you are using arbitrary effect F here.
It's possible to write an implementation of F that executes every action in a new thread.
2) Even if we assume that F is IO then the body of doSomething someone could do IO.shift. So the next actions including unlock would happen in another thread. Probably it's not possible with the current signature of doSomething but you get the idea.
Also, I'd like to know if there is some alternative in Scala that would allow something like this without the risk for deadlocks.
You can take a look at scalaz zio STM.
I'm still learning scala so this might be a question with an easy answer, but I've been stuck on writing a single method over and over for almost a day, unable to get this code to compile.
I'm playing with the Play Framework and a reactive mongo template to learn how Scala and Play work.
I have a controller with a few methods, endpoints for a REST service.
The issue is about the following method, which accepts a list of json objects and updates those objects using the mongo reactive driver. The class has one member, citiesFuture which is of type Future[JSONCollection].
The original class code which I'm adding this method to can be found here for context: CityController on github
def updateAll() = Action.async(parse.json) { request =>
Json.fromJson[List[City]](request.body) match {
case JsSuccess(givenCities, _) =>
citiesFuture onComplete[Future[Result]] { cities =>
val updateFutures: List[Future[UpdateWriteResult]] = for {
city <- givenCities
} yield cities.get.update(City.getUniqueQuery(city), Json.obj("$set" -> city))
val promise: Promise[Result] = Promise[Result] {
Future.sequence(updateFutures) onComplete[Result] {
case s#Success(_) =>
var count = 0
for {
updateWriteResult <- s.value
} yield count += updateWriteResult.n
promise success Ok(s"Updated $count cities")
case Failure(_) =>
promise success InternalServerError("Error updating cities")
}
}
promise.future
}
case JsError(errors) =>
Future.successful(BadRequest("Could not build a city from the json provided. " + Errors.show(errors)))
}
}
I've managed to get this far with alot of trial and error, but I'm starting to understand how some of the mechanics of scala and Futures work, I think :) I think I'm close, but my IDE still gives me a single Inspection error just at the single closing curly brace above the line promise.future.
The error reads: Expression of type Unit doesn't conform to expected type Nothing.
I've checked the expected return values for the Promise and onComplete code blocks, but I don't believe they expect Nothing as a return type.
Could somebody please explain to me what I'm missing, and also, I'm sure this can be done better, so let me know if you have any tips I can learn from!
You're kinda on the right track but as #cchantep said, once you're operating in Future-land, it would be very unusual to need to create your own with Promise.future.
In addition, it's actually quite unusual to see onComplete being used - idiomatic Scala generally favors the "higher-level" abstraction of mapping over Futures. I'll attempt to demonstrate how I'd write your function in a Play controller:
Firstly, the "endpoint" just takes care of one thing - interfacing with the outside world - i.e. the JSON-parsing part. If everything converts OK, it calls a private method (performUpdateAll) that actually does the work:
def updateAll() = Action.async(parse.json) { request =>
Json.fromJson[List[City]](request.body) match {
case JsSuccess(givenCities, _) =>
performUpdateAll(givenCities)
case JsError(errors) =>
Future.successful(BadRequest("Could not build a city from the json provided. "))
}
}
Next, we have the private function that performs the update of multiple cities. Again, trying to abide by the Single Responsibility Principle (in a functional sense - one function should do one thing), I've extracted out updateCity which knows how to update exactly one city and returns a Future[UpdateWriteResult]. A nice side-effect of this is code-reuse; you may find you'll be able to use such a function elsewhere.
private def performUpdateAll(givenCities:List[City]):Future[Result] = {
val updateFutures = givenCities.map { city =>
updateCity(city)
}
Future.sequence(updateFutures).map { listOfResults =>
if (listOfResults.forall(_.ok)) {
val count = listOfResults.map(_.n).sum
Ok(s"Updated $count cities")
} else {
InternalServerError("Error updating cities")
}
}
}
As far as I can tell, this will work in exactly the same way as you intended yours to work. But by using Future.map instead of its lower-level counterpart Future.onComplete and matching on Success and Failure you get much more succinct code where (in my opinion) it's much easier to see the intent because there's less boilerplate around it.
We still check that every update worked, with this:
if (listOfResults.forall(_.ok))
which I would argue reads pretty well - all the results have to be OK!
The other little trick I did to tidy up was replace your "counting" logic which used a mutable variable, with a one-liner:
var count = 0
for {
updateWriteResult <- s.value
} yield count += updateWriteResult.n
Becomes:
val count = listOfResults.map(_.n).sum
i.e. convert the list of results to a list of integers (the n in the UpdateWriteResult) and then use the built-in sum function available on lists to do the rest.
Getting stuck with a DAO layer I've created; works fine for the single case, but when needing to persist several bean instances in a transactional block, I find that I have coded myself into a corner. Why? Check out the DAO create method below:
def create(e: Entity): Option[Int] =
db.handle withSession { implicit ss: Session=>
catching( mapper.insert(e) ) option match {
case Some(success) => Some(Query(sequenceID))
case None => None
}
}
Queries that occur within a session block are set to auto commit, so I can't wrap several persistence operations in a transactional block. For example, here's a simplified for comprehension that processes new member subscriptions
val result = for{
u <- user.dao.create(ubean)
m <- member.dao.create(mbean)
o <- order.dao.create(obean)
} yield (u,m,o)
result match {
case Some((a,b,c)) => // all good
case _ => // failed, need to rollback here
}
I could manually perform the queries, but that gets ugly fast
db.handle withSession { implicit ss: Session=>
ss.withTransaction {
val result = for{
u <- safe( UserMapper.insert(ubean) )
...
}
def safe(q: Query[_]) =
catching( q ) option match {
case Some(success) => Some(Query(sequenceID))
case None => None
}
}
}
because I then wind up duplicating error handling, have to supply database, session, etc. all over the application, instead of encapsulating in the DAO layer
Anyone have some sage advice here for how to workaround this problem? I really like the concision of the for comprehension, Scala rocks ;-), ideas appreciated!
OK, unless someone has a better idea, here's what I'm rolling with:
Since DAO to Entity is a 1-to-1 relationship, and ScalaQuery auto commits queries executed in a session block, performing multiple inserts on separate entities is not possible via my DAO implementation.
The workaround is to create a GenericDAO, one not tied to a particular entity, which provides transactional query functionality, with error handling extracted into parent trait
def createMember(...): Boolean = {
db.handle withSession { implicit ss: Session=>
ss.withTransaction {
val result = for{
u <- safeInsert( UserMapper.insert(ubean) )(ss)
...
}
...
At the controller layer the implementation becomes dao.createMember(...) which is quite nice, imo, transactional inserts using safe for comprehension, cool stuff.