Background
I have Map[String,String] of configuration values. I want to extract a series of keys and provide meaningful error messages if any of them are missing. For example:
val a = Map("url"->"http://example.com", "user"->"bob", "password"->"12345")
Say I want to transform this into a case class:
case class HttpConnectionParams(url:String, user:String, password: String)
Now, I can simply use a for loop to extract the values:
for(url <- a.get("url");
user <- a.get("user");
password <- a.get("password")) yield {
HttpConnectionParams(url,user,password)
}
To get an Option[HttpConnectionParams]. This is nice and clean, except if I get a None then I don't know what was missing. I'd like to provide that information.
Validation with Scalaz
Enter scalaz. I'm using version 7.1.3.
From what I've been able to put together (a good reference is here) I can use disjunctions:
for(url <- a.get("url") \/> "Url must be supplied";
user <- a.get("user") \/> "Username must be supplied";
password <- a.get("password") \/> "Password must be supplied") yield {
HttpConnectionParams(url,user,password)
}
This is nice because now I get an error message, but this is railway oriented because it stops at the first failure. What if I want to get all of the errors? Let's use validation and the applicative builder (aka "|#|"):
val result = a.get("url").toSuccess("Url must be supplied") |#|
a.get("username").toSuccess("Username must be supplied") |#|
a.get("password").toSuccess("Password must be supplied")
result.tupled match {
case Success((url,user,password)) => HttpConnectionParams(url,user,password)
case Failure(m) => println("There was a failure"+m)
}
Questions
This does what I expect, but I have some questions about the usage:
Is there an easy to use alternative to scalaz for this use-case? I'd prefer to not open pandora's box and introduce scalaz if I don't have to.
One reason I'd like to not use scalaz is that it's really really hard to figure out what to do if you don't, like me, know the entire framework. For example, what is the list of implicits that you need to get the above code to work? import scalaz._ somehow didn't work for me.[1] How can I figure this out from the API docs?
Is there a more succinct way to express the validation use-case? I stumbled my way through until I arrived at something that worked and I have no idea if there are other, better ways of doing the same thing in scalaz.
[1] After much consternation I arrived at this set of imports for the applicative use-case. Hopefully this helps somebody:
import scalaz.std.string._
import scalaz.syntax.std.option._
import scalaz.syntax.apply._
import scalaz.Success
import scalaz.Failure
You can do this a little more nicely by defining a helper method and skipping the .tupled step by using .apply:
import scalaz._, Scalaz._
def lookup[K, V](m: Map[K, V], k: K, message: String): ValidationNel[String, V] =
m.get(k).toSuccess(NonEmptyList(message))
val validated: ValidationNel[String, HttpConnectionParams] = (
lookup(a, "url", "Url must be supplied") |#|
lookup(a, "username", "Username must be supplied") |#|
lookup(a, "password", "Password must be supplied")
)(HttpConnectionParams.apply)
Also, please don't be ashamed to use import scalaz._, Scalaz._. We all do it and it's just fine in the vast majority of cases. You can always go back and refine your imports later. I also still stand by this answer I wrote years ago—you shouldn't feel like you need to have a comprehensive understanding of Scalaz (or cats) in order to be able to use pieces of it effectively.
Related
Using cats.Semigroup one can write this:
import cats.Semigroup
import cats.implicits._
val l1: String Either Int = Left("error")
val r1: String Either Int = Right(1)
val r2: String Either Int = Right(2)
l1 |+| r1 // Left("error")
r1 |+| r2 // Right(3)
I would like to have an equally idiomatic operator (combine-like) that works like this:
if there is (at least) one Right in my computation, return a Right
if there are only Lefts, return a Left
E.g.:
Right(1) |+| Right(2) // Right(3)
Right(1) |+| Left("2") // Right(1)
Left("1") |+| Left("2") // Left("12") // in my particular case the wrapped value here does not really matter (could also be e.g. Left("1") or Left("2")), but I guess Left("12") would be the must logical result
Is there something like this already defined in e.g. cats on Either?
There are a bunch of lawful semigroup instances for Either, and which of them should be included in Cats was a matter of some debate. Cats, Scalaz, and Haskell all make different choices in this respect, and the instance you're describing (flipped but with both lefts and right combining) is different from all three of those, it doesn't have a specific name that I'm aware of, and it isn't provided under any name or in any form by Cats.
That's of course not a problem in itself, since as we'll see below it's pretty easy to verify that this instance is lawful, but there is one potential issue you should be aware of. You don't really explain your intended semantics, but if you ever want to promote this to a Monoid, the fact that you pick the Right when you have both a Left and a Right means that your zero will have to be Left. This might be kind of weird if you're thinking of rights as successes and lefts as errors that are safe to ignore when combining values.
You're asking about Semigroup, though, not Monoid, so let's just ignore that for now and show that this thing is lawful. First for the definition:
import cats.kernel.Semigroup
implicit def eitherSemigroup[A, B](implicit
A: Semigroup[A],
B: Semigroup[B]
): Semigroup[Either[A, B]] = Semigroup.instance {
case (Right(x), Right(y)) => Right(B.combine(x, y))
case (r # Right(_), Left(_)) => r
case (Left(_), r # Right(_)) => r
case (Left(x), Left(y)) => Left(A.combine(x, y))
}
And then the checking part:
import cats.instances.int._
import cats.instances.string._
import cats.kernel.instances.either.catsStdEqForEither
import cats.kernel.laws.discipline.SemigroupTests
import org.scalacheck.Test.Parameters
SemigroupTests(eitherSemigroup[String, Int]).semigroup.all.check(Parameters.default)
And yeah, it's fine:
+ semigroup.associative: OK, passed 100 tests.
+ semigroup.combineAllOption: OK, passed 100 tests.
+ semigroup.repeat1: OK, passed 100 tests.
+ semigroup.repeat2: OK, passed 100 tests.
Personally if I wanted something like this I'd probably use a wrapper to avoid confusing future readers of my code (including myself), but given that nobody really knows what the semigroup of Either should do, I don't think using a custom instance is as big of a problem as it is for most other types from the standard library.
I would like to run several queries in one transaction using a for-comprehension in doobie. Something like:
def addImage(path:String) : ConnectionIO[Image] = {
sql"INSERT INTO images(path) VALUES($path)".update.withUniqueGeneratedKeys('id', 'path')
}
def addUser(username: String, imageId: Optional[Int]) : ConnectionIO[User] = {
sql"INSERT INTO users(username, image_id) VALUES($username, $imageId)".update.withUniqueGeneratedKeys('id', 'username', 'image_id')
}
def createUser(username: String, imagePath: Optional[String]) : Future[User] = {
val composedIO : ConnectionIO[User] = for {
optImage <- imagePath.map { p => addImage(p) }
user <- addUser(username, optImage.map(_.id))
} yield user
composedIO.transact(xa).unsafeToFuture
}
I just started with doobie (and cats) so I'm not that familiar with FreeMonads. I've been trying different solutions but for the for-comprehension to work it looks like both blocks needs to return a cats.free.Free[doobie.free.connection.ConnectionOp,?].
If this is true, is there a way to transform my ConnectionIO[Image] (from the addImage call) into a cats.free.Free[doobie.free.connection.ConnectionOp,Option[Image]] ?
For your direct question, ConnectionIO is defined as type ConnectionIO[A] = Free[ConnectionOp, A], i.e. the two types are equivalent (no transformation required).
Your issue is different, and can be easily seen if we step through the code step by step. For simplicity, I will use Option where you used Optional.
imagePath.map { p => addImage(p) }:
imagePath is an Option, and map uses an A => B to convert Option[A] to Option[B].
Since addImage returns a ConnectionIO[Image], we now have an Option[ConnectionIO[Image]], i.e. this is an Option program, not a ConnectionIO program.
We can instead return a ConnectionIO[Option[Image]] by replacing map with traverse, which uses the Traverse typeclass, see https://typelevel.org/cats/typeclasses/traverse.html for some details on how this works. But a basic intuition is that where map would have given you an F[G[B]], traverse instead gives you a G[F[B]]. In a sense, it works similarly to Future.traverse from the standard library, but in a more general way.
addUser(username, optImage.map(_.id))
The issue here is that given optImage which is an Option[Image], and its id field, which is an Option[Int], the result of optImage.map(_.id) is an Option[Option[Int]], not the Option[Int] which your method expects.
One way of solving this (if it matches your requirements), is to change this part of code to
addUser(username, optImage.flatMap(_.id))
flatMap can "join" an Option with another created by its value (if it exists).
(note: you need to add import cats.implicits._ to get the syntax for traverse).
In general, some of the ideas here about Traverse, flatMap, etc., are useful to study, and two books for doing so are "Scala With Cats" (https://underscore.io/books/scala-with-cats/) and "Functional Programming with Scala" (https://www.manning.com/books/functional-programming-in-scala)
The author of doobie also recently gave a talk about "effects", which may be of use in improving your intuition about types like Option, IO, etc.: https://www.youtube.com/watch?v=po3wmq4S15A
If I got your intention right, you should use traverse instead of map:
val composedIO : ConnectionIO[User] = for {
optImage <- imagePath.traverse { p => addImage(p) }
user <- addUser(username, optImage.map(_.id))
} yield user
You might need to import cats.instances.option._ and/or cats.syntax.traverse._
In https://gist.github.com/satyagraha/897e427bfb5ed203e9d3054ac6705704 I have posted a Scala Cats validation scenario which seems reasonable, but I haven't found a very neat solution.
Essentially, there is a two-stage validation, where individual fields are validated, then a class constructor is called which may throw due to internal checks (in general this may not be under my control to change, hence the exception handling code). We wish to not to call the constructor if any field validation fails, but also combine any constructor failure into the final result. "Fail-fast" is definitely right here for the two-phase check.
This is a kind of flatMap problem, which the cats.data.Validated framework appears to handle via the cats.data.Validated#andThenoperation. However I couldn't find a particularly neat solution to the problem as you can see in the code. There are quite a limited number of operations available on a cats.syntax.CartesianBuilder and is wasn't clear to me how to link it with the andThen operation.
Any ideas welcome! Note there is a Cats issue https://github.com/typelevel/cats/issues/1343 which possibly is related, not sure.
For fail fast, chained validation it is easier to use Either than Validated. You can easily switch from Either to Validated or vice versa depending if you want error accumulation.
A possible solution to your problem would be to create a smart constructor for User which returns an Either[Message, User] and use this with Validated[Message, (Name, Date)].
import cats.implicits._
import cats.data.Validated
def user(name: Name, date: Date): Either[Message, User] =
Either.catchNonFatal(User(name, date)).leftMap(Message.toMessage)
// error accumulation -> Validated
val valids: Validated[Message, (Name, Date)] =
(validateName(nameRepr) |#| validateDate(dateDepr)).tupled
// error short circuiting -> either
val userOrMessage: Either[Message, User] =
valids.toEither.flatMap((user _).tupled)
// Either[Message,User] = Right(User(Name(joe),Date(now)))
I would make a helper second-order function to wrap the exception-throwing ones:
def attempt[A, B](f: A => B): A => Validated[Message, B] = a => tryNonFatal(f(a))
Also, default companions of case classes extend the FunctionN trait, so there's no need to do (User.apply _).tupled, it can be shortened to User.tupled (on custom companions, you need to write extends ((...) => ...)) but apply override will be autogenerated)
So we end up with that using andThen:
val valids = validateName(nameRepr) |#| validateDate(dateDepr)
val res: Validated[Message, User] = valids.tupled andThen attempt(User.tupled)
Applicative functors are often mentioned as an alternative to monads when your computation steps are independent. One of their often-mentioned advantages is that you don't need transformers when you want to stack applicatives, because F[G[X]] is always also an applicative.
Let's say I have following functions:
def getDataOption(): Option[Data]
def getUserFuture(): Future[User]
def process(data: Data, user: User)
I would like to have elegant stacking in order to get a Future[Option[User]] and Future[Option[Data]] and map that with process.
So far I only came up with this (using Cats):
Applicative[Future]
.compose[Option]
.map2(
Applicative[Future].pure(getDataOption()),
getUserFuture().map(Applicative[Option].pure))(process)
but I'm sure it's far from ideal. Is there a more elegant and generic way to achieve the same?
The most difficult thing is type inference here. This is the best I could do
// for the Applicative[Future[Option[?]]
import cats.Applicative
implicit val fo = {
import cats.std.future._
import cats.std.option._
Applicative[Future].compose[Option]
}
// for the |#| syntax
import cats.syntax.cartesian._
// to guide type inference
type FutureOption[A] = Future[Option[A]]
((Future(getDataOption): FutureOption[Data]) |#|
getUserFuture.map(Option.apply)).map(process _)
How do you use scalaz.WriterT for logging?
About monad transformers
This is a very short introduction. You may find more information on haskellwiki or this great slide by #jrwest.
Monads don't compose, meaning that if you have a monad A[_] and a monad B[_], then A[B[_]] can not be derived automatically. However in most cases this can be achieved by having a so-called monad transformer for a given monad.
If we have monad transformer BT for monad B, then we can compose a new monad A[B[_]] for any monad A. That's right, by using BT, we can put the B inside A.
Monad transformer usage in scalaz
The following assumes scalaz 7, since frankly I didn't use monad transformers with scalaz 6.
A monad transformer MT takes two type parameters, the first is the wrapper (outside) monad, the second is the actual data type at the bottom of the monad stack. Note: It may take more type parameters, but those are not related to the transformer-ness, but rather specific for that given monad (like the logged type of a Writer, or the error type of a Validation).
So if we have a List[Option[A]] which we would like to treat as a single composed monad, then we need OptionT[List, A]. If we have Option[List[A]], we need ListT[Option, A].
How to get there? If we have the non-transformer value, we can usually just wrap it with MT.apply to get the value inside the transformer. To get back from the transformed form to normal, we usually call .run on the transformed value.
So val a: OptionT[List, Int] = OptionT[List, Int](List(some(1)) and val b: List[Option[Int]] = a.run are the same data, just the representation is different.
It was suggested by Tony Morris that is best to go into the transformed version as early as possible and use that as long as possible.
Note: Composing multiple monads using transformers yields a transformer stack with types just the opposite order as the normal data type. So a normal List[Option[Validation[E, A]]] would look something like type ListOptionValidation[+E, +A] = ValidationT[({type l[+a] = OptionT[List, a]})#l, E, A]
Update: As of scalaz 7.0.0-M2, Validation is (correctly) not a Monad and so ValidationT doesn't exist. Use EitherT instead.
Using WriterT for logging
Based on your need, you can use the WriterT without any particular outer monad (in this case in the background it will use the Id monad which doesn't do anything), or can put the logging inside a monad, or put a monad inside the logging.
First case, simple logging
import scalaz.{Writer}
import scalaz.std.list.listMonoid
import scalaz._
def calc1 = Writer(List("doing calc"), 11)
def calc2 = Writer(List("doing other"), 22)
val r = for {
a <- calc1
b <- calc2
} yield {
a + b
}
r.run should be_== (List("doing calc", "doing other"), 33)
We import the listMonoid instance, since it also provides the Semigroup[List] instance. It is needed since WriterT needs the log type to be a semigroup in order to be able to combine the log values.
Second case, logging inside a monad
Here we chose the Option monad for simplicity.
import scalaz.{Writer, WriterT}
import scalaz.std.list.listMonoid
import scalaz.std.option.optionInstance
import scalaz.syntax.pointed._
def calc1 = WriterT((List("doing calc") -> 11).point[Option])
def calc2 = WriterT((List("doing other") -> 22).point[Option])
val r = for {
a <- calc1
b <- calc2
} yield {
a + b
}
r.run should be_== (Some(List("doing calc", "doing other"), 33))
With this approach, since the logging is inside the Option monad, if any of the bound options is None, we would just get a None result without any logs.
Note: x.point[Option] is the same in effect as Some(x), but may help to generalize the code better. Not lethal just did it that way for now.
Third option, logging outside of a monad
import scalaz.{Writer, OptionT}
import scalaz.std.list.listMonoid
import scalaz.std.option.optionInstance
import scalaz.syntax.pointed._
type Logger[+A] = WriterT[scalaz.Id.Id, List[String], A]
def calc1 = OptionT[Logger, Int](Writer(List("doing calc"), Some(11): Option[Int]))
def calc2 = OptionT[Logger, Int](Writer(List("doing other"), None: Option[Int]))
val r = for {
a <- calc1
b <- calc2
} yield {
a + b
}
r.run.run should be_== (List("doing calc", "doing other") -> None)
Here we use OptionT to put the Option monad inside the Writer. One of the calculations is Noneto show that even in this case logs are preserved.
Final remarks
In these examples List[String] was used as the log type. However using String is hardly ever the best way, just some convention forced on us by logging frameworks. It would be better to define a custom log ADT for example, and if needed to output, convert it to string as late as possible. This way you could serialize the log's ADT and easily analyse it later programmatically (instead of parsing strings).
WriterT has a host of useful methods to work with to ease logging, check out the source. For example given a w: WriterT[...], you may add a new log entry using w :++> List("other event"), or even log using the currently held value using w :++>> ((v) => List("the result is " + v)), etc.
There are many explicit and longish code (types, calls) in the examples. As always, these are for clarity, refactor them in your code by extracting common types and ops.
type OptionLogger[A] = WriterT[Option, NonEmptyList[String], A]
val two: OptionLogger[Int] = WriterT.put(2.some)("The number two".pure[NonEmptyList])
val hundred: OptionLogger[Int] = WriterT.put(100.some)("One hundred".pure[NonEmptyList])
val twoHundred = for {
a <- two
b <- hundred
} yield a * b
twoHundred.value must be equalTo(200.some)
val log = twoHundred.written map { _.list } getOrElse List() mkString(" ")
log must be equalTo("The number two One hundred")