My first steps with ZIO, I am trying to convert this readfile function with a compatible ZIO version.
The below snippet compiles, but I am not closing the source in the ZIO version. How do I do that?
def run(args: List[String]) =
myAppLogic.exitCode
val myAppLogic =
for {
_ <- readFileZio("C:\\path\\to\file.csv")
_ <- putStrLn("Hello! What is your name?")
name <- getStrLn
_ <- putStrLn(s"Hello, ${name}, welcome to ZIO!")
} yield ()
def readfile(file: String): String = {
val source = scala.io.Source.fromFile(file)
try source.getLines.mkString finally source.close()
}
def readFileZio(file: String): zio.Task[String] = {
val source = scala.io.Source.fromFile(file)
ZIO.fromTry[String]{
Try{source.getLines.mkString}
}
}
The simplest solution for your problem would be using bracket function, which in essence has similar purpose as try-finally block. It gets as first argument effect that closes resource (in your case Source) and as second effect that uses it.
So you could rewrite readFileZio like:
def readFileZio(file: String): Task[Iterator[String]] =
ZIO(Source.fromFile(file))
.bracket(
s => URIO(s.close),
s => ZIO(s.getLines())
)
Another option is to use ZManaged which is a data type that encapsulates the operation of opening and closing resource:
def managedSource(file: String): ZManaged[Any, Throwable, BufferedSource] =
Managed.make(ZIO(Source.fromFile(file)))(s => URIO(s.close))
And then you could use it like this:
def readFileZioManaged(file: String): Task[Iterator[String]] =
managedSource(file).use(s => ZIO(s.getLines()))
Related
I wrote simple callback(handler) function which i pass to async api and i want to wait for result:
object Handlers {
val logger: Logger = Logger("Handlers")
implicit val cs: ContextShift[IO] =
IO.contextShift(ExecutionContext.Implicits.global)
class DefaultHandler[A] {
val response: IO[MVar[IO, A]] = MVar.empty[IO, A]
def onResult(obj: Any): Unit = {
obj match {
case obj: A =>
println(response.flatMap(_.tryPut(obj)).unsafeRunSync())
println(response.flatMap(_.isEmpty).unsafeRunSync())
case _ => logger.error("Wrong expected type")
}
}
def getResponse: A = {
response.flatMap(_.take).unsafeRunSync()
}
}
But for some reason both tryPut and isEmpty(when i'd manually call onResult method) returns true, therefore when i calling getResponse it sleeps forever.
This is the my test:
class HandlersTest extends FunSuite {
test("DefaultHandler.test") {
val handler = new DefaultHandler[Int]
handler.onResult(3)
val response = handler.getResponse
assert(response != 0)
}
}
Can somebody explain why tryPut returns true, but nothing puts. And what is the right way to use Mvar/channels in scala?
IO[X] means that you have the recipe to create some X. So on your example, yuo are putting in one MVar and then asking in another.
Here is how I would do it.
object Handlers {
trait DefaultHandler[A] {
def onResult(obj: Any): IO[Unit]
def getResponse: IO[A]
}
object DefaultHandler {
def apply[A : ClassTag]: IO[DefaultHandler[A]] =
MVar.empty[IO, A].map { response =>
new DefaultHandler[A] {
override def onResult(obj: Any): IO[Unit] = obj match {
case obj: A =>
for {
r1 <- response.tryPut(obj)
_ <- IO(println(r1))
r2 <- response.isEmpty
_ <- IO(println(r2))
} yield ()
case _ =>
IO(logger.error("Wrong expected type"))
}
override def getResponse: IO[A] =
response.take
}
}
}
}
The "unsafe" is sort of a hint, but every time you call unsafeRunSync, you should basically think of it as an entire new universe. Before you make the call, you can only describe instructions for what will happen, you can't actually change anything. During the call is when all the changes occur. Once the call completes, that universe is destroyed, and you can read the result but no longer change anything. What happens in one unsafeRunSync universe doesn't affect another.
You need to call it exactly once in your test code. That means your test code needs to look something like:
val test = for {
handler <- TestHandler.DefaultHandler[Int]
_ <- handler.onResult(3)
response <- handler.getResponse
} yield response
assert test.unsafeRunSync() == 3
Note this doesn't really buy you much over just using the MVar directly. I think you're trying to mix side effects inside IO and outside it, but that doesn't work. All the side effects need to be inside.
I wanted to handle some exceptions in ZIO using catchAll or catchSome as the below :
object Test extends App {
def run(args: List[String]) =
myApp.fold(_ => 1, _ => 0)
val myApp =
for {
_ <- putStrLn(unsafeRun(toINT("3")).toString)
} yield ()
def toINT(s: String): IO[IOException, Int]= {
IO.succeed(s.toInt).map(v => v).catchAll(er =>IO.fail(er))
}
the code succeeded in case I passed a valid format number but it's unable to handle the exception in case I passed invalid format and idea ??
s.toInt gets evaluated outside of the IO monad. What happens is that you evaluate s.toInt first and try to pass the result of that to IO.succeed, but an exception has already been thrown before you can pass anything to IO.succeed. The name of succeed already basically says that you are sure that whatever you pass it is a plain value that cannot fail.
The docs suggest using Task.effect, IO.effect, or ZIO.effect for lifting an effect that can fail into ZIO.
Here is a program that worked for me:
val program =
for {
int <- toINT("3xyz")
_ <- putStrLn(int.toString)
} yield ()
def toINT(s: String): Task[Int] = {
ZIO.fromTry(Try(s.toInt))
}
rt.unsafeRun(program.catchAll(t => putStrLn(t.getMessage)))
def map[U: ClassTag](f: T => U): RDD[U] = withScope {
val cleanF = sc.clean(f)
new MapPartitionsRDD[U, T](this, (context, pid, iter) => iter.map(cleanF))
}
This code snippet is from spark 2.2 source code. I am not professional in scala, so just wondering can anyone explain this code in a programmatic perspective? I am not sure what the square bracket after map does. And also refer to https://www.tutorialspoint.com/scala/scala_functions.htm, a scala function should just have curly bracket after '=', but why in this code snippet there is a function named withScope after the '=' sign?
actually, a scala function can have no bracket after "=". eg.
def func():Int = 1
so you can think withScope{} is a function with return type RDD[U]，and the map function is to run the withScope function.
let's see the withScope's source code：
private[spark] def withScope[U](body: => U): U = RDDOperationScope.withScope[U](sc)(body)
see, it's a function here. let's go on:
private[spark] def withScope[T](
sc: SparkContext,
allowNesting: Boolean = false)(body: => T): T = {
val ourMethodName = "withScope"
val callerMethodName = Thread.currentThread.getStackTrace()
.dropWhile(_.getMethodName != ourMethodName)
.find(_.getMethodName != ourMethodName)
.map(_.getMethodName)
.getOrElse {
// Log a warning just in case, but this should almost certainly never happen
logWarning("No valid method name for this RDD operation scope!")
"N/A"
}
withScope[T](sc, callerMethodName, allowNesting, ignoreParent = false)(body)
}
let continue with the withScope at the end:
private[spark] def withScope[T](
sc: SparkContext,
name: String,
allowNesting: Boolean,
ignoreParent: Boolean)(body: => T): T = {
// Save the old scope to restore it later
val scopeKey = SparkContext.RDD_SCOPE_KEY
val noOverrideKey = SparkContext.RDD_SCOPE_NO_OVERRIDE_KEY
val oldScopeJson = sc.getLocalProperty(scopeKey)
val oldScope = Option(oldScopeJson).map(RDDOperationScope.fromJson)
val oldNoOverride = sc.getLocalProperty(noOverrideKey)
try {
if (ignoreParent) {
// Ignore all parent settings and scopes and start afresh with our own root scope
sc.setLocalProperty(scopeKey, new RDDOperationScope(name).toJson)
} else if (sc.getLocalProperty(noOverrideKey) == null) {
// Otherwise, set the scope only if the higher level caller allows us to do so
sc.setLocalProperty(scopeKey, new RDDOperationScope(name, oldScope).toJson)
}
// Optionally disallow the child body to override our scope
if (!allowNesting) {
sc.setLocalProperty(noOverrideKey, "true")
}
body
} finally {
// Remember to restore any state that was modified before exiting
sc.setLocalProperty(scopeKey, oldScopeJson)
sc.setLocalProperty(noOverrideKey, oldNoOverride)
}
}
in the end, it execute body parameter, int this case, body equals to
{
val cleanF = sc.clean(f)
new MapPartitionsRDD[U, T](this, (context, pid, iter) => iter.map(cleanF))
}
in conclusion, withScope is a Closure takes a function as argument, it first runs some code itself and the run the argument.
On the first line below, I want to know if there's anything I can or should put to the left of the = that will future-proof against bad editing. I had one Action.async, but now I want to have two choices (run a single MongoDB query or run multiple MongoDB queries) to arrive at the answer to the web query q. My code compiles as-is, but I think I should have to put something before the = to make the code more type-safe.
def q(arg: String) =
if (wantMultipleQueriesByDataSource)
runMultipleQueriesByDataSource(arg)
else
runSingleQuery(arg)
def runSingleQuery(arg: String) = Action.async {
if (oDb.isDefined) {
val collection: MongoCollection[Document] = oDb.get.getCollection(collectionName)
val fut = getMapData(collection, arg)
fut.map { docs: Seq[Document] =>
val docsSources = List(DocsSource(docs, "*"))
val pickedDocs = pickDocs(args, docsSources)
Ok(buildQueryAnswer(pickedDocs))
} recover {
case e => BadRequest("FAIL(rect): " + e.getMessage)
}
}
else Future.successful(Ok(buildQueryAnswer(Nil)))
}
def runMultipleQueriesByDataSource(arg: String) = Action.async {
if (oDb.isDefined) {
val collection: MongoCollection[Document] = oDb.get.getCollection(collectionName)
val dataSources = List("apples", "bananas", "cherries")
val futureCollections = dataSources map { getMapDataByDataSource(collection, _, arg) }
Future.sequence(futureCollections) map {
case docsGroupedByDataSource: Seq[Seq[Document]] =>
val docsSources = (docsGroupedByDataSource zip dataSources) map {x => DocsSource(x._1, x._2)}
val pickedDocs = pickDocs(args, docsSources)
Ok(buildQueryAnswer(pickedDocs))
case _ => Ok(buildQueryAnswer(Nil))
} recover {
case e => BadRequest("FAIL(rect): " + e.getMessage)
}
}
else Future.successful(Ok(buildQueryAnswer(Nil)))
}
You don't need to because Scala has type inference in this case. But if you want, you can change the method signature to the following:
def q(arg: String): Action[AnyContent]
I'm assuming here that you are using Playframework.
Edit: In fact, in this case, Action.async return an Action[AnyContent] and it receives a block that returns a Future[Result].
Anybody know a solution to this problem ? I rewrote try catch finally construct to a functional way of doing things, but I can't close the stream now :-)
import scala.util.control.Exception._
def gunzip() = {
logger.info(s"Gunziping file ${f.getAbsolutePath}")
catching(classOf[IOException], classOf[FileNotFoundException]).
andFinally(println("how can I close the stream ?")).
either ({
val is = new GZIPInputStream(new FileInputStream(f))
Stream.continually(is.read()).takeWhile(-1 !=).map(_.toByte).toArray
}) match {
case Left(e) =>
val msg = s"IO error reading file ${f.getAbsolutePath} ! on host ${Setup.smtpHost}"
logger.error(msg, e)
MailClient.send(msg, msg)
new Array[Byte](0)
case Right(v) => v
}
}
I rewrote it based on Senia's solution like this :
def gunzip() = {
logger.info(s"Gunziping file ${file.getAbsolutePath}")
def closeAfterReading(c: InputStream)(f: InputStream => Array[Byte]) = {
catching(classOf[IOException], classOf[FileNotFoundException])
.andFinally(c.close())
.either(f(c)) match {
case Left(e) => {
val msg = s"IO error reading file ${file.getAbsolutePath} ! on host ${Setup.smtpHost}"
logger.error(msg, e)
new Array[Byte](0)
}
case Right(v) => v
}
}
closeAfterReading(new GZIPInputStream(new FileInputStream(file))) { is =>
Stream.continually(is.read()).takeWhile(-1 !=).map(_.toByte).toArray
}
}
I prefer this construction for such cases:
def withCloseable[T <: Closeable, R](t: T)(f: T => R): R = {
allCatch.andFinally{t.close} apply { f(t) }
}
def read(f: File) =
withCloseable(new GZIPInputStream(new FileInputStream(f))) { is =>
Stream.continually(is.read()).takeWhile(-1 !=).map(_.toByte).toArray
}
Now you could wrap it with Try and recover on some exceptions:
val result =
Try { read(f) }.recover{
case e: IOException => recover(e) // logging, default value
case e: FileNotFoundException => recover(e)
}
val array = result.get // Exception here!
take "scala-arm"
take Apache "commons-io"
then do the following
val result =
for {fis <- resource.managed(new FileInputStream(f))
gis <- resource.managed(new GZIPInputStream(fis))}
yield IOUtils.toString(gis, "UTF-8")
result.acquireFor(identity) fold (reportExceptions _, v => v)
One way how to handle it would be to use a mutable list of things that are opened and need to be closed later:
val cs: Buffer[Closeable] = new ArrayBuffer();
def addClose[C <: Closeable](c: C) = { cs += c; c; }
catching(classOf[IOException], classOf[FileNotFoundException]).
andFinally({ cs.foreach(_.close()) }).
either ({
val is = addClose(new GZIPInputStream(new FileInputStream(f)))
Stream.continually(is.read()).takeWhile(-1 !=).map(_.toByte).toArray
}) // ...
Update: You could use scala-conduit library (I'm the author) for this purpose. (The library is currently not considered production ready.) The main aim of pipes (AKA conduids) is to construct composable components with well defined resource handling. Each pipe repeatedly receives input and produces input. Optionally, it also produces a final result when it finishes. Pips has finalizers that are run after a pipe finishes - either on its own or when its downstream pipe finishes. Your example could be reworked (using Java NIO) as follows:
/**
* Filters buffers until a given character is found. The last buffer
* (truncated up to the character) is also included.
*/
def untilPipe(c: Byte): Pipe[ByteBuffer,ByteBuffer,Unit] = ...
// Create a new source that chunks a file as ByteBuffer's.
// (Note that the buffer changes on every step.)
val source: Source[ByteBuffer,Unit] = ...
// Sink that prints bytes to the standard output.
// You would create your own sink doing whatever you want.
val sink: Sink[ByteBuffer,Unit]
= NIO.writeChannel(Channels.newChannel(System.out));
runPipe(source >-> untilPipe(-1) >-> sink);
As soon as untilPipe(-1) finds -1 and finishes, its upstream source pipe's finalizer is run and the input is closed. If an exception occurs anywhere in the pipeline, the input is closed as well.
The full example can be found here.
I have another one proposition for cases when a closeble object like java.io.Socket may run for a long time, so one have to wrap it in Future. This is handful when you also control a timeout, when Socket is not responding.
object CloseableFuture {
type Closeable = {
def close(): Unit
}
private def withClose[T, F1 <: Closeable](f: => F1, andThen: F1 => Future[T]): Future[T] = future(f).flatMap(closeable => {
val internal = andThen(closeable)
internal.onComplete(_ => closeable.close())
internal
})
def apply[T, F1 <: Closeable](f: => F1, andThen: F1 => T): Future[T] =
withClose(f, {c: F1 => future(andThen(c))})
def apply[T, F1 <: Closeable, F2 <: Closeable](f1: => F1, thenF2: F1 => F2, andThen: (F1,F2) => T): Future [T] =
withClose(f1, {c1:F1 => CloseableFuture(thenF2(c1), {c2:F2 => andThen(c1,c2)})})
}
After I open a java.io.Socket and java.io.InputStream for it, and then execute code which reads from WhoisServer, I got both of them closed finally. Full code:
CloseableFuture(
{new Socket(server.address, WhoisPort)},
(s: Socket) => s.getInputStream,
(socket: Socket, inputStream: InputStream) => {
val streamReader = new InputStreamReader(inputStream)
val bufferReader = new BufferedReader(streamReader)
val outputStream = socket.getOutputStream
val writer = new OutputStreamWriter(outputStream)
val bufferWriter = new BufferedWriter(writer)
bufferWriter.write(urlToAsk+System.getProperty("line.separator"))
bufferWriter.flush()
def readBuffer(acc: List[String]): List[String] = bufferReader.readLine() match {
case null => acc
case str => {
readBuffer(str :: acc)
}
}
val result = readBuffer(Nil).reverse.mkString("\r\n")
WhoisResult(urlToAsk, result)
}
)