Here is code in Scala:
def write() = {
try {
val out = new PrintWriter(new BufferedWriter(new FileWriter(fileName, true)))
out.println("123")
out.close
} catch {
case e: IOException => {}
}
//finally {
//out.close // ops, it's not visible in this context
//}
}
It would be better to have "out.close" in finally block, isn't it? But I don't want to use var.
My question is, how do I achieve that?
A variable defined in a block is local to that block. So if you insist on using try/finally manually you will have to move the val out of the block.
However, what you are trying to achieve is to create a resource, use it in a block, and call a close method on it when leaving the block, no matter whether you leave the block normally or abnormally via an exception. This is an extremely common problem, so there is already a library for it, called Scala ARM. ARM stands for automatic resource management.
Here is the basic usage:
import resource._
for(input <- managed(new FileInputStream("test.txt")) {
// Code that uses the input as a FileInputStream
}
There was some talk of moving this construct to the scala standard library, so in the future you probably won't even need an external dependency.
I would recommend using a library for something like this. It is just one more line in your build.sbt. But for educational purposes, here is how you would roll your own:
def managed[T <: AutoCloseable](resource:T) = new Traversable[T] {
def foreach[U](f:T=>U) {
try {
f(resource)
} finally {
resource.close()
}
}
}
And here is how to use it
scala> for(reader<-managed(new java.io.FileReader("/etc/passwd"))) { println(reader.read()) }
114
scala> for(reader<-managed(new java.io.FileReader("/etc/shadow"))) { println(reader.read()) }
java.io.FileNotFoundException: /etc/shadow (Permission denied)
...
You will still get the exception, but close will be called. Of course if close throws an exception as well you this will hide the original exception. Little details like this are probably handled better in scala ARM.
This is what I use to manage closable resources passed to function returning and not retuning futures
def withClosable[ T, C <: Closeable ]( closable: C )( f: C ⇒ T ) = try { f( closable ) } finally { IOUtils closeQuietly closable }
def withFutureClosable[ T <: Future[Any], C <: Closeable ]( closable: C )( f: C ⇒ T ) = f( closable ) andThen {
case _ => IOUtils closeQuietly closable
}
}
I use IOUtils from commons-io to simplify the call to actually close the resource. A simple try { closable.close() } catch { case _ => /* blah */ } would do
Example usage:
withClosable(new FileInpustream("f")) { stream => /* read the stream */ }
The loan pattern is more usual for this use case, but since anything goes on Stack Overflow, you can construct the expression you're looking for with Try.
Try deserves more exposure as a handy tool.
scala> import util._
import util._
scala> import io._
import io._
Try to open the file --
scala> def f =
| Try (Source.fromFile("foo.text")) map { in =>
then do something with it, packaging the result in a tuple with the i/o source --
note that when you do a value definition in a for-comprehension, this is what it does --
| (in, Try(in.getLines.mkString("/")))
| } flatMap {
then close the source and yield the result of the computation --
| case (in, res) =>
| in.close()
| res
| }
f: scala.util.Try[String]
Uncommented:
scala> def f =
| Try (Source.fromFile("foo.text")) map { in =>
| (in, Try(in.getLines.mkString("/")))
| } flatMap {
| case (in, res) =>
| in.close()
| res
| }
f: scala.util.Try[String]
scala> f
res1: scala.util.Try[String] = Failure(java.io.FileNotFoundException: foo.text (No such file or directory))
Create the test file with some classic humour text, then try again:
scala> f
res2: scala.util.Try[String] = Success(Now is the time/for all good dogs/to lie.)
You can sugarcoat it as a for-comprehension, though observe the extra flatten, since you get a map instead of flatMap from the yield:
scala> def g = (for {
| in <- Try (Source.fromFile("foo.text"))
| res = Try(in.getLines.mkString("/"))
| } yield {
| in.close()
| res
| }).flatten
g: scala.util.Try[String]
scala> g
res2: scala.util.Try[String] = Success(Now is the time/for all good dogs/to lie.)
What if we want to fail if the close fails?
I don't want to type in all that stuff into the REPL again!
scala> :hi // :history
[snip]
2490 def g = (for {
2491 in <- Try (Source.fromFile("foo.text"))
2492 res = Try(in.getLines.mkString("/"))
2493 } yield {
2494 in.close()
2495 res
2496 }).flatten
2497 :hi
scala> :edit 2490+7 // or just :edit 2490-
+import util._
+import io._
+def g = (for {
+ in <- Try (Source.fromFile("foo.text"))
+ res = Try(in.getLines.mkString("/"))
+} yield {
+ val ok = Try(in.close())
+ res transform (s => ok map (_ => s), new Failure(_))
+}).flatten
+
import util._
import io._
g: scala.util.Try[String]
The transform says that if the computation succeeded, convert that success to failure if the result of the close, ok, is a failure; and on a failed computation, keep that failure, though some people prefer to add up their failures.
Don't you know try/catch is so 1990s. </droll> (droll does not mean troll.)
Or just:
val is = new FileInputStream(file)
val result = try {
// do stuff
} finally {
is.close()
}
Because there's no way is can be null.
In your question code, just move val out outside try block. That way it will be identical to what Java AutoCloseable does, except for null case. But in Scala you don't have to deal with nullables.
Related
Let's say we have a fake data source which will return data it holds in batch
class DataSource(size: Int) {
private var s = 0
implicit val g = scala.concurrent.ExecutionContext.global
def getData(): Future[List[Int]] = {
s = s + 1
Future {
Thread.sleep(Random.nextInt(s * 100))
if (s <= size) {
List.fill(100)(s)
} else {
List()
}
}
}
object Test extends App {
val source = new DataSource(100)
implicit val g = scala.concurrent.ExecutionContext.global
def process(v: List[Int]): Unit = {
println(v)
}
def next(f: (List[Int]) => Unit): Unit = {
val fut = source.getData()
fut.onComplete {
case Success(v) => {
f(v)
v match {
case h :: t => next(f)
}
}
}
}
next(process)
Thread.sleep(1000000000)
}
I have mine, the problem here is some portion is more not pure. Ideally, I would like to wrap the Future for each batch into a big future, and the wrapper future success when last batch returned 0 size list? My situation is a little from this post, the next() there is synchronous call while my is also async.
Or is it ever possible to do what I want? Next batch will only be fetched when the previous one is resolved in the end whether to fetch the next batch depends on the size returned?
What's the best way to walk through this type of data sources? Are there any existing Scala frameworks that provide the feature I am looking for? Is play's Iteratee, Enumerator, Enumeratee the right tool? If so, can anyone provide an example on how to use those facilities to implement what I am looking for?
Edit----
With help from chunjef, I had just tried out. And it actually did work out for me. However, there was some small change I made based on his answer.
Source.fromIterator(()=>Iterator.continually(source.getData())).mapAsync(1) (f=>f.filter(_.size > 0))
.via(Flow[List[Int]].takeWhile(_.nonEmpty))
.runForeach(println)
However, can someone give comparison between Akka Stream and Play Iteratee? Does it worth me also try out Iteratee?
Code snip 1:
Source.fromIterator(() => Iterator.continually(ds.getData)) // line 1
.mapAsync(1)(identity) // line 2
.takeWhile(_.nonEmpty) // line 3
.runForeach(println) // line 4
Code snip 2: Assuming the getData depends on some other output of another flow, and I would like to concat it with the below flow. However, it yield too many files open error. Not sure what would cause this error, the mapAsync has been limited to 1 as its throughput if I understood correctly.
Flow[Int].mapConcat[Future[List[Int]]](c => {
Iterator.continually(ds.getData(c)).to[collection.immutable.Iterable]
}).mapAsync(1)(identity).takeWhile(_.nonEmpty).runForeach(println)
The following is one way to achieve the same behavior with Akka Streams, using your DataSource class:
import scala.concurrent.Future
import scala.util.Random
import akka.actor.ActorSystem
import akka.stream._
import akka.stream.scaladsl._
object StreamsExample extends App {
implicit val system = ActorSystem("Sandbox")
implicit val materializer = ActorMaterializer()
val ds = new DataSource(100)
Source.fromIterator(() => Iterator.continually(ds.getData)) // line 1
.mapAsync(1)(identity) // line 2
.takeWhile(_.nonEmpty) // line 3
.runForeach(println) // line 4
}
class DataSource(size: Int) {
...
}
A simplified line-by-line overview:
line 1: Creates a stream source that continually calls ds.getData if there is downstream demand.
line 2: mapAsync is a way to deal with stream elements that are Futures. In this case, the stream elements are of type Future[List[Int]]. The argument 1 is the level of parallelism: we specify 1 here because DataSource internally uses a mutable variable, and a parallelism level greater than one could produce unexpected results. identity is shorthand for x => x, which basically means that for each Future, we pass its result downstream without transforming it.
line 3: Essentially, ds.getData is called as long as the result of the Future is a non-empty List[Int]. If an empty List is encountered, processing is terminated.
line 4: runForeach here takes a function List[Int] => Unit and invokes that function for each stream element.
Ideally, I would like to wrap the Future for each batch into a big future, and the wrapper future success when last batch returned 0 size list?
I think you are looking for a Promise.
You would set up a Promise before you start the first iteration.
This gives you promise.future, a Future that you can then use to follow the completion of everything.
In your onComplete, you add a case _ => promise.success().
Something like
def loopUntilDone(f: (List[Int]) => Unit): Future[Unit] = {
val promise = Promise[Unit]
def next(): Unit = source.getData().onComplete {
case Success(v) =>
f(v)
v match {
case h :: t => next()
case _ => promise.success()
}
case Failure(e) => promise.failure(e)
}
// get going
next(f)
// return the Future for everything
promise.future
}
// future for everything, this is a `Future[Unit]`
// its `onComplete` will be triggered when there is no more data
val everything = loopUntilDone(process)
You are probably looking for a reactive streams library. My personal favorite (and one I'm most familiar with) is Monix. This is how it will work with DataSource unchanged
import scala.concurrent.duration.Duration
import scala.concurrent.Await
import monix.reactive.Observable
import monix.execution.Scheduler.Implicits.global
object Test extends App {
val source = new DataSource(100)
val completed = // <- this is Future[Unit], completes when foreach is done
Observable.repeat(Observable.fromFuture(source.getData()))
.flatten // <- Here it's Observable[List[Int]], it has collection-like methods
.takeWhile(_.nonEmpty)
.foreach(println)
Await.result(completed, Duration.Inf)
}
I just figured out that by using flatMapConcat can achieve what I wanted to achieve. There is no point to start another question as I have had the answer already. Put my sample code here just in case someone is looking for similar answer.
This type of API is very common for some integration between traditional Enterprise applications. The DataSource is to mock the API while the object App is to demonstrate how the client code can utilize Akka Stream to consume the APIs.
In my small project the API was provided in SOAP, and I used scalaxb to transform the SOAP to Scala async style. And with the client calls demonstrated in the object App, we can consume the API with AKKA Stream. Thanks for all for the help.
class DataSource(size: Int) {
private var transactionId: Long = 0
private val transactionCursorMap: mutable.HashMap[TransactionId, Set[ReadCursorId]] = mutable.HashMap.empty
private val cursorIteratorMap: mutable.HashMap[ReadCursorId, Iterator[List[Int]]] = mutable.HashMap.empty
implicit val g = scala.concurrent.ExecutionContext.global
case class TransactionId(id: Long)
case class ReadCursorId(id: Long)
def startTransaction(): Future[TransactionId] = {
Future {
synchronized {
transactionId += transactionId
}
val t = TransactionId(transactionId)
transactionCursorMap.update(t, Set(ReadCursorId(0)))
t
}
}
def createCursorId(t: TransactionId): ReadCursorId = {
synchronized {
val c = transactionCursorMap.getOrElseUpdate(t, Set(ReadCursorId(0)))
val currentId = c.foldLeft(0l) { (acc, a) => acc.max(a.id) }
val cId = ReadCursorId(currentId + 1)
transactionCursorMap.update(t, c + cId)
cursorIteratorMap.put(cId, createIterator)
cId
}
}
def createIterator(): Iterator[List[Int]] = {
(for {i <- 1 to 100} yield List.fill(100)(i)).toIterator
}
def startRead(t: TransactionId): Future[ReadCursorId] = {
Future {
createCursorId(t)
}
}
def getData(cursorId: ReadCursorId): Future[List[Int]] = {
synchronized {
Future {
Thread.sleep(Random.nextInt(100))
cursorIteratorMap.get(cursorId) match {
case Some(i) => i.next()
case _ => List()
}
}
}
}
}
object Test extends App {
val source = new DataSource(10)
implicit val system = ActorSystem("Sandbox")
implicit val materializer = ActorMaterializer()
implicit val g = scala.concurrent.ExecutionContext.global
//
// def process(v: List[Int]): Unit = {
// println(v)
// }
//
// def next(f: (List[Int]) => Unit): Unit = {
// val fut = source.getData()
// fut.onComplete {
// case Success(v) => {
// f(v)
// v match {
//
// case h :: t => next(f)
//
// }
// }
//
// }
//
// }
//
// next(process)
//
// Thread.sleep(1000000000)
val s = Source.fromFuture(source.startTransaction())
.map { e =>
source.startRead(e)
}
.mapAsync(1)(identity)
.flatMapConcat(
e => {
Source.fromIterator(() => Iterator.continually(source.getData(e)))
})
.mapAsync(5)(identity)
.via(Flow[List[Int]].takeWhile(_.nonEmpty))
.runForeach(println)
/*
val done = Source.fromIterator(() => Iterator.continually(source.getData())).mapAsync(1)(identity)
.via(Flow[List[Int]].takeWhile(_.nonEmpty))
.runFold(List[List[Int]]()) { (acc, r) =>
// println("=======" + acc + r)
r :: acc
}
done.onSuccess {
case e => {
e.foreach(println)
}
}
done.onComplete(_ => system.terminate())
*/
}
According to the Scala Language Specification (§6.19), "An enumerator sequence always starts with a generator". Why?
I sometimes find this restriction to be a hindrance when using for-comprehensions with monads, because it means you can't do things like this:
def getFooValue(): Future[Int] = {
for {
manager = Manager.getManager() // could throw an exception
foo <- manager.makeFoo() // method call returns a Future
value = foo.getValue()
} yield value
}
Indeed, scalac rejects this with the error message '<-' expected but '=' found.
If this was valid syntax in Scala, one advantage would be that any exception thrown by Manager.getManager() would be caught by the Future monad used within the for-comprehension, and would cause it to yield a failed Future, which is what I want. The workaround of moving the call to Manager.getManager() outside the for-comprehension doesn't have this advantage:
def getFooValue(): Future[Int] = {
val manager = Manager.getManager()
for {
foo <- manager.makeFoo()
value = foo.getValue()
} yield value
}
In this case, an exception thrown by foo.getValue() will yield a failed Future (which is what I want), but an exception thrown by Manager.getManager() will be thrown back to the caller of getFooValue() (which is not what I want). Other possible ways of handling the exception are more verbose.
I find this restriction especially puzzling because in Haskell's otherwise similar do notation, there is no requirement that a do block should begin with a statement containing <-. Can anyone explain this difference between Scala and Haskell?
Here's a complete working example showing how exceptions are caught by the Future monad in for-comprehensions:
import scala.concurrent._
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global
import scala.util.{Try, Success, Failure}
class Foo(val value: Int) {
def getValue(crash: Boolean): Int = {
if (crash) {
throw new Exception("failed to get value")
} else {
value
}
}
}
class Manager {
def makeFoo(crash: Boolean): Future[Foo] = {
if (crash) {
throw new Exception("failed to make Foo")
} else {
Future(new Foo(10))
}
}
}
object Manager {
def getManager(crash: Boolean): Manager = {
if (crash) {
throw new Exception("failed to get manager")
} else {
new Manager()
}
}
}
object Main extends App {
def getFooValue(crashGetManager: Boolean,
crashMakeFoo: Boolean,
crashGetValue: Boolean): Future[Int] = {
for {
manager <- Future(Manager.getManager(crashGetManager))
foo <- manager.makeFoo(crashMakeFoo)
value = foo.getValue(crashGetValue)
} yield value
}
def waitForValue(future: Future[Int]): Unit = {
val result = Try(Await.result(future, Duration("10 seconds")))
result match {
case Success(value) => println(s"Got value: $value")
case Failure(e) => println(s"Got error: $e")
}
}
val future1 = getFooValue(false, false, false)
waitForValue(future1)
val future2 = getFooValue(true, false, false)
waitForValue(future2)
val future3 = getFooValue(false, true, false)
waitForValue(future3)
val future4 = getFooValue(false, false, true)
waitForValue(future4)
}
Here's the output:
Got value: 10
Got error: java.lang.Exception: failed to get manager
Got error: java.lang.Exception: failed to make Foo
Got error: java.lang.Exception: failed to get value
This is a trivial example, but I'm working on a project in which we have a lot of non-trivial code that depends on this behaviour. As far as I understand, this is one of the main advantages of using Future (or Try) as a monad. What I find strange is that I have to write
manager <- Future(Manager.getManager(crashGetManager))
instead of
manager = Manager.getManager(crashGetManager)
(Edited to reflect #RexKerr's point that the monad is doing the work of catching the exceptions.)
for comprehensions do not catch exceptions. Try does, and it has the appropriate methods to participate in for-comprehensions, so you can
for {
manager <- Try { Manager.getManager() }
...
}
But then it's expecting Try all the way down unless you manually or implicitly have a way to switch container types (e.g. something that converts Try to a List).
So I'm not sure your premises are right. Any assignment you made in a for-comprehension can just be made early.
(Also, there is no point doing an assignment inside a for comprehension just to yield that exact value. Just do the computation in the yield block.)
(Also, just to illustrate that multiple types can play a role in for comprehensions so there's not a super-obvious correct answer for how to wrap an early assignment in terms of later types:
// List and Option, via implicit conversion
for {i <- List(1,2,3); j <- Option(i).filter(_ <2)} yield j
// Custom compatible types with map/flatMap
// Use :paste in the REPL to define A and B together
class A[X] { def flatMap[Y](f: X => B[Y]): A[Y] = new A[Y] }
class B[X](x: X) { def map[Y](f: X => Y): B[Y] = new B(f(x)) }
for{ i <- (new A[Int]); j <- (new B(i)) } yield j.toString
Even if you take the first type you still have the problem of whether there is a unique "bind" (way to wrap) and whether to doubly-wrap things that are already the correct type. There could be rules for all these things, but for-comprehensions are already hard enough to learn, no?)
Haskell translates the equivalent of for { manager = Manager.getManager(); ... } to the equivalent of lazy val manager = Manager.getManager(); for { ... }. This seems to work:
scala> lazy val x: Int = throw new Exception("")
x: Int = <lazy>
scala> for { y <- Future(x + 1) } yield y
res8: scala.concurrent.Future[Int] = scala.concurrent.impl.Promise$DefaultPromise#fedb05d
scala> Try(Await.result(res1, Duration("10 seconds")))
res9: scala.util.Try[Int] = Failure(java.lang.Exception: )
I think the reason this can't be done is because for-loops are syntactic sugar for flatMap and map methods (except if you are using a condition in the for-loop, in that case it's desugared with the method withFilter). When you are storing in a immutable variable, you can't use these methods. That's the reason you would be ok using Try as pointed out by Rex Kerr. In that case, you should be able to use map and flatMap methods.
Anybody know a solution to this problem ? I rewrote try catch finally construct to a functional way of doing things, but I can't close the stream now :-)
import scala.util.control.Exception._
def gunzip() = {
logger.info(s"Gunziping file ${f.getAbsolutePath}")
catching(classOf[IOException], classOf[FileNotFoundException]).
andFinally(println("how can I close the stream ?")).
either ({
val is = new GZIPInputStream(new FileInputStream(f))
Stream.continually(is.read()).takeWhile(-1 !=).map(_.toByte).toArray
}) match {
case Left(e) =>
val msg = s"IO error reading file ${f.getAbsolutePath} ! on host ${Setup.smtpHost}"
logger.error(msg, e)
MailClient.send(msg, msg)
new Array[Byte](0)
case Right(v) => v
}
}
I rewrote it based on Senia's solution like this :
def gunzip() = {
logger.info(s"Gunziping file ${file.getAbsolutePath}")
def closeAfterReading(c: InputStream)(f: InputStream => Array[Byte]) = {
catching(classOf[IOException], classOf[FileNotFoundException])
.andFinally(c.close())
.either(f(c)) match {
case Left(e) => {
val msg = s"IO error reading file ${file.getAbsolutePath} ! on host ${Setup.smtpHost}"
logger.error(msg, e)
new Array[Byte](0)
}
case Right(v) => v
}
}
closeAfterReading(new GZIPInputStream(new FileInputStream(file))) { is =>
Stream.continually(is.read()).takeWhile(-1 !=).map(_.toByte).toArray
}
}
I prefer this construction for such cases:
def withCloseable[T <: Closeable, R](t: T)(f: T => R): R = {
allCatch.andFinally{t.close} apply { f(t) }
}
def read(f: File) =
withCloseable(new GZIPInputStream(new FileInputStream(f))) { is =>
Stream.continually(is.read()).takeWhile(-1 !=).map(_.toByte).toArray
}
Now you could wrap it with Try and recover on some exceptions:
val result =
Try { read(f) }.recover{
case e: IOException => recover(e) // logging, default value
case e: FileNotFoundException => recover(e)
}
val array = result.get // Exception here!
take "scala-arm"
take Apache "commons-io"
then do the following
val result =
for {fis <- resource.managed(new FileInputStream(f))
gis <- resource.managed(new GZIPInputStream(fis))}
yield IOUtils.toString(gis, "UTF-8")
result.acquireFor(identity) fold (reportExceptions _, v => v)
One way how to handle it would be to use a mutable list of things that are opened and need to be closed later:
val cs: Buffer[Closeable] = new ArrayBuffer();
def addClose[C <: Closeable](c: C) = { cs += c; c; }
catching(classOf[IOException], classOf[FileNotFoundException]).
andFinally({ cs.foreach(_.close()) }).
either ({
val is = addClose(new GZIPInputStream(new FileInputStream(f)))
Stream.continually(is.read()).takeWhile(-1 !=).map(_.toByte).toArray
}) // ...
Update: You could use scala-conduit library (I'm the author) for this purpose. (The library is currently not considered production ready.) The main aim of pipes (AKA conduids) is to construct composable components with well defined resource handling. Each pipe repeatedly receives input and produces input. Optionally, it also produces a final result when it finishes. Pips has finalizers that are run after a pipe finishes - either on its own or when its downstream pipe finishes. Your example could be reworked (using Java NIO) as follows:
/**
* Filters buffers until a given character is found. The last buffer
* (truncated up to the character) is also included.
*/
def untilPipe(c: Byte): Pipe[ByteBuffer,ByteBuffer,Unit] = ...
// Create a new source that chunks a file as ByteBuffer's.
// (Note that the buffer changes on every step.)
val source: Source[ByteBuffer,Unit] = ...
// Sink that prints bytes to the standard output.
// You would create your own sink doing whatever you want.
val sink: Sink[ByteBuffer,Unit]
= NIO.writeChannel(Channels.newChannel(System.out));
runPipe(source >-> untilPipe(-1) >-> sink);
As soon as untilPipe(-1) finds -1 and finishes, its upstream source pipe's finalizer is run and the input is closed. If an exception occurs anywhere in the pipeline, the input is closed as well.
The full example can be found here.
I have another one proposition for cases when a closeble object like java.io.Socket may run for a long time, so one have to wrap it in Future. This is handful when you also control a timeout, when Socket is not responding.
object CloseableFuture {
type Closeable = {
def close(): Unit
}
private def withClose[T, F1 <: Closeable](f: => F1, andThen: F1 => Future[T]): Future[T] = future(f).flatMap(closeable => {
val internal = andThen(closeable)
internal.onComplete(_ => closeable.close())
internal
})
def apply[T, F1 <: Closeable](f: => F1, andThen: F1 => T): Future[T] =
withClose(f, {c: F1 => future(andThen(c))})
def apply[T, F1 <: Closeable, F2 <: Closeable](f1: => F1, thenF2: F1 => F2, andThen: (F1,F2) => T): Future [T] =
withClose(f1, {c1:F1 => CloseableFuture(thenF2(c1), {c2:F2 => andThen(c1,c2)})})
}
After I open a java.io.Socket and java.io.InputStream for it, and then execute code which reads from WhoisServer, I got both of them closed finally. Full code:
CloseableFuture(
{new Socket(server.address, WhoisPort)},
(s: Socket) => s.getInputStream,
(socket: Socket, inputStream: InputStream) => {
val streamReader = new InputStreamReader(inputStream)
val bufferReader = new BufferedReader(streamReader)
val outputStream = socket.getOutputStream
val writer = new OutputStreamWriter(outputStream)
val bufferWriter = new BufferedWriter(writer)
bufferWriter.write(urlToAsk+System.getProperty("line.separator"))
bufferWriter.flush()
def readBuffer(acc: List[String]): List[String] = bufferReader.readLine() match {
case null => acc
case str => {
readBuffer(str :: acc)
}
}
val result = readBuffer(Nil).reverse.mkString("\r\n")
WhoisResult(urlToAsk, result)
}
)
In python you can avoid try {} catch {} finally {} boilerplate with with (see What is the python keyword "with" used for?). I remember seeing an alternative to that in Scala, but I can't find it anymore.
It goes along the lines of:
def using[O](r: {def close()})(doit: () => O): O = try {
doit()
} finally {
r.close
}
using(myWriter){() => myWriter.println("something or another")}
Is it built into 2.10, or do I need a separate library for it?
It's almost trivial to make your own that covers almost all use cases (here using 2.10):
implicit class TidyUpAnything[A](val a: A) extends AnyVal {
def tidily[Z](g: A=>Any)(f: A=>Z) = try { f(a) } finally { g(a) }
}
If you want exceptions to pass through, use as is:
scala> Option(null: String).tidily(println){_.get} // Should print None
None
java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:313)
...
and if you want to handle exceptions, use it in conjunction with scala.util.Try:
scala> import scala.util._
scala> Try( Option(null: String).tidily(println){ _.get } )
None
res1: scala.util.Try[String] = Failure(java.util.NoSuchElementException: None.get)
Normally you would make g be something like _.close, but you can do arbitary resource cleanup with it. For example, here we back off a counter by one whenever we finish:
var i = 0
val a = Array(1,2)
a.tidily(_ => i -= 1){ _.foreach(_ => i += 1) }
scala> i
res2: Int = 1
I've defined 'using' function as following:
def using[A, B <: {def close(): Unit}] (closeable: B) (f: B => A): A =
try { f(closeable) } finally { closeable.close() }
I can use it like that:
using(new PrintWriter("sample.txt")){ out =>
out.println("hellow world!")
}
now I'm curious how to define 'using' function to take any number of parameters, and be able to access them separately:
using(new BufferedReader(new FileReader("in.txt")), new PrintWriter("out.txt")){ (in, out) =>
out.println(in.readLIne)
}
Starting Scala 2.13, the standard library provides a dedicated resource management utility: Using.
More specifically, the Using#Manager can be used when dealing with several resources.
In our case, we can manage different resources such as your PrintWriter or BufferedReader as they both implement AutoCloseable, in order to read and write from a file to another and, no matter what, close both the input and the output resource afterwards:
import scala.util.Using
import java.io.{PrintWriter, BufferedReader, FileReader}
Using.Manager { use =>
val in = use(new BufferedReader(new FileReader("input.txt")))
val out = use(new PrintWriter("output.txt"))
out.println(in.readLine)
}
// scala.util.Try[Unit] = Success(())
Someone has already done this—it's called Scala ARM.
From the readme:
import resource._
for(input <- managed(new FileInputStream("test.txt")) {
// Code that uses the input as a FileInputStream
}
I've been thinking about this and I thought maybe there was an other way to address this. Here is my take on supporting "any number" of parameters (limited by what tuples provide):
object UsingTest {
type Closeable = {def close():Unit }
final class CloseAfter[A<:Product](val x: A) {
def closeAfter[B](block: A=>B): B = {
try {
block(x);
} finally {
for (i <- 0 until x.productArity) {
x.productElement(i) match {
case c:Closeable => println("closing " + c); c.close()
case _ =>
}
}
}
}
}
implicit def any2CloseAfter[A<:Product](x: A): CloseAfter[A] =
new CloseAfter(x)
def main(args:Array[String]): Unit = {
import java.io._
(new BufferedReader(new FileReader("in.txt")),
new PrintWriter("out.txt"),
new PrintWriter("sample.txt")) closeAfter {case (in, out, other) =>
out.println(in.readLine)
other.println("hello world!")
}
}
}
I think I'm reusing the fact that 22 tuple/product classes have been written in the library... I don't think this syntax is clearer than using nested using (no pun intended), but it was an interesting puzzle.
using structural typing seems like a little overkill since java.lang.AutoCloseable is predestined for usage:
def using[A <: AutoCloseable, B](resource: A)(block: A => B): B =
try block(resource) finally resource.close()
or, if you prefer extension methods:
implicit class UsingExtension[A <: AutoCloseable](val resource: A) extends AnyVal {
def using[B](block: A => B): B = try block(resource) finally resource.close()
}
using2 is possible:
def using2[R1 <: AutoCloseable, R2 <: AutoCloseable, B](resource1: R1, resource2: R2)(block: (R1, R2) => B): B =
using(resource1) { _ =>
using(resource2) { _ =>
block(resource1, resource2)
}
}
but imho quite ugly - I would prefer to simply nest these using statements in the client code.
Unfortunately, there isn't support for arbitrary-length parameter lists with arbitrary types in standard Scala.
You might be able to do something like this with a couple of language changes (to allow variable parameter lists to be passed as HLists; see here for about 1/3 of what's needed).
Right now, the best thing to do is just do what Tuple and Function do: implement usingN for as many N as you need.
Two is easy enough, of course:
def using2[A, B <: {def close(): Unit}, C <: { def close(): Unit}](closeB: B, closeC: C)(f: (B,C) => A): A = {
try { f(closeB,closeC) } finally { closeB.close(); closeC.close() }
}
If you need more, it's probably worth writing something that'll generate the source code.
Here is an example that allows you to use the scala for comprehension as an automatic resource management block for any item that is a java.io.Closeable, but it could easily be expanded to work for any object with a close method.
This usage seems pretty close to the using statement and allows you to easily have as many resources defined in one block as you want.
object ResourceTest{
import CloseableResource._
import java.io._
def test(){
for( input <- new BufferedReader(new FileReader("/tmp/input.txt")); output <- new FileWriter("/tmp/output.txt") ){
output.write(input.readLine)
}
}
}
class CloseableResource[T](resource: =>T,onClose: T=>Unit){
def foreach(f: T=>Unit){
val r = resource
try{
f(r)
}
finally{
try{
onClose(r)
}
catch{
case e =>
println("error closing resource")
e.printStackTrace
}
}
}
}
object CloseableResource{
implicit def javaCloseableToCloseableResource[T <: java.io.Closeable](resource:T):CloseableResource[T] = new CloseableResource[T](resource,{_.close})
}
It is a good idea to detatch the cleanup algorithm from the program path.
This solution lets you accumulate closeables in a scope.
The scope cleanup will happen on after the block is executed, or the scope can be detached. The cleaning of the scope can then be done later.
This way we get the same convenience whitout being limited to single thread programming.
The utility class:
import java.io.Closeable
object ManagedScope {
val scope=new ThreadLocal[Scope]();
def managedScope[T](inner: =>T):T={
val previous=scope.get();
val thisScope=new Scope();
scope.set(thisScope);
try{
inner
} finally {
scope.set(previous);
if(!thisScope.detatched) thisScope.close();
}
}
def closeLater[T <: Closeable](what:T): T = {
val theScope=scope.get();
if(!(theScope eq null)){
theScope.closeables=theScope.closeables.:+(what);
}
what;
}
def detatchScope(): Scope={
val theScope=scope.get();
if(theScope eq null) null;
else {
theScope.detatched=true;
theScope;
}
}
}
class Scope{
var detatched=false;
var closeables:List[Closeable]=List();
def close():Unit={
for(c<-closeables){
try{
if(!(c eq null))c.close();
} catch{
case e:Throwable=>{};
}
}
}
}
The usage:
def checkSocketConnect(host:String, portNumber:Int):Unit = managedScope {
// The close later function tags the closeable to be closed later
val socket = closeLater( new Socket(host, portNumber) );
doWork(socket);
}
def checkFutureConnect(host:String, portNumber:Int):Unit = managedScope {
// The close later function tags the closeable to be closed later
val socket = closeLater( new Socket(host, portNumber) );
val future:Future[Boolean]=doAsyncWork(socket);
// Detatch the scope and use it in the future.
val scope=detatchScope();
future.onComplete(v=>scope.close());
}
This solution doesn't quite have the syntax you desire, but I think it's close enough :)
def using[A <: {def close(): Unit}, B](resources: List[A])(f: List[A] => B): B =
try f(resources) finally resources.foreach(_.close())
using(List(new BufferedReader(new FileReader("in.txt")), new PrintWriter("out.txt"))) {
case List(in: BufferedReader, out: PrintWriter) => out.println(in.readLine())
}
Of course the down side is you have to type out the types BufferedReader and PrintWrter in the using block. You might be able to add some magic so that you just need List(in, out) by using multiple ORed type bounds for type A in using.
By defining some pretty hacky and dangerous implicit conversions you can get around having to type List (and another way to get around specifying types for the resources), but I haven't documented the detail as it's too dangerous IMO.
here is my solution to the resource management in Scala:
def withResources[T <: AutoCloseable, V](r: => T)(f: T => V): V = {
val resource: T = r
require(resource != null, "resource is null")
var exception: Throwable = null
try {
f(resource)
} catch {
case NonFatal(e) =>
exception = e
throw e
} finally {
closeAndAddSuppressed(exception, resource)
}
}
private def closeAndAddSuppressed(e: Throwable,
resource: AutoCloseable): Unit = {
if (e != null) {
try {
resource.close()
} catch {
case NonFatal(suppressed) =>
e.addSuppressed(suppressed)
}
} else {
resource.close()
}
}
I used this in multiple Scala apps including managing resources in Spark executors. and one should be aware that we are other even better ways to manage resource like in CatsIO: https://typelevel.org/cats-effect/datatypes/resource.html. if you are ok with pure FP in Scala.
to answer your last question, you can definitely nest the resource like this:
withResource(r: File)(
r => {
withResource(a: File)(
anotherR => {
withResource(...)(...)
}
)
}
)
this way, not just that those resources are protected from leaking, they will also be released in the correct order(like stack). same behaviour like the Resource Monad from CatsIO.