scala functional way to store state - scala

I want to store a state (key -> value) in scala using functional way. I probably learned while ago in Odersky class but can not remember any more.
Here's my non-functional approach;
import org.scalatest.{FunSuite, Matchers}
trait EventHandler
class StatefulNonFn {
type EventName = String
private var state = Map.empty[EventName, EventHandler]
def update(name: String): EventHandler = {
state.get(name).fold {
val handler = new EventHandler {}
state += name -> handler
handler
}(eh => eh)
}
}
class NonFunctionalStateSpec extends FunSuite with Matchers {
test("stateful") {
val stateResult = new StatefulNonFn().update("MusicAdded")
stateResult.isInstanceOf[EventHandler] shouldBe true
}
}
One attempt I made is to make the state a "function of EventName and previousState" which makes sense but now I can't figure out how do I store those all states?
My first call is going to be fine because the state is empty in that case.
import org.scalatest.{FunSuite, Matchers}
trait EventHandler
class Stateful {
type EventName = String
private val stateFn = new ((String, Map[EventName, EventHandler]) => Map[EventName, EventHandler]) {
override def apply(name: String, prevState: Map[EventName, EventHandler]): Map[EventName, EventHandler] = {
val handler = new EventHandler {}
prevState + (name -> handler)
}
}
def initState = Map.empty[EventName, EventHandler]
def update(name: String, prevState: Map[EventName, EventHandler]) = stateFn(name, prevState)
}
class FunctionalStateSpec extends FunSuite with Matchers {
test("stateful") {
val stateHelper = new Stateful()
val stateResult = stateHelper.update("MusicAdded", stateHelper.initState)
stateResult.keys.size shouldBe 1
val stateResult1 = stateHelper.update("MusicDeleted", stateResult)
stateResult1.keys.size shouldBe 2
//what i obviously want is something like this without me wanting to store the previousStates
//stateHelper.update("MusicAdded1")
//stateHelper.update("MusicAdded2")
}
}
I am not sure, maybe something eventually has to be mutable. How do I Store the previous states in above case? without the client being the one to supply it in each call. Because state can be updated from 5 separate clients without knowing the previous state.

It just turns out that if you want to do some useful program, you need mutations and you need state. You need to do IO, like saving into the database (which could return some different ID every time you do it), or getting a random number, or the current timestamp, or printing into console, etc.
If you are doing functional programming, is not about doing pure, deterministic, total stuff. It's about isolating sideffects (like mutations and state) away.
So strictly pure languages like Haskell do it by returning actions (or plans, or descriptions of actions...) instead of performing them. The runtime performs those actions, so that way you have two things:
Your pure and sexy program
The runtime, in charge of doing the dirty stuff.
However Scala doesn't just expects you to return actions so that the runtime executes them... You need to do it yourself.
That being said, your solution is perfectly ok, if you don't want to go to dark places. Otherwise, I'd recommend you to read this, an article from John Degoes that basically explains how to do what you want (by simultaneously defining what a freemonad is).

Related

Akka; Passing around too many parameters with EventSourcedBehavior

We are building a event sourced system based on Akka-Typed. We quickly fall in a situation where our states requires many parameters passed as implicits parameters.
There is a solution on the style guide to use an enclosing class; https://doc.akka.io/docs/akka/current/typed/style-guide.html#passing-around-too-many-parameters.
However this is for "simple" behaviors, not for EventSourcedBehaviors. This technique of using a enclosing class that produce behaviors does not work for event sourced one because an event sourced behavior is made of one command handler and one event handler.
In our case we have a State trait that define one method to received commands and another to handle events. If we apply the enclosing class technique we have to create anonymous classes for all states, but those cannot be serialized.
object SampleEventSourced {
trait State extends CborSerializable {
def execute(cmd: Command): ReplyEffect
def apply(evt: Event): State
}
def apply(
persistenceId: PersistenceId,
config: Config,
): Behavior[Command] = {
EventSourcedBehavior
.withEnforcedReplies[Command, Event, State](
persistenceId,
new SampleEventSourced(config).empty(),
(state, cmd) => state.execute(cmd),
(state, evt) => state.apply(evt)
)// ...
}
}
class SampleEventSourced private(config: Config) {
import SampleEventSourced._
private def empty(): State = new State {
override def execute(cmd: Command): ReplyEffect = cmd match {
// ..
}
override def apply(evt: Event): State = evt match {
// ..
}
}
java.lang.IllegalArgumentException: Empty State [org.acme.SampleEventSourced$$anon$1#36d7d2fe] isn't serializable.
One solution would be to duplicate the creation of the event sourced behavior in each "state" method. But that will produce a lot of duplication.
object SampleEventSourced {
def apply(
persistenceId: PersistenceId,
config: Config,
): Behavior[Command] = new SampleEventSourced(config).empty()
}
class SampleEventSourced private(config: Config) {
import SampleEventSourced._
private def empty(): Behavior[Command] = EventSourcedBehavior
.withEnforcedReplies[Command, Event, State](
persistenceId,
new State(),
(state, cmd) => state.execute(cmd),
(state, evt) => state.apply(evt)
)// ...
}
Another would be to create concrete subclasses of State but we will have to pass the parameters across all of those states.
object SampleEventSourced {
class EmptyState extends State(config:Config, otherUselessParameter:Any) {
// ...
override def apply(evt: Event): evt match {
case _ => new OtherState(config, otherUselessParameter)
}
}
class OtherState extends State(config:Config, veryImportantParameter:Any) {
// ..
}
}
Putting those state classes inside the enclosing class won't work because those non-static inner classes cannot be de-serialized.
So, what's your solution for this case, how do you deal with EventSourcedBehavior with states that require many parameters ?
The problem is that that jackson-cbor, on top of being reflection based, isn't that effective at serializing things done in an idiomatic Scala way (the only real point in its favor is that it's better than Java serialization).
So the immediately apparent solution is to:
model the state as an Algebraic Data Type (generally a sealed trait extended by case classes and maybe a case object for the empty/initial state case).
don't serialize them via Jackson, but use a native-to-Scala serializer (e.g. play-json, circe, etc.) and an Akka serializer leveraging that serializer; it's likely easier to define a marker trait (e.g. CirceSerializable) so that in the configuration you only have to worry about one binding. Events & state will want to be marked with that trait: commands and their replies (at least the ones that could be sent over the network might also as well).
One subtlety to be aware of when defining your serializer is that, the native-to-Scala serializers will generally use some form of typeclass derivation at compile time, but this will be complicated if there's a need to serialize ActorRef, as a serializer for ActorRef is best done with the aid of the ActorSystem (EntityRef can now be inspected for serialization, and conceptually makes more sense for persistence, given the lifecycle, but that's going to be fairly custom): you'll therefore probably want to delay derivation of the ActorRef serializer/deserializer until the ActorSystem starts up and also delay derivation of the events/state/commands/replies containing ActorRefs until after then.
I'd probably tend to make the handler for each event a method in the sealed trait without letting the state have any knowledge of how events are reified. The benefit of this is that you can unit test the state transitions without needing events or commands.
case class ItemAdded(name: String) extends Event
sealed trait State {
def items: Set[String]
def addItem(name: String): State.NonEmpty
}
object State {
case object Empty extends State {
def items: Set[String] = Set.empty
def addItem(name: String): NonEmpty = NonEmpty(Set(name))
}
case class NonEmpty(items: Set[String]) extends State {
def addItem(name: String): NonEmpty = copy(items = items + name)
}
}
Then your event handler is pretty thin
(state, evt) => {
evt match {
case ItemAdded(name) => state.addItem(name)
}
}
Depending on how much validation your command handlers take on, those can get complex. The operative principle is that commands are interpreted into zero-or-more events, and events are interpreted into operations (methods on state which result in a new state) on the state. The state should thus have sufficient query methods (e.g. items) to allow the command-handler to consume a state.

Delayed variable initialization

Let's say I have a variable which I cannot initialize immediately, because I do not know its value (so lazy val won't work for me). I also cannot control the moment the object creation because it is handled by a framework (e.g. Activity in an Android app), so I cannot delay creation of whole object. I do NOT want to use var for this, as it brings the risk of incomplete initialization and invalid states.
So far I've been using approach like:
class MyClass extends InterfaceWithDelayedInitialization {
private val delayedValueP = Promise[Type]()
private val delayedValueF = delayedValueP.future
def onCreate(value: Type): Unit = {
delayedValueP.successful(value)
}
def someOtherMethod(): Unit = {
delayedValueF foreach { value =>
// do something with value
}
}
}
but I cannot help but feel that this approach is kind of smelly (not as smelly as using mutability and vars though).
Is there some standard solution for a problems like this one?

Task not serializable: java.io.NotSerializableException when calling function outside closure only on classes not objects

Getting strange behavior when calling function outside of a closure:
when function is in a object everything is working
when function is in a class get :
Task not serializable: java.io.NotSerializableException: testing
The problem is I need my code in a class and not an object. Any idea why this is happening? Is a Scala object serialized (default?)?
This is a working code example:
object working extends App {
val list = List(1,2,3)
val rddList = Spark.ctx.parallelize(list)
//calling function outside closure
val after = rddList.map(someFunc(_))
def someFunc(a:Int) = a+1
after.collect().map(println(_))
}
This is the non-working example :
object NOTworking extends App {
new testing().doIT
}
//adding extends Serializable wont help
class testing {
val list = List(1,2,3)
val rddList = Spark.ctx.parallelize(list)
def doIT = {
//again calling the fucntion someFunc
val after = rddList.map(someFunc(_))
//this will crash (spark lazy)
after.collect().map(println(_))
}
def someFunc(a:Int) = a+1
}
RDDs extend the Serialisable interface, so this is not what's causing your task to fail. Now this doesn't mean that you can serialise an RDD with Spark and avoid NotSerializableException
Spark is a distributed computing engine and its main abstraction is a resilient distributed dataset (RDD), which can be viewed as a distributed collection. Basically, RDD's elements are partitioned across the nodes of the cluster, but Spark abstracts this away from the user, letting the user interact with the RDD (collection) as if it were a local one.
Not to get into too many details, but when you run different transformations on a RDD (map, flatMap, filter and others), your transformation code (closure) is:
serialized on the driver node,
shipped to the appropriate nodes in the cluster,
deserialized,
and finally executed on the nodes
You can of course run this locally (as in your example), but all those phases (apart from shipping over network) still occur. [This lets you catch any bugs even before deploying to production]
What happens in your second case is that you are calling a method, defined in class testing from inside the map function. Spark sees that and since methods cannot be serialized on their own, Spark tries to serialize the whole testing class, so that the code will still work when executed in another JVM. You have two possibilities:
Either you make class testing serializable, so the whole class can be serialized by Spark:
import org.apache.spark.{SparkContext,SparkConf}
object Spark {
val ctx = new SparkContext(new SparkConf().setAppName("test").setMaster("local[*]"))
}
object NOTworking extends App {
new Test().doIT
}
class Test extends java.io.Serializable {
val rddList = Spark.ctx.parallelize(List(1,2,3))
def doIT() = {
val after = rddList.map(someFunc)
after.collect().foreach(println)
}
def someFunc(a: Int) = a + 1
}
or you make someFunc function instead of a method (functions are objects in Scala), so that Spark will be able to serialize it:
import org.apache.spark.{SparkContext,SparkConf}
object Spark {
val ctx = new SparkContext(new SparkConf().setAppName("test").setMaster("local[*]"))
}
object NOTworking extends App {
new Test().doIT
}
class Test {
val rddList = Spark.ctx.parallelize(List(1,2,3))
def doIT() = {
val after = rddList.map(someFunc)
after.collect().foreach(println)
}
val someFunc = (a: Int) => a + 1
}
Similar, but not the same problem with class serialization can be of interest to you and you can read on it in this Spark Summit 2013 presentation.
As a side note, you can rewrite rddList.map(someFunc(_)) to rddList.map(someFunc), they are exactly the same. Usually, the second is preferred as it's less verbose and cleaner to read.
EDIT (2015-03-15): SPARK-5307 introduced SerializationDebugger and Spark 1.3.0 is the first version to use it. It adds serialization path to a NotSerializableException. When a NotSerializableException is encountered, the debugger visits the object graph to find the path towards the object that cannot be serialized, and constructs information to help user to find the object.
In OP's case, this is what gets printed to stdout:
Serialization stack:
- object not serializable (class: testing, value: testing#2dfe2f00)
- field (class: testing$$anonfun$1, name: $outer, type: class testing)
- object (class testing$$anonfun$1, <function1>)
Grega's answer is great in explaining why the original code does not work and two ways to fix the issue. However, this solution is not very flexible; consider the case where your closure includes a method call on a non-Serializable class that you have no control over. You can neither add the Serializable tag to this class nor change the underlying implementation to change the method into a function.
Nilesh presents a great workaround for this, but the solution can be made both more concise and general:
def genMapper[A, B](f: A => B): A => B = {
val locker = com.twitter.chill.MeatLocker(f)
x => locker.get.apply(x)
}
This function-serializer can then be used to automatically wrap closures and method calls:
rdd map genMapper(someFunc)
This technique also has the benefit of not requiring the additional Shark dependencies in order to access KryoSerializationWrapper, since Twitter's Chill is already pulled in by core Spark
Complete talk fully explaining the problem, which proposes a great paradigm shifting way to avoid these serialization problems: https://github.com/samthebest/dump/blob/master/sams-scala-tutorial/serialization-exceptions-and-memory-leaks-no-ws.md
The top voted answer is basically suggesting throwing away an entire language feature - that is no longer using methods and only using functions. Indeed in functional programming methods in classes should be avoided, but turning them into functions isn't solving the design issue here (see above link).
As a quick fix in this particular situation you could just use the #transient annotation to tell it not to try to serialise the offending value (here, Spark.ctx is a custom class not Spark's one following OP's naming):
#transient
val rddList = Spark.ctx.parallelize(list)
You can also restructure code so that rddList lives somewhere else, but that is also nasty.
The Future is Probably Spores
In future Scala will include these things called "spores" that should allow us to fine grain control what does and does not exactly get pulled in by a closure. Furthermore this should turn all mistakes of accidentally pulling in non-serializable types (or any unwanted values) into compile errors rather than now which is horrible runtime exceptions / memory leaks.
http://docs.scala-lang.org/sips/pending/spores.html
A tip on Kryo serialization
When using kyro, make it so that registration is necessary, this will mean you get errors instead of memory leaks:
"Finally, I know that kryo has kryo.setRegistrationOptional(true) but I am having a very difficult time trying to figure out how to use it. When this option is turned on, kryo still seems to throw exceptions if I haven't registered classes."
Strategy for registering classes with kryo
Of course this only gives you type-level control not value-level control.
... more ideas to come.
I faced similar issue, and what I understand from Grega's answer is
object NOTworking extends App {
new testing().doIT
}
//adding extends Serializable wont help
class testing {
val list = List(1,2,3)
val rddList = Spark.ctx.parallelize(list)
def doIT = {
//again calling the fucntion someFunc
val after = rddList.map(someFunc(_))
//this will crash (spark lazy)
after.collect().map(println(_))
}
def someFunc(a:Int) = a+1
}
your doIT method is trying to serialize someFunc(_) method, but as method are not serializable, it tries to serialize class testing which is again not serializable.
So make your code work, you should define someFunc inside doIT method. For example:
def doIT = {
def someFunc(a:Int) = a+1
//function definition
}
val after = rddList.map(someFunc(_))
after.collect().map(println(_))
}
And if there are multiple functions coming into picture, then all those functions should be available to the parent context.
I solved this problem using a different approach. You simply need to serialize the objects before passing through the closure, and de-serialize afterwards. This approach just works, even if your classes aren't Serializable, because it uses Kryo behind the scenes. All you need is some curry. ;)
Here's an example of how I did it:
def genMapper(kryoWrapper: KryoSerializationWrapper[(Foo => Bar)])
(foo: Foo) : Bar = {
kryoWrapper.value.apply(foo)
}
val mapper = genMapper(KryoSerializationWrapper(new Blah(abc))) _
rdd.flatMap(mapper).collectAsMap()
object Blah(abc: ABC) extends (Foo => Bar) {
def apply(foo: Foo) : Bar = { //This is the real function }
}
Feel free to make Blah as complicated as you want, class, companion object, nested classes, references to multiple 3rd party libs.
KryoSerializationWrapper refers to: https://github.com/amplab/shark/blob/master/src/main/scala/shark/execution/serialization/KryoSerializationWrapper.scala
I'm not entirely certain that this applies to Scala but, in Java, I solved the NotSerializableException by refactoring my code so that the closure did not access a non-serializable final field.
Scala methods defined in a class are non-serializable, methods can be converted into functions to resolve serialization issue.
Method syntax
def func_name (x String) : String = {
...
return x
}
function syntax
val func_name = { (x String) =>
...
x
}
FYI in Spark 2.4 a lot of you will probably encounter this issue. Kryo serialization has gotten better but in many cases you cannot use spark.kryo.unsafe=true or the naive kryo serializer.
For a quick fix try changing the following in your Spark configuration
spark.kryo.unsafe="false"
OR
spark.serializer="org.apache.spark.serializer.JavaSerializer"
I modify custom RDD transformations that I encounter or personally write by using explicit broadcast variables and utilizing the new inbuilt twitter-chill api, converting them from rdd.map(row => to rdd.mapPartitions(partition => { functions.
Example
Old (not-great) Way
val sampleMap = Map("index1" -> 1234, "index2" -> 2345)
val outputRDD = rdd.map(row => {
val value = sampleMap.get(row._1)
value
})
Alternative (better) Way
import com.twitter.chill.MeatLocker
val sampleMap = Map("index1" -> 1234, "index2" -> 2345)
val brdSerSampleMap = spark.sparkContext.broadcast(MeatLocker(sampleMap))
rdd.mapPartitions(partition => {
val deSerSampleMap = brdSerSampleMap.value.get
partition.map(row => {
val value = sampleMap.get(row._1)
value
}).toIterator
})
This new way will only call the broadcast variable once per partition which is better. You will still need to use Java Serialization if you do not register classes.
I had a similar experience.
The error was triggered when I initialize a variable on the driver (master), but then tried to use it on one of the workers.
When that happens, Spark Streaming will try to serialize the object to send it over to the worker, and fail if the object is not serializable.
I solved the error by making the variable static.
Previous non-working code
private final PhoneNumberUtil phoneUtil = PhoneNumberUtil.getInstance();
Working code
private static final PhoneNumberUtil phoneUtil = PhoneNumberUtil.getInstance();
Credits:
https://learn.microsoft.com/en-us/answers/questions/35812/sparkexception-job-aborted-due-to-stage-failure-ta.html ( The answer of pradeepcheekatla-msft)
https://databricks.gitbooks.io/databricks-spark-knowledge-base/content/troubleshooting/javaionotserializableexception.html
def upper(name: String) : String = {
var uppper : String = name.toUpperCase()
uppper
}
val toUpperName = udf {(EmpName: String) => upper(EmpName)}
val emp_details = """[{"id": "1","name": "James Butt","country": "USA"},
{"id": "2", "name": "Josephine Darakjy","country": "USA"},
{"id": "3", "name": "Art Venere","country": "USA"},
{"id": "4", "name": "Lenna Paprocki","country": "USA"},
{"id": "5", "name": "Donette Foller","country": "USA"},
{"id": "6", "name": "Leota Dilliard","country": "USA"}]"""
val df_emp = spark.read.json(Seq(emp_details).toDS())
val df_name=df_emp.select($"id",$"name")
val df_upperName= df_name.withColumn("name",toUpperName($"name")).filter("id='5'")
display(df_upperName)
this will give error
org.apache.spark.SparkException: Task not serializable
at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:304)
Solution -
import java.io.Serializable;
object obj_upper extends Serializable {
def upper(name: String) : String =
{
var uppper : String = name.toUpperCase()
uppper
}
val toUpperName = udf {(EmpName: String) => upper(EmpName)}
}
val df_upperName=
df_name.withColumn("name",obj_upper.toUpperName($"name")).filter("id='5'")
display(df_upperName)
My solution was to add a compagnion class that handles all methods that are not seriazable within the class.

scala: Should private utility methods live in companion objects?

The question is self explanatory, but please allow me to provide an example:
I have the following:
class Foo {
def doAndPrint {
val result = doSomething()
val msg = message(result)
println(msg)
}
private def message(result: Result): String = {
"message formatted with %s".format(result)
}
}
In this context, the question is: Should def message(result: Result) live in object Foo?
The argument in favor is making explicit that def message(result: Result) does not depends on any state within class Foo.
The argument against is that the motivation of companion objects was to provide a place to put java public static methods.
The answer to this false dichotomy is neither. It should be a local method to doAndPrint.
class Foo {
def doAndPrint {
val result = doSomething()
def message(result: Result): String = s"message formatted with $result"
val msg = message(result)
println(msg)
}
}
In fact,
class Foo {
def doAndPrint {
val result = doSomething()
def message = s"message formatted with $result"
println(message)
}
}
Notice that it really depends on local state.
Edit: OK, as a nod to "self-explanatory," I would add, use the smallest scope that makes sense, and the example points out an asymmetry in the private relations between companions. Which begs for many puns which I haven't time to supply.
Answering more directly, from my observations, the companion module does not normally serve as a repository for FooUtil-style functions, though it does serve as a repository for implicit conversions, which have an arguably similar flavor, albeit public. Consider what winds up in the objects of collection types.
Consider a separation of concerns:
class Foo(f: String => Unit) {
def doSomethingAndDoSomethingWithIt {
val result = doSomething()
def message = s"message formatted with $result"
f(message)
}
}
You should put the methods where they belong. If you need to break things up for testing purposes, readability or even maintainability then you need to break them up. Even though Scala is influenced by FP concepts, FP patterns, and an FP mindset, it is still also an OO language.
Private helper methods are just that, helper methods to make your code easier to work with. If your class needs them, then there is no reason to spread out that logic in another class... just 'cause. Put them in the same place (and add in some means to access those methods for unit testing purposes like package visibility.)

Trouble extending the Future[T] trait in Scala

Please excuse my poor understanding of Scala. I'm just a Java developer that wants to get something working in the Play Framework. I even attempted to implement a trait using Java code, but I got even more obscure errors. I have the following Scala code:
package models
import scala.concurrent.Future
class SettableFuture[T](name: String) extends Future[T] {
var assignedValue: T
def set(newValue: T) =
assignedValue = newValue
//override abstract methods in Future[T]
def ready(atMost: scala.concurrent.duration.Duration)(implicit permit: scala.concurrent.CanAwait): models.SettableFuture[T] =
this
def result(atMost: scala.concurrent.duration.Duration)(implicit permit: scala.concurrent.CanAwait): T =
assignedValue
def isCompleted: Boolean =
false
def onComplete[U](func: scala.util.Try[T] => U)(implicit executor: scala.concurrent.ExecutionContext): Unit =
null
def value: Option[scala.util.Try[T]] =
null
}
And here's my error:
overriding method ready in trait Awaitable of type (atMost: scala.concurrent.duration.Duration)(implicit permit: scala.concurrent.CanAwait)SettableFuture.this.type; method ready has incompatible type
Ignore the return values of the methods for now, they're nonsensical because I'm just trying to fix all the compile errors.
I simply copied the method stubs from the compile time exception when extending a trait without overriding its abstract methods and pasted them into my source file. I don't understand why I'm still getting errors. I took a look at the signature for ready() in Awaitable and it looks like the return type should in fact be the class.
EDIT: the reason why I want to implement this is because in the Promise/Future Scala API, I can only find things that let me asynchronously execute long-running, blocking tasks. What I'm after is something that lets execution of a request be suspended until something of interest sets a value in the SettableFuture instance, which completes a Promise to send a response. In this way, it's somewhat like a continuation. Anyway, here is the working code that I ended up with:
package models
import java.util.concurrent.CountDownLatch
import java.util.concurrent.TimeUnit
import java.util.concurrent.atomic.AtomicInteger
import scala.concurrent.CanAwait
import scala.concurrent.ExecutionContext
import scala.concurrent.Future
import scala.concurrent.duration.Duration
import scala.util.Try
class SettableFuture[T]() extends Future[T] {
private final val ValueNotSet = 0
private final val ValueBeingSet = 1
private final val ValueSet = 2
private val valueStatus: AtomicInteger = new AtomicInteger(ValueNotSet)
private val onCompleteWaitHandle: CountDownLatch = new CountDownLatch(1)
private var onComplete: Try[T] => _ = _
private var assignedValue: T = _
/** Set a value and complete this Future.
*
* Returns false if the value has already been set by a past call to this method.
* Otherwise, marks this Future as complete, executes the function passed to
* onComplete, and finally returns true.
*/
def set(newValue: T): Boolean = {
//set value and trigger onComplete only once
if (valueStatus.compareAndSet(ValueNotSet, ValueBeingSet)) {
assignedValue = newValue
valueStatus.set(ValueSet)
onCompleteWaitHandle.countDown()
if (onComplete != null)
onComplete(Try(assignedValue))
true
}
false
}
//override abstract methods in the Future[T] trait
def ready(atMost: Duration)(implicit permit: CanAwait): this.type = {
onCompleteWaitHandle.await(atMost.length, atMost.unit)
this
}
def result(atMost: Duration)(implicit permit: CanAwait): T = {
ready(atMost)
assignedValue
}
def isCompleted: Boolean = (valueStatus.get() == ValueSet)
def onComplete[U](func: Try[T] => U)(implicit executor: ExecutionContext): Unit =
onComplete = func
def value: Option[Try[T]] = {
if (!isCompleted)
None
Some(Try(assignedValue))
}
}
In terms of the specific error you are getting, the ready method you are overriding has a return type of Awaitable.this.type - ie. the specific type of this instance of Awaitable (the super type of Future, so that in Future, this method is seen to have return type Future.this.type). For your SettableFuture class, this means the return type for the ready method needs to be models.SettableFuture.this.type.
Other minor problems you can expect to hit: the onComplete method's implementation should be {}, not null, as the latter is a return of type Null.type, not Unit, and the var assignedValue needs to be initialised in a non-abstract class, which can be done by adding = _ to the line defining the variable (although you really want to make it at least protected, and provide an accessor which will check if it has been set - maybe by changing the variable to an Option[T] initialised to None, or else maintaining a Boolean flag that can be checked in the accessor, and which is set to true by the set method).
In terms of what it looks like you are trying to achieve, however, you might just want to check out scala.concurrent.Promise, which represents a "promise of a future result". It has a method future that returns a Future, and various complete, completeWith, and similar methods that can be used to set the Promise's value, which in turn will cause the associated Future to become ready/completed.
The SettableFuture class unnecessarily mixes the two concerns that the Future and Promise traits were designed to separate:
asynchronous provision of a value (Promise), and
waiting for and reacting to that provision (Future)
Rather than thinking of a Future as an asynchronous, long-running, blocking computation, it may help to think of it simply as a value that may be provided at some future time. You can react to provision of that value in a number of ways, including registering a callback or mapping it to some other value. For example, in Play one will often suspend processing of a request with a pattern like this (in Scala):
def handleRequest = Action {
Async {
gimmeAFuture().map(value => Ok(value))
}
}
def gimmeAFuture(): Future[JsValue] = // ...
The gimmeAFuture method returns a Future, but the request handling code doesn't care about how the value is computed. It could be
computed immediately with Future.successful,
computed asynchronously with Future.apply, or
supplied by the completion of a Promise
As an example of the latter, the gimmeAFuture method might be implemented as follows:
def gimmeAFuture: Future[JsValue] = {
val p = Promise.apply[JsValue]()
// asynchronously complete the promise 30 seconds from now
scheduler.scheduleOnce(30.seconds)(p.complete(Success(someJsObject)))
// return the future immediately
p.future
}
Of course, you could implement that method however you want. The key is that something will need to hold on to that Promise object and complete it with a value in order to resume handling of the request. The request handler itself does not get a reference to the Promise object, since it only cares about the value that will be computed (i.e., the Promise's Future).
I revisited this question after finding this link and I realized why there was so much confusion.
I was planning on using the SettableFuture class because I couldn't find anything like it that already existed in the Play Java API. I wanted something equivalent to a TaskCompletionSource in .NET, and Aaron's answer made it really clear that Scala had exactly what I needed. Unfortunately, I couldn't find the equivalent in Play's Java API.
That link cleared up exactly why I was having so much difficulty over something that should have been so simple. Thanks all for answering my question!