Please pardon the length of this question.
I often need to create some contextual information at one layer of my code, and consume that information elsewhere. I generally find myself using implicit parameters:
def foo(params)(implicit cx: MyContextType) = ...
implicit val context = makeContext()
foo(params)
This works, but requires the implicit parameter to be passed around a lot, polluting the method signatures of layer after layout of intervening functions, even if they don't care about it themselves.
def foo(params)(implicit cx: MyContextType) = ... bar() ...
def bar(params)(implicit cx: MyContextType) = ... qux() ...
def qux(params)(implicit cx: MyContextType) = ... ged() ...
def ged(params)(implicit cx: MyContextType) = ... mog() ...
def mog(params)(implicit cx: MyContextType) = cx.doStuff(params)
implicit val context = makeContext()
foo(params)
I find this approach ugly, but it does have one advantage though: it's type safe. I know with certainty that mog will receive a context object of the right type, or it wouldn't compile.
It would alleviate the mess if I could use some form of "dependency injection" to locate the relevant context. The quotes are there to indicate that this is different from the usual dependency injection patterns found in Scala.
The start point foo and the end point mog may exist at very different levels of the system. For example, foo might be a user login controller, and mog might be doing SQL access. There may be many users logged in at once, but there's only one instance of the SQL layer. Each time mog is called by a different user, a different context is needed. So the context can't be baked into the receiving object, nor do you want to merge the two layers in any way (like the Cake Pattern). I'd also rather not rely on a DI/IoC library like Guice or Spring. I've found them very heavy and not very well suited to Scala.
So what I think I need is something that lets mog retrieve the correct context object for it at runtime, a bit like a ThreadLocal with a stack in it:
def foo(params) = ...bar()...
def bar(params) = ...qux()...
def qux(params) = ...ged()...
def ged(params) = ...mog()...
def mog(params) = { val cx = retrieveContext(); cx.doStuff(params) }
val context = makeContext()
usingContext(context) { foo(params) }
But that would fall as soon as asynchronous actor was involved anywhere in the chain. It doesn't matter which actor library you use, if the code runs on a different thread then it loses the ThreadLocal.
So... is there a trick I'm missing? A way of passing information contextually in Scala that doesn't pollute the intervening method signatures, doesn't bake the context into the receiver statically, and is still type-safe?
The Scala standard library includes something like your hypothetical "usingContext" called DynamicVariable. This question has some information about it When we should use scala.util.DynamicVariable? . DynamicVariable does use a ThreadLocal under the hood so many of your issues with ThreadLocal will remain.
The reader monad is a functional alternative to explicitly passing an environment http://debasishg.blogspot.com/2010/12/case-study-of-cleaner-composition-of.html. The Reader monad can be found in Scalaz http://code.google.com/p/scalaz/. However, the ReaderMonad does "pollute" your signatures in that their types must change and in general monadic programming can cause a lot of restructuring to your code plus extra object allocations for all the closures may not sit well if performance or memory is a concern.
Neither of these techniques will automatically share a context over an actor message send.
A little late to the party, but have you considered using implicit parameters to your classes constructors?
class Foo(implicit biz:Biz) {
def f() = biz.doStuff
}
class Biz {
def doStuff = println("do stuff called")
}
If you wanted to have a new biz for each call to f() you could let the implicit parameter be a function returning a new biz:
class Foo(implicit biz:() => Biz) {
def f() = biz().doStuff
}
Now you simply need to provide the context when constructing Foo. Which you can do like this:
trait Context {
private implicit def biz = () => new Biz
implicit def foo = new Foo // The implicit parameter biz will be resolved to the biz method above
}
class UI extends Context {
def render = foo.f()
}
Note that the implicit biz method will not be visible in UI. So we basically hide away those details :)
I wrote a blog post about using implicit parameters for dependency injection which can be found here (shameless self promotion ;) )
I think that the dependency injection from lift does what you want. See the wiki for details using the doWith () method.
Note that you can use it as a separate library, even if you are not running lift.
You asked this just about a year ago, but here's another possibility. If you only ever need to call one method:
def fooWithContext(cx: MyContextType)(params){
def bar(params) = ... qux() ...
def qux(params) = ... ged() ...
def ged(params) = ... mog() ...
def mog(params) = cx.doStuff(params)
... bar() ...
}
fooWithContext(makeContext())(params)
If you need all the methods to be externally visible:
case class Contextual(cx: MyContextType){
def foo(params) = ... bar() ...
def bar(params) = ... qux() ...
def qux(params) = ... ged() ...
def ged(params) = ... mog() ...
def mog(params) = cx.doStuff(params)
}
Contextual(makeContext()).foo(params)
This is basically the cake pattern, except that if all your stuff fits into a single file, you don't need all the messy trait stuff to combine it into one object: you can just nest them. Doing it this way also makes cx properly lexically scoped, so you don't end up with funny behavior when you use futures and actors and such. I suspect that if you use the new AnyVal, you could even do away with the overhead of allocating the Contextual object.
If you want to split your stuff into multiple files using traits, you only really need a single trait per file to hold everything and put the MyContextType properly in scope, if you don't need the fancy replaceable-components-via-inheritance thing most cake pattern examples have.
// file1.scala
case class Contextual(cx: MyContextType) with Trait1 with Trait2{
def foo(params) = ... bar() ...
def bar(params) = ... qux() ...
}
// file2.scala
trait Trait1{ self: Contextual =>
def qux(params) = ... ged() ...
def ged(params) = ... mog() ...
}
// file3.scala
trait Trait2{ self: Contextual =>
def mog(params) = cx.doStuff(params)
}
// file4.scala
Contextual(makeContext()).foo(params)
It looks kinda messy in a small example, but remember, you only need to split it off into a new trait if the code is getting too big to sit comfortable in one file. By that point your files are reasonably big, so an extra 2 lines of boilerplate on a 200-500 line file is not so bad really.
EDIT:
This works with asynchronous stuff too
case class Contextual(cx: MyContextType){
def foo(params) = ... bar() ...
def bar(params) = ... qux() ...
def qux(params) = ... ged() ...
def ged(params) = ... mog() ...
def mog(params) = Future{ cx.doStuff(params) }
def mog2(params) = (0 to 100).par.map(x => x * cx.getSomeValue )
def mog3(params) = Props(new MyActor(cx.getSomeValue))
}
Contextual(makeContext()).foo(params)
It Just Works using nesting. I'd be impressed if you could get similar functionality working with DynamicVariable.
You'd need a special subclass of Future that stores the current DynamicVariable.value when created, and hook into the ExecutionContext's prepare() or execute() method to extract the value and properly set up the DynamicVariable before executing the Future.
Then you'd need a special scala.collection.parallel.TaskSupport to do something similar in order to get parallel collections working. And a special akka.actor.Props in order to do something similar for that.
Every time there's a new mechanism of creating asynchronous tasks, DynamicVariable based implementations will break and you'll have weird bugs where you end up pulling up the wrong Context. Every time you add a new DynamicVariable to keep track of, you'll need to patch all your special executors to properly set/unset this new DynamicVariable. Using nesting you can just let lexical closure take care of all of this for you.
(I think Futures, collections.parallel and Props count as "layers in between that aren't my code")
Similar to the implicit approach, with Scala Macros you can do auto-wiring of objects using constructors - see my MacWire project (and excuse the self-promotion).
MacWire also has scopes (quite customisable, a ThreadLocal implementation is provided). However, I don't think you can propagate context across actor calls with a library - you need to carry some identifier around. This can be e.g. through a wrapper for sending actor messages, or more directly with the message.
Then as long as the identifier is unique per request/session/whatever your scope is, it's just a matter of looking things up in a map via a proxy (like the MacWire scopes do, the "identifier" here isn't needed as it is stored in the ThreadLocal).
Related
I have a following question. Our project has a lot of code, that runs tests in Scala. And there is a lot of code, that fills the fields like this:
production.setProduct(new Product)
production.getProduct.setUuid("b1253a77-0585-291f-57a4-53319e897866")
production.setSubProduct(new SubProduct)
production.getSubProduct.setUuid("89a877fa-ddb3-3009-bb24-735ba9f7281c")
Eventually, I grew tired from this code, since all those fields are actually subclasses of the basic class that has the uuid field, so, after thinking a while, I wrote the auxiliary function like this:
def createUuid[T <: GenericEntity](uuid: String)(implicit m : Manifest[T]) : T = {
val constructor = m.runtimeClass.getConstructors()(0)
val instance = constructor.newInstance().asInstanceOf[T]
instance.setUuid(uuid)
instance
}
Now, my code got two times shorter, since now I can write something like this:
production.setProduct(createUuid[Product]("b1253a77-0585-291f-57a4-53319e897866"))
production.setSubProduct(createUuid[SubProduct]("89a877fa-ddb3-3009-bb24-735ba9f7281c"))
That's good, but I am wondering, if I could somehow implement the function createUuid so the last bit would like this:
// Is that really possible?
production.setProduct(createUuid("b1253a77-0585-291f-57a4-53319e897866"))
production.setSubProduct(createUuid("89a877fa-ddb3-3009-bb24-735ba9f7281c"))
Can scala compiler guess, that setProduct expects not just a generic entity, but actually something like Product (or it's subclass)? Or there is no way in Scala to implement this even shorter?
Scala compiler won't infer/propagate the type outside-in. You could however create implicit conversions like:
implicit def stringToSubProduct(uuid: String): SubProduct = {
val n = new SubProduct
n.setUuid(uuid)
n
}
and then just call
production.setSubProduct("89a877fa-ddb3-3009-bb24-735ba9f7281c")
and the compiler will automatically use the stringToSubProduct because it has applicable types on the input and output.
Update: To have the code better organized I suggest wrapping the implicit defs to a companion object, like:
case class EntityUUID(uuid: String) {
uuid.matches("[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}") // possible uuid format check
}
case object EntityUUID {
implicit def toProduct(e: EntityUUID): Product = {
val p = new Product
p.setUuid(e.uuid)
p
}
implicit def toSubProduct(e: EntityUUID): SubProduct = {
val p = new SubProduct
p.setUuid(e.uuid)
p
}
}
and then you'd do
production.setProduct(EntityUUID("b1253a77-0585-291f-57a4-53319e897866"))
so anyone reading this could have an intuition where to find the conversion implementation.
Regarding your comment about some generic approach (having 30 types), I won't say it's not possible, but I just do not see how to do it. The reflection you used bypasses the type system. If all the 30 cases are the same piece of code, maybe you should reconsider your object design. Now you can still implement the 30 implicit defs by calling some method that uses reflection similar what you have provided. But you will have the option to change it in the future on just this one (30) place(s).
I'm attempting to write a method which accepts multiple generic types and takes as an argument a unit of work to execute.
The idea is that the unit of work is a common function that itself is generic. For the sake of example, let's say it's something like the following:
def loadModelRdd[T: TypeTag](sc: SparkContext): RDD[T] = {
...
}
loadModelRdd() will construct an RDD of the given type after some internal processing like loading the Model information, etc.
A prototype method I've been hacking on looks something like the following (non-working):
def forkAll[A : Manifest, B : Manifest](work: => RDD[_]): (RDD[A], RDD[B]) = {
def aFuture = Future { work } // How can I notify that this work call returns type A?
def bFuture = Future { work } // How can I notify that this work call returns type B?
val res = for {
a <- aFuture
b <- bFuture
} yield (a.asInstanceOf[A], b.asInstanceOf[B])
Await.result(res, 10.seconds)
}
This is a shortened version of the code I'm working on as I'm actually looking at accepting as many as 10 different types.
As you can see, the overall goal of the forkAll method is to wrap the unit of work in a Future, fork-join the execution of the unit of work for each type, then return the results as a Tuple'd result. An example consumer statement would be:
val (a, b) = forkAll[ClassA, ClassB](loadModelRdd)
i.e I want to fork-join at this point and wait for the results, but I want the executions to be executed in parallel and then collected back to the Driver (Spark Driver to be specific).
The problem is I'm not sure how to coerce the type returned by the unit of work within forkAll when constructing the Future {} blocks. Without the forkAll, the implementation looked like the following:
val resA = loadModelRdd[ClassA](sc)
val resB = loadModelRdd[ClassB](sc)
...
I am looking at doing this for two reasons:
To abstract the details of fork-join for any unit of work which matches this model.
A version of this code, which explicitly states what the unit of work is, is working in Production and was responsible for cutting execution of a long-running block by close to half. I have a couple of execution steps where this pattern could be applied
Is this something that is possible in Scala's type system? Or should I look at this problem from a different perspective? I've tried a couple of implementations (including one described here) but I haven't quite found one that fits my current view of the problem
Please let me know if there is any additional information needed.
Thanks!
Short answer: Scala does not allow functions with type parameters, so what you want is not exactly possible.
You are attempting to pass a method with a type parameter. Although methods are allowed to have type parameters, functions are not. When you try to pass a method, it acts like an anonymous function, so you must specify a type.
However, since methods do allow type parameters, you can take advantage of this by creating an abstract class that will do your fork/join
abstract class ForkJoin {
protected def work[T]: RDD[T]
def apply[A, B]: (RDD[A], RDD[B]) = {
// Write implementation of fork/join here
(work[A], work[B])
}
}
then overriding the type generic work method so that it does what you want, such as calling some other pre-defined method.
val forkJoin = new ForkJoin {
override protected def work[T]: RDD[T] =
loadModelRdd[T](sc)
}
val (intRdd, stringRdd) = forkJoin[Int, String]
Check out this for a prototype implementation that compiles and runs without issues.
Having used a Scala library that liberally exposes the reliance on implicits to the caller, I had experienced friction around this mechanism, as Scala makes it quite hard at times to debug implicit arguments, and because there's quite a bunch of places Scala would fill in values for implicit arguments from. (I could almost relate to it as "implicits hell" at one time).
At one time in my coding, Scala "complained" an implicit value could not be matched whereas in fact there was a "collision" of implicit values each coming from a different import.
Regardless of that perceived brittleness, it may at times feel borderline to an abuse of the context design pattern.
Why does it make sense to have implicit parameters in Scala?
In what scenarios would you use them and how would you avoid trouble?
As I'm not sure the experimentation-curve and potential for other team members getting totally confused are worth it, could you possibly suggest other scala idioms for sharing context between a multitude of Scala functions?
This questions is not for a specific implementation at hand, hopefully it's still a good fit for this site.
Generally, using a common type as an implicit parameter is a bad idea.
def badIdea(n: Int)(implicit s: String) = s * n
It doesn't take much to imagine why: you'll get conflicting implicits for the same thing if anyone else adopts this policy. Better to avoid it.
But who really wants to manually stuff in a scala.concurrent.ExecutionContext manually every time it's needed (which is practically everywhere)?
So the key is: when you have something with a specialized type, especially if it's bookkeeping that might need to be overridden manually but mostly should just do the right thing, then use implicit parameters. (This usually covers type classes as well.)
Then what do you do if you really need a string? Well, wrap it (at least formally--here it's a value class so in some contexts it will just pass the string around):
class MyWrappedString(val underlying: String) extends AnyVal {}
implicit val myString = new MyWrappedString("bird")
def decentIdea(n: Int)(implicit mws: MyWrappedString) = mws.underlying * n
scala> decentIdea(2) // In the bush?
res14: String = birdbird
Or if you think some additional logic is helpful, write a wrapper that takes an extra type parameter:
class ImplicitWithValue[K,V](val value: V) {
// Any extra generic logic goes here
}
object ImplicitWithValue {
class ValuePart[K] {
def apply[V](v: V) = new ImplicitWithValue[K,V](v)
}
private val genericValuePart = new ValuePart[Any]
private def typedValuePart[K] = genericValuePart.asInstanceOf[ValuePart[K]]
def apply[K] = typedValuePart[K]
}
Then you can
trait Marker1
implicit val implicit1 = ImplicitWithValue[Marker1]("fish")
def goodIdea(n: Int)(implicit ms: ImplicitWithValue[Marker1, String]) = ms.value * n
scala> goodIdea(3)
res17: String = fishfishfish
I'm writing a message parser. Suppose I have a superclass Message with two auxiliary constructors, one that accepts String raw messages and one that accepts a Map with datafields mapped out in key-value pairs.
class Message {
def this(s: String)
def this(m: Map[String, String])
def toRaw = { ... } # call third party lib to return the generated msg
def map # call third party lib to return the parsed message
def something1 # something common for all messages which would be overriden in child classes
def something2 # something common for all messages which would be overriden in child classes
...
}
There's good reason to do this as the library that does parsing/generating is kind of awkward and removing the complexity of interfacing with it into a separate class makes sense, the child class would look something like this:
class SomeMessage extends Message {
def something1 # ...
def something2 # ...
}
and the idea is to use the overloaded constructors in the child class, for example:
val msg = new SomeMessage(rawMessage) # or
val msg = new SomeMessage("fld1" -> ".....", "fld2" -> "....")
# and then be able to call
msg.something1
msg.something2 # ...
However, the way auxiliary constructors and inheritance seem to behave in Scala this pattern has proven to be pretty challenging, and the simplest solution I found so far is to create a method called constructMe, which does the work of the constructors in the above case:
val msg = new SomeMessage
msg.constructMe(rawMessage) # or
msg.constructMe("fld1" -> ".....", "fld2" -> "....")
which seems crazy to need a method called constructMe.
So, the question:
is there a way to structure the code so to simply use the overloaded constructors from the superclass? For example:
val msg = new SomeMessage(rawMessage) # or
val msg = new SomeMessage("fld1" -> ".....", "fld2" -> "....")
or am I simply approaching the problem the wrong way?
Unless I'm missing something, you are calling the constructor like this:
val msg = new SomeMessage(rawMessage)
But the Message class doesn't not take a parameter, your class should be defined so:
class Message(val message: String) {
def this(m: Map[String, String]) = this("some value from mapping")
}
Also note that the constructor in scala must call the primary constructor as first action, see this question for more info.
And then the class extending the Message class should be like this:
class SomeMessage(val someString: String) extends Message(someString) {
def this(m: Map[String, String]) = this("this is a SomeMessage")
}
Note that the constructor needs a code block otherwise your code won't compile, you can't have a definition like def this(someString: String) without providing the implementation.
Edit:
To be honest I don't quite get why you want to use Maps in your architecture, your class main point it to contain a String, having to do with complex types in constructors can lead to problems. Let's say you have some class which can take a Map[String, String] as a constructor parameter, what will you do with it? As I said a constructor must call himself as first instruction, what you could is something like this:
class A(someString: String) = {
def this(map: Map[String, String]) = this(map.toString)
}
And that's it, the restrictions in scala don't allow you to do anything more, you would want to do some validation, for example let's say you want to take always the second element in the map, this could throw exceptions since the user is not forced to provide a map with more than one value, he's not even forced to provide a filled map unless you start filling your class with requires.
In your case I probably would leave String as class parameter or maybe a List[String] where you can call mkString or toString.
Anyway if you are satisfied calling map.toString you have to give both constructor implementation to parent and child class, this is one of scala constructor restrictions (in Java you could approach the problem in a different way), I hope somebody will prove me wrong, but as far as I know there's no other way to do it.
As a side note, I personally find this kind of restriction to be correct (most of the time) since the force you to structure your code to be more rigorous and have a better architecture, think about the fact that allowing people to do whatever they want in a constructor (like in java) obfuscate their true purpose, that is return a new instance of a class.
I am using an external library in Scala, which uses a set of traits to pass around complex configuration options to different methods. This is Highcharts Scala API, but the problem seems to be more general.
The library defines a trait (HighchartsOptions in the actual usage), which is just a data transfer object that stores a number of fields and allows them to be passed around. Code simplified and generalized for clarity looks like this:
trait Opts {
def option1: Int = 3
def option2: String = "abc"
//Many more follow, often of more complex types
}
As long as the complete set of options can be generated in one place, this allows for a neat syntax:
val opts = new Opts() {
override val option1 = 5
//And so on for more fields
}
doSomething(opts)
However, there are a few situations where one piece of code prepares such a configuration but another piece of code needs to adjust just one option extra. It would be nice to be able to pass some Opts instance to a method and let the method modify a value or two.
Since the original trait is based on defs rather than vars, it's easy to override an option's value only if the type of the object is known, like in the example above. If a method receives only an instance of some anonymous subclass of Opts, how can it create another instance or modify the received one so that a call to e.g. option2 could return a different value? The desired operation is similar to what Mockito's spy does, however I feel there should be some less contrived way than using a mocking framework to achieve this effect.
PS: Actually I am a bit surprised by the use of such an interface by the library's authors, so perhaps I'm missing something and there is some completely different way of achieving my goal of building a single set of options from several different places in the code (e.g. some builder object that is mutable and that I can pass around instead of the finished HighchartsOptions)?
I would first check if using the Opts trait (solely) is an absolute necessity. Hopefully it's not and then you just extend the trait, overriding defs with vars, like you said.
When Opts is mandatory and you have its instance that you want to copy modifying some fields, here's what you could do:
Write a wrapper for Opts, which extends Opts, but delegates every call to the wrapped Opts excluding the fields that you want to modify. Set those fields to values you want.
Writing the wrapper for a broad-interface trait can be boring task, therefore you may consider using http://www.warski.org/blog/2013/09/automatic-generation-of-delegate-methods-with-macro-annotations/ to let macros generate most of it automatically.
The shortest, simplest way.
Define a case class:
case class Options(
option1: Int,
option2: String
/* ... */
) extends Opts
and implicit conversion from Opts to your Options
object OptsConverter {
implicit def toOptions(opts: Opts) = Options(
option1 = opts.option1,
option2 = opts.option2
/* ... */
)
}
That way you get all copy methods (generated by compiler) for free.
You can use it like that:
import OptsConverter.toOptions
def usage(opts: Opts) = {
val improvedOpts = opts.copy(option2 = "improved")
/* ... */
}
Note, that Options extends Opts, so you can use it whenever Opts is required. You'll be able to call copy to obtain a modified instance of Opts in every place where you import the implicit conversion.
The simplest solution is to allow the trait to define it's own "copy" method, and allow it's subclasses (or even base class) to work with that. However, the parameters can really only match the base class unless you recast it later. Incidentally this doesn't work as "mixed in" so your root might as well be an abstract class, but it works the same way. The point of this is that the subclass type keeps getting passed along as it's copied.
(Sorry I typed this without a compiler so it may need some work)
trait A {
type myType<:A
def option1: Int
def option2: String
def copyA(option1_:Int=option1, option2_String=option2):myType = new A {
def option1 = option_1
def option2 = option_2
}
}
trait B extends A { me=>
type myType = B
def option3: Double
//callable from A but properly returns B
override def copyA(option1_:Int=option1, option2_:String=option2):myType = new B {
def option1 = option_1
def option2 = option_2
def option3 = me.option3
}
//this is only callable if you've cast to type B
def copyB(option1_:Int=option1, option2_:String=option2, option3_:Double=option3):myType = new B {
def option1 = option_1
def option2 = option_2
def option3 = option_3
}
}