I've been trying to write a Scala (2.10.0) compiler plugin that analyzes some parts of a traversed code.
This is what I originally had:
class MyPlugin (val global: Global) extends Plugin {
import global._
val name = "myPlugin"
val components = List[PluginComponent](MyComponent)
private object MyComponent extends PluginComponent {
val global: MyPlugin.this.global.type = MyPlugin.this.global
val runsAfter = List ("refchecks")
val phaseName = "codeAnalysis"
def newPhase (_prev: Phase) = new AnalysisPhase (_prev)
class AnalysisPhase (prev: Phase) extends StdPhase (prev) {
override def name = phaseName
def apply (unit: CompilationUnit) {
codeTraverser traverse unit.body
printLinesToFile(counters.map{case (k,v) => k + "\t" + v},out)
}
def codeTraverser = new ForeachTreeTraverser (tree => /* Analyze tree */)
}
}
}
This code works as expected, however I don't like it because I cannot decouple the code traverser method from this object. I would like to write a separate CodeTraverser class that will perform the analysis on a given Tree. This, among other things may help me test this code better.
The main problem is that unit.body is of an internal Tree type inside scala.reflect.internal.Trees. If I could work with scala.reflect.api.Trees#Tree instead of the internal version I could decouple the traverser functionality and even test it quite easily.
I've tried to find a way to convert between the two, but to no avail. Is it even possible? From looking at their source code, many things looks too similar for this to be impossible.
You're probably struggling with the cake pattern that the compiler is implemented with and a lot of path-dependency that comes with it. I've gone through this some time ago when I was writing some really beefy macro and wanted to refactor a bunch of functions out of macro implementation into separate utility class. I found this to be quite an annoying issue.
Here's how I would implement your Traverser in a separate class:
class MyPluginUtils[G <: Global with Singleton](global: G) {
import global._
class AnalyzingTraverser extends ForeachTreeTraverser(tree => /* analyze */)
}
Now, inside your plugin you have to use it like this:
val utils = new MyPluginUtils[global.type](global)
import utils.{global => _, _}
val traverser = new AnalyzingTraverser
As you can see, it's not the most intuitive thing in the world (i.e. this is confusing as hell), but this is the best that I could come up with that actually worked, and I tried a lot of things before finally settling on this one. I would be really happy to see some nicer way to do this.
AFAIK, such extensibility is one of the general problems with the cake pattern (as used in scalac implementation). I've seen other people also complain about this.
The component (the SubComponent or PluginComponent) must be created with the global member initialized early (that is, as an early definition).
Don't forget to review the one-question faq. I may go set google calendar to remind me to do that every Monday morning.
For an example, see the continuations plugin.
The component is defined with a utility class mixed in.
The utility class follows the usual cake recipe. (Leave it as an abstract dependency and let the compiler ensure that everything was mixed correctly.)
Here is a recent edit showing more early definitions, as a demonstration that this usage is not anomalous.
val anfPhase = new {
val global = SelectiveCPSPlugin.this.global
val cpsEnabled = pluginEnabled
override val enabled = cpsEnabled
} with SelectiveANFTransform {
val runsAfter = List("pickler")
}
(In future, they plan to deprecate early definitions in favor of parameterized traits when they are available in the language.)
More generally, global, i.e., "the compiler", is routinely instantiated for testing the compiler itself. I haven't seen it mocked, but computeInternalPhases is the template method for picking what phases are assembled modulo plugins.
There is a current effort to reduce internal dependencies, for the purpose of testing, as a window onto the difficulties involved.
Related
I am trying to refactor some code for a program which uses an ActorSystem as the backbone for Http calls.
My specific goal is to make my code more modular so I can write libraries of functions which make http calls using an ActorSystem where the ActorSystem is later expected to be provided by the application.
This is a general question though as I tend to run into this problem a reasonable amount.
I have two goals:
Minimize the number of ActorSystems I create to simplify tracking of them (one per top level application is the goal)
Avoid explicitly passing around the ActorSystem and context everywhere it's needed.
Conceptually - the code below illustrates how I'm thinking about it (of course this code would not compile).
import akka.actor.ActorSystem
import intermediateModule._
import scala.concurrent.ExecutionContextExecutor
object MyApp extends App {
// Create the actorsystem and place into scope
implicit val system = ActorSystem()
implicit val context = system.dispatcher
intermediateFunc1(300)
}
// Elsewhere in the intermediate module
object intermediateModule {
import expectsActorSystemModule._
def intermediateFunc1(x: Int) = {
// Relies on ActorSystem and Execution context,
// but won't compile because, of course the application ActorSystem and
// ec is not in scope
usesActorSystem(x)
}
}
// In this modiule, usesActorSystem needs an ActorSystem
object expectsActorSystemModule {
def usesActorSystem(x: Int)
(implicit system: ActorSystem, context: ExecutionContextExecutor) = ???
//... does some stuff like sending http requests with ActorSystem
}
Is there a way to "trickle down" implicits through the sub-modules to achieve the goal of the top level application providing the needed implicits?
Can this be done in a way such that the "depth" of module imports doesn't matter (e.g. if I added a few more intermediate libraries in between the top level app and the module which requires the ActorSystem)?
The answer here is dependency injection. Every object that has dependencies on other objects should receive them as constructor parameters. The important thing here is that higher layers only get their own dependencies, and not their dependencies' dependencies.
In your example, IntermediateModule doesn't use the ActorSystem itself; it only needs it to pass it on to ExpectsActorSystemModule. This is bad, because if the latter changes and requires another dependency, you will need to change the former as well – that is too much coupling. You can refactor it like so:
import akka.actor.ActorSystem
import scala.concurrent.ExecutionContextExecutor
object MyApp extends App {
// Create the actorsystem and place into scope
// wire everything together
implicit val system = ActorSystem()
implicit val context = system.dispatcher
val expectsActorSystemModule = new ExpectsActorSystemModule
val intermediateModule = new IntermediateModule(expectsActorSystemModule)
// run stuff
intermediateModule.intermediateFunc1(300)
}
// Elsewhere in the intermediate module
class IntermediateModule(expectsActorSystemModule: ExpectsActorSystemModule) {
def intermediateFunc1(x: Int) = {
// Note: no ActorSystem or ExecutionContext is needed, because they were
// injected into expectsActorSystemModule
expectsActorSystemModule.usesActorSystem(x)
}
}
// In this module, usesActorSystem needs an ActorSystem
class ExpectsActorSystemModule(
implicit system: ActorSystem,
context: ExecutionContextExecutor) {
def usesActorSystem(x: Int) = ???
//... does some stuff like sending http requests with ActorSystem
}
Note that IntermediateModule no longer needs an ActorSystem or an ExecutionContext, because those were provided directly to ExpectsActorSystemModule.
The slightly annoying part is that at some point you have to instantiate all these objects in your application and wire them all together. In the above example it's only 4 lines in MyApp, but it will get significantly longer for more substantial programs.
There are libraries like MacWire or Guice to help with this, but I would recommend against using them. They make it much less transparent what is going on, and they don't save all that much code either – in my opinion, it's a bad tradeoff. And these two specifically have more downsides. Guice comes from the Java world and gives you basically no compile-time guarantees, meaning that your code might compile just fine and then fail to start because Guice. MacWire is better in that regard (everything is done at compile time), but it's not future-proof because it's implemented as a Scala 2 macro – it will not work on Scala 3 in its current form.
Another approach that is popular among the purely functional programming community is to use ZIO's ZLayer. But since you're working on an existing codebase that is based on the Lightbend tech stack, this is unlikely to be the means of choice in this particular case.
In particular situations, you need to have some utility methods that are required across different classes. To solve this situation, you create an Util object wherein you place all these methods
object AggregatorUtil {
def aggregateValues(list : List[BigDecimal]) = //some logic...
}
// Import everything in the Utilities object
import AggregatorUtil._
and then import whichever members of util are required in your class. However, the downside to this is that, as all your methods are inside the singleton object and it becomes tricky to mock the object and unit test methods of the class that use utility methods.
To solve this problem again, the only solution that came to mind was Extracting the functionality out to a trait and then mocking the trait.
Please let me know if there is any other approach for handling and testing of util methods and which one is a rather cleaner approach.
Thanks in advance !!!
Note: -I am using scalatest and mockito in my project.
If you need to mock, putting this all in a mocked-out trait is the way forward. If mocking is unnecessary though, avoid it. Mocking unnecessarily is... unnecessary. You'll just be wasting time and effort for something which provides no additional value.
Mocking is best used when you have complex functionality or functionality in other files which you want to treat as a black box and just assume it works as expected (you'd then typically unit test this stuff separately). If you can avoid it and use the functions' actual functionality though, you'll get a much more realistic view of what your application does and will spot new bugs/breaking changes quicker (if you have mocked out functionality and forget to update your mocks, you might not any spot new bugs you introduce).
A good example of when mocking is necessary is when you're mocking calls to a database in a MVC application (e.g. a Scala Play microservice). You obviously don't want to have to run an actual database when testing your code, so you'd typically mock out your connector layer and return dummy/mocked data from your connector functions.
An example of something you wouldn't mock is something like:
trait MyTrait {
def toInt(str: String): Int
}
val mockedTrait = mock[MyTrait]
when(mockedTrait.toInt(eq("3")).thenReturn(3)
It's a bit of a silly example, but I think it explains the point clearly - doing something like this would be ridiculous. Mocking isn't always the answer.
I mostly mock with Test-Implementation which I find more readable and you don't have to learn a mocking framework.
Here an example:
The Interface:
trait DataRepo {
def persist(data: DataObject): Future[DataObject]
def idents(): Future[List[String]]
def insertData(dataCont: DataObject): Future[Int]
...
}
The Mocked Interface:
object DataRepoMock extends DataRepo {
def persist(data: DataObject): Future[DataObject] = ??? // only implement when needed
def idents(): Future[List[String]] = Future.successful((0 to 10).map(_=>Random.nextInt(100)))
def insertData(dataCont: DataObject): Future[Int] = Future.successful(Random.nextInt(100))
...
}
You can also use all the Scala goodies, like Pattern Matching, to make your Mock react differently to input.
Here is an example that this is not just used by myself;):
EPFLx: scala-reactiveX see Lecture 2.5 Testing Actor Systems:
def fakeGetter(url:String, depth: Int):Props =
Props(new Getter(url, depth){
override def webClient: WebClient = FakeWebClient
})
I have a complex project which reads configurations from a DB through the object ConfigAccessor which implements two basic APIs: getConfig(name: String) and storeConfig(c: Config).
Due to how the project is currently designed, almost every component needs to use the ConfigAccessor to talk with the DB. Thus, being this component an object it is easy to just import it and call its static methods.
Now I am trying to build some unit tests for the project in which the configurations are stored in a in-memory hashMap. So, first of all I decoupled the config accessor logic from its storage (using the cake pattern). In this way I can define my own ConfigDbComponent while testing
class ConfigAccessor {
this: ConfigDbComponent =>
...
The "problem" is that now ConfigAccessor is a class, which means I have to instantiate it at the beginning of my application and pass it everywhere to whoever needs it. The first way I can think of for passing this instance around would be through other components constructors. This would become quite verbose (adding a parameter to every constructor in the project).
What do you suggest me to do? Is there a way to use some design pattern to overcome this verbosity or some external mocking library would be more suitable for this?
Yes, the "right" way is passing it in constructors. You can reduce verbosity by providing a default argument:
class Foo(config: ConfigAccessor = ConfigAccessor) { ... }
There are some "dependency injection" frameworks, like guice or spring, built around this, but I won't go there, because I am not a fan.
You could also continue utilizing the cake pattern:
trait Configuration {
def config: ConfigAccessor
}
trait Foo { self: Configuration => ... }
class FooProd extends Foo with ProConfig
class FooTest extends Foo with TestConfig
Alternatively, use the "static setter". It minimizes changes to existing code, but requires mutable state, which is really frowned upon in scala:
object Config extends ConfigAccessor {
#volatile private var accessor: ConfigAccessor = _
def configurate(cfg: ConfigAccessor) = synchronized {
val old = accessor
accessor = cfg
old
}
def getConfig(c: String) = Option(accessor).fold(
throw new IllegalStateException("Not configurated!")
)(_.getConfig(c))
You can retain a global ConfigAccessor and allow selectable accessors like this:
object ConfigAccessor {
private lazy val accessor = GetConfigAccessor()
def getConfig(name: String) = accessor.getConfig(name)
...
}
For production builds you can put logic in GetConfigAccessor to select the appropriate accessor based on some global config such as typesafe config.
For unit testing you can have a different version of GetConfigAccessor for different test builds which return the appropriate test implementation.
Making this value lazy allows you to control the order of initialisation and if necessary do some non-functional mutable stuff in the initialisation code before creating the components.
Update following comments
The production code would have an implementation of GetConfigAccessor something like this:
object GetConfigAccessor {
private val useAws = System.getProperties.getProperty("accessor.aws") == "true"
def apply(): ConfigAccessor =
if (useAws) {
return new AwsConfigAccessor
} else {
return new PostgresConfigAccessor
}
}
Both AwsConfigAccessor and PostgresConfigAccessor would have their own unit tests to prove that they conform to the correct behaviour. The appropriate accessor can be selected at runtime by setting the appropriate system property.
For unit testing there would be a simpler implementation of GetConfigAccessor, something like this:
def GetConfigAccessor() = new MockConfigAccessor
Unit testing is done within a unit testing framework which contains a number of libraries and mock objects that are not part of the production code. These are built separately and are not compiled into the final product. So this version of GetConfigAccessor would be part of that unit testing code and would not be part of the final product.
Having said all that, I would only use this model for reading static configuration data because that keeps the code functional. The ConfigAccessor is just a convenient way to access global constants without having them passed down in the constructor.
If you are also writing data then this is more like a real DB than a configuration. In that case I would create custom accessors for each component that give access to different parts of the DB. That way it is clear which parts of the data are updated by each component. These accessors would be passed down to the component and can then be unit tested with the appropriate mock implementation as normal.
You may need to partition your data into static config and dynamic config and handle them separately.
I am building a market simulator using Scala/Akka/Play. I have an Akka actor with two children. The children need to have specific types which I would like to specify as parameters.
Suppose that I have the following class definition...
case class SecuritiesMarket[A <: AuctionMechanismLike, C <: ClearingMechanismLike](instrument: Security) extends Actor
with ActorLogging {
val auctionMechanism: ActorRef = context.actorOf(Props[A], "auction-mechanism")
val clearingMechanism: ActorRef = context.actorOf(Props[C], "clearing-mechanism")
def receive: Receive = {
case order: OrderLike => auctionMechanism forward order
case fill: FillLike => clearingMechanism forward fill
}
}
Instances of this class can be created as follows...
val stockMarket = SecuritiesMarket[DoubleAuctionMechanism, CCPClearingMechanism](Security("GOOG"))
val derivativesMarket = SecuritiesMarket[BatchAuctionMechanism, BilateralClearingMechanism](Security("SomeDerivative"))
There are many possible combinations of auction mechanism types and clearing mechanism types that I might use when creating SecuritiesMarket instance for a particular model/simulation.
Can I specify the type parameters that I wish to use in a given simulation in the application.conf file?
I see two questions here.
Can I get a Class instance from a String?
Yes.
val cls: Class[DoubleAuctionMechanism] = Class.forName("your.app.DoubleAuctionMechanism").asInstanceOf[Class[DoubleAuctionMechanism]]
You would still need the cast, as forName returns Class[_].
Can I instantiate a type with type parameters are not known compile time?
Well sort of, but not really.
object SecuritiesMarket {
def apply[A, C](clsAuc: Class[A], clsClr: Class[C])(security: Security): SecuritiesMarket[A, C] = {
SecuritiesMarket[A, C](security)
}
}
I think auction mechanisms and clearing mechanisms are dependencies for SecurityMarket. I'm guessing you instantiate them in its constructor somehow (how?). If that's the case why not just pass them in as a constructor parameter?
Edit:
I don't see how I could create the child actors inside SecurityMarket
Answering this in the comments; Props[T] can also be written as Props[T](classOfT), which can be simplified as Props(classOfT). Those three are the same. So the following code:
val auctionMechanism: ActorRef = context.actorOf(Props[A], "auction-mechanism")
Can be replaced with:
val classOfA = Class.forName("path.to.A")
val auctionMechanism: ActorRef = context.actorOf(Props(classOfA), "auction-mechanism")
First, application.conf is a runtime artifact and its contents are as far as I know not normally parsed at compile time. When the file is parsed at runtime, the parser creates an instance of the class Config which then controls the Akka setup.
The Typesafe Config library project readme is quite nice and the linked documentation has all of the details:
https://github.com/typesafehub/config/blob/master/README.md.
Second, since template parameters are not available at runtime because of type erasure, you can't normally use application.conf to control templating. You could create a custom build step to parse application.conf and modify your code before compilation, but this is maybe not what you want. (And if you do want a custom build step, perhaps a different .conf would be appropriate.)
Instead you might try simply eliminating the type parameters for the securities market class. Then create a single, simple implementation of the auction and clearing actors. Implement these actors by reading the names of the respective mechanisms from application.conf, instantiating the configured mechanism reflectively, and delegating to the instantiated mechanism. The mechanism classes could be independent of Akka, which is perhaps nice if that's where you keep most of your logic?
I was searching a way of doing dependency injection in Scala kind of like Spring or Unity in C# and I found nothing really interesting.
MacWire: I don't understand the benefit as we have to give the class in wire[CASS]. So what's the point if you give the implementation when you call wire? I can do new CASS it will be the same.
Cake pattern with self type: Seems to not answer what I'm searching for.
So I decided to make my implementation and ask you what do you think because it's surprising me that nothing like this has been done before. Maybe my implementation have lot's of issues in real life also.
So here is an example:
trait Messenger {
def send
}
class SkypeMessenger extends Messenger {
def send = println("Skype")
}
class ViberMessenger extends Messenger {
def send = println("Viber")
}
I want here to inject everywhere in my app the implementation configured in only one place:
object App {
val messenger = Inject[Messenger]
def main(args: Array[String]) {
messenger.send
}
}
Note the Inject[Messenger] that I define like below with the config I want (prod or dev):
object Inject extends Injector with DevConfig
trait ProdConfig {
this: Injector =>
register[Messager](new SkypeMessager)
register[Messager](new ViberMessager, "viber")
}
trait DevConfig {
this: Injector =>
register[Messager](new ViberMessager)
register[Messager](new ViberMessager, "viber")
}
And finally here is the Injector which contains all methods apply and register:
class Injector {
var map = Map[String, Any]()
def apply[T: ClassTag] =
map(classTag[T].toString).asInstanceOf[T]
def apply[T: ClassTag](id: String) =
map(classTag[T].toString + id).asInstanceOf[T]
def register[T: ClassTag](instance: T, id: String = "") = {
map += (classTag[T].toString + id -> instance)
instance
}
}
To summaries:
I have a class Injector which is a Map between interfaces/traits (eventually also an id) and an instance of the implementation.
We define a trait for each config (dev, prod...) which contains the registers. It also have a self reference to Injector.
And we create an instance of the Injector with the Config we want
The usage is to call the apply method giving the Interface type (eventually also an id) and it will return the implementation's instance.
What do you think?
You code looks a lot like dependency injection in Lift web framework. You can consult Lift source code to see how it's implemented or just use the framework. You don't have to run a Lift app to use its libraries. Here is a small intro doc. Basically you should be looking at this code in Lift:
package net.liftweb.http
/**
* A base trait for a Factory. A Factory is both an Injector and
* a collection of FactorMaker instances. The FactoryMaker instances auto-register
* with the Injector. This provides both concrete Maker/Vender functionality as
* well as Injector functionality.
*/
trait Factory extends SimpleInjector
You can also check this related question: Scala - write unit tests for objects/singletons that extends a trait/class with DB connection where I show how Lift injector is used.
Thanks guys,
So I make my answer but the one from Aleksey was very good.
I understand better the Cake Pattern with this sample:
https://github.com/freekh/play-slick/tree/master/samples/play-slick-cake-sample
Take a look also to the other implementations without DI and compare:
https://github.com/freekh/play-slick/tree/master/samples/
And so the cake pattern doesn't have a centralized config like we can have with my shown lift style DI. I will anyway use the Cake pattern as it fits well with Slick.
What I didn't like with Subcut is the implicits everywhere. I know there is a way to avoid them but it looks like a fix to me.
Thanks
To comment on MacWire, you are right that you could just use new - and that's the whole point :). MacWire is there only to let you remove some boilerplate from your code, by not having to enumerate all the dependencies again (which is already done in the constructor).
The main idea is that you do the wiring at "the end of the world", where you assemble your application (or you could divide that into trait-modules, but that's optional). Otherwise you just use constructors to express dependencies. No magic, no frameworks.