I'm writing a system that is deployed in several places and each site needs its own configurations and settings. A "configuration" is a named value that is necessary to a particular site (e.g., the database URL, S3 bucket name); every configuration is necessary, there is not usually a default, and it's typically string-valued. A setting is a named value but it just tweaks the behavior of the system; it's often numeric or Boolean, and there's usually some default.
So far, I've been using property files or thing like them, but it's a terrible solution. Several times, a developer has added a requirement for a configuration but not added the value to file for the live configuration, so the new release passed all the tests, then failed when released to live.
Better, of course, for every file to be compiled — so if there's a missing configuration, or one of the wrong type, it won't get past the compiler — and inject the site-specific class into the build for each site. As a bones, a Scala file can easy model more complex values, especially lists, but also maps and tuples.
The downside is, the files are sometimes maintained by people who aren't developers, so it has to be pretty self-explanatory, which was the advantage of property files. (Someone explain XML configurations to me: all the complexity of a compilable file but the run-time risk of a property file.)
What I'm looking for is an easy pattern for defining a group required names and allowable values. Any suggestions?
How about using Mixin composition? Since traits are applied from right to left we can:
Define traits:
default property:
trait PropertyA {
val userName = "default"
def useUserName() : Unit = {
println(userName)
}
}
some other property:
trait SomePropertyA extends PropertyA {
val userName = "non-default"
def useUserName() : Unit = {
println(userName)
}
}
Define an abstract class with default property:
trait HasPropertyA {
val prop : PropertyA = new PropertyA
}
Define an abstract class with non - default property:
trait HasSomeOtherPropertyA extends HasPropertyA {
override val prop:PropertyA = new SomePropertyA {}
}
In your class use default one:
trait MyClass extends PropertyA {
doSomethingWith(prop.userName)
}
or in the other situation mix it with some other property:
if(env.isSome) {
val otherProps = MyClass with HasSomeOtherPropertyA
doSomethingWith(prop.userName)// userName == non-default!
}
Read in more detail in the paper Scalable Component Abstractions
Although Lift is a web framework in its essence, it has some utilities as well. One of them is Dependency Injection, see : http://simply.liftweb.net/index-8.2.html#toc-Section-8.2. So you can for example create a base trait with default values and then subclass the Runtime, Development, Test... environment values. And I think that it's easy for someone without knowledge of scala to put some override def defaultValue = "new value" in a file.
Related
I have a complex project which reads configurations from a DB through the object ConfigAccessor which implements two basic APIs: getConfig(name: String) and storeConfig(c: Config).
Due to how the project is currently designed, almost every component needs to use the ConfigAccessor to talk with the DB. Thus, being this component an object it is easy to just import it and call its static methods.
Now I am trying to build some unit tests for the project in which the configurations are stored in a in-memory hashMap. So, first of all I decoupled the config accessor logic from its storage (using the cake pattern). In this way I can define my own ConfigDbComponent while testing
class ConfigAccessor {
this: ConfigDbComponent =>
...
The "problem" is that now ConfigAccessor is a class, which means I have to instantiate it at the beginning of my application and pass it everywhere to whoever needs it. The first way I can think of for passing this instance around would be through other components constructors. This would become quite verbose (adding a parameter to every constructor in the project).
What do you suggest me to do? Is there a way to use some design pattern to overcome this verbosity or some external mocking library would be more suitable for this?
Yes, the "right" way is passing it in constructors. You can reduce verbosity by providing a default argument:
class Foo(config: ConfigAccessor = ConfigAccessor) { ... }
There are some "dependency injection" frameworks, like guice or spring, built around this, but I won't go there, because I am not a fan.
You could also continue utilizing the cake pattern:
trait Configuration {
def config: ConfigAccessor
}
trait Foo { self: Configuration => ... }
class FooProd extends Foo with ProConfig
class FooTest extends Foo with TestConfig
Alternatively, use the "static setter". It minimizes changes to existing code, but requires mutable state, which is really frowned upon in scala:
object Config extends ConfigAccessor {
#volatile private var accessor: ConfigAccessor = _
def configurate(cfg: ConfigAccessor) = synchronized {
val old = accessor
accessor = cfg
old
}
def getConfig(c: String) = Option(accessor).fold(
throw new IllegalStateException("Not configurated!")
)(_.getConfig(c))
You can retain a global ConfigAccessor and allow selectable accessors like this:
object ConfigAccessor {
private lazy val accessor = GetConfigAccessor()
def getConfig(name: String) = accessor.getConfig(name)
...
}
For production builds you can put logic in GetConfigAccessor to select the appropriate accessor based on some global config such as typesafe config.
For unit testing you can have a different version of GetConfigAccessor for different test builds which return the appropriate test implementation.
Making this value lazy allows you to control the order of initialisation and if necessary do some non-functional mutable stuff in the initialisation code before creating the components.
Update following comments
The production code would have an implementation of GetConfigAccessor something like this:
object GetConfigAccessor {
private val useAws = System.getProperties.getProperty("accessor.aws") == "true"
def apply(): ConfigAccessor =
if (useAws) {
return new AwsConfigAccessor
} else {
return new PostgresConfigAccessor
}
}
Both AwsConfigAccessor and PostgresConfigAccessor would have their own unit tests to prove that they conform to the correct behaviour. The appropriate accessor can be selected at runtime by setting the appropriate system property.
For unit testing there would be a simpler implementation of GetConfigAccessor, something like this:
def GetConfigAccessor() = new MockConfigAccessor
Unit testing is done within a unit testing framework which contains a number of libraries and mock objects that are not part of the production code. These are built separately and are not compiled into the final product. So this version of GetConfigAccessor would be part of that unit testing code and would not be part of the final product.
Having said all that, I would only use this model for reading static configuration data because that keeps the code functional. The ConfigAccessor is just a convenient way to access global constants without having them passed down in the constructor.
If you are also writing data then this is more like a real DB than a configuration. In that case I would create custom accessors for each component that give access to different parts of the DB. That way it is clear which parts of the data are updated by each component. These accessors would be passed down to the component and can then be unit tested with the appropriate mock implementation as normal.
You may need to partition your data into static config and dynamic config and handle them separately.
I am building a market simulator using Scala/Akka/Play. I have an Akka actor with two children. The children need to have specific types which I would like to specify as parameters.
Suppose that I have the following class definition...
case class SecuritiesMarket[A <: AuctionMechanismLike, C <: ClearingMechanismLike](instrument: Security) extends Actor
with ActorLogging {
val auctionMechanism: ActorRef = context.actorOf(Props[A], "auction-mechanism")
val clearingMechanism: ActorRef = context.actorOf(Props[C], "clearing-mechanism")
def receive: Receive = {
case order: OrderLike => auctionMechanism forward order
case fill: FillLike => clearingMechanism forward fill
}
}
Instances of this class can be created as follows...
val stockMarket = SecuritiesMarket[DoubleAuctionMechanism, CCPClearingMechanism](Security("GOOG"))
val derivativesMarket = SecuritiesMarket[BatchAuctionMechanism, BilateralClearingMechanism](Security("SomeDerivative"))
There are many possible combinations of auction mechanism types and clearing mechanism types that I might use when creating SecuritiesMarket instance for a particular model/simulation.
Can I specify the type parameters that I wish to use in a given simulation in the application.conf file?
I see two questions here.
Can I get a Class instance from a String?
Yes.
val cls: Class[DoubleAuctionMechanism] = Class.forName("your.app.DoubleAuctionMechanism").asInstanceOf[Class[DoubleAuctionMechanism]]
You would still need the cast, as forName returns Class[_].
Can I instantiate a type with type parameters are not known compile time?
Well sort of, but not really.
object SecuritiesMarket {
def apply[A, C](clsAuc: Class[A], clsClr: Class[C])(security: Security): SecuritiesMarket[A, C] = {
SecuritiesMarket[A, C](security)
}
}
I think auction mechanisms and clearing mechanisms are dependencies for SecurityMarket. I'm guessing you instantiate them in its constructor somehow (how?). If that's the case why not just pass them in as a constructor parameter?
Edit:
I don't see how I could create the child actors inside SecurityMarket
Answering this in the comments; Props[T] can also be written as Props[T](classOfT), which can be simplified as Props(classOfT). Those three are the same. So the following code:
val auctionMechanism: ActorRef = context.actorOf(Props[A], "auction-mechanism")
Can be replaced with:
val classOfA = Class.forName("path.to.A")
val auctionMechanism: ActorRef = context.actorOf(Props(classOfA), "auction-mechanism")
First, application.conf is a runtime artifact and its contents are as far as I know not normally parsed at compile time. When the file is parsed at runtime, the parser creates an instance of the class Config which then controls the Akka setup.
The Typesafe Config library project readme is quite nice and the linked documentation has all of the details:
https://github.com/typesafehub/config/blob/master/README.md.
Second, since template parameters are not available at runtime because of type erasure, you can't normally use application.conf to control templating. You could create a custom build step to parse application.conf and modify your code before compilation, but this is maybe not what you want. (And if you do want a custom build step, perhaps a different .conf would be appropriate.)
Instead you might try simply eliminating the type parameters for the securities market class. Then create a single, simple implementation of the auction and clearing actors. Implement these actors by reading the names of the respective mechanisms from application.conf, instantiating the configured mechanism reflectively, and delegating to the instantiated mechanism. The mechanism classes could be independent of Akka, which is perhaps nice if that's where you keep most of your logic?
I am building a large agent-based / multi-agent model of a stock exchange using Akka/Play/Scala, etc and I am struggling a bit to understand how to configure my application. Below is a snippet of code that illustrates an example of the type of problem I face:
class Exchange extends Actor {
val orderRoutingLogic = new OrderRoutingLogic()
val router = {
val marketsForSecurities = securities.foreach { security =>
val marketForSecurity = context.actorOf(Props[DoubleAuctionMarket](
new DoubleAuctionMarket(security) with BasicMatchingEngine), security.name
)
orderRoutingLogic.addMarket(security, marketForSecurity)
}
Router(orderRoutingLogic)
}
In the snippet above I inject a BasicMatchingEngine into the DoubleAuctionMarket. However I have written a number of different matching engines and I would like to be able to configure the type of matching engine injected into DoubleAuctionMarket in the application configuration file.
Can this level of application configuration be done using typesafe Config and HOCON configuration files?
interesting case. If I understood you right, you want to configure Market actor mixing in some MatchingEngine type specified in config?
Some clarification: you can't simply mix in dynamic type. I mean if you move MatchingEngine type to config - it will be known only at runtime, when config is parsed. And at that time you'll not be able to instantiate new DoubleAuctionMarket(security) with ???SomeClassInstance???. But maybe you could replace inheritance with aggregation. Maybe an instance of MatchingEngine can be passed to Market as parameter?
Now, how to obtain an instance of MatchingEngine from config? In short - Typesafe Config has no parser for FQCN properties, but it's not hard to do it yourself using reflection. This technique is used in many places in Akka. Look here first. provider property set as fqcn string and can be changed to other provider (i.e. RemoteActorRefProvider) in other configurations. Now look at how it's processed to obtain Provider instance. First it's just being read as string here. Then ProviderClass is used to instantiate actual (runtime) provider here. DynamicAccess is a utility helping with reflexive calls. It's not publicly accessible via context.system, but just take a piece of it or instantiate yourself, I don't think it's a big issue.
With some modifications, your code may look:
class Exchange extends Actor {
val orderRoutingLogic = new OrderRoutingLogic()
val matchingEngineClass = context.system.settings.config.getString("stocks.matching-engine")
val matchingEngine = DynamicAccess.createInstance[MatchingEngine](matchingEngineClass)
val router = {
val marketsForSecurities = securities.foreach { security =>
val marketForSecurity = context.actorOf(DoubleAuctionMarket.props(security, matchingEngine))
orderRoutingLogic.addMarket(security, marketForSecurity)
}
Router(orderRoutingLogic)
}
I've moved props to companion object of DoubleAuctionMarket as stated in Recommended Preactices of akka docs. Usage of Props(new Actor()) is dangerous practice.
I've been trying to write a Scala (2.10.0) compiler plugin that analyzes some parts of a traversed code.
This is what I originally had:
class MyPlugin (val global: Global) extends Plugin {
import global._
val name = "myPlugin"
val components = List[PluginComponent](MyComponent)
private object MyComponent extends PluginComponent {
val global: MyPlugin.this.global.type = MyPlugin.this.global
val runsAfter = List ("refchecks")
val phaseName = "codeAnalysis"
def newPhase (_prev: Phase) = new AnalysisPhase (_prev)
class AnalysisPhase (prev: Phase) extends StdPhase (prev) {
override def name = phaseName
def apply (unit: CompilationUnit) {
codeTraverser traverse unit.body
printLinesToFile(counters.map{case (k,v) => k + "\t" + v},out)
}
def codeTraverser = new ForeachTreeTraverser (tree => /* Analyze tree */)
}
}
}
This code works as expected, however I don't like it because I cannot decouple the code traverser method from this object. I would like to write a separate CodeTraverser class that will perform the analysis on a given Tree. This, among other things may help me test this code better.
The main problem is that unit.body is of an internal Tree type inside scala.reflect.internal.Trees. If I could work with scala.reflect.api.Trees#Tree instead of the internal version I could decouple the traverser functionality and even test it quite easily.
I've tried to find a way to convert between the two, but to no avail. Is it even possible? From looking at their source code, many things looks too similar for this to be impossible.
You're probably struggling with the cake pattern that the compiler is implemented with and a lot of path-dependency that comes with it. I've gone through this some time ago when I was writing some really beefy macro and wanted to refactor a bunch of functions out of macro implementation into separate utility class. I found this to be quite an annoying issue.
Here's how I would implement your Traverser in a separate class:
class MyPluginUtils[G <: Global with Singleton](global: G) {
import global._
class AnalyzingTraverser extends ForeachTreeTraverser(tree => /* analyze */)
}
Now, inside your plugin you have to use it like this:
val utils = new MyPluginUtils[global.type](global)
import utils.{global => _, _}
val traverser = new AnalyzingTraverser
As you can see, it's not the most intuitive thing in the world (i.e. this is confusing as hell), but this is the best that I could come up with that actually worked, and I tried a lot of things before finally settling on this one. I would be really happy to see some nicer way to do this.
AFAIK, such extensibility is one of the general problems with the cake pattern (as used in scalac implementation). I've seen other people also complain about this.
The component (the SubComponent or PluginComponent) must be created with the global member initialized early (that is, as an early definition).
Don't forget to review the one-question faq. I may go set google calendar to remind me to do that every Monday morning.
For an example, see the continuations plugin.
The component is defined with a utility class mixed in.
The utility class follows the usual cake recipe. (Leave it as an abstract dependency and let the compiler ensure that everything was mixed correctly.)
Here is a recent edit showing more early definitions, as a demonstration that this usage is not anomalous.
val anfPhase = new {
val global = SelectiveCPSPlugin.this.global
val cpsEnabled = pluginEnabled
override val enabled = cpsEnabled
} with SelectiveANFTransform {
val runsAfter = List("pickler")
}
(In future, they plan to deprecate early definitions in favor of parameterized traits when they are available in the language.)
More generally, global, i.e., "the compiler", is routinely instantiated for testing the compiler itself. I haven't seen it mocked, but computeInternalPhases is the template method for picking what phases are assembled modulo plugins.
There is a current effort to reduce internal dependencies, for the purpose of testing, as a window onto the difficulties involved.
I know it's not directly possible to serialize a function/anonymous class to the database but what are the alternatives? Do you know any useful approach to this?
To present my situation: I want to award a user "badges" based on his scores. So I have different types of badges that can be easily defined by extending this class:
class BadgeType(id:Long, name:String, detector:Function1[List[UserScore],Boolean])
The detector member is a function that walks the list of scores and return true if the User qualifies for a badge of this type.
The problem is that each time I want to add/edit/modify a badge type I need to edit the source code, recompile the whole thing and re-deploy the server. It would be much more useful if I could persist all BadgeType instances to a database. But how to do that?
The only thing that comes to mind is to have the body of the function as a script (ex: Groovy) that is evaluated at runtime.
Another approach (that does not involve a database) might be to have each badge type into a jar that I can somehow hot-deploy at runtime, which I guess is how a plugin-system might work.
What do you think?
My very brief advice is that if you want this to be truly data-driven, you need to implement a rules DSL and an interpreter. The rules are what get saved to the database, and the interpreter takes a rule instance and evaluates it against some context.
But that's overkill most of the time. You're better off having a little snippet of actual Scala code that implements the rule for each badge, give them unique IDs, then store the IDs in the database.
e.g.:
trait BadgeEval extends Function1[User,Boolean] {
def badgeId: Int
}
object Badge1234 extends BadgeEval {
def badgeId = 1234
def apply(user: User) = {
user.isSufficientlyAwesome // && ...
}
}
You can either have a big whitelist of BadgeEval instances:
val weDontNeedNoStinkingBadges = Map(
1234 -> Badge1234,
5678 -> Badge5678,
// ...
}
def evaluator(id: Int): Option[BadgeEval] = weDontNeedNoStinkingBadges.get(id)
def doesUserGetBadge(user: User, id: Int) = evaluator(id).map(_(user)).getOrElse(false)
... or if you want to keep them decoupled, use reflection:
def badgeEvalClass(id: Int) = Class.forName("com.example.badge.Badge" + id + "$").asInstanceOf[Class[BadgeEval]]
... and if you're interested in runtime pluggability, try the service provider pattern.
You can try and use Scala Continuations - they can give you the ability to serialize the computation and run it at later time or even on another machine.
Some links:
Continuations
What are Scala continuations and why use them?
Swarm - Concurrency with Scala Continuations
Serialization relates to data rather than methods. You cannot serialize functionality because it is a class file which is designed to serialize that and object serialization serializes the fields of an object.
So like Alex says, you need a rule engine.
Try this one if you want something fairly simple, which is string based, so you can serialize the rules as strings in a database or file:
http://blog.maxant.co.uk/pebble/2011/11/12/1321129560000.html
Using a DSL has the same problems unless you interpret or compile the code at runtime.