How should I organize implicits in my Scala application? - scala

Having written a few scala tools, I'm trying to come to grips with the best way to arrange my code - particularly implicits. I have 2 goals:
Sometimes, I want to be able to import just the implicits I ask for.
Othertimes, I want to just import everything.
To avoid duplicating the implicits, I've come up with this structure (similar to the way scalaz is arranged):
case class StringW(s : String) {
def contrived = s + "?"
}
trait StringWImplicits {
implicit def To(s : String) = StringW(s)
implicit def From(sw : StringW) = sw.s
}
object StringW extends StringWImplicits
// Elsewhere on Monkey Island
object World extends StringWImplicits with ListWImplicits with MoreImplicits
This allows me to just
import StringW._ // Selective import
or (in most cases)
import World._. // Import everything
How does everyone else do it?

I think that implicit conversions are dangerous if you don't know where they are coming from. In my case, I put my implicits in a Conversions class and import it as close to the use as possible
def someMethod(d: Date) ; Unit {
import mydate.Conversions._
val tz = TimeZone.getDefault
val timeOfDay = d.getTimeOfDay(tz) //implicit used here
...
}
I'm not sure I like "inheriting" implicits from various traits for the same reason it was considered bad Java practice to implement an interface so you could use its constants directly (static imports are preferred instead).

I usually had implicit conversions in an object which clearly signals that what it is imported is an implicit conversion.
For example, if I have a class com.foo.bar.FilthyRichString, the implicit conversions would go into com.foo.bar.implicit.FilthyRichStringImplicit. I know the names are a bit long, but that's why we have IDEs (and Scala IDE support is getting better). The way I do this is that I feel it is important that all the implicit conversions can be clearly viewed in a 10 second code review. I could look at the following code:
// other imports
import com.foo.bar.FilthyRichString
import com.foo.bar.util.Logger
import com.foo.bar.util.FileIO
import com.foo.bar.implicits.FilthyRichStringImplicit._
import com.foo.bar.implicits.MyListImplicit._
// other implicits
and at a glance see all the implicit conversions that are active in this source file. They would also be all gathered together, if you use the convention that imports are grouped by packages, with a new line between different packages.
Along the lines of the same argument, I wouldn't like a catch-all object that holds all of the implicit conversions. In a big project, would you really use all of the implicit conversions in all your source files? I think that doing that means very tight coupling between different parts of your code.
Also, a catch-all object is not very good for documentation. In the case of explicitly writing all the implicit conversions used in a file, one can just look at your import statements and straight away jump to the documentation of the implicit class. In the case of a catch-all object, one would have to look at that object (which in a big project might be huge) and then search for the implicit conversion they are after.
I agree with oxbow_lakes that having implicit conversion in traits is bad because of the temptation of inheriting from it, which is, as he said, bad practice. Along those lines, I would make the objects holding the implicit conversions final just to avoid the temptation altogether. His idea of importing them as close to the use as possible is very nice as well, if implicit conversions are just used sparingly in the code.
-- Flaviu Cipcigan

Related

A safe way of 'overwriting'/'forcing' a certain implicit resolution

I have a custom formatter (say MyFormat) for org.joda.DateTime which provides a Format[org.joda.DateTime].
Play provides a default formatter for the same class in the play.api.libs.json package.
I would like to use MyFormat across my application, and not the one Play provides. I have done this through explicit imports/mixins which does the trick, however during changes or omissions when these get removed, serialization defaults to the Play formatter and I end up with runtime errors. This seems very error prone.
Ultimately I would like for my code not to compile if there is no MyFormat for org.joda.DateTime in scope whenever one is required.
Is there a nice and safe way of doing this?
You have a couple options, but neither of them will be exactly what you want. The crux of the problem is that Scala will look for implicits within related companion objects. For example, if it needs to find an implicit Format[DateTime], it will look within the companion object of Format and the companion object of DateTime. In this case, there is an implicit def within Format that will produce the desired (or in your case, undesired) Format[DateTime] and there's nothing you can do to get rid of it, short of forking and hacking Play.
Workaround 1: Create an object with your custom formatters, and import it globally:
package my.project
object Formats {
implicit val dateTimeFormat: Format[DateTime] = ???
}
(in other files)
import my.project.Formats._
The Format[DateTime] imported into a local scope will supersede the one resolved from the Format object. However, you can't prevent your code from compiling without it (unless you have some sort of scalastyle rule or something that requires the import).
Workaround 2: Create a wrapper for DateTime with it's own Format, and give it an implicit conversion to DateTime.
case class MyDateTime(dt: DateTime) {
implicit def toDateTime: DateTime = dt
}
object MyDateTime {
implicit val fmt: Format[MyDateTime] = ???
implicit def fromDateTime(dt: DateTime): MyDateTime = MyDateTime(dt)
}
This is not a complete example, but an idea as to how to implement this. DateTime can be swapped with MyDateTime is most places, however it may require other implicits revolving around MyDateTime. For example, when I tried using this with Anorm, I needed an implicit ToStatement[MyDateTime], and probably a few others--so that's the obvious disadvantage. However, this would explicitly avoid using the default Format[DateTime].

Extracting the complete call graph of a scala project (tough one)

I would like to extract from a given Scala project, the call graph of all methods which are part of the project's own source.
As I understand, the presentation compiler doesn't enable that, and it requires going down all the way down to the actual compiler (or a compiler plugin?).
Can you suggest complete code, that would safely work for most scala projects but those that use the wackiest dynamic language features? for the call graph, I mean a directed (possibly cyclic) graph comprising class/trait + method vertices where an edge A -> B indicates that A may call B.
Calls to/from libraries should be avoided or "marked" as being outside the project's own source.
EDIT:
See my macro paradise derived prototype solution, based on #dk14's lead, as an answer below. Hosted on github at https://github.com/matanster/sbt-example-paradise.
Here's the working prototype, which prints the necessary underlying data to the console as a proof of concept. http://goo.gl/oeshdx.
How This Works
I have adapted the concepts from #dk14 on top boilerplate from macro paradise.
Macro paradise lets you define an annotation that will apply your macro over any annotated object in your source code. From there you have access to the AST that the compiler generates for the source, and scala reflection api can be used to explore the type information of the AST elements. Quasiquotes (the etymology is from haskell or something) are used to match the AST for the relevant elements.
More about Quasiquotes
The generally important thing to note is that quasiquotes work over an AST, but they are a strange-at-first-glance api and not a direct representation of the AST (!). The AST is picked up for you by paradise's macro annotation, and then quasiquotes are the tool for exploring the AST at hand: you match, slice and dice the abstract syntax tree using quasiquotes.
The practical thing to note about quasiquotes is that there are fixed quasiquote templates for matching each type of scala AST - a template for a scala class definition, a template for a scala method definition, etc. These tempaltes are all provided here, making it very simple to match and deconstruct the AST at hand to its interesting constituents. While the templates may look daunting at first glance, they are mostly just templates mimicking the scala syntax, and you may freely change the $ prepended variable names within them to names that feel nicer to your taste.
I still need to further hone the quasiquote matches I use, which currently aren't perfect. However, my code seems to produce the desired result for many cases, and honing the matches to 95% precision may be well doable.
Sample Output
found class B
class B has method doB
found object DefaultExpander
object DefaultExpander has method foo
object DefaultExpander has method apply
which calls Console on object scala of type package scala
which calls foo on object DefaultExpander.this of type object DefaultExpander
which calls <init> on object new A of type class A
which calls doA on object a of type class A
which calls <init> on object new B of type class B
which calls doB on object b of type class B
which calls mkString on object tags.map[String, Seq[String]](((tag: logTag) => "[".+(Util.getObjectName(tag)).+("]")))(collection.this.Seq.canBuildFrom[String]) of type trait Seq
which calls map on object tags of type trait Seq
which calls $plus on object "[".+(Util.getObjectName(tag)) of type class String
which calls $plus on object "[" of type class String
which calls getObjectName on object Util of type object Util
which calls canBuildFrom on object collection.this.Seq of type object Seq
which calls Seq on object collection.this of type package collection
.
.
.
It is easy to see how callers and callees can be correlated from this data, and how call targets outside the project's source can be filtered or marked out. This is all for scala 2.11. Using this code, one will need to prepend an annotation to each class/object/etc in each source file.
The challenges that remain are mostly:
Challenges remaining:
This crashes after getting the job done. Hinging on https://github.com/scalamacros/paradise/issues/67
Need to find a way to ultimately apply the magic to entire source files without manually annotating each class and object with the static annotation. This is rather minor for now, and admittedly, there are benefits for being able to control classes to include and ignore anyway. A preprocessing stage that implants the annotation before (almost) every top level source file definition, would be one nice solution.
Honing the matchers such that all and only relevant definitions are matched - to make this general and solid beyond my simplistic and cursory testing.
Alternative Approach to Ponder
acyclic brings to mind a quite opposite approach that still sticks to the realm of the scala compiler - it inspects all symbols generated for the source, by the compiler (as much as I gather from the source). What it does is check for cyclic references (see the repo for a detailed definition). Each symbol supposedly has enough information attached to it, to derive the graph of references that acyclic needs to generate.
A solution inspired by this approach may, if feasible, locate the parent "owner" of every symbol rather than focus on the graph of source files connections as acyclic itself does. Thus with some effort it would recover the class/object ownership of each method. Not sure if this design would not computationally explode, nor how to deterministically obtain the class encompassing each symbol.
The upside would be that there is no need for macro annotations here. The downside is that this cannot sprinkle runtime instrumentation as the macro paradise rather easily allows, which could be at times useful.
It requires more precise analysis, but as a start this simple macro will print all possible applyies, but it requires macro-paradise and all traced classess should have #trace annotation:
class trace extends StaticAnnotation {
def macroTransform(annottees: Any*) = macro tracerMacro.impl
}
object tracerMacro {
def impl(c: Context)(annottees: c.Expr[Any]*): c.Expr[Any] = {
import c.universe._
val inputs = annottees.map(_.tree).toList
def analizeBody(name: String, method: String, body: c.Tree) = body.foreach {
case q"$expr(..$exprss)" => println(name + "." + method + ": " + expr)
case _ =>
}
val output = inputs.head match {
case q"class $name extends $parent { ..$body }" =>
q"""
class $name extends $parent {
..${
body.map {
case x#q"def $method[..$tt] (..$params): $typ = $body" =>
analizeBody(name.toString, method.toString, body)
x
case x#q"def $method[..$tt]: $typ = $body" =>
analizeBody(name.toString, method.toString, body)
x
}
}
}
"""
case x => sys.error(x.toString)
}
c.Expr[Any](output)
}
}
Input:
#trace class MyF {
def call(param: Int): Int = {
call2(param)
if(true) call3(param) else cl()
}
def call2(oaram: Int) = ???
def cl() = 5
def call3(param2: Int) = ???
}
Output (as compiler's warnings, but you may output to file intead of println):
Warning:scalac: MyF.call: call2
Warning:scalac: MyF.call: call3
Warning:scalac: MyF.call: cl
Of course, you might want to c.typeCheck(input) it (as now expr.tpe on found trees is equals null) and find which class this calling method belongs to actually, so the resulting code may not be so trivial.
P.S. macroAnnotations give you unchecked tree (as it's on earlier compiler stage than regular macroses), so if you want something typechecked - the best way is surround the piece of code you want to typecheck with call of some your regular macro, and process it inside this macro (you can even pass some static parameters). Every regular macro inside tree produced by macro-annotation - will be executed as usual.
Edit
The basic idea in this answer was to bypass the (pretty complex) Scala compiler completely, and extract the graph from the generated .class files in the end. It appeared that a decompiler with sufficiently verbose output could reduce the problem to basic text manipulation. However, after a more detailed examination it turned out that this is not the case. One would just get back to square one, but with obfuscated Java code instead of the original Scala code. So this proposal does not really work, although there is some rationale behind working with the final .class files instead of intermediate structures used internally by the Scala compiler.
/Edit
I don't know whether there are tools out there that do it out of the box (I assume that you have checked that). I have only a very rough idea what the presentation compiler is. But if all that you want is to extract a graph with methods as nodes and potential calls of methods as edges, I have a proposal for a quick-and-dirty solution. This would work only if you want to use it for some sort of visualization, it doesn't help you at all if you want to perform some clever refactoring operations.
In case that you want to attempt building such a graph-generator yourself, it might turn out much simpler than you think. But for this, you need to go all the way down, even past the compiler. Just grab your compiled .class files, and use something like the CFR java decompiler on it.
When used on a single compiled .class file, CFR will generate list of classes that the current class depends on (here I use my little pet project as example):
import akka.actor.Actor;
import akka.actor.ActorContext;
import akka.actor.ActorLogging;
import akka.actor.ActorPath;
import akka.actor.ActorRef;
import akka.actor.Props;
import akka.actor.ScalaActorRef;
import akka.actor.SupervisorStrategy;
import akka.actor.package;
import akka.event.LoggingAdapter;
import akka.pattern.PipeToSupport;
import akka.pattern.package;
import scala.Function1;
import scala.None;
import scala.Option;
import scala.PartialFunction;
...
(very long list with all the classes this one depends on)
...
import scavenger.backend.worker.WorkerCache$class;
import scavenger.backend.worker.WorkerScheduler;
import scavenger.backend.worker.WorkerScheduler$class;
import scavenger.categories.formalccc.Elem;
Then it will spit out some horribly looking code, that might look like this (small excerpt):
public PartialFunction<Object, BoxedUnit> handleLocalResponses() {
return SimpleComputationExecutor.class.handleLocalResponses((SimpleComputationExecutor)this);
}
public Context provideComputationContext() {
return ContextProvider.class.provideComputationContext((ContextProvider)this);
}
public ActorRef scavenger$backend$worker$MasterJoin$$_master() {
return this.scavenger$backend$worker$MasterJoin$$_master;
}
#TraitSetter
public void scavenger$backend$worker$MasterJoin$$_master_$eq(ActorRef x$1) {
this.scavenger$backend$worker$MasterJoin$$_master = x$1;
}
public ActorRef scavenger$backend$worker$MasterJoin$$_masterProxy() {
return this.scavenger$backend$worker$MasterJoin$$_masterProxy;
}
#TraitSetter
public void scavenger$backend$worker$MasterJoin$$_masterProxy_$eq(ActorRef x$1) {
this.scavenger$backend$worker$MasterJoin$$_masterProxy = x$1;
}
public ActorRef master() {
return MasterJoin$class.master((MasterJoin)this);
}
What one should notice here is that all methods come with their full signature, including the class in which they are defined, for example:
Scheduler.class.schedule(...)
ContextProvider.class.provideComputationContext(...)
SimpleComputationExecutor.class.fulfillPromise(...)
SimpleComputationExecutor.class.computeHere(...)
SimpleComputationExecutor.class.handleLocalResponses(...)
So if you need a quick-and-dirty solution, it might well be that you could get away with just ~10 lines of awk,grep,sort and uniq wizardry to get nice adjacency lists with all your classes as nodes and methods as edges.
I've never tried it, it's just an idea. I cannot guarantee that Java decompilers work well on Scala code.

Why global ExecutionContext is not a default parameter in future block?

That's really annoying when you write some highly concurrent code with Futures or Actors response and you have manually import ExecutionContext.Implicits.global. Tried to find some good explanation why it's not made as a default parameter, like it is done with Strategy in Scalaz Concurrent. That would be very helpful, not to insert/remove all those imports in the code. Or am i missing some logic?
The general trend seems to be to require the user to explicitly import things like implicits, extra operators or DSLs. I think this is a good thing since it makes things less "magic" and more understandable.
But nothing stops you from defining a package-wide implicit value for your code. Note that if the implicit ExecutionContext was always imported by default, you wouldn't be able to do this.
In the package object:
package object myawsomeconcurrencylibrary {
implicit def defaultExecutionContext = scala.concurrent.ExecutionContext.global
}
In any class in the same package:
package myawsomeconcurrencylibrary
object Bla {
future { ... } // implicit from package object is used unless you explicitly provide your own
}

how to understand the following scala call

I have a quite puzzling question. I am playing with squeryl, and found when I used:
package models
import org.squeryl.{Schema, KeyedEntity}
object db extends Schema {
val userTable = table[User]("User")
on(userTable)(b => declare(
b.email is(unique,indexed("idxEmailAddresses"))
))
}
I had to import import org.squeryl.PrimitiveTypeMode._
But this does not make sense to me. Here is is defined in org.squeryl.dsl.NonNumericalExpression, but why do I have to include the seemingly irrelevant import org.squeryl.PrimitiveTypeMode._?
Thank you.
I agree with #sschaef that this is due to required implicit conversions. When APIs (like squeryl) decide to build a DSL (domain specific language) in order to get a slick looking way to code against their API, implicit conversions will be required. The core API probably takes certain types of objects that it might be cumbersome/ugly to be instantiating directly in the code. Thus, they will use implicit conversions to do some of the lifting for you and keep the DSL as clean as possible. If you check out the Scaladoc for the PrimitiveTypeMode object, you can see the many implicit defs that are defined on it. Implicit conversions (used in pimping libraries) will 'upconvert' from one type into another to gain access to more functionality on the pimped out class. When the code is the implicit things are explicitly included into the final compiled code.
http://squeryl.org/api/index.html#org.squeryl.PrimitiveTypeMode$
Also, I believe the implicit conversion you are looking for is:
import org.squeryl.PrimitiveTypeMode.string2ScalarString
which is inherited from org.squeryl.dsl.QueryDsl.

How to import several implicit at once?

I have several implicit context for my application.
like
import scala.collection.JavaConversions._
import HadoopConversion._
etc
Right now I have to copy paste all those imports at each file. Is it possible to combine them in one file and make only one import?
A good technique that some libraries provide by default is bundling up implicits into a trait. This way you can compose sets of implicits by defining a trait that extends other implicit bundling traits. And then you can use it at the top of your scala file with the following.
import MyBundleOfImplicits._
Or be more selective by mixing it in only where you need it.
object Main extends App with MyBundleOfImplicits {
// ...
}
Unfortunately with something like JavaConversions, to use this method you will need to redefine all the implicits you want to use inside a trait.
trait JavaConversionsImplicits {
import java.{lang => jl}
import java.{util => ju}
import scala.collection.JavaConversions
implicit def asJavaIterable[A](i : Iterable[A]): jl.Iterable[A] = JavaConversions.asJavaIterable(i)
implicit def asJavaIterator[A](i : Iterator[A]): ju.Iterator[A] = JavaConversions.asJavaIterator(i)
}
trait MyBundleOfImplicits extends JavaConversionsImplicits with OtherImplicits
Scala does not have first-class imports. So the answer to your question is no. But there is an exception for the scala REPL. You can put all your imports in a file and then just tell the REPL where it is located. See this question.
The other answers/comments are already comprehensive. But if you just want to reduce COPY/PASTEs, all mainstream IDE/text-editors support text templating ('live template' in IntelliJ IDEA, 'template' in Eclipse, 'snippets' in TextMate ...) that will definitely make your life easier.