What is a nice approach for collecting and using useful Scala utility functions across projects. The focus here on really simple, standalone functions like:
def toBinary(i: Int, digits: Int = 8) =
String.format("%" + digits + "s", i.toBinaryString).replace(' ', '0')
def concat(ss: String*) = ss filter (_.nonEmpty) mkString ", "
concat: (ss: String*)String
This question is basic, I know ;-) but, I've learned that there is always an optimum way to do something. For example, reusing code from within the Scala interactive shell, Idea, Eclipse, with or without SBT, having the library hosed on GitHub, ect, could quickly introduce optimal, and non-optimal approaches to such a simple problem.
You might want to put such methods in a package object.
You could also put them in a normal object and import everything in the object when you need those methods.
object Utilities {
def toBinary(i: Int, digits: Int = 8) = // ...
}
// Import everything in the Utilities object
import Utilities._
If you want it trivially accessible from everywhere, your best bet is actually to stick it inside the scala-library jar. Personally, I have all my custom stuff in /jvm/S.jar (or something like that) and add it to the classpath every time I need it.
Repacking the Scala library jar is really easy--unpack it, move your class hierarchy in, and pack it up again. (You should have it inside some package and/or package object.)
Related
I have an object in Scala that returns a Set[String]:
object MyObject {
val setOfStrings = Set[String]("string1","string2")
}
I can successfully refer to that val from my Python code (using Py4J) with this:
jvm.com.package.MyObject.setOfStrings()
however I can't figure out how to do anything useful with it. I want to convert it into a python set.
I managed to do it using this:
eval(
str(
jvm.com.package.MyObject.setOfStrings()
)
.replace("Set(", "{\"")
.replace(")", "\"}")
.replace(", ", "\", \"")
)
but that's a horrifically brittle way of achieving it and I'm assuming Py4J provides a much better way.
I'm aware that SetConverter can convert a python set to a Java set, but I want to do the opposite. Anyone know how?
You could use a java.util.Set on the Scala side, which will be converted by Py4J into the Python equivalent MutableSet.
If you don't want to use Java collections in your Scala code base, then you might want to convert a scala.collection.immutable.Set into its Java equivalent when you want to communicate with Python.
I didn't run this code, but you could probably do this:
object YourObject {
val setOfStrings = Set[String]("string1","string2")
}
object ViewObject {
import scala.jdk.CollectionConverters._
def setOfStrings: java.util.Set[String] = YourObject.setOfStrings.asJava
}
and use it in Python:
jvm.com.package.ViewObject.setOfStrings()
Note that in this case, you won't be able to modify the collection. If you want to be able to update the view in Python and have the changes reflected back in Scala, then you'll have to use a scala.collection.mutable.Set in your Scala code.
Is it possible to generate the with... type method with IntelliJ for Scala?
Example:
case class Person(name: String, age: Int)
I would like to find the tool to auto-generate the method of type:
def withName(name: String): Person = this.copy(name=name)
def withAge(age: Int): Person = this.copy(age=age)
is it possible?
Thank you.
There is no such thing out-of-the-box, but you can create a scala template of your own:
Select Settings/Preferences | Editor | Live Templates.
From options on the right, open the list of Scala templates.
Click + to add a new template.
You can see an example here
I you want to auto-generate these methods rather than writing them explicitly (even with an IntelliJ template), you can do that with an annotation macro that will run at compile-time.
In particular, you can check the scalameta project for informations about this. Note however that macros are an experimental feature that is likely to change in trivial ways when Scala 3 is released. In my opinion, you should think hard about whether writing withName(name) rather than copy(name=name) is worth the trouble of defining all these methods (whether its manually, through IntelliJ snippets, or using macros), and only go for macros if it will save you a lot of trouble down the line.
I would like to extract from a given Scala project, the call graph of all methods which are part of the project's own source.
As I understand, the presentation compiler doesn't enable that, and it requires going down all the way down to the actual compiler (or a compiler plugin?).
Can you suggest complete code, that would safely work for most scala projects but those that use the wackiest dynamic language features? for the call graph, I mean a directed (possibly cyclic) graph comprising class/trait + method vertices where an edge A -> B indicates that A may call B.
Calls to/from libraries should be avoided or "marked" as being outside the project's own source.
EDIT:
See my macro paradise derived prototype solution, based on #dk14's lead, as an answer below. Hosted on github at https://github.com/matanster/sbt-example-paradise.
Here's the working prototype, which prints the necessary underlying data to the console as a proof of concept. http://goo.gl/oeshdx.
How This Works
I have adapted the concepts from #dk14 on top boilerplate from macro paradise.
Macro paradise lets you define an annotation that will apply your macro over any annotated object in your source code. From there you have access to the AST that the compiler generates for the source, and scala reflection api can be used to explore the type information of the AST elements. Quasiquotes (the etymology is from haskell or something) are used to match the AST for the relevant elements.
More about Quasiquotes
The generally important thing to note is that quasiquotes work over an AST, but they are a strange-at-first-glance api and not a direct representation of the AST (!). The AST is picked up for you by paradise's macro annotation, and then quasiquotes are the tool for exploring the AST at hand: you match, slice and dice the abstract syntax tree using quasiquotes.
The practical thing to note about quasiquotes is that there are fixed quasiquote templates for matching each type of scala AST - a template for a scala class definition, a template for a scala method definition, etc. These tempaltes are all provided here, making it very simple to match and deconstruct the AST at hand to its interesting constituents. While the templates may look daunting at first glance, they are mostly just templates mimicking the scala syntax, and you may freely change the $ prepended variable names within them to names that feel nicer to your taste.
I still need to further hone the quasiquote matches I use, which currently aren't perfect. However, my code seems to produce the desired result for many cases, and honing the matches to 95% precision may be well doable.
Sample Output
found class B
class B has method doB
found object DefaultExpander
object DefaultExpander has method foo
object DefaultExpander has method apply
which calls Console on object scala of type package scala
which calls foo on object DefaultExpander.this of type object DefaultExpander
which calls <init> on object new A of type class A
which calls doA on object a of type class A
which calls <init> on object new B of type class B
which calls doB on object b of type class B
which calls mkString on object tags.map[String, Seq[String]](((tag: logTag) => "[".+(Util.getObjectName(tag)).+("]")))(collection.this.Seq.canBuildFrom[String]) of type trait Seq
which calls map on object tags of type trait Seq
which calls $plus on object "[".+(Util.getObjectName(tag)) of type class String
which calls $plus on object "[" of type class String
which calls getObjectName on object Util of type object Util
which calls canBuildFrom on object collection.this.Seq of type object Seq
which calls Seq on object collection.this of type package collection
.
.
.
It is easy to see how callers and callees can be correlated from this data, and how call targets outside the project's source can be filtered or marked out. This is all for scala 2.11. Using this code, one will need to prepend an annotation to each class/object/etc in each source file.
The challenges that remain are mostly:
Challenges remaining:
This crashes after getting the job done. Hinging on https://github.com/scalamacros/paradise/issues/67
Need to find a way to ultimately apply the magic to entire source files without manually annotating each class and object with the static annotation. This is rather minor for now, and admittedly, there are benefits for being able to control classes to include and ignore anyway. A preprocessing stage that implants the annotation before (almost) every top level source file definition, would be one nice solution.
Honing the matchers such that all and only relevant definitions are matched - to make this general and solid beyond my simplistic and cursory testing.
Alternative Approach to Ponder
acyclic brings to mind a quite opposite approach that still sticks to the realm of the scala compiler - it inspects all symbols generated for the source, by the compiler (as much as I gather from the source). What it does is check for cyclic references (see the repo for a detailed definition). Each symbol supposedly has enough information attached to it, to derive the graph of references that acyclic needs to generate.
A solution inspired by this approach may, if feasible, locate the parent "owner" of every symbol rather than focus on the graph of source files connections as acyclic itself does. Thus with some effort it would recover the class/object ownership of each method. Not sure if this design would not computationally explode, nor how to deterministically obtain the class encompassing each symbol.
The upside would be that there is no need for macro annotations here. The downside is that this cannot sprinkle runtime instrumentation as the macro paradise rather easily allows, which could be at times useful.
It requires more precise analysis, but as a start this simple macro will print all possible applyies, but it requires macro-paradise and all traced classess should have #trace annotation:
class trace extends StaticAnnotation {
def macroTransform(annottees: Any*) = macro tracerMacro.impl
}
object tracerMacro {
def impl(c: Context)(annottees: c.Expr[Any]*): c.Expr[Any] = {
import c.universe._
val inputs = annottees.map(_.tree).toList
def analizeBody(name: String, method: String, body: c.Tree) = body.foreach {
case q"$expr(..$exprss)" => println(name + "." + method + ": " + expr)
case _ =>
}
val output = inputs.head match {
case q"class $name extends $parent { ..$body }" =>
q"""
class $name extends $parent {
..${
body.map {
case x#q"def $method[..$tt] (..$params): $typ = $body" =>
analizeBody(name.toString, method.toString, body)
x
case x#q"def $method[..$tt]: $typ = $body" =>
analizeBody(name.toString, method.toString, body)
x
}
}
}
"""
case x => sys.error(x.toString)
}
c.Expr[Any](output)
}
}
Input:
#trace class MyF {
def call(param: Int): Int = {
call2(param)
if(true) call3(param) else cl()
}
def call2(oaram: Int) = ???
def cl() = 5
def call3(param2: Int) = ???
}
Output (as compiler's warnings, but you may output to file intead of println):
Warning:scalac: MyF.call: call2
Warning:scalac: MyF.call: call3
Warning:scalac: MyF.call: cl
Of course, you might want to c.typeCheck(input) it (as now expr.tpe on found trees is equals null) and find which class this calling method belongs to actually, so the resulting code may not be so trivial.
P.S. macroAnnotations give you unchecked tree (as it's on earlier compiler stage than regular macroses), so if you want something typechecked - the best way is surround the piece of code you want to typecheck with call of some your regular macro, and process it inside this macro (you can even pass some static parameters). Every regular macro inside tree produced by macro-annotation - will be executed as usual.
Edit
The basic idea in this answer was to bypass the (pretty complex) Scala compiler completely, and extract the graph from the generated .class files in the end. It appeared that a decompiler with sufficiently verbose output could reduce the problem to basic text manipulation. However, after a more detailed examination it turned out that this is not the case. One would just get back to square one, but with obfuscated Java code instead of the original Scala code. So this proposal does not really work, although there is some rationale behind working with the final .class files instead of intermediate structures used internally by the Scala compiler.
/Edit
I don't know whether there are tools out there that do it out of the box (I assume that you have checked that). I have only a very rough idea what the presentation compiler is. But if all that you want is to extract a graph with methods as nodes and potential calls of methods as edges, I have a proposal for a quick-and-dirty solution. This would work only if you want to use it for some sort of visualization, it doesn't help you at all if you want to perform some clever refactoring operations.
In case that you want to attempt building such a graph-generator yourself, it might turn out much simpler than you think. But for this, you need to go all the way down, even past the compiler. Just grab your compiled .class files, and use something like the CFR java decompiler on it.
When used on a single compiled .class file, CFR will generate list of classes that the current class depends on (here I use my little pet project as example):
import akka.actor.Actor;
import akka.actor.ActorContext;
import akka.actor.ActorLogging;
import akka.actor.ActorPath;
import akka.actor.ActorRef;
import akka.actor.Props;
import akka.actor.ScalaActorRef;
import akka.actor.SupervisorStrategy;
import akka.actor.package;
import akka.event.LoggingAdapter;
import akka.pattern.PipeToSupport;
import akka.pattern.package;
import scala.Function1;
import scala.None;
import scala.Option;
import scala.PartialFunction;
...
(very long list with all the classes this one depends on)
...
import scavenger.backend.worker.WorkerCache$class;
import scavenger.backend.worker.WorkerScheduler;
import scavenger.backend.worker.WorkerScheduler$class;
import scavenger.categories.formalccc.Elem;
Then it will spit out some horribly looking code, that might look like this (small excerpt):
public PartialFunction<Object, BoxedUnit> handleLocalResponses() {
return SimpleComputationExecutor.class.handleLocalResponses((SimpleComputationExecutor)this);
}
public Context provideComputationContext() {
return ContextProvider.class.provideComputationContext((ContextProvider)this);
}
public ActorRef scavenger$backend$worker$MasterJoin$$_master() {
return this.scavenger$backend$worker$MasterJoin$$_master;
}
#TraitSetter
public void scavenger$backend$worker$MasterJoin$$_master_$eq(ActorRef x$1) {
this.scavenger$backend$worker$MasterJoin$$_master = x$1;
}
public ActorRef scavenger$backend$worker$MasterJoin$$_masterProxy() {
return this.scavenger$backend$worker$MasterJoin$$_masterProxy;
}
#TraitSetter
public void scavenger$backend$worker$MasterJoin$$_masterProxy_$eq(ActorRef x$1) {
this.scavenger$backend$worker$MasterJoin$$_masterProxy = x$1;
}
public ActorRef master() {
return MasterJoin$class.master((MasterJoin)this);
}
What one should notice here is that all methods come with their full signature, including the class in which they are defined, for example:
Scheduler.class.schedule(...)
ContextProvider.class.provideComputationContext(...)
SimpleComputationExecutor.class.fulfillPromise(...)
SimpleComputationExecutor.class.computeHere(...)
SimpleComputationExecutor.class.handleLocalResponses(...)
So if you need a quick-and-dirty solution, it might well be that you could get away with just ~10 lines of awk,grep,sort and uniq wizardry to get nice adjacency lists with all your classes as nodes and methods as edges.
I've never tried it, it's just an idea. I cannot guarantee that Java decompilers work well on Scala code.
There are 2 reasons for me to ask:
1. I'd like a better code fragmentation to facilitate version control on per-function level
2. I struggle from some attention deficit disorder and it is hard for me to work with long pieces of code such as big class files
To address these problems I used to use include directives in C++ and partial class definitions and manually-definable foldable regions in C#. Are there any such things available in Scala 2.8?
I've tried to use editor-fold tag in NetBeans IDE, but it does not work in Scala editor unfortunately :-(
UPDATE: As far as I understand, there are no such facilities in Scala. So I'd like to ask: someone who has any connection to Scala authors, or an account on their Bugzilla (or whatever they use), please, suggest them an idea - they should probably think of introducing something of such (I was fascinated by C# regions and partial classes for example, and plain old includes also look like a convenient tool to have) to make Scala even more beautiful through laconicity, IMHO.
How about doing it with traits? You define it like this:
trait Similarity
{
def isSimilar(x: Any): Boolean
def isNotSimilar(x: Any): Boolean = !isSimilar(x)
}
...and then you use it like so:
class Point(xc: Int, yc: Int) extends Similarity
{
var x: Int = xc
var y: Int = yc
def isSimilar(obj: Any) =
obj.isInstanceOf[Point] &&
obj.asInstanceOf[Point].x == x
}
If the class Point were bigger, you could split it further into traits, resulting in the division that you want. Please note, however, that I don't think this is advisable, as it will make it very difficult to get a good overview of your code, unless you already know it by heart. If you can break it in a nice way, however, you might be able to get some nice, reusable blocks out of it, so in the end it might still be worth doing.
Best of luck to you!
//file A.scala
trait A { self: B =>
....
}
//file B.scala
trait B { self: A =>
....
}
//file C.scala
class C extends A with B
I suggest to read white paper by Martin at this link. In this white paper 'Case Sudy: The Scala Compiler' chapter will give you idea about how you can achieve component based design having code in several separate files.
Scala code folding works properly in IDEA.
The version control tools I work with (bzr or git, mostly) have no trouble isolating changes line-by-line. What use case do you have--that's common enough to worry about--where line-level isolation (which allows changes to independent methods to be merged without user intervention) is not enough?
Also, if you can't focus on something as large as one class with many methods, use more classes. A method generally requires you to know what the other methods are, what the fields are, and so on. Having that split across separate files is just asking for trouble. Instead, encapsulate your problem in a different way so you can work with smaller self-contained chunks at a time.
What kind of constructs need 'scalac' compile and how to make an equivalent that will work in interpreter?
Edit: I want to use scala instead of python as a scripting language. (with #!/usr/bin/scala)
You ought to be able to do anything in the REPL that you can do in outside code. Keep in mind that:
Things that refer to each other circularly need to be inside one block. So the following can't be entered as-is; you have to wrap it inside some other object:
class C(i : Int) { def succ = C(i+1) }
object C { def apply(i: Int) = new C(i) }
The execution environment is somewhat different, so benchmarking timings will not always come out the same way as if you run them from compiled code.
You enter the execution path a different way; if you want to call a main method, though, you certainly can from inside the REPL.
You can't just cut-and-paste an entire library into the REPL and have it work exactly like the library did; the REPL has a different structure than normal packages do. So drop the "package" declarations during testing.
EDIT: after reading your question again, I have to admit, that I didn't really answer it ;). But maybe it still helps.
I know of two limitations of interpreter (or REPL), when it comes to loading scala files (in order to interactively test them).
You can't load scala files with package definitions in them. REPL complains and does not load all class is the scala file to be loaded. It read that it has to do with the fact that files that are loaded into the REPL are treated as an object . . . which of course can't have any package definitions in them.
REPL is strange (or a little bit unpredictable) when there are class files of loaded scala files on the classpath. Check out this question by myself and especially my 2 last comments to the second answer.
Furthermore there is a problem with circular dependencies, that I don't know a workaround for: Suppose there is a Class A that uses Class B which again needs A to do it's job. Of course you can't define A since there is no definition of B and vice versa. Providing a dummy for one of those doesn't work either:
scala> class A {
| def alterString(s:String) = s
| def printStuff(s:String) = println(alterString(s))
| }
defined class A
scala> class B {
| val prefix = "this is a test: "
| def doJob() = new A() printStuff "1 2 3"
| }
defined class B
scala> class A {
| def alterString(s:String) = new B().prefix + s
| def printStuff(s:String) = println(alterString(s))
| }
defined class A
scala> new B().doJob()
1 2 3
scala>
Although I already provided a newer definition of A, class B still used the one that was present when I defined it.
I think everything in Scala will work from the interpreter as well (which I believe just calls the compiler under the hood anyway).
Do you suspect something to not work? Or is this a trick/interview question (I suppose anything that want to directly interact with the class loader or class files may behave differently in the two environments, but I have no idea really).