I would like to write a macro that compiles code that it receives as a String literal and detect a typecheck error in the compiled code that is due to a macro expansion failure (either the macro was aborted, or the expanded macro failed to typecheck)
I was thinking something like this:
def myMacro(c: Context)(codeStringLiteral: c.Expr[String]): c.Expr[Unit] = {
val codeString = getString(codeStringLiteral) // this part is easy
val ast = c.parse(code)
val actualCode = util.Try(c.typecheck(ast)).recover{ case t: TypecheckException =>
if(t.isMacroExpansionFailure) doOneThing
else doOtherThing
}
c.Expr(actualCode.get)
}
Is this possible?
Context
Such a macro would make testing other macros much more pleasant by deferring to runtime a failure that is due to a macro expansion, thus allowing one to execute the entire test suite even when a test case for your macro is broken.
Of course, it is easy enough to simply differ typechecking entirely to runtime, but it would be really nifty to only differ errors that are due to the macro you are writing under test and fail at compile time if it's the test code itself that is at fault.
Of course, it's possible for an unrelated macro to fail, but it's unlikely to happen very often.
There's an obscure flag for c.typecheck, called withMacrosDisabled. If you pass true there, that should prevent any macro from being expanded. Now you could compare the status of c.typecheck(withMacrosDisabled = false) and c.typecheck(withMacrosDisabled = true) and dispatch accordingly.
That won't work for whitebox macros, because withMacrosDisabled = false may make legit code using whitebox macros to fail typechecking, but for blackbox macros it should be more or less okay.
Related
I'm trying to perform the equivalent of the eval method from Python in nim.
I was under the impression that parseStmt from the macros package should help me with this, but I'm facing a compilation issue that I don't understand.
import macros
echo parseStmt("1 + 2")
I would have expected this to print 3 when executed, but instead the compilation complains that
Error: request to generate code for .compileTime proc: $
I found this thread, and the examples there work, and following this, I was able to make the following program that works as I would expect:
import macros
import strformat
macro eval(value: string): untyped =
result = parseStmt fmt"{value}"
echo eval("1+2")
But I don't undertand why it needs to be written in this way at all. If I inline the statement, let value = "1 + 2"; echo parseStmt fmt"{value}", I get the same compile error as above.
Also, why is parseStmt value different from parseStmt fmt"{value}", in the context of the eval macro above?
What am I missing here?
Thank you in advance for any clarifications!
Unlike Python which is an interpreted language, Nim is compiled. This means that all code is parsed and turned into machine code on compile-time and the program that you end up with doesn't really know anything about Nim at all (at least as long as you don't import the Nim compiler as a module, which is possible). So parseStmt and all macro/template expansion stuff in Nim is done completely during compilation. The error, although maybe a bit hard to read, is trying to tell you that what was passed to $ (which is the convert-to-string operator in Nim, and called by echo on all its arguments) is a compile-time thing that can't be used on runtime. In this case it's because parseStmt doesn't return "3", it returns something like NimNode(kind: nnkIntLit, intVal: 3), and the NimNode type is only available during compile-time. Nim however allows you to run code on compile-time to return other code, this is what a macro does. The eval macro you wrote there takes value which is any statement that resolves to a string during runtime, passed as a NimNode. This is also why result = parseStmt value won't work in your case, because value is not yet a string, but could be something like reading a string from standard input which would result in a string during runtime. However the use of strformat here is a bit confusing and overkill. If you change your code to:
import macros
macro eval(value: static[string]): untyped =
result = parseStmt value
echo eval("1+2")
It will work just fine. This is because we have now told Nim that value needs to be a static i.e. known during compile-time. In this case the string literal "1+2" is obviously known at compile-time, but this could also be a call to a compile-time procedure, or even the output of staticRead which reads a file during compilation.
As you can see Nim is very powerful, but the barrier between compile-time and run-time can sometimes be a bit confusing. Note also that your eval procedure doesn't actually evaluate anything at compile-time, it simply returns the Nim code 1 + 2 so your code ends up being echo 1 + 2. If you want to actually run the code at compile-time you might want to look into compile-time procedures.
Hope this helps shed some light on your issue.
Note: while this answer outlines why this happens, keep in mind that what you're trying to do probably won't result in what you want (which I assumed to be runtime evaluation of expressions).
You're trying to pass a NimNode to parseStmt which expects a string. The fmt macro automatically stringifies anything in the {}, you can omit the fmt by doing $value to turn the node into a string.
As I already noted, this will not work as it does in Python: Nim does not have runtime evaluation. The expression in the string is going to be evaluated at compile time, so a simple example like this will not do what you want:
import std/rdstdin
let x = readLineFromStdin(">")
echo eval(x)
Why?
First of all, because you're stringifying the AST you pass to the eval, it's not the string behind the x variable that's going to get passed to the macro - it's going to be the symbol that denotes the x variable. If you stringify a symbol, you get the underlying identifier, which means that parseStmt will receive "x" as its parameter. This will effect in the string stored in x being printed out, which is wrong.
What you want instead is the following:
import std/rdstdin
import std/macros
macro eval(value: static string): untyped =
result = parseStmt(value)
echo eval("1 + 2")
This prevents runtime-known values from being passed to the macro. You can only pass consts and literals to it now, which is the correct behavior.
I've written a macro which generates an ADT, and would like to be able to write a test proving that I've done so in a manner which allows exhaustivity checking to work. Both specs2 (via typecheck) and Shapeless (via illTyped) allow one to test whether code is well typed or not, however this doesn't seem to get me anywhere for detecting a non-exhaustive pattern match (even when compiling with -Xfatal-warnings).
Is there a way to detect a non-exhaustive pattern match that I can use in my test?
As far as I understand it, exhaustivity checking is not really a feature of the type system, i.e. if you look at the typing rules, its not part of them, but something done on top of a type-checked program.
I guess what you could do, is to use a macro to generate code as part of your test cases. Your tests would then fail during compilation of the test suit. Or you use the compiler api to compile the code during test execution. It should be possible to get those warnings there.
I figured out that you can do this using the toolbox API:
import scala.reflect.runtime.currentMirror
import scala.tools.reflect.ToolBox
import scala.reflect.runtime.universe._
val code = q"""
val s: Option[Int] = Some(1)
s match{case Some(x)=>}
"""
val toolbox = currentMirror.mkToolBox(new tools.reflect.FrontEnd {
override def display(info: Info) = println(info)
override def interactive() {}
})
toolbox.eval(code)
This prints the following warning
Info(NoPosition,match may not be exhaustive.
It would fail on the following input: None,WARNING)
You can also parse the code you want to test-compile from some string or build the AST up splicing parts into quasiquotes or whatever else suits you.
I wonder if this:
object Foo {
val regex = "some complex regex".r
def foo() {
// use regex
}
}
and this:
object Foo {
def foo() {
val regex = "some complex regex".r
// use regex
}
}
will have any performance difference. i.e., will scala compiler recognize that "some complex regex".r is a constant and cache it, so that it will not recompile every time?
It will have a difference in runtime. Expression from first example will be calculated only once. Expression from second - every time you call Foo.foo(). Calculation here means applying implicitly added function "r" (from scala-library) to the string:
scala> ".*".r
res40: scala.util.matching.Regex = .*
This function actually compiles the regular expression every time you call it (no caching).
Btw, any naive caching of regexps in runtime is vulnerable to OutOfMemory - however, I believe it's possible to implement it safely with WeakHashMap, but current Java's Pattern implementation (which is underlying to scala's Regex) doesn't implement it actually, probably because such implementation may not have predictable effect on performance (GC may have to remove most of cached values every time it's running). Cache with eviction is more predictable, but still not so easy way (who's gonna choose timeout/size for it?). Talking about scala-way, some smart macro could do optimization in compile-time (do caching only for 'string constant'-based regexps), but by default:
Scala compiler also doesn't have any optimizations about regexps because regexp is not a part of the scala language.
So it's better to move static "".r constructions out of the function.
What is the proposed way of performing tests on scala macros?
I realize that one needs two projects due to the necessity of separate compilation. This step, if necessary, is acceptable and mostly clear.
But how do you assert a macro expansion fails when it should? Without some special facility, the test case won't compile and therefore the whole test project won't compile.
I think this assert would require another macro of the form
errors(code: => _): List[CompileError]
which returns the compile errors of the inner macro. The same would be required for testing that warnings occur if they should and so on...
Are there some existing testing facilities for Scala macros?
You can use assertDoesNotCompile from ScalaTest. Here is an example of using this to test the Scala typechecker:
import org.scalatest.FunSuite
class TypeTest extends FunSuite {
test("String vals can't hold ints") {
assertDoesNotCompile("val x: String = 1")
}
}
You can pass a string containing an example of when your macro expansion should fail to assertDoesNotCompile. Note that that there is also assertCompiles, if you feel the need for that.
you could use twitter eval to check if the code copiles
https://github.com/twitter/util/blob/master/util-eval/src/main/scala/com/twitter/util/Eval.scala#L247
With intention of reducing the boilerplate for the end-user when deriving instances of a certain typeclass (let's take Showable for example), I aim to write a macro, which will autogenerate the instance names. E.g.:
// calling this:
Showable.derive[Int]
Showable.derive[String]
// should expand to this:
// implicit val derivedShowableInstance0 = new Showable[Int] { ... }
// implicit val derivedShowableInstance1 = new Showable[String] { ... }
I tried to approach the problem the following way, but the compiler complained that the expression should return a < no type > instead of Unit:
object Showable {
def derive[a] = macro Macros.derive[a]
object Macros {
private var instanceNameIndex = 0
def derive[ a : c.WeakTypeTag ]( c : Context ) = {
import c.universe._
import Flag._
val newInstanceDeclaration = ...
c.Expr[Unit](
ValDef(
Modifiers(IMPLICIT),
newTermName("derivedShowableInstance" + nameIndex),
TypeTree(),
newInstanceDeclaration
)
)
}
}
}
I get that a val declaration is not exactly an expression and hence Unit might not be appropriate, but then how to make it right?
Is this at all possible? If not then why, will it ever be, and are there any workarounds?
Yes, that's right. Declarations/definitions aren't expressions, so they need to be wrapped into Unit-returning blocks to become ones. Typically Scala does this automatically, but in this particular case you need to do it yourself.
However if you wrap a definition in a block, then it becomes invisible from the outside. That's the limitation of the current macro system that strictly follows the metaphor of "macro application is very much similar to a typed function call". Function calls don't bring new members into scope, so neither do def macros - both for technical and philosophical reasons. As shown in the example #3 of my recent "What Are Macros Good For?" talk, by using structural types def macros can work around this limitation, but this doesn't look particularly related to your question, so I'll omit the details.
On the other hand, there are some ideas how to overcome this limitation with new macro flavors. Macro annotations show that it's technically feasible for the macro engine to hook into new member creation, but we'd like to get more experience with macro annotations before bringing them into trunk. Some details about this can be found in my "Philosophy of Scala Macros" presentation. This can be useful in your situation, but I still won't go into details, because I think there's a much better solution for this particular case (feel free to ask for elaboration in comments, though!).
What I'd like to recommend you is to use materialization as described in http://docs.scala-lang.org/overviews/macros/implicits.html. With materializing macros you can automatically generate type class instances for the user without having the user write any code at all. Would that fit your use case?