I am trying to write a DSL with Scala. I would initially like to be able to write things like
defType "foo"
when using it.
I had thought that the following should work:
src/main/scala/Test.scala
class Dsl {
def defType(name: String) = "dummy"
}
object Dsl {
def apply() = new Dsl()
}
class UseDsl {
def foo() = {
val dsl = Dsl()
import dsl._
dsl defType "foo"
defType("foo")
defType "foo"
}
}
This fails to compile:
[error] Test.scala:15:17: ';' expected but string literal found.
[error] defType "foo"
[error] ^
[error] one error found
[error] (Compile / compileIncremental) Compilation failed
Explicitly giving dsl works with spaces to separate the method names and arguments.
Implicitly using dsl and parentheses to indicate arguments vs method names works.
Attempting to use both of them together fails.
Is there a way to get that to work?
Once this is working, I plan to extend the DSL to support things like
defType "foo"
-- "bar1" :: "quz"
-- "bar2" :: "quz"
which would be equivalent to
dsl.defType("foo").
--(ImplicitClass("bar1", dsl).::("quz")).
--(ImplicitClass("bar2", dsl).::("quz"))
Is this something that I will be able to get to work? I think the ImplicitClass would work with a declaration like
def implicit ImplicitClass(a: String, implicit dsl: Dsl) = ...
but clearly, my understanding of how you can get Scala to add things to your code is imperfect.
If it won't work, what are some minimal additions that would let it work?
build.sbt
ThisBuild / organization := "test"
ThisBuild / version := "0.0.1-SNAPSHOT"
ThisBuild / scalaVersion := "2.12.8"
//
// Projects
//
lazy val root = (project in file("."))
No, method argument is not valid. For infix method calls without parentheses, you have to do
val1 method1 val2 method2 val3 ...
This chain can end in a method without arguments or in the last method's argument, and the first one is not too safe.
Even for members of current type, you need to do this method1 ... and can't omit this as in this.method1(...).
Related
In a Scala 3 macro that takes a type parameter T, you can use TypeRepr.of[T] and the new Scala 3 reflection API to explore the companionClass of T, and find the Symbol for an arbitrary method on that companion class (eg companionClass.declarations.find(_.name == "list") to find a list() method).
Given the Symbol for a companion object method, how would you then invoke that method within a quoted code block?
I'm guessing I would need to convert that Symbol to a Expr[T], but I don't know how to do that!
In a Scala 2 macro, the invocation of a listMethod of type c.universe.Symbol in a q"..." quasiquote seems pretty simple - just say $listMethod, and then you can start mapping on the resulting list, eg:
q"""
$listMethod.map(_.toString)
"""
Trying to do a similar thing in a Scala 3 macro gets an error like this:
[error] 27 | ${listMethod}.map(_.toString)
[error] | ^^^^^^^^^^
[error] | Found: (listMethod : x$1.reflect.Symbol)
[error] | Required: quoted.Expr[Any]
What is the correct code to get this working in Scala 3?
You can see more code context in the AvroSerialisableMacro classes (Scala 2 compiles, Scala 3 currently nowhere near!) here: https://github.com/guardian/marley/pull/77/files
First, let's talk how to call a method using symbol name in general.
You might need Select. You can call obtain it in a a few different ways, e.g.:
New(TypeTree.of[YourType]).select(primaryConstructor) // when you want to create something
expression.asTerm.select(method) // when you want to call it on something
Once you selected method you can provide arguments:
select.appliedToArgs(args) // if there is only 1 argument list
select.appliedToArgss(args) // if there is more than one argument list
// (type parameter list is listed in paramSymss
// but shouldn't be used here, so filter it out!)
select.appliedToNone // if this is a method like "def method(): T"
// (single, but empty, parameter list)
select.appliedToArgss(Nil) // is this is a method like "def method: T"
// (with not even empty parameter list)
There are also other methods like appliedToType, appliedToTypeTrees, but if you have a method name as a Symbol and want to use it to call something this should be a good starting point.
And remember that source code of Quotes is your friend, so even when your IDE doesn't give you any suggestions, it can point you towards some solution.
In theory these methods are defined on Term rather than Select (<: Term) but your use case will be most likely picking an expression and calling a method on it with some parameters. So a full example could be e.g.
val expression: Expr[Input]
val method: Symbol
val args: List[Term]
// (input: Input).method(args) : Output
expression // Expr[Input]
.asTerm // Term
.select(method) // Select
.appliedToArgs(args) // Term
.asExpr // Expr[?]
.asExprOf[Output] // Expr[Output]
Obviously, proving that the expression can call method and making sure that types of Terms in args match allowed types of values that you pass to the method, is on you. It is a bit more hassle than it was in Scala 2 since quotes allow you to work with Type[T] and Expr[T] only, so anything that doesn't fall under that category has to be implemented with macros/Tasty ADT until you get to the point that you can return Expr inside ${}.
That said, the example you linked shows that these calls are rather hardcoded, so you don't have to look up Symbols and call them. Your code will most likely do away with:
// T <: ThriftEnum
// Creating companion's Expr can be done with asExprOf called on
// Ref from Dmytro Mitin's answer
def findCompanionOfThisOrParent(): Expr[ThriftEnumObject[T]] = ...
// _Expr_ with the List value you constructed instead of Symbol!
val listOfValues: Expr[List[T]] = '{
${ findCompanionOfThisOrParent() }.list
}
// once you have an Expr you don't have to do any magic
// to call a method on it, Quotes works nice
'{
...
val valueMap = Map(${ listOfValues }.map(x => x ->
org.apache.avro.generic.GenericData.get.createEnum(
com.gu.marley.enumsymbols.SnakesOnACamel.toSnake(x.name), schemaInstance)
): _*)
...
}
Difference between Scala 2 quasiquotes and Scala 3 quotations is that the former must compile during compile time of the main code using macros (i.e. during macro expansion, macro runtime) while the latter must compile earlier, at macro compile time. So Scala 3 quotations '{...}/${...} are more like Scala 2 reify{...}/.splice than Scala 2 quasiquotes q"..."/${...}.
`tq` equivalent in Scala 3 macros
You have to re-create AST. Let's see what shape AST should have:
object B:
def fff(): Unit = ()
import scala.quoted.*
inline def foo(): Unit = ${fooImpl}
def fooImpl(using Quotes): Expr[Unit] =
import quotes.reflect.*
println('{B.fff()}.asTerm.show(using Printer.TreeStructure))
'{()}
foo() // ... Apply(Select(Ident("B"), "fff"), Nil)
So in order to re-create AST try to use Apply(...) and Select.unique(..., "list"):
import scala.quoted.*
inline def foo[T](): Unit = ${fooImpl[T]}
def fooImpl[T: Type](using Quotes): Expr[Unit] =
import quotes.reflect.*
val sym = TypeRepr.of[T].typeSymbol
'{
println("aaa")
${
Apply(
Select.unique(
Ref(sym.companionModule),
"list"
),
Nil
).asExprOf[Unit]
}
}
Testing (in a different file):
class A
object A {
def list(): Unit = println("list")
}
foo[A]()
//scalac: {
// scala.Predef.println("aaa")
// A.list()
//}
// prints at runtime:
// aaa
// list
Using method symbol rather than its name and using convenience methods rather than AST nodes directly, you can rewrite fooImpl as
def fooImpl[T: Type](using Quotes): Expr[Unit] =
import quotes.reflect.*
val sym = TypeRepr.of[T].typeSymbol
val listMethod = sym.companionClass.declarations.find(_.name == "list").get
'{
println("aaa")
${
Ref(sym.companionModule)
.select(listMethod)
.appliedToArgs(Nil)
.asExprOf[Unit]
}
}
This is just an example how to create an AST. You should use your actual return type of def list() instead of Unit in .asExprOf[Unit].
How to get the list of default fields values for typed case class?
scala 3 macro how to implement generic trait
Here is a simple example in shapeless:
it("from Witness") {
val ttg = implicitly[TypeTag[Witness.Lt[String]]]
val ctg = implicitly[ClassTag[Witness.Lt[String]]]
}
it("... from macro") {
val ttg = implicitly[TypeTag[Witness.`1`.T]]
val ctg = implicitly[ClassTag[Witness.`1`.T]]
}
it("... doesn't work") {
val ttg = implicitly[TypeTag[w1.T]] // failed!
val ctg = implicitly[ClassTag[w2.T]]
}
The second and third it block have very similar bytecode (after the Witness.? macro has been invoked), yet one succeed and one failed:
[Error] /home/peng/git-spike/scalaspike/common/src/test/scala/com/tribbloids/spike/scala_spike/Reflection/InferTypeTag.scala:55: No TypeTag available for com.tribbloids.spike.scala_spike.Reflection.InferTypeTag.w1.T
What could have caused this? And how do I circumvent this problem?
If you switch on scalacOptions += "-Xlog-implicits" you'll see
val w1 = Witness(1)
val ttg = implicitly[TypeTag[w1.T]] // doesn't compile
//materializing requested reflect.runtime.universe.type.TypeTag[App.w1.T] using scala.reflect.api.`package`.materializeTypeTag[App.w1.T](scala.reflect.runtime.`package`.universe)
//scala.reflect.api.`package`.materializeTypeTag[App.w1.T](scala.reflect.runtime.`package`.universe) is not a valid implicit value for reflect.runtime.universe.TypeTag[App.w1.T] because:
//failed to typecheck the materialized tag:
//cannot create a TypeTag referring to type shapeless.Witness.<refinement>.T local to the reifee: use WeakTypeTag instead
//No TypeTag available for App.w1.T
So try to use WeakTypeTag as recommended
val w1 = Witness(1)
val ttg2 = implicitly[WeakTypeTag[w1.T]] // compiles
In Scala, why it is impossible to infer TypeTag from type alias or dependent type?
Why there is no TypeTag available in nested instantiations (when interpreted by scala code runner)?
Typetags not working inside of code block scope?
Type aliases screw up type tags?
How can I prevent the usage of a specific implicit in my scala code?
For example, I was recently bit by the default Codec provided by https://github.com/scala/scala/blob/68bad81726d15d03a843dc476d52cbbaf52fb168/src/library/scala/io/Codec.scala#L76.
Is there a way to ensure that any code that calls for an implicit codec: Codec never uses the one provided by fallbackSystemCodec?
Alternatively, is it possible to block all implicit Codecs?
Is this something that should be doable using scalafix?
Scalafix can inspect implicit arguments using SemanticTree. Here is an example solution by defining a custom scalafix rule.
Given
import scala.io.Codec
object Hello {
def foo(implicit codec: Codec) = 3
foo
}
we can define a custom rule
class ExcludedImplicitsRule(config: ExcludedImplicitsRuleConfig)
extends SemanticRule("ExcludedImplicitsRule") {
...
override def fix(implicit doc: SemanticDocument): Patch = {
doc.tree.collect {
case term: Term if term.synthetic.isDefined => // TODO: Use ApplyTree(func, args)
val struct = term.synthetic.structure
val isImplicit = struct.contains("implicit")
val excludedImplicit = config.blacklist.find(struct.contains)
if (isImplicit && excludedImplicit.isDefined)
Patch.lint(ExcludedImplicitsDiagnostic(term, excludedImplicit.getOrElse(config.blacklist.mkString(","))))
else
Patch.empty
}.asPatch
}
}
and corresponding .scalafix.conf
rule = ExcludedImplicitsRule
ExcludedImplicitsRuleConfig.blacklist = [
fallbackSystemCodec
]
should enable sbt scalafix to raise the diagnostic
[error] /Users/mario/IdeaProjects/scalafix-exclude-implicits/example-project/scalafix-exclude-implicits-example/src/main/scala/example/Hello.scala:7:3: error: [ExcludedImplicitsRule] Attempting to pass excluded implicit fallbackSystemCodec to foo'
[error] foo
[error] ^^^
[error] (Compile / scalafix) scalafix.sbt.ScalafixFailed: LinterError
Note the output of println(term.synthetic.structure)
Some(ApplyTree(
OriginalTree(Term.Name("foo")),
List(
IdTree(SymbolInformation(scala/io/LowPriorityCodecImplicits#fallbackSystemCodec. => implicit lazy val method fallbackSystemCodec: Codec))
)
))
Clearly the above solution is not efficient as it searches strings, however it should give some direction. Perhaps matching on ApplyTree(func, args) would be better.
scalafix-exclude-implicits-example shows how to configure the project to use ExcludedImplicitsRule.
You can do this by using a new type altogether; this way, nobody will be able to override it in your dependencies. It's essentially the answer I posted to create an ambiguous low priority implicit
It may not be practical though, if for example you can't change the type.
I have the following example build.sbt that uses sbt-assembly. (My assembly.sbt and project/assembly.sbt are set up as described in the readme.)
import AssemblyKeys._
organization := "com.example"
name := "hello-sbt"
version := "1.0"
scalaVersion := "2.10.3"
val hello = taskKey[Unit]("Prints hello")
hello := println(s"hello, ${assembly.value.getName}")
val hey = taskKey[Unit]("Prints hey")
hey <<= assembly map { (asm) => println(s"hey, ${asm.getName}") }
//val hi = taskKey[Unit]("Prints hi")
//hi <<= assembly { (asm) => println(s"hi, $asm") }
Both hello and hey are functionally equivalent, and when I run either task from sbt, they run assembly first and print a message with the same filename. Is there a meaningful difference between the two? (It seems like the definition of hello is "slightly magical", since the dependency on assembly is only implied there, not explicit.)
Lastly, I'm trying to understand why hey needs the map call. Obviously it results in a different object getting passed into asm, but I'm not quite sure how to fix this type error in the definition of hi:
sbt-hello/build.sbt:21: error: type mismatch;
found : Unit
required: sbt.Task[Unit]
hi <<= assembly { (asm) => println(s"hi, $asm") }
^
[error] Type error in expression
It looks like assembly here is a [sbt.TaskKey[java.io.File]][2] but I don't see a map method defined there, so I can't quite figure out what's happening in the type of hey above.
sbt 0.12 syntax vs sbt 0.13 syntax
Is there a meaningful difference between the two?
By meaningful difference, if you mean semantic difference as in observable difference in the behavior of the compiled code, they are the same.
If you mean, any intended differences in code, it's about the style difference between sbt 0.12 syntax sbt 0.13 syntax. Conceptually, I think sbt 0.13 syntax makes it easier to learn and code since you deal with T instead of Initialize[T] directly. Using macro, sbt 0.13 expands x.value into sbt 0.12 equivalent.
anatomy of <<=
I'm trying to understand why hey needs the map call.
That's actually one of the difference macro now is able to handle automatically.
To understand why map is needed in sbt 0.12 style, you need to understand the type of sbt DSL expression, which is Setting[_]. As Getting Started guide puts it:
Instead, the build definition creates a huge list of objects with type Setting[T] where T is the type of the value in the map. A Setting describes a transformation to the map, such as adding a new key-value pair or appending to an existing value.
For tasks, the type of DSL expression is Setting[Task[T]]. To turn a setting key into Setting[T], or to turn a task key into Setting[Task[T]], you use <<= method defined on respective keys. This is implemented in Structure.scala (sbt 0.12 code base has simpler implementation of <<= so I'll be using that as the reference.):
sealed trait SettingKey[T] extends ScopedTaskable[T] with KeyedInitialize[T] with Scoped.ScopingSetting[SettingKey[T]] with Scoped.DefinableSetting[T] with Scoped.ListSetting[T, Id] { ... }
sealed trait TaskKey[T] extends ScopedTaskable[T] with KeyedInitialize[Task[T]] with Scoped.ScopingSetting[TaskKey[T]] with Scoped.ListSetting[T, Task] with Scoped.DefinableTask[T] { ... }
object Scoped {
sealed trait DefinableSetting[T] {
final def <<= (app: Initialize[T]): Setting[T] = setting(scopedKey, app)
...
}
sealed trait DefinableTask[T] { self: TaskKey[T] =>
def <<= (app: Initialize[Task[T]]): Setting[Task[T]] = Project.setting(scopedKey, app)
...
}
}
Note the types of app parameters. Setting key's <<= takes Initialize[T] whereas the task key's <<= takes Initialize[Task[T]]. In other words, depending on the the type of lhs of an <<= expression the type of rhs changes. This requires sbt 0.12 users to be aware of the setting/task difference in the keys.
Suppose you have a setting key like description on the lhs, and suppose you want to depend on name setting and create a description. To create a setting dependency expression you use apply:
description <<= name { n => n + " is good." }
apply for a single key is implemented in Settings.scala:
sealed trait Keyed[S, T] extends Initialize[T]
{
def transform: S => T
final def apply[Z](g: T => Z): Initialize[Z] = new GetValue(scopedKey, g compose transform)
}
trait KeyedInitialize[T] extends Keyed[T, T] {
final val transform = idFun[T]
}
Next, instead of description, suppose you want to create a setting for jarName in assembly. This is a task key, so rhs of <<= takes Initialize[Task[T]], so apply is not good. This is where map comes in:
jarName in assembly <<= name map { n => n + ".jar" }
This is implemented in Structure.scala as well:
final class RichInitialize[S](init: Initialize[S]) {
def map[T](f: S => T): Initialize[Task[T]] = init(s => mktask(f(s)) )
}
Because a setting key extends KeyedInitialize[T], which is Initialize[T], and because there's an implicit conversion from Initialize[T] to RichInitialize[T] the above is available to name. This is an odd way of defining map since maps usually preserves the structure.
It might make more sense, if you see similar enrichment class for task keys:
final class RichInitializeTask[S](i: Initialize[Task[S]]) extends RichInitTaskBase[S, Task] {...}
sealed abstract class RichInitTaskBase[S, R[_]] {
def map[T](f: S => T): Initialize[R[T]] = mapR(f compose successM)
}
So for tasks, map maps from a task of type S to T. For settings, we can think of it as: map is not defined on a setting, so it implicitly converts itself to a task and maps that. In any case, this let's sbt 0.12 users to think: Use apply for settings, map for tasks. Note that apply ever goes away for task keys as they extend Keyed[Task[T], Task[T]]. This should explain:
sbt-hello/build.sbt:21: error: type mismatch;
found : Unit
required: sbt.Task[Unit]
Then there's the tuple issue. So far I've discussed dependencies to a single setting. If you want to depend on more, sbt implicitly adds apply and map to Tuple2..N to handle it. Now it's expanded to 15, but it used to be up till only Tuple9. Seeing from a new user's point of view, the idea of invoking map on a Tuple9 of settings so it generates a task-like Initialize[Task[T]] would appear alien. Without changing the underlying mechanism, sbt 0.13 provides much cleaner surface to get started.
I'm trying to use the new scope filtering API on tasks to execute all tests for a project and it's dependencies.
val select = ScopeFilter(inDependencies(p), inConfigurations(Test))
val agg = executeTests.all(select).map(aggregateTestOutput)
executeTests in Test := agg.value
But I'm getting the error
[error] Runtime reference to undefined setting:
[error]
[error] proj/test:executeTests from proj/test:executeTests
With ScopeFilter(inDependencies(p, includeRoot=false), inConfigurations(Test)), it will run the tests from the projects dependencies. I've tried even just making the scope filter ScopeFilter(inProject(p), inConfigurations(Test)) and it fails with the same error.
What do I need to do to make this work?
It could be a problem with the dead code elimination in the settings processor. all is implemented with flatMap, so all of the dependencies aren't statically known. The dead code elimination incorrectly discards the default executeTests because it isn't used statically. Try explicitly referencing the previous value, like:
val select = ScopeFilter(inDependencies(p, includeRoot=false), inConfigurations(Test))
val agg = executeTests.all(select)
executeTests in Test := {
val outs = (executeTests in Test).value +: agg.value
aggregateTestOutput(outs)
}