When using a scala macro, why the compilation error stacktrace looks so incoherent? - scala

Question

I'd like to experiment scala on some unreliable macro. So I mimic a simple error in shapeless invocation:
import shapeless.Witness
val v = Witness(Random.nextInt())
Then I compile it with gradle build --info --stacktrace, expecting to find an exception that starts with 'shapeless.Witness' or something similar. Instead I got this:
> Task :common:compileTestScala
[Error] /home/peng/git-spike/scalaspike/common/src/test/scala/com/tribbloids/spike/shapeless_spike/RefinedType/Creation.scala:12: Expression scala.util.Random.nextInt() does not evaluate to a constant or a stable reference value
one error found
> Task :common:compileTestScala FAILED
5 actionable tasks: 1 executed, 4 up-to-date
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':common:compileTestScala'.
> Compilation failed
* Try:
Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':common:compileTestScala'.
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$executeIfValid$1(ExecuteActionsTaskExecuter.java:205)
at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:263)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:203)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:184)
at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:114)
at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)
at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:62)
at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:56)
at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:416)
at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:406)
at org.gradle.internal.operations.DefaultBuildOperationExecutor$1.execute(DefaultBuildOperationExecutor.java:165)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:250)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:158)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:102)
at org.gradle.internal.operations.DelegatingBuildOperationExecutor.call(DelegatingBuildOperationExecutor.java:36)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:41)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:372)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:359)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:352)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:338)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.lambda$run$0(DefaultPlanExecutor.java:127)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:191)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.executeNextNode(DefaultPlanExecutor.java:182)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:124)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
Caused by: org.gradle.api.internal.tasks.compile.CompilationFailedException: Compilation failed
at org.gradle.api.internal.tasks.scala.ZincScalaCompiler.execute(ZincScalaCompiler.java:151)
at org.gradle.api.internal.tasks.scala.ZincScalaCompilerFacade.execute(ZincScalaCompilerFacade.java:47)
at org.gradle.api.internal.tasks.scala.ZincScalaCompilerFacade.execute(ZincScalaCompilerFacade.java:32)
at org.gradle.api.internal.tasks.compile.daemon.AbstractDaemonCompiler$CompilerWorkAction.execute(AbstractDaemonCompiler.java:135)
at org.gradle.workers.internal.DefaultWorkerServer.execute(DefaultWorkerServer.java:63)
at org.gradle.workers.internal.AbstractClassLoaderWorker$1.create(AbstractClassLoaderWorker.java:47)
at org.gradle.workers.internal.AbstractClassLoaderWorker$1.create(AbstractClassLoaderWorker.java:41)
at org.gradle.internal.classloader.ClassLoaderUtils.executeInClassloader(ClassLoaderUtils.java:98)
at org.gradle.workers.internal.AbstractClassLoaderWorker.executeInClassLoader(AbstractClassLoaderWorker.java:41)
at org.gradle.workers.internal.IsolatedClassloaderWorker.execute(IsolatedClassloaderWorker.java:49)
at org.gradle.workers.internal.WorkerDaemonServer.execute(WorkerDaemonServer.java:84)
at org.gradle.process.internal.worker.request.WorkerAction$1.call(WorkerAction.java:129)
at org.gradle.process.internal.worker.child.WorkerLogEventListener.withWorkerLoggingProtocol(WorkerLogEventListener.java:41)
at org.gradle.process.internal.worker.request.WorkerAction.run(WorkerAction.java:126)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:412)
... 3 more
Caused by: Compilation failed
at sbt.internal.inc.AnalyzingCompiler.call(AnalyzingCompiler.scala:253)
at sbt.internal.inc.AnalyzingCompiler.compile(AnalyzingCompiler.scala:122)
at sbt.internal.inc.AnalyzingCompiler.compile(AnalyzingCompiler.scala:95)
at sbt.internal.inc.MixedAnalyzingCompiler.$anonfun$compile$4(MixedAnalyzingCompiler.scala:91)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at sbt.internal.inc.MixedAnalyzingCompiler.timed(MixedAnalyzingCompiler.scala:186)
at sbt.internal.inc.MixedAnalyzingCompiler.$anonfun$compile$3(MixedAnalyzingCompiler.scala:82)
at sbt.internal.inc.MixedAnalyzingCompiler.$anonfun$compile$3$adapted(MixedAnalyzingCompiler.scala:77)
at sbt.internal.inc.JarUtils$.withPreviousJar(JarUtils.scala:215)
at sbt.internal.inc.MixedAnalyzingCompiler.compileScala$1(MixedAnalyzingCompiler.scala:77)
at sbt.internal.inc.MixedAnalyzingCompiler.compile(MixedAnalyzingCompiler.scala:146)
at sbt.internal.inc.IncrementalCompilerImpl.$anonfun$compileInternal$1(IncrementalCompilerImpl.scala:343)
at sbt.internal.inc.IncrementalCompilerImpl.$anonfun$compileInternal$1$adapted(IncrementalCompilerImpl.scala:343)
at sbt.internal.inc.Incremental$.doCompile(Incremental.scala:120)
at sbt.internal.inc.Incremental$.$anonfun$compile$4(Incremental.scala:100)
at sbt.internal.inc.IncrementalCommon.recompileClasses(IncrementalCommon.scala:180)
at sbt.internal.inc.IncrementalCommon.cycle(IncrementalCommon.scala:98)
at sbt.internal.inc.Incremental$.$anonfun$compile$3(Incremental.scala:102)
at sbt.internal.inc.Incremental$.manageClassfiles(Incremental.scala:155)
at sbt.internal.inc.Incremental$.compile(Incremental.scala:92)
at sbt.internal.inc.IncrementalCompile$.apply(Compile.scala:75)
at sbt.internal.inc.IncrementalCompilerImpl.compileInternal(IncrementalCompilerImpl.scala:348)
at sbt.internal.inc.IncrementalCompilerImpl.$anonfun$compileIncrementally$1(IncrementalCompilerImpl.scala:301)
at sbt.internal.inc.IncrementalCompilerImpl.handleCompilationError(IncrementalCompilerImpl.scala:168)
at sbt.internal.inc.IncrementalCompilerImpl.compileIncrementally(IncrementalCompilerImpl.scala:248)
at sbt.internal.inc.IncrementalCompilerImpl.compile(IncrementalCompilerImpl.scala:74)
at org.gradle.api.internal.tasks.scala.ZincScalaCompiler.execute(ZincScalaCompiler.java:147)
... 21 more
How is this supposed to help me? I can only see the code of shapeless.Witness, not the entire sbt.internal.inc.AnalyzingCompiler. What should I do to see the error that can be related to the macro that actually reported the error?
UPDATE 1 On behalf of Dmytro Mitin I'd like to also attach my build configurations:
allprojects {
apply(plugin = "java")
apply(plugin = "java-library")
apply(plugin = "scala")
...
dependencies {
implementation("org.scala-lang:scala-compiler:${vs.scalaV}")
implementation("org.scala-lang:scala-library:${vs.scalaV}")
implementation("org.scala-lang:scala-reflect:${vs.scalaV}")
<vs.scalaV=2.12.11>
...
<witness is introduced in s submodule>
}
tasks {
val jvmTarget = JavaVersion.VERSION_1_8.toString()
withType<ScalaCompile> {
targetCompatibility = jvmTarget
scalaCompileOptions.loggingLevel = "debug"
scalaCompileOptions.additionalParameters = listOf(
"-encoding", "utf8",
"-unchecked",
"-deprecation",
"-feature",
"-Xfatal-warnings",
"-Xlog-implicits",
"-Yissue-debug"
)
}
}
}
UPDATE 2 on maven:
[ERROR] /home/peng/git-dev/scala-2.11/datapassports/spookystuff/core/src/test/scala/com/tribbloids/spookystuff/utils/WitnessSpike.scala:11: Expression scala.util.Random.nextInt() does not evaluate to a constant or a stable reference value
[ERROR] Witness(Random.nextInt())
[ERROR] ^
...
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.4.6:testCompile (scala-test-compile-first) on project spookystuff-core: Execution scala-test-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.4.6:testCompile failed.: CompileFailed -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.4.6:testCompile (scala-test-compile-first) on project spookystuff-core: Execution scala-test-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.4.6:testCompile failed.
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:190)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:186)
at java.util.concurrent.FutureTask.run (FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:511)
at java.util.concurrent.FutureTask.run (FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
at java.lang.Thread.run (Thread.java:748)
Caused by: org.apache.maven.plugin.PluginExecutionException: Execution scala-test-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.4.6:testCompile failed.
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:148)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:190)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:186)
at java.util.concurrent.FutureTask.run (FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:511)
at java.util.concurrent.FutureTask.run (FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
at java.lang.Thread.run (Thread.java:748)
Caused by: sbt.compiler.CompileFailed
at sbt.compiler.AnalyzingCompiler.call (AnalyzingCompiler.scala:111)
at sbt.compiler.AnalyzingCompiler.compile (AnalyzingCompiler.scala:53)
at sbt.compiler.AnalyzingCompiler.compile (AnalyzingCompiler.scala:47)
at sbt.compiler.AggressiveCompile$$anonfun$3$$anonfun$compileScala$1$1.apply$mcV$sp (AggressiveCompile.scala:95)
at sbt.compiler.AggressiveCompile$$anonfun$3$$anonfun$compileScala$1$1.apply (AggressiveCompile.scala:95)
at sbt.compiler.AggressiveCompile$$anonfun$3$$anonfun$compileScala$1$1.apply (AggressiveCompile.scala:95)
at sbt.compiler.AggressiveCompile.sbt$compiler$AggressiveCompile$$timed (AggressiveCompile.scala:168)
at sbt.compiler.AggressiveCompile$$anonfun$3.compileScala$1 (AggressiveCompile.scala:94)
at sbt.compiler.AggressiveCompile$$anonfun$3.apply (AggressiveCompile.scala:142)
at sbt.compiler.AggressiveCompile$$anonfun$3.apply (AggressiveCompile.scala:84)
at sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply (Compile.scala:66)
at sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply (Compile.scala:64)
at sbt.inc.IncrementalCommon.cycle (IncrementalCommon.scala:32)
at sbt.inc.Incremental$$anonfun$1.apply (Incremental.scala:72)
at sbt.inc.Incremental$$anonfun$1.apply (Incremental.scala:71)
at sbt.inc.Incremental$.manageClassfiles (Incremental.scala:99)
at sbt.inc.Incremental$.compile (Incremental.scala:71)
at sbt.inc.IncrementalCompile$.apply (Compile.scala:54)
at sbt.compiler.AggressiveCompile.compile2 (AggressiveCompile.scala:159)
at sbt.compiler.AggressiveCompile.compile1 (AggressiveCompile.scala:68)
at com.typesafe.zinc.Compiler.compile (Compiler.scala:207)
at com.typesafe.zinc.Compiler.compile (Compiler.scala:189)
at com.typesafe.zinc.Compiler.compile (Compiler.scala:180)
at com.typesafe.zinc.Compiler.compile (Compiler.scala:171)
at sbt_inc.SbtIncrementalCompiler.compile (SbtIncrementalCompiler.java:94)
at scala_maven.ScalaCompilerSupport.incrementalCompile (ScalaCompilerSupport.java:320)
at scala_maven.ScalaCompilerSupport.compile (ScalaCompilerSupport.java:136)
at scala_maven.ScalaCompilerSupport.doExecute (ScalaCompilerSupport.java:116)
at scala_maven.ScalaMojoSupport.execute (ScalaMojoSupport.java:574)
at scala_maven.ScalaTestCompileMojo.execute (ScalaTestCompileMojo.java:50)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:190)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:186)
at java.util.concurrent.FutureTask.run (FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:511)
at java.util.concurrent.FutureTask.run (FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
at java.lang.Thread.run (Thread.java:748)
Oops, equally useless. Can't believe I had high hope for it. Relevenat build plugin configuration looks like this:
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.4.6</version>
<executions>
<execution>
<id>scala-compile-first</id>
<phase>process-resources</phase>
<goals>
<goal>compile</goal>
</goals>
</execution>
<execution>
<id>scala-test-compile-first</id>
<phase>process-test-resources</phase>
<goals>
<goal>testCompile</goal>
</goals>
</execution>
<!--required for publishing to maven central-->
<execution>
<id>add-source</id>
<goals>
<goal>add-source</goal>
</goals>
</execution>
<execution>
<id>attach-scaladocs</id>
<goals>
<goal>doc-jar</goal>
</goals>
<configuration>
<args>
<!-- see https://github.com/dmlc/xgboost/issues/4000-->
<arg>-nobootcp</arg>
</args>
</configuration>
</execution>
</executions>
<configuration>
<scalaVersion>${scala.version}</scalaVersion>
<scalaCompatVersion>${scala.binary.version}</scalaCompatVersion>
<recompileMode>incremental</recompileMode>
<!--https://tpolecat.github.io/2017/04/25/scalac-flags.html-->
<args>
<arg>-target:jvm-${java.version}</arg>
<arg>-unchecked</arg>
<arg>-deprecation</arg>
<arg>-feature</arg>
<arg>-g:vars</arg>
<arg>-encoding</arg>
<arg>UTF-8</arg>
<arg>-Xlint</arg>
</args>
<javacArgs>
<javacArg>-source</javacArg>
<javacArg>${java.version}</javacArg>
<javacArg>-target</javacArg>
<javacArg>${java.version}</javacArg>
</javacArgs>
<excludes>
<exclude>**/.ipynb_checkpoints/*</exclude>
</excludes>
</configuration>
</plugin>
UPDATE 3 I finally tried on sbt, but apparently the macro stacktrace option cannot be enabled. This is the scalac configuration in sbt build file:
scalacOptions ++= Seq(
"-unchecked",
"-deprecation",
"-feature",
// "-Xfatal-warnings",
"-Xlint:poly-implicit-overload",
"-Xlint:option-implicit",
// "-Xlog-implicits",
// "-Xlog-implicit-conversions",
"-Yissue-debug"
)
scalacOptions in (Compile, doc) ++= Seq("-groups")
/** Enable for debugging */
testOptions in Test += Tests.Argument("-oF")
Then it shows even less information than maven & gradle:
> run
[warn] Credentials file /home/peng/.bintray/.credentials does not exist
[warn] Credentials file /home/peng/.ivy2/.sbtcredentials does not exist
[info] Compiling 235 Scala sources and 30 Java sources to /home/peng/git-release/spark-nlp/target/scala-2.11/classes...
[error] /home/peng/git-release/spark-nlp/src/main/scala/com/johnsnowlabs/collections/SearchTrie.scala:86: Expression scala.util.Random.nextInt() does not evaluate to a constant or a stable reference value
[error] Witness(Random.nextInt())
[error] ^
[warn] /home/peng/git-release/spark-nlp/src/main/scala/com/johnsnowlabs/ml/tensorflow/TensorflowSpell.scala:134: inferred existential type (org.tensorflow.Tensor[?0], org.tensorflow.Tensor[?0], org.tensorflow.Tensor[?0]) forSome { type ?0; type ?0; type ?0 }, which cannot be expressed by wildcards, should be enabled
... (skipping irrelevant warnings)
[warn] 9 warnings found
[error] one error found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 5 s, completed 28-Jul-2020 12:44:30 AM
>
>
>
> last run
>
>
Do I miss something in any of these build system?

Answer 1

Do other compiler errors give you useful stacktraces?
Stack traces are fundamentally a runtime thing: the only stacktrace you can have at compile time is the stacktrace within the compiler (e.g. the sbt.compiler package in the stack traces in the question).
In general, to debug macro issues, depending on the issue, you'll either:
Run the Scala compiler inside a debugger (there are instructions) if the macro in question is failing to expand
Dumping the generated code with the -Ymacro-debug-lite compiler option, which will give you the generated Scala code and the abstract syntax tree representation; this would be useful only in the case where the macro expands but either itself fails to compile or exhibits a runtime bug