object scala in compiler mirror not found - running Scala compiler programatically [no sbt - no IDE] [duplicate] - scala

I'm trying to run a Scala application packed as JAR (including dependencies) but this fails until the Scala library is added by using the -Xbootclasspath/p option.
Failing invocation:
java -jar /path/to/target/scala-2.10/application-assembly-1.0.jar
After the application did some of its intended output, the console shows:
Exception in thread "main"
scala.reflect.internal.MissingRequirementError: object scala.runtime
in compiler mirror not found.
at scala.reflect.internal.MissingRequirementError$.signal(MissingRequirementError.scala:16)
at scala.reflect.internal.MissingRequirementError$.notFound(MissingRequirementError.scala:17)
at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:48)
at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:40)
at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:61)
at scala.reflect.internal.Mirrors$RootsBase.getPackage(Mirrors.scala:172)
at scala.reflect.internal.Mirrors$RootsBase.getRequiredPackage(Mirrors.scala:175)
at scala.reflect.internal.Definitions$DefinitionsClass.RuntimePackage$lzycompute(Definitions.scala:181)
at scala.reflect.internal.Definitions$DefinitionsClass.RuntimePackage(Definitions.scala:181)
at scala.reflect.internal.Definitions$DefinitionsClass.RuntimePackageClass$lzycompute(Definitions.scala:182)
at scala.reflect.internal.Definitions$DefinitionsClass.RuntimePackageClass(Definitions.scala:182)
at scala.reflect.internal.Definitions$DefinitionsClass.AnnotationDefaultAttr$lzycompute(Definitions.scala:1015)
at scala.reflect.internal.Definitions$DefinitionsClass.AnnotationDefaultAttr(Definitions.scala:1014)
at scala.reflect.internal.Definitions$DefinitionsClass.syntheticCoreClasses$lzycompute(Definitions.scala:1144)
at scala.reflect.internal.Definitions$DefinitionsClass.syntheticCoreClasses(Definitions.scala:1143)
at scala.reflect.internal.Definitions$DefinitionsClass.symbolsNotPresentInBytecode$lzycompute(Definitions.scala:1187)
at scala.reflect.internal.Definitions$DefinitionsClass.symbolsNotPresentInBytecode(Definitions.scala:1187)
at scala.reflect.internal.Definitions$DefinitionsClass.init(Definitions.scala:1252)
at scala.tools.nsc.Global$Run.(Global.scala:1290)
at extract.ScalaExtractor$Compiler$2$.(ScalaExtractor.scala:24)
Working invocation:
java -Xbootclasspath/p:/path/to/home/.sbt/boot/scala-2.10.2/lib/scala-library.jar -jar /path/to/target/scala-2.10/application-assembly-1.0.jar
The strange thing about it is that the application-assembly-1.0.jar was built so that it includes all dependencies including the Scala library. When one extracts the JAR file it can be verified that the class files in the scala.runtime package have been included.
Creation of the JAR file
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.9.1") was added to project/plugins.sbt and the assembly target was invoked. A JAR file of about 25MB results.
Building the JAR with proguard shows the same runtime behavior as seen with assembly's JAR file.
Application code that triggers the MissingRequirementError
Some application code works fine and the previously described exception is triggered as soon as the new Run from the following fragment executes.
import scala.reflect.internal.util.BatchSourceFile
import scala.reflect.io.AbstractFile
import scala.reflect.io.Path.jfile2path
import scala.tools.nsc.Global
import scala.tools.nsc.Settings
…
import scala.tools.nsc._
object Compiler extends Global(new Settings()) {
new Run // This is line 24 from the stack trace!
def parse(path: File) = {
val code = AbstractFile.getFile(path)
val bfs = new BatchSourceFile(code, code.toCharArray)
val parser = new syntaxAnalyzer.UnitParser(new CompilationUnit(bfs))
parser.smartParse()
}
}
val ast = Compiler.parse(file)
Among others, scala-library, scala-compiler and scala-reflect are defined as dependencies in build.sbt.
For the curios / background information
The aim of the application is to aid in localization of Java and Scala programs. The task of the code fragment above is to get an AST from a Scala file in order to find method calls in there.
The questions
Given the Scala library is included in the JAR file, why is necessary to call the JAR using -Xbootclasspath/p:scala-library.jar?
Why do other parts of the application run just fine even though scala.runtime is reported as missing later?

The easy way to configure the settings with familiar keystrokes:
import scala.tools.nsc.Global
import scala.tools.nsc.Settings
def main(args: Array[String]) {
val s = new Settings
s processArgumentString "-usejavacp"
val g = new Global(s)
val r = new g.Run
}
That works for your scenario.
Even easier:
java -Dscala.usejavacp=true -jar ./scall.jar
Bonus info, I happened to come across the enabling commit message:
Went ahead and implemented classpaths as described in email to
scala-internals on the theory that at this point I must know what I'm
doing.
** PUBLIC SERVICE ANNOUNCEMENT **
If your code of whatever kind stopped working with this commit (most
likely the error is something like "object scala not found") you can
get it working again with either of:
passing -usejavacp on the command line
set system property "scala.usejavacp" to "true"
Either of these will alert scala that you want the java application
classpath to be utilized by scala as well.

Related

Running multiple Scala apps from one jar on JVM

I have a Scala application that successfully runs on the JVM using an uber jar via the command: java -jar myapp.jar. I need to create a separate, but related Scala job that utilizes many of the same objects/functions/dependencies as the first, making it a great candidate to keep in the same code repository & uber jar. Please note that these Scala jobs do not utilize Spark, so spark2-submit is out of the equation.
Question: How can I run 2 separate Scala jobs from the same uber jar on the JVM? (I am using Scala 2.11.8 and SBT for jar assembly)
Additional Context:
I've already looked into related StackOverflow discussions, namely this post about specifying Java classes using java -cp myapp.jar MyClass and this post, which only presented the solution of running the Scala equivalent using scala -classpath myapp.jar MyClass.
While the scala -classpath solution may have worked for the OP of the second linked discussion, I'll be deploying my code to an environment that doesn't have executables for scala or sbt, only java.
Let's say these are the 2 Scala jobs I want to run:
// MyClass.scala
package mypackage
object MyClass {
def main(args: Array[String]): Unit = {
println("Hello, World!")
}
}
// MyClass2.scala
package mypackage
object MyClass2 {
def main(args: Array[String]): Unit = {
println("Hello, World! This is the second job!")
}
}
Is there a way to run Scala code using java -cp myapp.jar MyClass?
I've tried this and receive the following error:
Error: Could not find or load main class MyClass
The main alternative I can think of would be to create a Scala object that serves as a main entry point and takes a parameter to determine which job gets run. I'd like to avoid that solution if possible, but it would allow me to continue using java -jar myapp.jar, which has been working fine.
You need to use a fully qualified name for the App instance:
java -cp myapp.jar mypackage.MyClass

Properly adding a SBT subproject to a Scala Play project

I'm trying to leverage some existing code with a new Scala Play project. I'm attempting to do this by adding the existing project as a sub-project to my Scala Play project by following the instructions here.
However, I am unable to figure out how to access objects in the subproject from the parent project. For example, I'm trying to create Scala Play singleton object that provides an instance of an object from the subproject.
package services
import javax.inject.Singleton
#Singleton
class EngineProvider {
var instance: Engine = _
def getEngine: Engine = {
if (instance == null) {
instance = new ScalaPlayEngine()
}
instance
}
}
However, my Intellij cannot find a way to import Engine, or ScalaPlayEngine (both objects are from my subproject). I can "trick" Intellij into removing it's errors by importing the following:
import _root_.Engine
import _root_.ScalaPlayEngine
But when I actually try to run the Play project, I still get an error:
play.sbt.PlayExceptions$CompilationException: Compilation error[_root_ cannot be imported]
at play.sbt.PlayExceptions$CompilationException$.apply(PlayExceptions.scala:27)
at play.sbt.PlayExceptions$CompilationException$.apply(PlayExceptions.scala:27)
at scala.Option.map(Option.scala:145)
at play.sbt.run.PlayReload$$anonfun$taskFailureHandler$1.apply(PlayReload.scala:49)
at play.sbt.run.PlayReload$$anonfun$taskFailureHandler$1.apply(PlayReload.scala:44)
at scala.Option.map(Option.scala:145)
at play.sbt.run.PlayReload$.taskFailureHandler(PlayReload.scala:44)
at play.sbt.run.PlayReload$.compileFailure(PlayReload.scala:40)
at play.sbt.run.PlayReload$$anonfun$compile$1.apply(PlayReload.scala:17)
at play.sbt.run.PlayReload$$anonfun$compile$1.apply(PlayReload.scala:17)
My parent (Scala Play) project build.sbt file contains this:
lazy val root = (project in file("."))
.enablePlugins(PlayScala)
.aggregate(engine)
.dependsOn(engine)
lazy val engine = project
so I would have expected it to be able to pick up any objects from the sub project (since it depends on it). My project structure looks something like this:
ParentScalaPlayProject
build.sbt
app
services
EngineProvider.scala
.
.
.
engine (child project)
build.sbt
src
main
scala
Engine.scala
ScalaPlayEngine.scala
.
.
.
I'm very new to both SBT and Scala Play, so it's very likely I'm missing something obvious here. Thanks in advance!
Your error message says: Compilation error[_root_ cannot be imported]
According to the Scala Language Specification, Section 9.2:
Top-level definitions outside a packaging are assumed to be injected into a special empty package. That package cannot be named and therefore cannot be imported. However, members of the empty package are visible to each other without qualification.
And indeed, your classes Engine and ScalaPlayEngine don't live in a package (src/main/scala doesn't count as a package). If you move them to a proper package, it should work.

"scala.runtime in compiler mirror not found" but working when started with -Xbootclasspath/p:scala-library.jar

I'm trying to run a Scala application packed as JAR (including dependencies) but this fails until the Scala library is added by using the -Xbootclasspath/p option.
Failing invocation:
java -jar /path/to/target/scala-2.10/application-assembly-1.0.jar
After the application did some of its intended output, the console shows:
Exception in thread "main"
scala.reflect.internal.MissingRequirementError: object scala.runtime
in compiler mirror not found.
at scala.reflect.internal.MissingRequirementError$.signal(MissingRequirementError.scala:16)
at scala.reflect.internal.MissingRequirementError$.notFound(MissingRequirementError.scala:17)
at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:48)
at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:40)
at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:61)
at scala.reflect.internal.Mirrors$RootsBase.getPackage(Mirrors.scala:172)
at scala.reflect.internal.Mirrors$RootsBase.getRequiredPackage(Mirrors.scala:175)
at scala.reflect.internal.Definitions$DefinitionsClass.RuntimePackage$lzycompute(Definitions.scala:181)
at scala.reflect.internal.Definitions$DefinitionsClass.RuntimePackage(Definitions.scala:181)
at scala.reflect.internal.Definitions$DefinitionsClass.RuntimePackageClass$lzycompute(Definitions.scala:182)
at scala.reflect.internal.Definitions$DefinitionsClass.RuntimePackageClass(Definitions.scala:182)
at scala.reflect.internal.Definitions$DefinitionsClass.AnnotationDefaultAttr$lzycompute(Definitions.scala:1015)
at scala.reflect.internal.Definitions$DefinitionsClass.AnnotationDefaultAttr(Definitions.scala:1014)
at scala.reflect.internal.Definitions$DefinitionsClass.syntheticCoreClasses$lzycompute(Definitions.scala:1144)
at scala.reflect.internal.Definitions$DefinitionsClass.syntheticCoreClasses(Definitions.scala:1143)
at scala.reflect.internal.Definitions$DefinitionsClass.symbolsNotPresentInBytecode$lzycompute(Definitions.scala:1187)
at scala.reflect.internal.Definitions$DefinitionsClass.symbolsNotPresentInBytecode(Definitions.scala:1187)
at scala.reflect.internal.Definitions$DefinitionsClass.init(Definitions.scala:1252)
at scala.tools.nsc.Global$Run.(Global.scala:1290)
at extract.ScalaExtractor$Compiler$2$.(ScalaExtractor.scala:24)
Working invocation:
java -Xbootclasspath/p:/path/to/home/.sbt/boot/scala-2.10.2/lib/scala-library.jar -jar /path/to/target/scala-2.10/application-assembly-1.0.jar
The strange thing about it is that the application-assembly-1.0.jar was built so that it includes all dependencies including the Scala library. When one extracts the JAR file it can be verified that the class files in the scala.runtime package have been included.
Creation of the JAR file
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.9.1") was added to project/plugins.sbt and the assembly target was invoked. A JAR file of about 25MB results.
Building the JAR with proguard shows the same runtime behavior as seen with assembly's JAR file.
Application code that triggers the MissingRequirementError
Some application code works fine and the previously described exception is triggered as soon as the new Run from the following fragment executes.
import scala.reflect.internal.util.BatchSourceFile
import scala.reflect.io.AbstractFile
import scala.reflect.io.Path.jfile2path
import scala.tools.nsc.Global
import scala.tools.nsc.Settings
…
import scala.tools.nsc._
object Compiler extends Global(new Settings()) {
new Run // This is line 24 from the stack trace!
def parse(path: File) = {
val code = AbstractFile.getFile(path)
val bfs = new BatchSourceFile(code, code.toCharArray)
val parser = new syntaxAnalyzer.UnitParser(new CompilationUnit(bfs))
parser.smartParse()
}
}
val ast = Compiler.parse(file)
Among others, scala-library, scala-compiler and scala-reflect are defined as dependencies in build.sbt.
For the curios / background information
The aim of the application is to aid in localization of Java and Scala programs. The task of the code fragment above is to get an AST from a Scala file in order to find method calls in there.
The questions
Given the Scala library is included in the JAR file, why is necessary to call the JAR using -Xbootclasspath/p:scala-library.jar?
Why do other parts of the application run just fine even though scala.runtime is reported as missing later?
The easy way to configure the settings with familiar keystrokes:
import scala.tools.nsc.Global
import scala.tools.nsc.Settings
def main(args: Array[String]) {
val s = new Settings
s processArgumentString "-usejavacp"
val g = new Global(s)
val r = new g.Run
}
That works for your scenario.
Even easier:
java -Dscala.usejavacp=true -jar ./scall.jar
Bonus info, I happened to come across the enabling commit message:
Went ahead and implemented classpaths as described in email to
scala-internals on the theory that at this point I must know what I'm
doing.
** PUBLIC SERVICE ANNOUNCEMENT **
If your code of whatever kind stopped working with this commit (most
likely the error is something like "object scala not found") you can
get it working again with either of:
passing -usejavacp on the command line
set system property "scala.usejavacp" to "true"
Either of these will alert scala that you want the java application
classpath to be utilized by scala as well.

SBT: Plugin dependencies and project classpath

How does one add a external dependency to a SBT plugin and make it available on both the project and plugin classpath ?:
Specifically I have a simple plugin that should run our TestNG test suites and do some post processing. Here is a simplified version:
import sbt._
import java.util.ArrayList
import Keys._
import org.testng._
object RunTestSuitesPlugin extends Plugin {
lazy val runTestSuites = TaskKey[Unit]("run-test-suites", "runs TestNG test suites")
lazy val testSuites = SettingKey[Seq[String]]("test-suites", "list of test suites to run")
class JavaListWrapper[T](val seq: Seq[T]) {
def toJavaList = seq.foldLeft(new java.util.ArrayList[T](seq.size)) { (al, e) => al.add(e); al }
}
implicit def listToJavaList[T](l: Seq[T]) = new JavaListWrapper(l)
def runTestSuitesTask = runTestSuites <<= (target, streams, testSuites) map {
(targetDirectory, taskStream, suites) =>
import taskStream.log
log.info("running test suites: " + suites)
runSuites(suites)
}
private def runSuites(testSuites: Seq[String]) = {
var tester = new TestNG
tester.setTestSuites(testSuites.toJavaList)
tester.run()
}
def testSuiteSettings = {
inConfig(Compile)(Seq(
runTestSuitesTask,
testSuites := Seq("testsuites/mysuite.xml"),
libraryDependencies += "org.testng" % "testng" % "5.14"))
}
}
The problem is that when I add this plugin to a project and run it with run-test-suites then it fails with java.lang.NoClassDefFoundError: org/testng/TestNG even though show full-classpath shows that testng.jar is on the classpath.
So somehow the classpath used when executing the plugin differs from the one in my project, so how do I make a plugin dependency appear in both places ?
I'll try an answer, but I'm not very familiar with the inner details of sbt.
Normally, the path for the build system (as opposed to your program) is under project, as explained here. That would typically be in a project/plugins.sbt. Sounds right, as there is no reason that the application you develop should be concerned by what libraries your build system uses, nor the other way round.
When your plugin run the application code, that may not be so simple and there could well be classpath/classloader issues. I'm not sure that it will work. Normally, your plugin should implement a testing Framework rather than define its own task. Documentation of testing for sbt is limited.
A testing framework should implement org.scalatools.testing.Framework, in test-interface. Your build will take it into account after you add
testFrameworks += new TestFramework("full.class.name")
When you run the normal test command, it let every framework recognize the test classes it deals with (two criteria available: extending some base class or having some annotation) and run them. The framework run in the build, it is given a class loader to access the application code.
You may have a look at the framework implementation for junit (shipped with sbt). Also there is a TestNG implementation. I don't know it, according to its doc, it is a little bit unorthodox, hopefully it will work for you.
The error was fixed by adding TestNG directly to unmanagedJars in Compile in the project that uses the plugin.
I have not found any resources explaining the structure of the SBT class path during plugin execution so any attempt at explaining why this step is necessary will be greatly appreciated.

AbstractMethodError when running standalone jar built with SBT and ProGuard

I've written a simple Scala application that I'd like to distribute in the form of a standalone, executable jar to servers without the Scala runtime. Everything works fine when invoked through SBT run, but not java -jar.
When I run the jar through java, I get the following unhandled exception:
Exception in thread "main" java.lang.AbstractMethodError: java.util.logging.Handler.publish(Ljava/util/logging/LogRecord;)V
at java.util.logging.Logger.log(Logger.java:458)
at net.lag.logging.Logger.log(Logger.scala:108)
at net.lag.logging.Logger.log(Logger.scala:91)
at net.lag.logging.Logger.info(Logger.scala:121)
at com.rentawebgeek.sitewiki.SiteWiki$.main(SiteWiki.scala:29)
at com.rentawebgeek.sitewiki.SiteWiki.main(SiteWiki.scala)
Exception in thread "Thread-0" java.lang.AbstractMethodError: java.util.logging.Handler.close()V
at java.util.logging.LogManager.resetLogger(LogManager.java:682)
at java.util.logging.LogManager.reset(LogManager.java:665)
at java.util.logging.LogManager$Cleaner.run(LogManager.java:223)
I'm using Configgy and it's Logger, and, per the javadocs for AbstractMethodError, thought it might be related to Scala/SBT using a different Java version than what I'm invoking from my shell. However, java -version and $JAVA_HOME/bin/java -version (what /usr/local/bin/scala uses) both match up as 1.6.0_22.
My ProGuard options are:
//program entry point
override def mainClass: Option[String] = Some("com.rentawebgeek.sitewiki.SiteWiki")
//proguard
override def proguardOptions = List(
"-keepclasseswithmembers public class * { public static void main(java.lang.String[]); }",
"-dontoptimize",
"-dontobfuscate",
"-keep class *",
proguardKeepLimitedSerializability,
proguardKeepAllScala,
"-keep interface scala.ScalaObject"
)
override def proguardInJars = Path.fromFile(scalaLibraryJar) +++ super.proguardInJars
How can I resolve this error? Or find another way to build an executable jar from an SBT project for a Scala-less deployment?
Check out what call you are making at line 29 in SiteWiki.scala; that is the offending call. You're probably calling a trait/class there with an abstract method. Most probably the method that should implement the abstract method is ripped away by proguard (or there the Scala override doesn't match up (I've seen that happen)).
If the line is long to find the offending call; try to decompose over multiple lines.
The latest ProGuard releases contain a sample configuration for processing a Scala application plus the Scala runtime:
http://proguard.sourceforge.net/manual/examples.html#scala
If that doesn't work, the output of -printconfiguration and the console output might help finding the root cause.