I wrote a sample compiler plugin for Scala. I tested it on two following sample files:
scalac -Xplugin: Test.jar *.scala
A:
package pkg {
class A {
def compute(): Int = 42
}
}
B:
import pkg._
class B {
def fun(): Unit = {
new A().compute()
}
}
I need to test this plugin on Akka scala files(https://github.com/akka/akka/tree/v2.6.10/akka-actor/src/main/scala/akka), but it shows thousands of errors because it can not compile the Akka files. In another word, how can I compile Akka actor files with this command:
scalac *.scala
To compile the Akka project directly with scalac you have to add all of the options sbt injects and somehow provide the additional sources that sbt generates at build time(e.g. the Version and more).
You can still easily inject a compiler plugin re-using the sbt build.
You can take inspiration on how it is done here for Scala 3 plugin, or, similarly, here for a Scala 2 one.
Related
I have a Scala application that successfully runs on the JVM using an uber jar via the command: java -jar myapp.jar. I need to create a separate, but related Scala job that utilizes many of the same objects/functions/dependencies as the first, making it a great candidate to keep in the same code repository & uber jar. Please note that these Scala jobs do not utilize Spark, so spark2-submit is out of the equation.
Question: How can I run 2 separate Scala jobs from the same uber jar on the JVM? (I am using Scala 2.11.8 and SBT for jar assembly)
Additional Context:
I've already looked into related StackOverflow discussions, namely this post about specifying Java classes using java -cp myapp.jar MyClass and this post, which only presented the solution of running the Scala equivalent using scala -classpath myapp.jar MyClass.
While the scala -classpath solution may have worked for the OP of the second linked discussion, I'll be deploying my code to an environment that doesn't have executables for scala or sbt, only java.
Let's say these are the 2 Scala jobs I want to run:
// MyClass.scala
package mypackage
object MyClass {
def main(args: Array[String]): Unit = {
println("Hello, World!")
}
}
// MyClass2.scala
package mypackage
object MyClass2 {
def main(args: Array[String]): Unit = {
println("Hello, World! This is the second job!")
}
}
Is there a way to run Scala code using java -cp myapp.jar MyClass?
I've tried this and receive the following error:
Error: Could not find or load main class MyClass
The main alternative I can think of would be to create a Scala object that serves as a main entry point and takes a parameter to determine which job gets run. I'd like to avoid that solution if possible, but it would allow me to continue using java -jar myapp.jar, which has been working fine.
You need to use a fully qualified name for the App instance:
java -cp myapp.jar mypackage.MyClass
I'm trying to run a Scala application packed as JAR (including dependencies) but this fails until the Scala library is added by using the -Xbootclasspath/p option.
Failing invocation:
java -jar /path/to/target/scala-2.10/application-assembly-1.0.jar
After the application did some of its intended output, the console shows:
Exception in thread "main"
scala.reflect.internal.MissingRequirementError: object scala.runtime
in compiler mirror not found.
at scala.reflect.internal.MissingRequirementError$.signal(MissingRequirementError.scala:16)
at scala.reflect.internal.MissingRequirementError$.notFound(MissingRequirementError.scala:17)
at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:48)
at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:40)
at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:61)
at scala.reflect.internal.Mirrors$RootsBase.getPackage(Mirrors.scala:172)
at scala.reflect.internal.Mirrors$RootsBase.getRequiredPackage(Mirrors.scala:175)
at scala.reflect.internal.Definitions$DefinitionsClass.RuntimePackage$lzycompute(Definitions.scala:181)
at scala.reflect.internal.Definitions$DefinitionsClass.RuntimePackage(Definitions.scala:181)
at scala.reflect.internal.Definitions$DefinitionsClass.RuntimePackageClass$lzycompute(Definitions.scala:182)
at scala.reflect.internal.Definitions$DefinitionsClass.RuntimePackageClass(Definitions.scala:182)
at scala.reflect.internal.Definitions$DefinitionsClass.AnnotationDefaultAttr$lzycompute(Definitions.scala:1015)
at scala.reflect.internal.Definitions$DefinitionsClass.AnnotationDefaultAttr(Definitions.scala:1014)
at scala.reflect.internal.Definitions$DefinitionsClass.syntheticCoreClasses$lzycompute(Definitions.scala:1144)
at scala.reflect.internal.Definitions$DefinitionsClass.syntheticCoreClasses(Definitions.scala:1143)
at scala.reflect.internal.Definitions$DefinitionsClass.symbolsNotPresentInBytecode$lzycompute(Definitions.scala:1187)
at scala.reflect.internal.Definitions$DefinitionsClass.symbolsNotPresentInBytecode(Definitions.scala:1187)
at scala.reflect.internal.Definitions$DefinitionsClass.init(Definitions.scala:1252)
at scala.tools.nsc.Global$Run.(Global.scala:1290)
at extract.ScalaExtractor$Compiler$2$.(ScalaExtractor.scala:24)
Working invocation:
java -Xbootclasspath/p:/path/to/home/.sbt/boot/scala-2.10.2/lib/scala-library.jar -jar /path/to/target/scala-2.10/application-assembly-1.0.jar
The strange thing about it is that the application-assembly-1.0.jar was built so that it includes all dependencies including the Scala library. When one extracts the JAR file it can be verified that the class files in the scala.runtime package have been included.
Creation of the JAR file
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.9.1") was added to project/plugins.sbt and the assembly target was invoked. A JAR file of about 25MB results.
Building the JAR with proguard shows the same runtime behavior as seen with assembly's JAR file.
Application code that triggers the MissingRequirementError
Some application code works fine and the previously described exception is triggered as soon as the new Run from the following fragment executes.
import scala.reflect.internal.util.BatchSourceFile
import scala.reflect.io.AbstractFile
import scala.reflect.io.Path.jfile2path
import scala.tools.nsc.Global
import scala.tools.nsc.Settings
…
import scala.tools.nsc._
object Compiler extends Global(new Settings()) {
new Run // This is line 24 from the stack trace!
def parse(path: File) = {
val code = AbstractFile.getFile(path)
val bfs = new BatchSourceFile(code, code.toCharArray)
val parser = new syntaxAnalyzer.UnitParser(new CompilationUnit(bfs))
parser.smartParse()
}
}
val ast = Compiler.parse(file)
Among others, scala-library, scala-compiler and scala-reflect are defined as dependencies in build.sbt.
For the curios / background information
The aim of the application is to aid in localization of Java and Scala programs. The task of the code fragment above is to get an AST from a Scala file in order to find method calls in there.
The questions
Given the Scala library is included in the JAR file, why is necessary to call the JAR using -Xbootclasspath/p:scala-library.jar?
Why do other parts of the application run just fine even though scala.runtime is reported as missing later?
The easy way to configure the settings with familiar keystrokes:
import scala.tools.nsc.Global
import scala.tools.nsc.Settings
def main(args: Array[String]) {
val s = new Settings
s processArgumentString "-usejavacp"
val g = new Global(s)
val r = new g.Run
}
That works for your scenario.
Even easier:
java -Dscala.usejavacp=true -jar ./scall.jar
Bonus info, I happened to come across the enabling commit message:
Went ahead and implemented classpaths as described in email to
scala-internals on the theory that at this point I must know what I'm
doing.
** PUBLIC SERVICE ANNOUNCEMENT **
If your code of whatever kind stopped working with this commit (most
likely the error is something like "object scala not found") you can
get it working again with either of:
passing -usejavacp on the command line
set system property "scala.usejavacp" to "true"
Either of these will alert scala that you want the java application
classpath to be utilized by scala as well.
Can SBT be used to launch applicative Scala batches?
I mean, a Scala applicative code that would decrement all the user's balance for exemple.
I can imagine a special SBT project for that with a build dependency on the applicative code, so that some SBT tasks of the project can launch this applicative code.
Would you do such a thing? Why?
What are the alternatives to ease the launch of applicative batches in Scala? With Maven I used to use the appassembler plugin which would generate sh scripts and all the classpath related stuff.
A simple way is to use sbt run.
object Main {
def main(args: Array[String]) {
// decrement all user balances.
};
};
Then sbt run from shell, jenkins, etc.
Use command line args to expand the functionality.
How does one add a external dependency to a SBT plugin and make it available on both the project and plugin classpath ?:
Specifically I have a simple plugin that should run our TestNG test suites and do some post processing. Here is a simplified version:
import sbt._
import java.util.ArrayList
import Keys._
import org.testng._
object RunTestSuitesPlugin extends Plugin {
lazy val runTestSuites = TaskKey[Unit]("run-test-suites", "runs TestNG test suites")
lazy val testSuites = SettingKey[Seq[String]]("test-suites", "list of test suites to run")
class JavaListWrapper[T](val seq: Seq[T]) {
def toJavaList = seq.foldLeft(new java.util.ArrayList[T](seq.size)) { (al, e) => al.add(e); al }
}
implicit def listToJavaList[T](l: Seq[T]) = new JavaListWrapper(l)
def runTestSuitesTask = runTestSuites <<= (target, streams, testSuites) map {
(targetDirectory, taskStream, suites) =>
import taskStream.log
log.info("running test suites: " + suites)
runSuites(suites)
}
private def runSuites(testSuites: Seq[String]) = {
var tester = new TestNG
tester.setTestSuites(testSuites.toJavaList)
tester.run()
}
def testSuiteSettings = {
inConfig(Compile)(Seq(
runTestSuitesTask,
testSuites := Seq("testsuites/mysuite.xml"),
libraryDependencies += "org.testng" % "testng" % "5.14"))
}
}
The problem is that when I add this plugin to a project and run it with run-test-suites then it fails with java.lang.NoClassDefFoundError: org/testng/TestNG even though show full-classpath shows that testng.jar is on the classpath.
So somehow the classpath used when executing the plugin differs from the one in my project, so how do I make a plugin dependency appear in both places ?
I'll try an answer, but I'm not very familiar with the inner details of sbt.
Normally, the path for the build system (as opposed to your program) is under project, as explained here. That would typically be in a project/plugins.sbt. Sounds right, as there is no reason that the application you develop should be concerned by what libraries your build system uses, nor the other way round.
When your plugin run the application code, that may not be so simple and there could well be classpath/classloader issues. I'm not sure that it will work. Normally, your plugin should implement a testing Framework rather than define its own task. Documentation of testing for sbt is limited.
A testing framework should implement org.scalatools.testing.Framework, in test-interface. Your build will take it into account after you add
testFrameworks += new TestFramework("full.class.name")
When you run the normal test command, it let every framework recognize the test classes it deals with (two criteria available: extending some base class or having some annotation) and run them. The framework run in the build, it is given a class loader to access the application code.
You may have a look at the framework implementation for junit (shipped with sbt). Also there is a TestNG implementation. I don't know it, according to its doc, it is a little bit unorthodox, hopefully it will work for you.
The error was fixed by adding TestNG directly to unmanagedJars in Compile in the project that uses the plugin.
I have not found any resources explaining the structure of the SBT class path during plugin execution so any attempt at explaining why this step is necessary will be greatly appreciated.
I am trying to GET a response from an API, using scala and dispatch. However, I get this error, after building. I googled for a solution, and tried cleaning, and resarting eclipse, but the error wont go away. What seems to be the problem? I use eclipse Helios (ie 3.6) and Scala v2.8.1, with Scala IDE v1.0.0.201104170033, installed from the Eclipse market.
dispatch{dispatch.type}.Http{object dispatch.Http} of type object dispatch.Http does not take parameters
This is my code.
class getList {
def main(args: Array[String]){
Http("http://foo.com/" >>> System.out)
}
}
What am I doing wrong?
What libraries have you downloaded? Are you sure the dependencies are set correctly? I tried with dispatch_http, version 8.0 for Scala 2.8.1, and it worked.
What imports are you using? I used these imports to make it work:
import dispatch.Http
import dispatch.HandlerVerbs._
Finally... class getList??? I assume this is a result of cut&pasting from actual code, but you should strive to produce a compilable example of your problem. Scala doesn't run programs from class, only from object, and it follows Java style of having classes start with an uppercase letter.
Here's the minimal code I used with SBT to get a working example.
Initializing:
~/test$ sbt
Project does not exist, create new project? (y/N/s) y
Name: test
Organization: test
Version [1.0]:
Scala version [2.7.7]: 2.8.1
sbt version [0.7.4]:
~/test$ cat project/build/TestProject.scala
import sbt._
class TestProject(info: ProjectInfo) extends DefaultProject(info) {
val dvers = "0.8.0"
val http = "net.databinder" %% "dispatch-http" % dvers
}
~/test$ cat src/main/scala/GetList.scala
import dispatch.Http
import dispatch.HandlerVerbs._
object GetList {
def main(args: Array[String]){
Http("http://foo.com/" >>> System.out)
}
}
~/test# sbt update run