I'm going through the tutorial and it looks like Scala.js only runs under sbt.
Are there bits of Scala.js (or the Scala environment generally) that are not written in Scala? Or cannot all the necessary bits go through Scala.js for some other reason? What am I missing?
Mostly, this is because the Scala compiler uses too many parts of the JDK that have not been ported to Scala.js (yet). Some of these parts, most notably related to reading files (in the classpath, and source files), which cannot be implemented in JavaScript as such (though could be implemented for one particular platform, such as Node.js).
There is also the dependency on ASM, a Java bytecode manipulation library written in Java. Even though Scala.js compiles to JavaScript, the Java bytecode is still used for separate compilation (symbol lookup in previously compiled parts, such as libraries).
So, even though the Scala.js specific parts are written in a platform-independent way (e.g., we test that the Scala.js optimizer can optimize itself), there are a lot of parts in scalac that do not work out-of-the-box in Scala.js.
Related
I succeed compiling a scala project to machine code using Scala Native.
But I want to generate some executable code at runtime (I plan to implement a standalone compiler from a scala-like language to machine code).
The goal is to have a self-hosted language, independent of JVM.
Is it possible to somehow embed the Scala Native compiler in my project?
As described in https://www.scala-native.org/en/v0.4.0/contrib/build.html,
The build of Scala Native contains the following JVM-based portions of which the 1st, 3rd, and 4th seem like they would be necessary for a Scala Native compiler embedded in your own compiler:
The Scala Native sbt plugin and its dependencies (directory names are in parentheses). These are JVM projects.
sbtScalaNative (sbt-scala-native)
tools
nir, util
nirparser
testRunner (test-runner)
So Scala Native is not independent of JVM as OP's question seeks. Conversely, studying the NIR (scala-Native Intermediate Representation) portions of the Scala Native codebase might indicate a point (somewhere near the emission of NIR onward) to factor out a nonJVM NIR-to-LLVM backend. Then OP's “self-hosted language” that compiles NIR to LLVM IR to machine code “from a scala-like language to machine code” as OP's question seeks might be possible, as derived from some backend extract/fragment of Scala Native's codebase after the parser, perhaps after the AST, which is dependent on Scala(-proper)'s JVM-based parser, whereas from NIR onward is in the JVM simply because the parser and AST were already within the JVM.
In the search for ways to make Java libraries accessible to both the JavaScript and JVM sides of Scala.js cross-built projects, please consider the following experiment:
Imagine that a Scala.js project needs advanced matrix math capabilities such as Singular Value Decomposition. Although the JavaScript world has Numeric.js and the JVM world has many options, JAMA not least among them, no cross building Scala.js solution exists at the time of this question's formulation.
What options do we have?
Write anew or port a matrix library for Scala.js.
Wrap Numeric.js facades and JAMA into a common Scala.js interface.
Write facades for Numeric.js, then compile it with Nashorn for JVM support.
Transpile JAMA to JavaScript with JSweet and fashion appropriate Scala.js facades.
This question reflects option 4.
After reconditioning JAMA for the JSweet transpiler, publishing the transpiled JavaScript as a CommonJS module through npm, and writing Scala.js facades for the CommonJS module, Scala code can now access Jama on the JVM side, and a port of it on the JS side.
Unfortunately, the core data structure on the JVM side has type: double[][], Array[Array[Double]] in Scala syntax, but JSweet converts it to the JavaScript array type, js.Array[js.Array[Double]] in Scala.js syntax.
Now, from the Scala.js cross built perspective, two identically named, equivalently functional, but utterly distinct and separate libraries exist.
From Scala syntax, we can construct the 3D identity matrix on the JS side as so:
new Matrix(
js.Array[js.Array[Double]](
new js.Array[Double](1.0, 0.0, 0.0),
new js.Array[Double](0.0, 1.0, 0.0),
new js.Array[Double](0.0, 0.0, 1.0)
)
)
On the JVM side, we write:
new Matrix(
Array[Array[Double]](
new Array[Double](1.0, 0.0, 0.0),
new Array[Double](0.0, 1.0, 0.0),
new Array[Double](0.0, 0.0, 1.0)
)
)
How could we unify these two implementations?
Is there a trick to equate js.Array and Array?
Would you suggest an entirely different approach to make Java libraries accessible to cross-built Scala.js projects?
Let me start with the easy question:
Is there a trick to equate js.Array and Array?
No, there is no such "trick". Those data structures are fundamentally different. js.Array's length is variable, and can be patched with additional properties. Array is Java's fixed-length array. If there was any way to equate them, Scala.js would have done that for you, like it does for Double and number.
One relatively easy way to unify the APIs would be to rebuild the API of JAMA in Scala.js code, where every class is a wrapper for the facaded JS class coming from the JSweet-compiled library. This allows the Scala-level API of JAMA to be exactly equivalent between Scala/JVM and Scala.js. It does require some amount of code writing to replicate the APIs, but at least the body of the methods need not be rewritten.
A completely different approach, very radical and requiring an insane amount of man·hours would be fork the JSweet compiler to emit Scala.js IR (.sjsir) files instead of .js files. That way, you could link the JSweet-generated .sjsir files for JAMA together with the Scala.js-generated .sjsir files for your application. This would give the maximum performance, since the Scala.js optimizer would be able to optimize across the app/library border.
A recipe for porting Java libraries to Scala.js:
Procure Java source code.
Strip out IO and other JSweet incompatible functionality.
Compile it with JSweet as a Node.js module.
Hand roll a package.json file and publish the JSweet transpiler output on npm.
Add ScalaJS-bundler to a cross compiled Scala.js project; include the CommonJS module from step five.
Include the Java version in the JVM dependencies of build.sbt.
Write a Scala.js facade for the CommonJS module.
Write a wrapper for the Scala.js facade with the exact interface from the original Java library. Add in whatever IO and other JSweet incompatible functionality if Scala.js supports it.
My understanding is that it is quite simple to create & parse an external DSL in Scala (e.g. representing rules). Is my assumption correct that the DSL can only be interpreted during runtime but does not support code generation (like ANTLR) for archiving better performance ?
EDIT: To be more precise, my question is if I could achieve this (create an external domain specific language and generate java/scala code) with built-in Scala tools/libraries (e.g. http://www.artima.com/pins1ed/combinator-parsing.html). Not writing a whole parser / code generator completely by yourself in scala. It's also clear that you can achieve this with third-party tools but you have to learn additional stuff and have additional dependencies. I'm new in the area of implementing DSLs, so I have no gutfeeling so far when to use external tools like ANTLR and what you can (with a reasonable effort) do with Scala on-board stuff.
Is my assumption correct that the DSL can only be interpreted during runtime but does not support code generation (like ANTLR) for archiving better performance ?
No, this is wrong. It is possible to write a compiler in Scala, after all, Scala is Turing-complete (i.e. you can write anything), and you don't even need Turing-completeness for a compiler.
Some examples of compilers written in Scala include
the Scala compiler itself (in all its variations, Scala-JVM, Scala.js, Scala-native, Scala-virtualized, Typelevel Scala, the abandoned Scala.NET, …)
the Dotty compiler
Scalisp
Scalispa
… and many others …
I'm building a tool that will receive unpredictable data structure, and I want to generate case class to accomplish the structure of the received data.
I'm trying to figure out if it's possible to generate case class at runtime? This structure will be know only at runtime.
It's something similar to what macro does, but in runtime.
I've found this project on the internet
mars
Which is very close to what I want to do ,but I couldn't find if it was successful of not.
Another way of doing it is generate the code, compile and put the result in the classpath, like IScala is doing to use the code in an iterative way. But I don't think that this will scale.
Does anybody has already done something like runtime code generation?
This question was also posted in scala-user mailing list
UPDATE: (as per the comments)
If all you want is throw-away code generated at runtime to be fed into to a library that cannot work with just lists and maps, and not code to be stored and used later, it would make sense to look for solutions to this problem for Java or JVM. That is, unless the library requires some Scala specific features not available to vanilla JVM bytecode (Scala adds some extras to the bytecode, which Java code doesn't need/have).
what is the benefit of generating statically typed code dynamically? as opposed to using a dynamic data structure.
I would not attempt that at all. Just use a structure such as nested lists and maps.
Runtime code generation is one of the purposes of the Mars Project. Mars is under development, at the moment there is no release version. Mars requires its own toolchain to expand macros at runtime and should use several features unique to scala.meta (http://scalameta.org/), for example, AST interpretation and AST persistence. Currently we are working on ASTs typechecking in scala-reflect, required for runtime macros expansion.
I already have an interpreter for my language.
It is implemented with:
parser -> scala parser combinators;
AST -> scala case classes;
evaluator -> scala pattern matching.
Now I want to compile AST to native code and hopefully Java bytecode.
I am thinking of two main options to accomplish at least one of these two tasks:
generate LLVM IR code;
generate C code and/or Java code;
obs.: GCJ and SLEM seem to be unusable (GCJ works with simple code, as I could test)
Short Answer
I'd go with Java Bytecode.
Long Answer
The thing is, the higher-level the language you compile to,
The slower and more cumbersome the compilation process is
The more flexibility you get
For instance, if you compile to C, you can then get a lot of possible backends for C compilers - you can generate Java Bytecode, LLVM IR, asm for many architectures, etc., but you basically compile twice. If you choose LLVM IR you're already halfway to compiling to asm (parsing LLVM IR is far faster than parsing a language such as C), but you'll have a very hard time getting Java Bytecode from that. Both intermediate languages can compile to native, though.
I think compiling to some intermediate representation is preferable to compiling to a general-purpose programming language. Between LLVM IR and Java Bytecode I'd go with Java Bytecode - even though I personally like LLVM IR better - because you wrote that you basically want both, and while you can sort of convert Java Bytecode to LLVM IR, the other direction is very difficult.
The only remaining difficulty is translating your language to Java Bytecode. This related question about tools that can make it easier might help.
Finally, another advantage of Java Bytecode is that it'll play well with your interpreter, effectively allowing you to easily generate a hotspot-like JITter (or even a trace compiler).
I agree with #Oak about the choice of ByteCode as the most simple target. A possible Scala library to generate ByteCode is CafeBabe by #psuter.
You cannot do everything with it, but for small project it could be sufficient. The syntax is also very clear. Please see the project Wiki for more information.