There is a myriad of development tools and terms in the ecosystem, for example, language server, build server, Metals, BSP, LSP, Bloop, Zinc, Coursier, incremental compiler, presentation compiler, etc.
I was wondering if someone could demonstrate how they fit together and briefly explain relations and differences. Specifically I am hoping for a diagram and answer along the lines of Making sense of Scala FP Libraries. For example, here is my attempt
(Concept) (Example implementation)
--------------------------------------------------------------------
IDE Visual Studio Code
| |
Scala IDE plugin Metals Visual Studio Extension
| |
Language Server Protocol Microsoft LSP
| |
Scala language server Metals
| |
Build Server Protocol BSP from JetBrains and Scala Center
| |
Scala build server Bloop
| |
Build tool sbt
| |
Dependency resolution Coursier
| |
Incremental compiler Zinc
| |
Presentation compiler parser and typer phases of scalac
| |
Bytecode generation remaining compiler phases
IDEs like Intellij or (once) Scala IDE are intended for "smart" development, where the editor tells you if your code is correct, suggest fixes, autogenerate some code, provide navigation around code, refactoring - in other words many features that expand way beyond simple text edition with perhaps only syntax highlighting.
Intellij has a Scala extension which makes use of their own reimplementation of a Scala compiler - they needed that for better partial compilation and intellisense working even if part of the code is broken. The import build definition from other build tool (e.g. sbt or bloop) and from then Intellij doesn't reply on anything external (unless you use some option like "build with sbt").
Scala IDE relied on Scala presentation compiler for intellisense. As you can read on scala-ide.org:
The Scala IDE for Eclipse uses the Scala Presentation Compiler, a faster asynchronous version of the Scala Compiler. The presentation compiler only runs the phases up until and including the typer phase, that is, the first 4 of the 27 scala compilation phases. The IDE uses the presentation compiler to provide semantic features such as live error markers, inferred type hovers, and semantic highlighting. This document describes the key classes you need to know in order to understand how the Scala IDE uses the presentation compiler and provides some examples of interactions between the IDE and the presentation compiler.
Every other editor/IDE is intended to use Language Server Protocol - LSP is Microsoft's invention in order to standardize a way of supporting languages within different editors (though they invented it for the sake of VS Code) that would allow them to provide IDE features. Metals (from ScalaMeta Language
Server) is LSP implementation for Scala. As you can read here:
Code completions, type at point and parameter hints are implemented using the Scala presentation compiler, which is maintained by the Scala compiler team at Lightbend.
You can add it to VS Code using Scala Metals extension.
sbt, gradle, mill, fury, cbt, etc are build tools which use something like ivy2 or coursier to resolve and download dependencies, and then use Zinc incremental compiler in order to provide ability to (re)build things incrementally to normal compiler. Build tools can run tests, generate artifacts and deploy them to repository.
bloop is a solution to problem that compiletion is fast if JVM is hot, and JVM gets cold every time you kill your build tool/IDE. For that reason you use nailgun to keep some JVM warm, running build tasks in background. On its own bloop cannot generate configuration and in general it is something that should be generated by other build tool to speed up compilation during development. Protocol used to communicate with bloop server running in background is build server protocol (bsp).
Coursier, while is used primarily for dependency resolution, can also be used to install scala programs. Some of the noteworthy programs that you can install are:
scalafmt - scala formatter
ammonite - alternative REPL to scala which provides a lot of nice features
scalafix - code-rewriting tool which is used to provide automatic code migrations
I gave up on describing thing in table, as it would much better be shown on a graph, but since SO doesn't support that visuals I just resorted to a plain text.
Related
Although IntelliJ Scala Plugin uses Scala compiler proper to generate the actual bytecode, it seems to use its own implementation of presentation compiler to provide real-time type-aware error highlighting in the editor:
syntax highlighting feature that is implemented by Idea's Scala
plugin and it requires immediate re-processing of all the files you
change in a way similar but now exactly the same as what the real
compiler does.
This might be the reason why sometimes projects build successfully and yet IntelliJ shows false positive red underlines in the editor. This seems to be the case even if use sbt shell is selected as evident by open ticket SCL-11079
By default, the sbt compile should mark additional errors it finds in
the editor and project view. False positive errors reported by the
internal annotator should be removed if possible. There should be an
option to use sbt output as the "single source of truth" for
highlighting errors, at the cost of instant error highlighting.
or if project is imported as BSP project
sbt shell indeed won't help with error highlighting in the editor, nor
will BSP (for the time being). In-ide highlighting uses a method that
is completely separate from regular compilation
Is there anyway to configure IntelliJ to use Scala out-of-the-box presentation compiler instead of its own custom implementation such that we can decrease the chance false positive errors akin to what Metals provides?
According to show bsp diagnostics in editor SCL-16610 there exists a IntelliJ registry flag
scala.highlighting.compiler.errors.in.editor
which enables experimental feature
As error highlighting using the built-in parser and type-checker
continues to be a problem for the Scala plugin, we can experiment with
using compiler output from BSP... Potential downsides: Less rich error highlighting on type mismatch
problems: BSP and the compiler expose only raw string messages and
positions. We would lose type diffs and type error hints.
Here are the steps that made it work on my machine
Install JetBrains Toolbox which enables installation of Early Access Programs alongside releases (such that you can revert to your working system if anything goes wrong)
Install 2020.1
Make sure Nightly Builds update channel is set under Preferences | Languages & Frameworks | Scala | Updates
Remove .idea from the root of the project: rm -fr .idea
Re-import the project and select BSP project
There should be .bloop directory under project root if all OK
Open IntelliJ registry: Help | Find Action... | Registry
Set key scala.highlighting.compiler.errors.in.editor
Suspicious that at least a part of this post might be an April Fool's joke, I set about installing Dotty on my Linux laptop and was pleased and surprised to discover that it seems to be completely straight - I was able to get the Fibonacci function as given in German working fine from the REPL.
Now I want to code in Dotty/Scala in German in the IntelliJ IDE, but have not so far managed to achieve this, or for that matter English Language 'Hello World' for which will the Dotty compile fails with an SBT error, see below (there are no errors in the IntelliJ code window for the object).
Opening a new Dotty project via the IntelliJ new project dialogue puts a new instance of Dotty in my .ivy2 directory, distinct from that I use in the REPL. I have found Felix Mulder's Dotty plugin but I'm not sure how to use a plugin from source. After the project has set itself up:
German keywords are not recognised in any new Scala file I create in src > main.
In build.sbt some operators and keywords are not recognised
In build.sbt each entry gives the error Expression Def.Setting[SomeType] must conform to DslEntry in SBT file
I get this error log warning (twice):
Binary version (0.1.1-20170410-0bd7821-NIGHTLY) for dependency
ch.epfl.lamp#scala-library;0.1.1-20170410-0bd7821-NIGHTLY in
default#skala_2.11;0.1-SNAPSHOT differs from Scala binary version in
project (2.11).
which I think might be a consequence of the skala and dotty paths in my PATH environment variable.
Searching on the error message in point 3 led me here: I can switch to an unbundled SBT I know to be good easily enough, but HW will still not compile, instead giving a quite lengthy SBT error.
Unfortunately I don't understand the syntax of the entries in build.sbt well enough to change them according to the advice of murmelssonic. If I just have to learn that then OK, I'm sure I can.
I believe I have read posts which suggest that the build.sbt errors can be ignored as they do not actually lead to compile errors, but I can't confirm this as I can't compile anything yet. Even if this be true, I want to eliminate these, the better not to obscure legitimate warnings and errors.
My question might be better expresses as simply "How do I install the Dotty plugin for IntelliJ?", but I'm not sure that's the complete solution to this collection of symptoms.
System info:
OS Kubuntu 16.10
Java 1.8_121
IntelliJ 2017.1
Dotty v0.1.1-20170410
SBT 0.13.13 and 0.13.15 (each attempted) via IntellliJ dialogue
SBT 0.1-SNAPSHOT (from sbt version on command line in sbt dir, used by changing IntelliJ settings).
While Skala is a natural development for the Scala language, seeing how its creator Martin Odersky is a German and should have implemented German keywords in the first place, JetBrains is a company founded by Russians. As such, we are committed to exporting cyrillic keywords into the world of programming.
In a way, IntelliJ-Scala is already an alternative compiler for the Scala programming language specification. This will be increasingly true with the release of the all-new Скала plugin for IntelliJ!
This is a totally official statement by the IntelliJ-Скала team's only German member.
What's a prefereed way to download Scala , via scala-lang.org, can it be added directly by the Eclipse IDE or how to add Scala to Eclipse IDE? Is there a convention on how to add a JVM language?
Update
I could add Clojure and Scala from Help...install new software so now I can create those kinds of projects:
You typically install a plugin for that language using Eclipse's build in plugin language.
for instance for Clojure you can search the plugin manager for "counter clockwise" to get Clojure support. for Scala it looks like http://scala-ide.org/ is a good starting point for detailed instructions and a nice screencast on setting this up. It's worth noting that many JVM languages like Clojure don't need to be explicitly installed, they are downloaded as required by standard Java build tools like Maven. Other languages like Groovy are more easily used if you install them explicity.
If you're an IDEA user, by far the best way is to use SBT to manage your projects and use its sbt-idea plug-in (which adds the gen-idea task) to create IDEA projects files and directories.
Ideally, use Paul Phillips' SBT launcher (to get SBT version flexibility if you're ever going to build 3rd-party projects whose specified SBT versions can be all over the map) and configure non-project-specific plugins, such as sbt-idea, in your per-user shared ~/.sbt directory. That way all your projects have it automatically yet it does not intrude on the project definition itself.
Optionally, there is an IDEA plug-in that integrates SBT's interactive console with the IDEA app. The best part of that is compilation errors become hyperlinks to your code.
I don't know if there's an Eclipse counterpart to sbt-idea, but even if there's not, SBT is the way to go.
The JVM languages typically have plugins for Eclipse, e.g. Counterclockwise for Clojure.
In many cases you can also use JVM languages without a plugin in a regular Java project. For example the following approach works with Clojure:
Include clojure-1.4.0.jar as a dependency (using Maven or similar)
Include clojure source files as regular resources in your project
Write a short piece of Java code that executes the clojure code by calling the appropriate methods in clojure.lang.RT (a class provided in Clojure's jar file)
I am trying to generate some boilerplate with SBT (tool which is totally new to me). I am using shapeless sbt files as my main reference for the task. I have seen that this project uses code generation from scratch, but my case is slightly different, since I would like to generate some classes from another ones. I pretend to use the new Scala 2.10.0-M4 reflection capabilities for doing so. What basic configuration is needed to have reflection available from a SBT build?
By now, the sbt is unable to find the scala.reflect.runtime.universe package, and I do not know if the problem comes either from the new Scala jar division or from a bad configuration. Besides, my sbt about says:
[info] This is sbt 0.13.0-20120530-052139
[info] The current project is {file:/home/jlg/sandbox/abc/}abc
[info] The current project is built against Scala 2.10.0-SNAPSHOT
[info]
[info] sbt, sbt plugins, and build definitions are using Scala 2.9.2
By the way, does anybody know other projects using SBT to generate source code?
Current SBT releases are based on Scala 2.9, and source code generation runs together with SBT with the same libraries. There are basically two choices:
be extremely bleeding-edge: get an SBT release running on Scala 2.10 (not even the 0.13 branch does), or waiting for it. The biggest problem is not just that you'd have to recompile SBT yourself, it's recompiling every single SBT plugin you'll need for Scala 2.10. In the long-term, this is maybe the best strategy to do what you ask, but it might be a lot of effort for now. However, beware that you cannot use reflection on your compiled code without evil tricks, since code generation is supposed to happen before compilation. If you need to do that, consider instead generating code at compile-time within the program using macros. This excludes SBT and is much more standard, but I'm not sure if you can generate complete classes in this release (this is I think planned for the future).
go with the old: stick with Scala 2.9 and use scalap's capabilities (ScalaSigParser) for compile-time reflection. The problem is that the API is different (not sure how deeply) and not really supported for public use, although various people have been using it for ages. For a project I'm running, a colleague implemented approach and I integrated it within SBT for my project (https://github.com/ps-mr/LinqOnSteroids/); on top of that, I use Scalate to write the templates to use for code generation, which is quite powerful.
See in particular build.sbt, which invokes
project/Generator.scala and project/src/main/scala/ivm/generation/ScalaSigHelpers.scala (some non-fully-generic wrappers for ScalaSigParser). Scalate Templates for generated code are in
src/main/resources, the most relevant here is src/main/resources/WrappedClassInlined.ssp.
Even more stuff is involved, I fear you'll pratically need a checkout and playing with it to see what it does exactly—but feel free to ask questions.
Please note that the code is protected by a BSD license, so you need to keep the original copyright if you copy the code.
Note: all the links (except the license) are to the current HEAD for stability, so that they won't disappear so easily even if the files are moved/removed in future versions.
If you're using 2.10.0-SNAPSHOT, then you should go for scala.reflect.runtime.universe. Take a look at http://dcsobral.blogspot.ch/2012/07/json-serialization-with-reflection-in.html for more information.
I'm looking for a guide for polyglot programming in this two languages.
I know that interop between them is possible, since they are both running on the same Java VM, and both got compiled to the same bytecode.
But there is some cumbersomes:
Can I use sbt to compile Clojure code?
Can I use leiningen to compile Scala code?
(
Yeah, yeah, I know I can just plug in jar from one language to project in other lang.)
Is there some libraries that could save me from writing boilerplate
interop code?
and so on
So the question is How to setup polyglot development in Scala and Clojure?
The short anser: use Maven3
maven3 with the zi plugin for clojure and the maven-scala-plugin will allow you to have a nicely integrated polyglot project where you can use any language you want as long as its Java ;)
One of the big changes in Maven3 was a push toward polyglot JVM programming including the ability to write POMs in many languages. You loose the polish of leiningen when taking this route, though you gain plenty in return.
I've not used it, but Leiningen does have scalac support.
Personally, I'd stay away from using Maven directly despite being the accepted answer. Here is the simple Leiningen project definition which:
generates Java sources from ANTLR4 grammar (as a bonus);
compiles Java sources;
compiles Scala sources (I prefer to use Scala on top of generated ANTLR4 Java code since it's much more concise and pleasant to work with);
continues with Clojure stuff.
(defproject polyglot "0.1.0-SNAPSHOT"
:description "FIXME: write description"
:url "http://example.com/FIXME"
:license {:name "Eclipse Public License"
:url "http://www.eclipse.org/legal/epl-v10.html"}
:plugins [[lein-antlr4 "0.1.0-SNAPSHOT"]
[lein-scalac "0.1.0"]]
:dependencies [[org.clojure/clojure "1.5.1"]
[org.antlr/antlr4 "4.0"]
[org.scala-lang/scala-library "2.10.1"]]
:warn-on-reflection true
:antlr4-source-paths ["antlr4"]
:antlr4-options {:package "parser" :listener true}
:antlr4-compile-path "target/antlr4/parser"
:java-source-paths ["target/antlr4/parser" "src/java"]
:scala-source-path "src/scala"
:prep-tasks ["antlr4" "javac" "scalac" "compile"])
To use ANTLR4 plugin download and 'lein install' the lein-antlr4 plugin. If you don't need it, just remove relevant lines from the project definition.
To use Scala plugin I needed to download it and change
[org.scala-lang/scala-compiler "2.9.1"]
to
[org.scala-lang/scala-compiler "2.10.1"]
in plugin's project.clj, then 'lein install' it locally. With older dependency version I was getting
java.lang.Error: typeConstructor inapplicable for <none>
from Scala compiler.
As a maven alternative: gradle is able to compile both languages.
Scala's Dynamic trait was created to enable easier integration with dynamically typed languages, but you'll have to write your own forwarder. Also, you'll have to rewrite it with Scala 2.10, since its been changed a bit. Perhaps, in fact, you should use Scala 2.10 (milestone 2 is available). Scala 2.10 CAT support may also help you with integration on the Scala side.
You can use SBT to compile Clojure, since SBT is very flexibly, but, more pragmatically, there is no Clojure-compilation support readily available. You can look at this plugin as an example of how to add compilation of other languages.
I find it curious that so little integration between Scala and Clojure seems to exist. They are often compared, and you'll often see people saying it choose one over the other.