I just read the document of eclipse xtext, and found the language it uses is similar to java, but with a few differences:
http://www.eclipse.org/Xtext/documentation/2_0_0/040-first-code-generator.php
For example, it has such a code snippet:
def compile(Entity e) '''
package «e.eContainer.fullyQualifiedName»;
public class «e.name» {
}
'''
What's the language?
The language you are referring to is called Xtend2 and is a Java-like programming language that has been tailored for code generation. It features some very useful concepts such as closures, dynamic dispatch, type inference and the rich strings you've already seen. Xtend2 code can be executed by the JVM since it's been translated to human-readable Java code each time you hit save. Sven Efftinge blogged about Xtend2's core ideas a while ago. There's a tutorial on the Xtext website (the page you mentioned in your question), too.
Also, Xtend2 shares most ideas of Xbase, a reusable foundation for programming languages built with Xtext.
It appears to be some sort of a notation for code generation (which of course one could surmise by reading the page title). And another "duhh!": They appear to call the overall package Xtext. However, I've not found anything that names the 2-3 different languages they use for grammar, semantics, etc.
Looks like Scala + some template with so called merge codes.
http://www.scala-lang.org/
Related
In JavaScript we have estree which is the AST definition evolving from Mozilla's implementation. But nowadays if you build an AST transformer in JS of a JS AST, you probably use this structure. Do we have anything like this for Swift, of the Swift AST?
I see we have a grammar, but what about an AST? I guess I can make one from that, but still.
If nothing standard, do we have any AST examples?
The Swift project offers SwiftSyntax as a package for working with Swift source code, in Swift. Under the hood, it's powered by the compiler's own libSyntax, written in C++.
Note that this currently isn't the representation that the Swift compiler proper uses for actually compiling Swift code — libSyntax focuses on source code itself for rewriting, formatting, transformations, etc., but is largely void of the semantic information that you may find in a compiler AST necessary for transforming the source into machine code. If you're just looking to operate on the AST without those semantics, this may be sufficient for your use case.
The repo README should have some info to get you started, an example, and some real-world use cases showing concrete usage of the library.
I want to ask the advantages mps and xtext have over each other and the main features when writing a language. I know when working with mps you are directly editing the AST and xtext uses a parser. I have read an advantage of using a AST allows for multiple languages to be extended for the language you are making, I don’t really understand what this means, could this be explained further and why would someone want to extend multiple language ?
Also i have read that the AST cut out ambiguous code, how does it do this?
I know that both MPS and xtext have features like underlining and highlighting code is their any other feature relating to code validation ?
Any other main differences and general feature of them are welcome ?
I have no practical experience with Xtext, so I will talk mostly about MPS.
LWB
Both Xtext and MPS are language workbenches, so they have their own schema used to metamodel the abstract syntax (structure of concepts), some way to define the concrete syntax (notations) and some way to define generators (M2M or M2T transformations) or less usually interpreters. Then they have provide the IDE itself with highlighting, smart actions like refactoring and contextual error fixes, advanced search and navigation (go to declaration etc.), checking for errors (type errors, static code analysis, checking of defined constraints & rules, checking cardinality, dataflow analysis), ... So yes, lots of options for validation. I have mentioned things, that are in MPS, not sure if Xtext provides everything. However, all of these features are organised in so-called aspects which you can check out in a summary table which shortly describes each aspect.
Projectional editor
As you have mentioned, MPS uses a projectional editor. You directly manipulate the AST, parser-based post-IntelliJ smart IDEs are able to provide you with intelligent actions like recaftoring and go to declaration etc. only because they parse the language in memory and construct an AST behind the scenes anyway. Projectional editors skip the parsing step.
Dodging ambiguity
It uses no parser at all, so all of the downsides of having a parser are gone. First of all, the language developer does not need to be an expert in syntax analysis, so you don't need to hire them specifically. But the best win is to have infinite language composability. This is achieved, as you have mentioned, by totally avoiding ambiguities which could appear in grammars (MPS does not use a grammar, but a model). Let's say you use language A and language B. For demonstration, let's say both languages extend BaseLanguage (abbr. BL, the MPS-equivalent of Java) and they have both defined a statement to log. Concept a logs to stderr and b logs to a file. However, both a and b have an identical concrete syntax (i.e. editor definition in MPS) which just says log. Now if you had a parser and it encounters the token log it cannot decide from which language the concept is, so it's ambiguous - not even a look-ahead parser can do it. In a projectional editor this cannot happen, because only the projection is identical and under the hood the AST has an instance of either a or b (you can think of it as always using the whole FQN of a class in Java, just the package is hidden in the IDE, so you can use identically named classes from different packages). The "ambiguity" is resolved at the time of writing by the user: when he writes log a dropdown menu appears clearly showing that one of them is a and the other is b (maybe even shows a description which would say "Log to file" / "Log to stderr").
Modularity
Consequently, MPS has very good modularity, composability and extensibility of languages. You have mentioned
allows for multiple languages to be extended for the language you are making [...]
why would someone want to extend multiple language
You need to differentiate between using a language and extending it
(if you are interested more Völter talks about 4 kinds of composition techniques regarding languages: referencing, extension, reuse and embedding). Using a lanugage is just the ability to write programs in it. If you extend a language, it's kind of like inheritance, you add new concepts to it, f.e. create a new type of Java (BL) statement. And it has been done in the standard languages shipped with MPS too. You have for example the checkedDots language which extends BL with an operation .? which is null-safe (similarly to null-conditional operator ?. in C#). So why extend a language? Because you can use new constructs, add new functionality or syntactic sugar. Another ready-to-use language in BL is the tuples language, which has both indexed and named tuples. Then there is the collections language, which kind-of replaces the Java Stream API. All of these little languages are extensions which you can start using with a simple Ctrl+L. You could also embed another language to your language - use a regex inside an SQL statement inside your Java code.
Generation
Another kind of language dependency in MPS is to have a "generation target" language. Generators in MPS work in a way that you transform your language sentence (i.e. model) into another MPS language. You can invent your own little language, or implement LOLcode and setup the generator to transform it into valid Java code. However, this language must already exist in MPS, so you cannot generate it to Python, if there is no Python implementation in MPS. The other alternative is to generate text (M2T), this way you could theoretically generate Python source code, or just print the LOLcode as-is.
Multiple notations
The second great difference in projectional vs. parser-based editors is that the latter inherently supports only textual notation. Maybe there are some external tools you can use. On the other side, MPS provides textual, tabular, symbolic (math symbols) and graphical (diagrams) notations. There is a possibility to swap your view from one notation to another, per concept or for the whole "file" (program).
Drawbacks
It's not all roses though. Projectional editors have some limitations, or challenges to solve. There is an analysis of challenges in projectional editors which points out mainly usability and infrastructure integration. They are mostly solved in MPS, f.e. regarding infrastructure you have a good VCS diff/merge tool. For automatic/cmd builds there is a language that generates Ant. Gradle or Maven does not work with MPS directly, but through Ant. Regarding usability "MPS takes a
while to get used to, but then its usability is comparable to ParEs."3 You should use a language called GrammarCells (available through MPS-extensions or mbeddr.platform) which makes it easy to build good editors (mainly for arithemtic expressions), otherwise by default you must enter concepts in prefix order (+ first, not the number). Comments in MPS cannot be placed willy-nilly. Cannot establish references to non-existing nodes... (see the Table 1. in 3)
MPS currently does not have a web-based version. There are some planned, though. Jetbrains works on WebMPS, then there is modelix.
Portability
Generally, you are stuck to working in MPS. By default it is not really portable, unless you explicitly define generators which produce portable output. If you want to input a program , you can code a paste-handler where you could put your parser, or you can change the format in which the AST is stored (from XML to maybe directly your language, but this would again require a parser to read). I am currently working on a solution which enables to import an MPS language from a YAJCo model (model-based parser generator, where the input is not a grammar, but Java classes representing the semantic model). Then you can import a sentence (file) which creates and populates a model (AST). From the program in MPS you can generate Java source code which fills the original Java classes if you need it.
BTW the mbeddr project has implemented importing from ECore check here
Dictionary
M2M = model to model
M2T = model to text
Given a Scala AST, is there a way to generate Scala source code?
I'm looking into ways to autogenerate Scala source by parsing/analyzing other Scala source. Any tips would be appreciated!
I have been successfully using Scala-Refactoring by Mirko Stocker for this task.
For synthetically constructing ASTs, it relies strongly on the existing Tree DSL of Scala's NSC.
Although the code is a bit messy, you can find an example usage in my project ScalaCollider-UGens.
I have also come across a very useful class by Johannes Rudolph.
See our DMS Software Reengineering Toolkit.
DMS provides a complete ecosystem for parsing/analyzing/optimizing/transforming source code in many languages. It achieves this by provide generic machinery for these tasks as its core capabilities, and specializing those according to explicitly supplied language definitions ("front ends"). DMS has front ends for many languages (C, C++, C#, Java, COBOL, ...) that have been used in anger, and a process for defining others very quickly.
We work on expanding the language set more or less continuously. DMS already has parts of a Scala front end implemented, and we know how to finish it based on the other 30+ front ends we have built, with special emphasis on knowledge of Java.
I realize that it is impossible to have one language that is best for everything.
But there is a class of simple programs, whose source code looks virtually identical in any language.
I am thinking not just "hello world", but also arithmetics, maybe string manipulation, basic stuff that you would typically see in utility classes.
I would like to keep my utilities in this meta-language and have it automatically translated to a bunch of popular languages. I do this by hand right now.
Again, I do not ask for translation of every single possible program. I am thinking a very limited, simple language, but superportable.
Do you know of anything like that? Is there a reason why it should not exist?
Check Haxe, and its Wikipedia page. It's open source and its main purpose is what you describe: generating code in many languages from only one source.
Just about any language that you choose is going to have some feature that doesn't map to another in a natural way. The closest thing I can think of is probably a useful subset of JavaScript. Of course, if you are the language author you can limit it as much as you want, providing only constructs that are common to just about any language (loops, conditionals, etc.)
For purposes of mutability, an XML representation would be best, but you wouldn't want to code in it.
If you find that there is no universal language, you can try a pragmatic model-driven development approach, using a template-based code generator.
On the template you keep the underlying concepts of an algorithm. Then, you would add code for this algorithm in one or more specific languages (C++,Java,JS,Python) when necessary. You would have to do it anyway, whatever the language or approach you choose. A configuration switch would pick the correct language for any template you apply.
AtomWeaver is a code generator that works with templates and employs ABSE as the modeling approach.
I did some looking and found this.
https://www.indiegogo.com/projects/universal-programming-language
looks interesting
A classic Pascal is very simple. Oberon is another similar option. Or you could invent your own derivative language similar to the pseudocode from the computer science textbooks. It's trivial to implement a translator from one of that languages into any decent modern imperative language.
Both of those frameworks deal with meta-model:
XText (Eclipse)
MPS (JetBrain)
Do you have example of practical applications based on meta-model transformation with those tools?
We created whole bug tracker using MPS. Code generation is not the goal but mean to get some executable code. The goal is to give a tool to developer that allows creating DSLs with minimum effort.
Cool thing about MPS is that it also provides you with an IDE for your language. And different DSLs you create are compatible, i.e. you can create DSL that extends Java with closures and another DSL that enables external methods, and these extensions will work together.
They are different in term of document storing the metamodel.
Regarding XText, this article illustrates one usage, when it comes to y create your own programming languages and domain-specific languages (DSLs).
Once you have a language, you want to process it and this means usually to transform your model into another representation.
The facility responsible for this transformation is called generator and consists of a bunch of transformation templates (e.G. XPand) and some code executing them. On some event, the model is read in and the transformations are applied to produce code.
Example of such a model transformation:
dot3zest, which comes with a DOT to Zest interpreter (which now uses the Xtext switch API generated for the DOT grammar) is support for ad-hoc DOT edge definitions.
Regarding MPS, you have here a serie of practical examples,
like this code generation to GPL such as Java, C#, C++ or XML:
(source: googlecode.com)
I think the main usage of XText is firstly to create a DSL from the grammer you defined and an eclipse workbench auto-generated for you. Secondly, it can transform the scrpit written in your DSL into java. The built-in expressions from XText2 is a plus.
The framework gives you a free IDE to support your writing DSL you created. And the DSL is the ulimate product to provide. It can be used to abstract the rules and logics from the real world. For example, in our project, the product config rule. Only specialist knows them, so they write some in the DSL you create.