ANLTR not walking whole tree when certain syntax appears - scala

Question

I am working on a project which reads source code of various languages. The project itself is written in scala but what I'm doing should be familiar if you know antlr. I have used the scala.g4 grammar on github to generate the Parser, Lexer etc for antlr4. I have written a subclass of ScalaBaseListener which simply prints on overridden the Enter methods
eg
override def enterClassDef(ctx: ScalaParser.ClassDefContext): Unit = {
println(ctx.getText)
}
In my application's main I am attempting to walk the whole tree from a file source like so:
import ScalaLexer._
import org.antlr.v4.runtime._
import org.antlr.v4.runtime.tree._
import scala.io.Source
object Main extends App {
val fileContents = Source.fromFile(args(0)).getLines.mkString
val charStream = new ANTLRInputStream(fileContents)
val lexer = new ScalaLexer(charStream)
val tokens = new CommonTokenStream(lexer)
val parser = new ScalaParser(tokens)
val tree = parser.compilationUnit
ParseTreeWalker.DEFAULT
.walk(new ScalaMySubclassListener(), tree)
}
I have found that if the source file is say, just a couple of classes:
class Foo {
def bar = {
1
}
def baz = 1
}
class Foo1 {
def bar = {
1
}
def baz = 1
}
I can see from the output of my program that every leaf in the tree is walked.
However, if I were to add an import statement at the top of the file (as there will often be in a scala source file)
import Thing._
class Foo {
def bar = {
1
}
def baz = 1
}
class Foo1 {
def bar = {
1
}
def baz = 1
}
only the leaves in the import statement get walked. The rest of the file gets ignored.
When I parse the source file using the antlr4 GUI, the whole tree is visible.

Answer 1

The first thing to do when the parse tree seems cut off is to check whether there are any syntax errors as that would be the most common cause. Since you didn't mess with error handling at all in your code, that means that any syntax errors should be printed to stderr. Since none are, there apparently weren't any syntax errors.
But let's not give up on the idea of there being a syntax error just yet. One common pitfall when it comes to syntax errors in ANTLR is if your start rule does not end with an EOF. If that's the case, ANTLR will simply try to find a prefix of the input that's syntactically valid and ignore the rest. That is, it will stop at the first syntax error without actually producing an error message (as long as there's a valid program leading up to that error - since many grammars accept empty programs, that is very often the case). And sure enough: if we look at Scala.g4 there's no EOF anywhere in the grammar (at the time of this writing anyway). So let's add EOF at the end of the compilationUnit rule. Now if we recompile everything and run your code again, we finally get a syntax error:
line 1:20 mismatched input 'Foo' expecting {<EOF>, '.', ',', 'implicit', 'lazy', 'case', '#', 'override', 'abstract', 'final', 'sealed', 'private', 'protected', 'import', 'class', 'object', 'trait', 'package'}
Now there's two things that might strike you as curious:
Why does ANTLR detect a syntax error when run from your code, but not from the TestRig GUI (even after adding the EOF, the GUI will still show a correct tree).
Why does the error message claim that Foo appears on column 20 of line 1 when it's actually on line 3?
The answer to both of these questions is the same: The input that you're feeding ANTLR is not what's in your test file. To verify this, try printing fileContents after you read it in. You'll see that the all input is on a single line, starting with import Thing._class Foo, which clearly isn't correct syntax.
The reason that happens is that getLines gives you a list of lines without line endings and mkString joins them together without any separator. The quick fix would be to simply pass "\n" as the separator to mkString, but the better solution is to not read the file yourself at all.
Instead you can make ANTLR do it by creating your input stream using CharStreams.fromFileName. This will also get rid of the warning about ANTLRInputStream being deprecated.