What's wrong with this simplest jflex code? - jflex

Question

I'm learning jflex, and wrote a simplest jflex code, which makes a single character #:
import com.intellij.lexer.FlexLexer;
import com.intellij.psi.tree.IElementType;
%%
%class PubspecLexer
%implements FlexLexer
%unicode
%type IElementType
%function advance
%debug
Comment = "#"
%%
{Comment} { System.out.println("Found comment!"); return PubTokenTypes.Comment; }
. { return PubTokenTypes.BadCharacter; }
Then I generate a PubspecLexer class, and try it:
public static void main(String[] args) throws IOException {
PubspecLexer lexer = new PubspecLexer(new StringReader("#!!!!!"));
for (int i = 0; i < 3; i++) {
IElementType token = lexer.advance();
System.out.println(token);
}
}
But it prints 3 nulls:
null
null
null
Why it neither return Comment nor BadCharacter?

Answer 1

It's not jflex problem, actually, it's because the idea-flex changes original usage.
When use jflex to write intellij-idea plugins, we are using a patched "JFlex.jar" and "idea-flex.skeleton", later defines the zzRefill method as:
private boolean zzRefill() throws java.io.IOException {
return true;
}
Instead of original:
private boolean zzRefill() throws java.io.IOException {
// ... ignore some code
/* finally: fill the buffer with new input */
int numRead = zzReader.read(zzBuffer, zzEndRead,
zzBuffer.length-zzEndRead);
// ... ignore some code
// numRead < 0
return true;
}
Notice there is a zzReader in the code, and which holds the string #!!!!! I passed in. But in the idea-flex version, which is never used.
So to work with idea-flex version, I should use it like this:
public class MyLexer extends FlexAdapter {
public MyLexer() {
super(new PubspecLexer((Reader) null));
}
}
Then:
public static void main(String[] args) {
String input = "#!!!!!";
MyLexer lexer = new MyLexer();
lexer.start(input);
for (int i = 0; i < 3; i++) {
System.out.println(lexer.getTokenType());
lexer.advance();
}
}
Which prints:
match: --#--
action [19] { System.out.println("Found comment!"); return PubTokenTypes.Comment(); }
Found comment!
Pub:Comment
match: --!--
action [20] { return PubTokenTypes.BadCharacter(); }
Pub:BadCharacter
match: --!--
action [20] { return PubTokenTypes.BadCharacter(); }
Pub:BadCharacter