Eclipse editor plug-in: syntax highlighting - eclipse

Question

I'm working on an editor plugin for a custom language and I've managed to set it up so all the necessary keywords highlight. The problem is the words become highlighted even if they are part of another word.
For example: let's say public is a keyword and I initialize a variable called publicVar so it looks like this public int publicVar. public highlights as expected but the 'public' part of publicVar is also highlighted which is not what I want.
public WFSPartitionScanner()
{
int index = 0;
int numOfRules = 5 + reversedWords.length + commonFunctions.length+directives.length +
BIFs.length + operators.length + strongOperators.length;
IToken string = new Token(WFS_STRING);
IToken comment = new Token(WFS_COMMENT);
IToken reversedWord = new Token(WFS_REVERSED_WORD);
IToken commonFunction = new Token(WFS_COMMON_FUNCTION);
IToken directive = new Token(WFS_DIRECTIVE);
IToken bif = new Token(WFS_BIF);
IToken operator = new Token(WFS_OPERATOR);
IToken strongOperator = new Token(WFS_STRONG_OPERATOR);
IToken numberToken = new Token(WFS_NUMBER);
IPredicateRule[] rules= new IPredicateRule[numOfRules];
rules[index] = new MultiLineRule("\"","\"", string, '\\');
rules[++index] = new MultiLineRule("\'", "\'", string, '\\');
rules[++index] = new SingleLineRule("//","\n", comment);
rules[++index] = new MultiLineRule("/*", "*/", comment);
rules[++index] = new WFSNumberRule(numberToken);
for(int i = 0; i < reversedWords.length; i++)
{
rules[++index] = new WordPatternRule(new WordDetector(reversedWords[i]), reversedWords[i], "",reversedWord);
}
for(int i = 0; i < commonFunctions.length; i++)
{
rules[++index] = new WordPatternRule(new WordDetector(commonFunctions[i]), commonFunctions[i], "",commonFunction);
}
for(int i = 0; i < BIFs.length;i++)
{
rules[++index] = new WordPatternRule(new WordDetector(BIFs[i]), BIFs[i], "",bif);
}
for(int i = 0; i < directives.length;i++)
{
rules[++index] = new WordPatternRule(new WordDetector(directives[i]), directives[i], "",directive);
}
for(int i=0; i < operators.length; i++)
{
rules[++index]= new WordPatternRule(new WordDetector(operators[i]), operators[i], "", operator);
}
for(int i=0; i < strongOperators.length; i++)
{
rules[++index]= new WordPatternRule(new WordDetector(strongOperators[i]), strongOperators[i], "", strongOperator);
}
setPredicateRules(rules);
}
public class WordDetector implements IWordDetector{
private char start;
private char[] part;
public WordDetector(String word)
{
this.start = word.charAt(0);
this.part = new char[word.length() - 1];
for(int i = 1; i < word.length(); i++)
{
part[i-1] = word.charAt(i);
}
}
#Override
public boolean isWordPart(char c) {
for(int i = 0; i < part.length; i++)
{
if(c == part[i])
{
return true;
}
}
return false;
}
#Override
public boolean isWordStart(char c) {
return (c == start);
}
}
I've also tried changing the WordPatternRule from
WordPatternRule(new WordDetector('KEYWORD'), 'KEYWORD', "",reversedWord);
to
WordPatternRule(new WordDetector('KEYWORD'), 'FIRST LETTER OF KEYWORD', 'LAST LETTER OF KEYWORD,reversedWord);
but I got the same results.

Answer 1

Ok I figured out part of the problem. To fix it i made my own WORDRULE
public class WFSWordRule extends WordRule implements IPredicateRule{
private IToken successToken;
public WFSWordRule(IWordDetector detector, String[] keywords, IToken token ) {
super(detector);
this.successToken= token;
for (String word : keywords) {
addWord(word, token);
}
}
#Override
public IToken evaluate(ICharacterScanner scanner, boolean arg1) {
return super.evaluate(scanner);
}
#Override
public IToken getSuccessToken() {
return successToken;
}
}
this doesn't completely solve the problem though. Now if there is a keyword at the end of a word the keyword is still highlighted. Using the same example as before, if i have a keyword 'Public' and i have a variable called 'varPublic' public is highlighted in both cases. But if i have a variable called 'PublicVar' public is not highlighted. any tips?

Answer 2

If you're trying to highlight words, not parts of words, I think you should use WordRule instead of WordPatternRule, as you have. From what I understand, WordPatternRule is used for finding patterns within a word, whereas WordRule is used for finding individual words.
I've used WordRules in an Eclipse plug in which I've been working on, and I don't have the problem of words being highlighted within other words. You can look at its code and use it as an example. Basically, it takes an IWordDetector implementation and you use its addWord method to add all the words you want it to detect.
Unlike CombinedWordRule, which greg-449 mentioned, it's a public part of the JFace framework which you are already using.

Answer 3

Not sure, if this is still useful, but I fixed this issue by overriding the method nextToken in my Scanner:
public IToken nextToken() {
if (this.fContentType == null || this.fRules == null) {
// don't try to resume
this.fTokenOffset = this.fOffset;
this.fColumn = RuleBasedScanner.UNDEFINED;
if (this.fRules != null) {
for (IRule fRule : this.fRules) {
IToken token = fRule.evaluate(this);
try {
if (!token.isUndefined()) {
if (this.fTokenOffset > 0 && (isWordEnd(this.fDocument.getChar(this.fTokenOffset - 1))
|| isSpecialKey(this.fDocument.getChar(this.fTokenOffset)))) {
this.fContentType = null;
return token;
}
}
} catch (BadLocationException ex) {
ex.printStackTrace();
}
}
}
if (this.read() == ICharacterScanner.EOF) {
return Token.EOF;
}
return this.fDefaultReturnToken;
}
}
isWordEnd checks for:
c == (char) 0 || c == ' ' || c == '\t' || c == '\n' || c == '\r' || c == '(' || c == ')' || c == ',' || c == ';'
isSpecialKey checks for:
c == '%'
For the other Issue, you've already fixed, I reimplemented the endSequenceDetected method in my rule:
#Override
protected boolean endSequenceDetected(ICharacterScanner scanner) {
int c = scanner.read();
scanner.unread();
return isWordEnd((char) c);
}