Operator Precedence with Scala Parser Combinators - scala

I am working on a Parsing logic that needs to take operator precedence into consideration. My needs are not too complex. To start with I need multiplication and division to take higher precedence than addition and subtraction.
For example: 1 + 2 * 3 should be treated as 1 + (2 * 3). This is a simple example but you get the point!
[There are couple more custom tokens that I need to add to the precedence logic, which I may be able to add based on the suggestions I receive here.]
Here is one example of dealing with operator precedence: http://jim-mcbeath.blogspot.com/2008/09/scala-parser-combinators.html#precedencerevisited.
Are there any other ideas?

This is a bit simpler that Jim McBeath's example, but it does what you say you need, i.e. correct arithmetic precdedence, and also allows for parentheses. I adapted the example from Programming in Scala to get it to actually do the calculation and provide the answer.
It should be quite self-explanatory. There is a heirarchy formed by saying an expr consists of terms interspersed with operators, terms consist of factors with operators, and factors are floating point numbers or expressions in parentheses.
import scala.util.parsing.combinator.JavaTokenParsers
class Arith extends JavaTokenParsers {
type D = Double
def expr: Parser[D] = term ~ rep(plus | minus) ^^ {case a~b => (a /: b)((acc,f) => f(acc))}
def plus: Parser[D=>D] = "+" ~ term ^^ {case "+"~b => _ + b}
def minus: Parser[D=>D] = "-" ~ term ^^ {case "-"~b => _ - b}
def term: Parser[D] = factor ~ rep(times | divide) ^^ {case a~b => (a /: b)((acc,f) => f(acc))}
def times: Parser[D=>D] = "*" ~ factor ^^ {case "*"~b => _ * b }
def divide: Parser[D=>D] = "/" ~ factor ^^ {case "/"~b => _ / b}
def factor: Parser[D] = fpn | "(" ~> expr <~ ")"
def fpn: Parser[D] = floatingPointNumber ^^ (_.toDouble)
}
object Main extends Arith with App {
val input = "(1 + 2 * 3 + 9) * 2 + 1"
println(parseAll(expr, input).get) // prints 33.0
}

Related

Why the expr parser only can parse the first item of it?

I have a Praser
package app
import scala.util.parsing.combinator._
class MyParser extends JavaTokenParsers {
import MyParser._
def expr =
plus | sub | multi | divide | num
def num = floatingPointNumber ^^ (x => Value(x.toDouble).e)
def plus = num ~ rep("+" ~> num) ^^ {
case num ~ nums => nums.foldLeft(num.e) {
(x, y) => Operation("+", x, y)
}
}
def sub = num ~ rep("-" ~> num) ^^ {
case num ~ nums => nums.foldLeft(num.e){
(x, y) => Operation("-", x, y)
}
}
def multi = num ~ rep("*" ~> num) ^^ {
case num ~ nums => nums.foldLeft(num.e){
(x, y) => Operation("*", x, y)
}
}
def divide = num ~ rep("/" ~> num) ^^ {
case num ~ nums => nums.foldLeft(num.e){
(x, y) => Operation("/", x, y)
}
}
}
object MyParser {
sealed trait Expr {
def e = this.asInstanceOf[Expr]
def compute: Double = this match {
case Value(x) => x
case Operation(op, left, right) => (op : #unchecked) match {
case "+" => left.compute + right.compute
case "-" => left.compute - right.compute
case "*" => left.compute * right.compute
case "/" => left.compute / right.compute
}
}
}
case class Value(x: Double) extends Expr
case class Operation(op: String, left: Expr, right: Expr) extends Expr
}
and I use it to parse the expression something
package app
object Runner extends App {
val p = new MyParser
println(p.parseAll(p.expr, "1 * 11"))
}
it prints
[1.3] failure: end of input expected
1 * 11
^
but if I parse the expression 1 + 11, it will succeed in parsing it.
[1.7] parsed: Operation(+,Value(1.0),Value(11.0))
and I can parse something through the plus , multi , divide , num , sub combinator , but the expr combinator only can parse the first item of the or combinator .
so why it only can parse the first item of the expr parser? And how can I change the definition of the parsers to make the parse successful ?
The problem is that you're using rep which matches zero or more times.
def rep[T](p: => Parser[T]): Parser[List[T]] = rep1(p) | success(List())
you need to use rep1 instead which would require at least one match.
If you replace all rep with rep1, your code works.
Check out the changes on scastie
Run an experiment:
println(p.parseAll(p.expr, "1 + 11"))
println(p.parseAll(p.expr, "1 - 11"))
println(p.parseAll(p.expr, "1 * 11"))
println(p.parseAll(p.expr, "1 / 11"))
What will happen?
[1.7] parsed: Operation(+,Value(1.0),Value(11.0))
[1.3] failure: end of input expected
1 - 11
^
[1.3] failure: end of input expected
1 * 11
^
[1.3] failure: end of input expected
1 / 11
+ is consumed, but everything else fails. Let's change def expr definition
def expr =
multi | plus | sub | divide | num
[1.3] failure: end of input expected
1 + 11
^
[1.3] failure: end of input expected
1 - 11
^
[1.7] parsed: Operation(*,Value(1.0),Value(11.0))
[1.3] failure: end of input expected
1 / 11
^
By moving multi to the beginning, * case passed, but + failed.
def expr =
num | multi | plus | sub | divide
[1.3] failure: end of input expected
1 + 11
^
[1.3] failure: end of input expected
1 - 11
^
[1.3] failure: end of input expected
1 * 11
^
[1.3] failure: end of input expected
1 / 11
With num as the first case everything fails. It is apparent now that this code
num | multi | plus | sub | divide
is NOT matching if any of its parts match, but only if the first one matches.
What does docs says about it?
/** A parser combinator for alternative composition.
*
* `p | q` succeeds if `p` succeeds or `q` succeeds.
* Note that `q` is only tried if `p`s failure is non-fatal (i.e., back-tracking is allowed).
*
* #param q a parser that will be executed if `p` (this parser) fails (and allows back-tracking)
* #return a `Parser` that returns the result of the first parser to succeed (out of `p` and `q`)
* The resulting parser succeeds if (and only if)
* - `p` succeeds, ''or''
* - if `p` fails allowing back-tracking and `q` succeeds.
*/
def | [U >: T](q: => Parser[U]): Parser[U] = append(q).named("|")
Important note: back tracking has to be allowed. If it isn't, then failure to match the first parser, will results in failing the alternative without trying the second parser at all.
How to make your parser backtracking? Well, you would have to use PackratParsers as this is the only parser in the library that supports backtracking. Or rewrite your code to not rely on backtracking in the first place.
Personally, I recommend not using Scala Parser Combinators and instead use a library where you explicitly decide when you can still backtrack, and when you should not allow it, like e.g. fastparse.

Why doesn't parser combinator backtrack?

Consider
import util.parsing.combinator._
object TreeParser extends JavaTokenParsers {
lazy val expr: Parser[String] = decimalNumber | sum
//> expr: => TreeParser.Parser[String]
lazy val sum: Parser[String] = expr ~ "+" ~ expr ^^ {case a ~ plus ~ b => s"($a)+($b)"}
//> sum: => TreeParser.Parser[String]
println(parseAll(expr, "1 + 1")) //> TreeParser.ParseResult[String] = [1.3] failure: string matching regex
//| `\z' expected but `+' found
}
The same story with fastparse
import fastparse.all._
val expr: P[Any] = P("1" | sum)
val sum: P[Any] = expr ~ "+" ~ expr
val top = expr ~ End
println(top.parse("1+1")) // Failure(End:1:2 ..."+1")
Parsers are great to discover that taking the first literal is a bad idea but do not try to fall back to the sum production. Why?
I understand that parser takes the first branch that can successfully eat up a part of input string and exits. Here, "1" of expression matches the first input char and parsing completes. In order to grab more, we need to make sum the first alternative. However, plain stupid
lazy val expr: Parser[String] = sum | "1"
endы up with stack overflow. The library authors therefore approach it from another side
val sum: P[Any] = P( num ~ ("+".! ~/ num).rep )
val top: P[Any] = P( sum ~ End )
Here, we start sum with terminal, which is fine but this syntax is more verbose and, furthermore, it produces a terminal, followed by a list, which is good for a reduction operator, like sum, but is difficult to map to a series of binary operators.
What if your language defines expression, which admits a binary operator? You want to match every occurrence of expr op expr and map it to a corresponding tree node
expr ~ "op" ~ expr ^^ {case a ~ _ ~ b => BinOp(a,b)"}
How do you do that? In short, I want a greedy parser, that consumes the whole string. This is what I mean by 'greedy' rather than greedy algorigthm that jumps into the first wagon and ends up in a dead end.
As I have found here, we need to replace | alternative operator with secret |||
//lazy val expr: Parser[String] = decimalNumber | sum
lazy val backtrackGreedy: Parser[String] = decimalNumber ||| sum
lazy val sum: Parser[String] = decimalNumber ~ "+" ~ backtrackGreedy ^^ {case a ~ plus ~ b => s"($a)+($b)"}
println(parseAll(backtrackGreedy, "1 + 1")) // [1.6] parsed: (1)+(1)
The order of alternatives does not matter with this operator. To stop stack overflow, we need to eliminate the left recursion, sum = expr + expr => sum = number + expr.
Another answer says that we need to normalize, that is instead of
def foo = "foo" | "fo"
def obar = "obar"
def foobar = foo ~ obar
we need to use
def workingFooBar = ("foo" ~ obar) | ("fo" ~ obar)
But first solution is more striking.
The parser does backtrack. Try val expr: P[String] = P(("1" | "1" ~ "+" ~ "1").!) and expr.parse("1+1") for example.
The problem is in your grammar. expr parses 1 and it is a successful parsing by your definition. Then sum fails and now you want to blame the dutiful expr for what happened?
There are plenty of examples on how to deal with binary operators. For example, the first example here: http://lihaoyi.github.io/fastparse/

Parser combinator grammar not yielding correct associativity

I am working on a simple expression parser, however given the following parser combinator declarations below, I can't seem to pass my tests and a right associative tree keeps on popping up.
def EXPR:Parser[E] = FACTOR ~ rep(SUM|MINUS) ^^ {case a~b => (a /: b)((acc,f) => f(acc))}
def SUM:Parser[E => E] = "+" ~ EXPR ^^ {case "+" ~ b => Sum(_, b)}
def MINUS:Parser[E => E] = "-" ~ EXPR ^^ {case "-" ~ b => Diff(_, b)}
I've been debugging hours for this. I hope someone can help me figure it out it's not coming out right.
"5-4-3" would yield a tree that evaluates to 4 instead of the expected -2.
What is wrong with the grammar above?
I don't work with Scala but do work with F# parser combinators and also needed associativity with infix operators. While I am sure you can do 5-4 or 2+3, the problem comes in with a sequence of two or more such operators of the same precedence and operator, i.e. 5-4-2 or 2+3+5. The problem won't show up with addition as (2+3)+5 = 2+(3+5) but (5-4)-2 <> 5-(4-2) as you know.
See: Monadic Parser Combinators 4.3 Repetition with meaningful separators. Note: The separators are the operators such as "+" and "*" and not whitespace or commas.
See: Functional Parsers Look for the chainl and chainr parsers in section 7. More parser combinators.
For example, an arithmetical expressions, where the operators that
separate the subexpressions have to be part of the parse tree. For
this case we will develop the functions chainr and chainl. These
functions expect that the parser for the separators yields a function
(!);
The function f should operate on an element and a list of tuples, each
containing an operator and an element. For example, f(e0; [(1; e1);
(2; e2); (3; e3)]) should return ((eo 1 e1) 2 e2) 3 e3. You may
recognize a version of foldl in this (albeit an uncurried one), where
a tuple (; y) from the list and intermediate result x are combined
applying x y.
You need a fold function in the semantic parser, i.e. the part that converts the tokens from the syntactic parser into the output of the parser. In your code I believe it is this part.
{case a~b => (a /: b)((acc,f) => f(acc))}
Sorry I can't do better as I don't use Scala.
"-" ~ EXPR ^^ {case "-" ~ b => Diff(_, b)}
for 5-4-3, it expands to
Diff(5, 4-3)
which is
Diff(5, Diff(4, 3))
however, what you need is:
Diff(Diff(5, 4), 3))
// for 5 + 4 - 3 it should be
Diff(Sum(5, 4), 3)
you need to involve stack.
It seems using "+" ~ EXPR made the answer incorrect. It should have been FACTOR instead.

Scala rep() and associativity

I want to express this grammar in scala StdTokenParsers:
expr -> expr ("+"|"-") ~ muldivexpr | muldivexpr
"+" and "-" is left associative.
The grammar is left recursive so it caused infinite recursions. I can rewrite to remove left recursion, but it will change to right associativity.
Now I am planning to use scala rep() to rewrite it as:
expr -> rep(muldivexpr ("+"|"-")) ~ muldivexpr
but will rep() that change the associativity? How does rep() work in this case?
I am asking this question because I have to output the AST in the future.
You are most likely looking for:
chainl1[T](p: => Parser[T], q: => Parser[(T, T) => T]): Parser[T]
The general idea is p is an operand, and q is a separator yielding a function which can combine two operands, e.g.
chainl1(muldivexpr,
"+" ^^^ { (l: Expr, r: Expr) => Addition(l, r) }
| "-" ^^^ { (l: Expr, r: Expr) => Subtraction(l, r) }
)

Scala combinator parsers - distinguish between number strings and variable strings

I'm doing Cay Horstmann's combinator parser exercises, I wonder about the best way to distinguish between strings that represent numbers and strings that represent variables in a match statement:
def factor: Parser[ExprTree] = (wholeNumber | "(" ~ expr ~ ")" | ident) ^^ {
case a: wholeNumber => Number(a.toInt)
case a: String => Variable(a)
}
The second line there, "case a: wholeNumber" is not legal. I thought about a regexp, but haven't found a way to get it to work with "case".
I would split it up a bit and push the case analysis into the |. This is one of the advantages of combinators and really LL(*) parsing in general:
def factor: Parser[ExprTree] = ( wholeNumber ^^ { Number(_.toInt) }
| "(" ~> expr <~ ")"
| ident ^^ { Variable(_) } )
I apologize if you're not familiar with the underscore syntax. Basically it just means "substitute the nth parameter to the enclosing function value". Thus { Variable(_) } is equivalent to { x => Variable(x) }.
Another bit of syntax magic here is the ~> and <~ operators in place of ~. These operators mean that the parsing of that term should include the syntax of both the parens, but the result should be solely determined by the result of expr. Thus, the "(" ~> expr <~ ")" matches exactly the same thing as "(" ~ expr ~ ")", but it doesn't require the extra case analysis to retrieve the inner result value from expr.