I'm trying to define a simple tokenizer for a language in lex.
Basically , I want to define tokens for parenthesis, comma, comparison ops, in/con/ncon ops, and logical ops. And I want any other token to match the 'keywords' regexp, as that would represent a STRINGARG in my language.
Everytime I try to feed it a string like 'A_FIELD', it gives me a LEXER ERROR. I want it to match 'keywords' and return a STRINGARG token.
Here is my .l file :
%{
#include "y.tab.h"
%}
lparen "("
rparen ")"
comma ","
comparison ("=="|"!="|">"|"<"|">="|"<=")
intok ("in"|"IN")
conncontok ("con"|"CON"|"ncon"|"NCON")
logical ("and"|"or"|"AND"|"OR"|"&"|"|")
keywords ( "(" | ")" | "," | "==" | "!=" | ">" | "<" | ">=" | "<=" | "in" | "IN" | "con" | "CON" | "ncon" | "NCON" | "and" | "AND" | "&" | "or"\
| "OR" | "|" )
%%
" " /* ignore whitespace */
{lparen} { return LPAREN; }
{rparen} { return RPAREN; }
{comma} { return COMMA; }
{comparison} { yylval.str = yytext; return COMPARISON; }
{intok} { return IN; }
{conncontok} { yylval.str = yytext; return CONNCON; }
{logical} { return LOGICAL; }
^keywords { yylval.str = yytext; return STRINGARG; }
. { printf("LEXER ERROR."); exit(1); }
%%
#ifndef yywrap
int yywrap() { return 1; }
#endif
I found the answer to this problem.
Basically I wanted a stringarg to be anything other than one of the recognized tokens. So when I set up my lex definition as follows, everything worked out fine. I should have been using character classes, not tokens in the last rule :
%%
" " /* ignore whitespace */
{lparen} { return LPAREN; }
{rparen} { return RPAREN; }
{comma} { return COMMA; }
{comparison} { yylval.str = yytext; return COMPARISON; }
{intok} { return IN; }
{conncontok} { yylval.str = yytext; return CONNCON; }
{logical} { return LOGICAL; }
**[^ \t\n]+ { yylval.str = yytext; return STRINGARG; }**
. { printf( "Lexer error." ); exit(1); }
%%
Related
I have a variable that stores a switch statement
$com = '
switch ($_)
{
1 {"It is one."}
2 {"It is two."}
3 {"It is three."}
4 {"It is four."}
}
'
I am trying to pipe in the number to run the switch statement
something like:
1 | iex($com)
Your options are:
A scriptblock or function with a process block:
$com = {
process {
switch ($_) {
1 { "one." }
2 { "two." }
3 { "three." }
}
}
}
function thing {
process {
switch ($_) {
1 { "one." }
2 { "two." }
3 { "three." }
}
}
}
1..3 | & $com
1..3 | thing
A filter, exactly the same functionality:
filter thing {
switch ($_) {
1 { "one." }
2 { "two." }
3 { "three." }
}
}
1..3 | thing
Using ScriptBlock.Create method (this would require a process block in the string expression):
$com = '
process {
switch ($_) {
1 { "one." }
2 { "two." }
3 { "three." }
}
}
'
1..3 | & ([scriptblock]::Create($com))
Using ScriptBlock.InvokeWithContext method and the automatic variable $input, this technique does not stream and also requires an outer scriptblock to work, it's just to showcase and should be discarded as an option:
$com = '
switch ($_) {
1 { "one." }
2 { "two." }
3 { "three." }
}
'
1..3 | & { [scriptblock]::Create($com).InvokeWithContext($null, [psvariable]::new('_', $input)) }
Using Invoke-Expression, also requires an outer scriptblock with a process block (should be discarded - from all techniques displayed above this is the worst one, the string expression is being evaluated per item passed through the pipeline):
$com = '
switch ($_) {
1 { "one." }
2 { "two." }
3 { "three." }
}
'
1..3 | & { process { Invoke-Expression $com } }
I am trying to calculate the number of combinations that it is possible to do in content spinning.
I do it in dart for an application coded with Flutter
For the following character string:
{hello | hi} {world | everyone}
There are 4 possible combinations.
For the following one:
{hi | {john | jane}}
There are 3 of them.
I succeed for the following character string:
{hello | hi} {world | everyone}
with the following agorithm:
final String pattern = "{ hello|hi } { world|everyone }";
RegExp regExp = RegExp(r"\{([^\{\}]*)\}");
final Iterable<RegExpMatch> matches = regExp.allMatches(pattern);
List<int> nMaches = [];
for (int i = 0; i < matches.length; i++) {
final List<String> pin = matches.elementAt(i).group(1)!.split("|");
nMaches.add(pin.length);
}
int possibilities= nMaches.fold(1, (previous, current) => previous * current);
print(possibilities);
But I'm having trouble designing the same code for the following string:
{hi | {john | jane}}
Could you help me ?
thanks in advance
Maybe something like this:
int getVariantsCount(String pattern) {
int count(
String partialPattern,
int from,
int accumulator,
) {
var endGroupIndex = 0;
var stop = false;
for (var i = 0; i < partialPattern.length; i++) {
if (partialPattern[i] == '}') {
if (i == partialPattern.length - 1 || partialPattern[i + 1] == '{') {
endGroupIndex = i + 1;
stop = i == partialPattern.length - 1;
break;
}
}
}
var substring = partialPattern.substring(0, endGroupIndex);
print(substring);
final inGroup = substring.codeUnits.fold(
1,
(previousValue, element) => element == '|'.codeUnits[0]
? (previousValue + 1)
: previousValue);
if (stop) {
return accumulator * inGroup;
}
return count(
partialPattern.substring(endGroupIndex),
endGroupIndex,
accumulator * inGroup,
);
}
return count(pattern.replaceAll(' ', ''), 0, 1);
}
Some tests:
print(getVariantsCount('{hello | hi}') == 2);
print(getVariantsCount('{nice | good {job | goal}}') == 3);
print(getVariantsCount('{hello}{world|everyone}') == 2);
print(getVariantsCount('{ hello|hi } { world|everyone }') == 4);
print(getVariantsCount('{ hello|hi|ola } { world|everyone }') == 6);
print(getVariantsCount('{hello | hi} {nice | good {job | goal}}') == 6);
print(
getVariantsCount('{hello | hi} {cute | nice | good {job | goal}}') == 8);
print(
getVariantsCount('{hello | hi | ok} {cute | nice | good {job | goal}}') ==
12);
print(getVariantsCount('{hello} {cute | nice | good {job | goal}}') == 4);
print(getVariantsCount('{hello | hi} {cute | nice | good} {job | goal}') == 12);
I'm trying to write a macro that will rewrite certain Rust control flow, but I'm having difficulty matching an if expression. The problem is that the predicate is an expression, but an expr is not permitted to be followed by a block or {.
The best I've got is to use tt:
macro_rules! branch {
(
if $pred:tt
$r1:block
else
$r2:block
) => {
if $pred {
$r1
} else {
$r2
}
};
}
Which works fine with single-token or grouped predicates:
branch! {
if (foo == bar) {
1
} else {
2
}
}
But fails if the predicate was not grouped:
branch! {
if foo == bar {
1
} else {
2
}
}
error: no rules expected the token `==`
I also tried to use a repeating pattern of tt in the predicate:
macro_rules! branch {
(
if $($pred:tt)+
$r1:block
else
$r2:block
) => {
if $($pred)+ {
$r1
} else {
$r2
}
};
}
But this produces an error because it's now ambiguous whether subsequent block should match the tt too:
error: local ambiguity: multiple parsing options: built-in NTs tt ('pred') or block ('r1').
Is there a way to do this, or am I stuck with inventing special syntax to use in the macro?
You could use a TT muncher to parse the predicate:
macro_rules! branch {
{
if $($rest:tt)*
} => {
branch_parser! {
predicate = ()
rest = ($($rest)*)
}
};
}
macro_rules! branch_parser {
{
predicate = ($($predicate:tt)*)
rest = ({ $($then:tt)* } else { $($else:tt)* })
} => {
println!("predicate: {}", stringify!($($predicate)*));
println!("then: {}", stringify!($($then)*));
println!("else: {}", stringify!($($else)*));
};
{
predicate = ($($predicate:tt)*)
rest = ($next:tt $($rest:tt)*)
} => {
branch_parser! {
predicate = ($($predicate)* $next)
rest = ($($rest)*)
}
};
}
fn main() {
branch! {
if foo == bar {
1
} else {
2
}
}
}
Output:
predicate: foo == bar
then: 1
else: 2
I'm converting VBScript to PowerShell and I have an issue with Select Case. I'm trying to find a way to do a Case Else.
switch($IncomingText.SubString(0,1)) {
"1" {
switch($IncomingText.SubString(12,3)) {
"601" {
$TempString = ($IncomingText.SubString(0,75))
if (RemoveSpaces($IncomingText.SubString(75,10)) = True) {
$TempString = $TempString + (StrReverse($IncomingText.SubString(75,10))) + ($IncomingText.SubString(85,10))
} else {
$TempString = $TempString + ($IncomingText.SubString(75,20))
}
return $TempString
Case Else
if (RemoveSpaces($IncomingText.SubString(155,10)) = True) {
$TempString = $TempString + (StrReverse($IncomingText.SubString(155,10))) + ($IncomingText.SubString(165))
} else {
$TempString = $TempString + ($IncomingText.SubString(155))
}
return $TempString
}
}
}
}
In VBScript there's this Case Else. I want to stay in the "601" and proceed with the next section. How do I do it in PowerShell? Since Case Else doesn't seems to be working for me.
Taken from Windows Powershell site
https://technet.microsoft.com/en-us/library/ff730937.aspx
So use default instead of case else.
$a = 5
switch ($a)
{
1 {"The color is red."}
2 {"The color is blue."}
3 {"The color is green."}
4 {"The color is yellow."}
5 {"The color is orange."}
6 {"The color is purple."}
7 {"The color is pink."}
8 {"The color is brown."}
default {"The color could not be determined."}
}
You're looking for default:
Switch ($something) {
1 { .. }
2 { .. }
default { .. }
}
I'd like to point out that the Switch command will evaluate scriptblocks similar to an If statement, so you could just create more complex cases:
switch($IncomingText) {
{$_.SubString(0,1) -eq '1' -and $IncomingText.SubString(12,3) -eq '601'}
{
$TempString = $IncomingText.SubString(0,75)
if (![string]::IsNullOrWhiteSpace($IncomingText.SubString(75,10))) {
$TempString = $TempString + (StrReverse($IncomingText.SubString(75,10))) + ($IncomingText.SubString(85,10))
} else {
$TempString = $TempString + ($IncomingText.SubString(75,20))
}
}
default
{
if (RemoveSpaces($IncomingText.SubString(155,10)) = True) {
$TempString = $TempString + (StrReverse($IncomingText.SubString(155,10))) + ($IncomingText.SubString(165))
} else {
$TempString = $TempString + ($IncomingText.SubString(155))
}
return $TempString
}
}
I would also like to note that it looks like you are removing spaces from a 10 character string, and testing to see if anything is left. If there is you reverse those 10 characters, then add the next 10 characters, if not you simply add all 20 characters as they are. My point is that if the first 10 characters are all spaces then reversing them is harmless, so you may as well just always reverse them. With that in mind your code gets a lot simpler:
Function StrReverse ($MyString){
$arrMyString = $MyString.ToCharArray()
[array]::Reverse($arrMyString)
$arrMyString -join ''
}
switch($IncomingText) {
{$_.SubString(0,1) -eq '1' -and $IncomingText.SubString(12,3) -eq '601'}
{
$TempString = $IncomingText.SubString(0,75) + (StrReverse $IncomingText.SubString(75,10)) + $IncomingText.SubString(85,10)
}
default
{
$TempString + (StrReverse $IncomingText.SubString(155,10)) + $IncomingText.SubString(165)
}
}
I am using JavaCC to print an AST in a particular format.
I need it to be like this :
LetNode( Identier(X), ExprNode( PlusNode( IntegerLiteral(8), IntegerLiteral(2) )))
but I am getting:
Start(LetNode(Identifier(x)(ExprNode(IntegerLiteral(5)(PlusNode(IntegerLiteral(5)()))
I am using the dump method to print this:
public void dump(String prefix) {
System.out.print(toString(prefix));
System.out.print("(");
if (children != null) {
for (int i = 0; i < children.length; ++i) {
SimpleNode n = (SimpleNode)children[i];
if (n != null) {
n.dump(prefix +"");
}
}
System.out.print(")");
}
}
}
The output without any modification is :
Start
Sxl
Statement
VariableDecl
Identifier
Type
Expression
SimpleExpression
Term
Factor
Literal
IntegerLiteral
AdditiveOp
Term
Factor
Literal
IntegerLiteral
My method:
void AdditiveOp():
{}
{
(Plus() /*#Add(2)*/ | Minus() | Or())
}
void Plus():
{
}
{
(< PLUS >)
#PlusNode
}
If I remove the /* */ it does not work as it says it expects something else not +
Any help would be greatly appreciated!
The usual way to make an AST with JJT for an expression grammar is to do something like this
void AdditiveExpression() #void :
{ Token t ; }
{
MultiplicativeExpression()
( "+" MultiplicativeExpression() #Add(2)
| "-" MultiplicativeExpression() #Subtract(2)
)*
}