Perl 5: How to suppress #var interpolation in regex patterns - perl

Question

This perl script doesn't substitute the input text_str as expected:
my $text_str ='public class ZipUtilTest extends TestCase {}';
my $find = '^(public class \\w+) extends TestCase \\{';
my $replace = '#RunWith(JUnit4.class)\\n\\1 {';
eval '$text_str =~ ' . "s#$find#$replace#mg";
say "$text_str";
Output (wrong):
(JUnit4.class)
public class ZipUtilTest {}
This revised perl script (with '#' in 'replace' escaped) substitutes as expected:
my $text_str ='public class ZipUtilTest extends TestCase {}';
my $find = '^(public class \\w+) extends TestCase \\{';
my $replace = '#RunWith(JUnit4.class)\\n\\1 {';
$replace =~ s/#/\\#/g; # Escape '#' to avoid Perl #var interpolation
eval '$text_str =~ ' . "s#$find#$replace#mg";
say "$text_str";
Output (correct):
#RunWith(JUnit4.class)
public class ZipUtilTest {}
It looks like '#RunWith' in the 'replace' pattern is treated as a Perl #variable and interpolated to an empty string.
Is there a better way to handle this than escaping the '#' character in patterns? If we have to do this, any other '#'-like characters need to be escaped?
(Note: This has nothing to do with using \Q\E to suppress the magic of regex meta characters. Please don't close this because of existing questions of that nature.)

Answer 1

You can use a positive-lookahead to match the { without capturing it in $1. Then the replacement string does not need to contain the $1.
When building a regex, it's better to use the regex quoting operator qr{} than strings; it will quote like a regex, not a string. This can avoid subtle bugs.
use v5.10;
my $text_str = 'public class ZipUtilTest extends TestCase {}';
# Use a positive-look ahead to match, but not capture, the {
# Quote as regex to avoid subtle quoting issues.
my $find = qr'^(public class \w+) extends TestCase(?>\s*\{)';
# Use double-quotes to interpolate the \n, but escape the \#.
my $replace = "\#RunWith(JUnit4.class)\n";
# Add the $1 to the end of the replacement.
$text_str =~ s{$find}{$replace$1};
say $text_str;
Demonstration.

Answer 2

It seems you want to load language-agnostic search and replace patterns from a configuration file, and then apply them via a Perl script.
If that is your goal, then using eval is not appropriate since Perl has syntax that you do not want to support, as you found out.
It is not reasonable to try to work around those Perl-specific parts by trying to escape them, since that can get rather complex. For example, you considered escaping occurrences of # as they can introduce an array name, but what if that character already is backslash-escaped? Handling this properly would require an almost complete re-implementation of Perl's string literal syntax, which doesn't sound like fun.
What I would do is to define a replacement string syntax of our own, so that we're completely independent from Perl's syntax.
For example, we might define our replacement string syntax to be entirely verbatim, except that we support certain backslash-escapes. Let's say that the syntax '\' DIGIT such as \1 replaces a capture, and that the usual backlash escapes are supported (\b \t \n \v \f \r \" \' \\ \x0A), which is the common subset of JavaScript string literals, Python 3 string literals, and Perl escapes, minus octal escapes. Note that these languages do not agree on a syntax for Unicode characters.
We can implement an interpreter for this string replacement language as follows: we parse the replacement string into an array of opcodes, alternating a literal string with the number of a capture. For example, the replacement pattern abc\1def would be parsed into ['abc', 1, 'def']:
sub parse_replacement_pattern {
my ($pattern) = #_;
my #ops = (''); # init with empty string
# use m//gc style parsing which lets us anchor patterns at the current "pos"
pos($pattern) = 0;
while (pos $pattern < length $pattern) {
if ($pattern =~ /\G([^\\]+)/gc) {
$ops[-1] .= $1;
}
elsif ($pattern =~ /\G\\n/gc) {
$ops[-1] .= "\n";
}
... # and so on for the basic escapes
elsif ($pattern =~ /\G\\x([0-9a-fA-F]{2})/gc) {
$ops[-1] .= chr $1;
}
elsif ($pattern =~ /\G\\([1-9])/gc) {
push #ops, $1, ''; # add replacement opcode + empty string
}
else {
die "invalid syntax";
}
}
return \#ops;
}
We can apply such a replacement pattern by looping through the operations, appending the literal string or the capture contents as appropriate.
sub apply_replacement_pattern {
my ($ops) = #_;
my $output = '';
my $is_capture = 0;
for my $op (#$ops) {
if ($is_capture) {
# we know that $op must be the number of a capture buffer
$output .= ${^CAPTURE}[$op - 1]; # like eval "\$$op"
}
else {
# we know that $op must be a literal string
$output .= $op;
}
$is_capture = !$is_capture;
}
return $output;
}
We can now use these functions in your test case:
my $text_str ='public class ZipUtilTest extends TestCase {}';
my $find = '^(public class \\w+) extends TestCase \\{';
my $replace = '#RunWith(JUnit4.class)\\n\\1 {';
my $replace_ops = parse_replacement_pattern($replace);
$text_str =~ s{$find}{apply_replacement_pattern($replace_ops)}mge;
say $text_str;
This produces the expected output
#RunWith(JUnit4.class)
public class ZipUtilTest {}

Answer 3

Here is a version that works: use $1 directly in the replacement side, not in a pre-made variable for it. It saves us some hassle.
use warnings;
use strict;
use feature 'say';
my $text_str = 'public class ZipUtilTest extends TestCase {}';
#say $text_str;
my $re = '^(public class \w+) extends TestCase \{';
#say $re;
my $replace = "\#RunWith(JUnit4.class)\n";
#say $replace;
$text_str =~ s/$re/${replace}$1 {/;
say $text_str;
Update with comments
The variables for the pattern and the replacement string are read from a configuration file. Then the "hassle" I mention becomes more serious.
If $1 is to be prepared in the replacement-string variable, it must be a mere string (of characters $ and 1) there while it need become a variable, and be evaluated, in the regex.
That means the variable must be eval-ed (or regex run with /ee), and that is the problem with the string form of eval -- input from outside: eval will evaluate (run) anything, any code. We don't need malicious action regarding text-to-become-code in config files, just consider typos.
As for nicely escaping (only) what need be escaped, one can prepare for that, a hash for example:
my %esc_char = ( at => '\#' ); # etc
and use this when composing the variable with the replacement string.
If both the pattern and replacement must come from config files and must be non-specific to Perl, as a comment says, then I am not sure how to improve the code offered in the question. Except that it should be heavily protected against running (accidentally, say) bad code.