When I run this without using the extra-escape for the "\n", hexdump doesn't print the 0a for the embedded newline.
Why does the "\n" need here an extra-treatment?
(While searching for an answer I found String::ShellQuote which does the escaping.)
#!/usr/bin/env perl
use warnings;
use 5.012;
use utf8;
binmode STDOUT, ':utf8';
use charnames qw(:full);
use IPC::System::Simple qw(system);
for my $i ( 0x08 .. 0x0d ) {
printf "0x%02x - %s\n", $i, '\N{' . charnames::viacode( $i ) . '}';
my $string = "It" . chr( $i ) . "s";
$string =~ s/\n/\\n/g;
system( "echo -e \Q$string\E | hexdump -C" );
say "";
}
When you don't convert the newline to the two characters \n, you're executing the command
echo -e \
| hexdump -C
To sh, that's equivalent to
echo -e | hexdump -C
When you convert the newline to the two characters \n, you're executing the command
echo -e \\n | hexdump -C
That passes the two characters \n to echo, for which it outputs a newline under -e.
You don't need to use -e and to create escapes for -e. You could create a proper shell command. That command would be:
echo '
' | hexdump -C
You can do that a number of ways. You could roll out your own solution.
(my $sh_literal = $string) =~ s/'/'\\''/g;
$sh_literal = "'$sh_literal'";
system( "echo $sh_literal | hexdump -C" );
There is String::ShellQuote.
use String::ShellQuote qw( shell_quote );
my $sh_literal = shell_quote($string);
system( "echo $sh_literal | hexdump -C" );
Finally, you could avoid the shell entirely.
open(my $fh, "|-", "hexdump", "-vC")
or die("Could not start hexdump: $!\n");
print($fh $string);
as #mugenkenichi commented echo is interpreting your strings too, so you have to escape special characters twice, once for perl and once for echo.
Instead this approach might be more convenient:
#!/usr/bin/env perl
use warnings;
use 5.012;
use utf8;
binmode STDOUT, ':utf8';
use charnames qw(:full);
use IPC::System::Simple qw(system);
for my $i ( 0x08 .. 0x0d ) {
printf "0x%02x - %s\n", $i, '\N{' . charnames::viacode($i) . '}';
my $string = "It" . chr($i) . "s";
open( my $fh, "| hexdump -vC" )
or die "could not talk to hexdump";
print $fh $string;
say "";
}
Related
I basically would like to do this:
$_ = "some content that need to be escaped &>|\"$\'`\s\\";
qx{echo $_ | foo}
There are two problems here. First the content of $_ needs to be escaped as it can contain binary data. Second, invoking echo might be slightly inefficient.
How can I simply pipe some content as STDIN to a command in Perl?
The following assume #cmd contains the program and its arguments (if any).
my #cmd = ('foo');
If you want to capture the output, you can use any of the following:
use String::ShellQuote qw( shell_quote );
my $cmd1 = shell_quote('printf', '%s', $_);
my $cmd2 = shell_quote(#cmd);
my $output = qx{$cmd1 | $cmd2};
use IPC::Run3 qw( run3 );
run3(\#cmd, \$_, \my $output);
use IPC::Run qw( run );
run(\#cmd, \$_, \my $output);
If you don't want to capture the output, you can use any of the following:
use String::ShellQuote qw( shell_quote );
my $cmd1 = shell_quote('printf', '%s', $_);
my $cmd2 = shell_quote(#cmd);
system("$cmd1 | $cmd2");
system('/bin/sh', '-c', 'printf "%s" "$0" | "$#"', $_, #cmd);
use String::ShellQuote qw( shell_quote );
my $cmd = shell_quote(#cmd);
open(my $pipe, '|-', $cmd);
print($pipe $_);
close($pipe);
open(my $pipe, '|-', '/bin/sh', '-c', '"$#"', 'dummy', #cmd);
print($pipe $_);
close($pipe);
use IPC::Run3 qw( run3 );
run3(\#cmd, \$_);
use IPC::Run qw( run );
run(\#cmd, \$_);
If you don't want to capture the output, but you don't want to see it either, you can use any of the following:
use String::ShellQuote qw( shell_quote );
my $cmd1 = shell_quote('printf', '%s', $_);
my $cmd2 = shell_quote(#cmd);
system("$cmd1 | $cmd2 >/dev/null");
system('/bin/sh', '-c', 'printf "%s" "$0" | "$#" >/dev/null', $_, #cmd);
use String::ShellQuote qw( shell_quote );
my $cmd = shell_quote(#cmd);
open(my $pipe, '|-', "$cmd >/dev/null");
print($pipe $_);
close($pipe);
open(my $pipe, '|-', '/bin/sh', '-c', '"$#" >/dev/null', 'dummy', #cmd);
print($pipe $_);
close($pipe);
use IPC::Run3 qw( run3 );
run3(\#cmd, \$_, \undef);
use IPC::Run qw( run );
run(\#cmd, \$_, \undef);
Notes:
The solutions using printf will impose a limit on the size of the data to pass to the program's STDIN.
The solutions using printf are unable to pass a NUL to the program's STDIN.
The presented solutions that use IPC::Run3 and IPC::Run don't involve a shell. This avoids problems.
You should probably use system and capture from IPC::System::Simple instead of the builtin system and qx to get "free" error checking.
This answer is a very naive approach. It's prone to deadlock. Don't use it!
ikegami explains in a comment below:
If the parent writes enough to the pipe attached to the child's STDIN, and if the child outputs enough to the pipe attached to its STDOUT before it reads from its STDIN, there will be a deadlock. (This can be as little as 4KB on some systems.) The solution involved using something like select, threads, etc. The better solution is to use a tool that has already solved the problem for you (IPC::Run3 or IPC::Run). IPC::Open2 and IPC::Open3 are too low-level to be useful in most circumstances
I'll leave the original answer, but encourage readers to pick the solution from one of the other answers instead.
You can use open2 from IPC::Open2 to read and write to the same process.
Now you don't need to care about escaping anything.
use IPC::Open2;
use FileHandle;
my $writer = FileHandle->new;
my $reader = FileHandle->new;
my $pid = open2( $reader, $writer, 'wc -c' );
# write to the pipe
print $writer 'some content that need to be escaped &>|\"$\'`\s\\';
# tell it you're done
$writer->close;
# read the out of the pipe
my $line = <$reader>;
print $line;
This will print 48.
Note that you can't use double quotes "" for the exact input you showed because the number of backslashes \ is wrong.
See perldoc open and perlipc for more information.
I like the solution provided by #simbabque since it avoids calling the Shell. Anyway, for comparison, a shorter solution can be obtained using Bash (but avoiding echo) by using a Bash Here string:
$_ = q{some content that need to be escaped &>|\"$\'`\s\\};
$_ =~ s/'/'"'"'/g; # Bash needs single quotes to be escaped
system 'bash', '-c', "foo <<< '$_'";
And, if you need to capture the output of the command:
use Capture::Tiny 'capture_stdout';
my $res = capture_stdout { system 'bash', '-c', "foo <<< '$_'" };
I'm able to do this on the command line and it works :
~/Tools/perl/edif_extr_cell.pl design.edif nmos1p8v | perl -p -e 's/^/\n/ if /portImplementation|figure\s+device/;' | perl -n -000 -e 'print if /portImplementation/;'
(basically, extracting a section of the EDIF file).
Now, I want to make a utility of this. And my script is below. Question : can this code be more efficient? If feel like it's very inelegant. I could pipe streams easily on the command line but, in a script, I feel lost.
#!/usr/bin/perl -w -p
BEGIN{ $file = '';}
s/^/\n/ if /portImplementation|figure\s+device/;
$file .= $_;
END{
$cmd = q{\rm -f /tmp/dump}.$$.'.txt';
system( $cmd );
open( OUT, ">/tmp/dump$$.txt");
print OUT $file;
close OUT;
$out = `perl -n -000 -e 'print if /portImplementation/;' /tmp/dump$$.txt`;
system( $cmd );
print $out;
}
If I understand correct, you want to be able to do
~/Tools/perl/edif_extr_cell.pl design.edif nmos1p8v | myfilter
Ideally, you'd merge the two Perl scripts into one rather than having one script launch two instances of Perl, but this turns out to be rather hard because of the change to $/ (via -00) and because you insert newlines in the first filter.
The simplest answer:
#!/bin/sh
perl -pe's/^/\n/ if /portImplementation|figure\s+device/' |
perl -00ne'print if /portImplementation/'
It appears that you were trying to write the equivalent of that sh script in Perl. It would look like the following:
#!/usr/bin/perl
use strict;
use warnings;
use IPC::Open qw( open3 );
# open3 has issues with lexical file handles.
pipe(local *PIPE_READER, local *PIPE_WRITER)
or die($!);
my $pid1 = open3('<&STDIN', '>&PIPE_WRITER', '>&STDERR',
'perl', '-pes/^/\n/ if /portImplementation|figure\s+device/');
my $pid2 = open3('<&PIPE_READER', '>&STDOUT', '>&STDERR',
'perl', '-00neprint if /portImplementation/');
waitpid($pid1);
waitpid($pid2);
I'd normally recommend IPC::Run3 or IPC::Run for launching and interfacing with child processes, but low-level open3 does the trick nicely in this particular situation.
I downloaded a random EDIF file from GitHub, running the following script on it gives the same output as your code:
#! /usr/bin/perl
use warnings;
use strict;
my #buffer;
my $found;
my $prepend = q();
while (<>) {
if (/portImplementation|figure\s+device/) {
if ($found && #buffer) {
print $prepend, #buffer;
$prepend = "\n";
}
undef $found;
#buffer = ();
}
$found ||= /portImplementation/;
push #buffer, $_;
}
# Don't forget to output the last paragraph!
print $prepend, #buffer if $found && #buffer;
I use Perl scripts to run commands for several users in several VOBs on ClearCase. I have a list of the VOBs which I read in from a text file. I then loop on that list and do whatever ClearCase command I am trying to do. However, this time the script does not seem to work. If I print out the command to the screen then go and copy and paste it at the prompt it works fine. It just will not executed from the Perl script. The only difference I saw was the fmt characters but even when I remove that it does not execute. I tried first putting the fmt on the line directly then tried setting them to variables. You will see the first comment line is the one that failed but I left it there as an example of what I tried. The last two comments are from another script that I run like this that does work.
Code:
#! /usr/local/bin/perl -w
use strict;
open(VOBS,"vobs.txt") || die "Can't open: !$\n";
my $u = '%u';
my $a ='%Ad';
my $n ='%N/n';
my $user='john';
my $ct = '/usr/atria/bin/cleartool';
while(my $newvobs=<VOBS>){
chomp($newvobs);
my $tag = $newvobs;
print "\n $tag \n";
print " $ct lstype -kind brtype -invob $tag | grep $user ";
`$ct lstype -kind brtype -invob $tag | grep $user`;
# `/usr/atria/bin/cleartool lstype -kind brtype -invob $tag -fmt '%u %Ad %N/\n' `;
# print "\n cleartool rmtag -view $tag \n";
#`/usr/atria/bin/cleartool rmtag -view $tag `;
}
close(VOBS);
Actually Your program runs, but is does not print anything.
Example:
#!/usr/bin/perl
use strict;
use warnings;
my $cmd = "cat";
`$cmd $0 | grep warning`;
Output: (nothing)
Easiest to fix. Last line
print `$cmd $0 | grep warning`
Output:
use warnings;
print `$cmd $0 | grep warning`;
If You need the exit code, replace last line with
my $exit = system("$cmd $0 | grep warning");
print $exit;
Output:
use warnings;
my $exit = system("$cmd $0 | grep warning");
0
Or use open to process output:
open my $fh, "$cmd $0 | grep warning|" or die;
while (<$fh>) { print $_; }
close $fh;
Output:
use warnings;
open my $fh, "$cmd $0 | grep warning|" or die;
But I could suggest something like bellow. Using AUTOLOAD the clearcase commands can be used as internal perl commands.
#!/usr/bin/perl
use strict;
use warnings;
sub AUTOLOAD {
(my $sub = $::AUTOLOAD) =~ s/.*:://;
print "---\n";
system("time $sub #_");
print "---\n";
}
my $cmd = "cat";
eval "$cmd($0)";
Output:
---
#!/usr/bin/perl
use strict;
use warnings;
sub AUTOLOAD {
(my $sub = $::AUTOLOAD) =~ s/.*:://;
print "---\n";
system("time $sub #_");
print "---\n";
}
cat($0);
0.00user 0.00system 0:00.00elapsed 400%CPU (0avgtext+0avgdata 2112maxresident)k
0inputs+0outputs (0major+174minor)pagefaults 0swaps
---
For reference, here is a perl script which uses -fmt, "Finding the latest baseline for a component":
Example:
ccperl StreamComp.pl mystream#\pvobtag | findstr {component}
Script "streamcomp.pl":
#!/usr/bin/perl -w
my $cmdout = `cleartool desc -fmt '%[latest_bls]CXp' stream:$ARGV[0]`;
my #baselines = split(/,/,$cmdout);
foreach $baseline (#baselines)
{
$compname=`cleartool desc -fmt '%[component]p' $baseline`;
printf("%-30s \t %s\n", $compname, $baseline);
}
It is a program working for a ClearCase UCM environment, but it could give you an idea of the kind of working statements (try without grep first) you could try to reproduce in your own base ClearCase program.
I'm looking for a simple/elegant way to grep a file such that every returned line must match every line of a pattern file.
With input file
acb
bc
ca
bac
And pattern file
a
b
c
The command should return
acb
bac
I tried to do this with grep -f but that returns if it matches a single pattern in the file (and not all). I also tried something with a recursive call to perl -ne (foreach line of the pattern file, call perl -ne on the search file and try to grep in place) but I couldn't get the syntax parser to accept a call to perl from perl, so not sure if that's possible.
I thought there's probably a more elegant way to do this, so I thought I'd check. Thanks!
===UPDATE===
Thanks for your answers so far, sorry if I wasn't clear but I was hoping for just a one-line result (creating a script for this seems too heavy, just wanted something quick). I've been thinking about it some more and I came up with this so far:
perl -n -e 'chomp($_); print " | grep $_ "' pattern | xargs echo "cat input"
which prints
cat input | grep a | grep b | grep c
This string is what I want to execute, I just need to somehow execute it now. I tried an additional pipe to eval
perl -n -e 'chomp($_); print " | grep $_ "' pattern | xargs echo "cat input" | eval
Though that gives the message:
xargs: echo: terminated by signal 13
I'm not sure what that means?
One way using perl:
Content of input:
acb
bc
ca
bac
Content of pattern:
a
b
c
Content of script.pl:
use warnings;
use strict;
## Check arguments.
die qq[Usage: perl $0 <input-file> <pattern-file>\n] unless #ARGV == 2;
## Open files.
open my $pattern_fh, qq[<], pop #ARGV or die qq[ERROR: Cannot open pattern file: $!\n];
open my $input_fh, qq[<], pop #ARGV or die qq[ERROR: Cannot open input file: $!\n];
## Variable to save the regular expression.
my $str;
## Read patterns to match, and create a regex, with each string in a positive
## look-ahead.
while ( <$pattern_fh> ) {
chomp;
$str .= qq[(?=.*$_)];
}
my $regex = qr/$str/;
## Read each line of data and test if the regex matches.
while ( <$input_fh> ) {
chomp;
printf qq[%s\n], $_ if m/$regex/o;
}
Run it like:
perl script.pl input pattern
With following output:
acb
bac
Using Perl, I suggest you read all the patterns into an array and compile them. Then you can read through your input file using grep to make sure all of the regexes match.
The code looks like this
use strict;
use warnings;
open my $ptn, '<', 'pattern.txt' or die $!;
my #patterns = map { chomp(my $re = $_); qr/$re/; } grep /\S/, <$ptn>;
open my $in, '<', 'input.txt' or die $!;
while (my $line = <$in>) {
print $line unless grep { $line !~ $_ } #patterns;
}
output
acb
bac
Another way is to read all the input lines and then start filtering by each pattern:
#!/usr/bin/perl
use strict;
use warnings;
open my $in, '<', 'input.txt' or die $!;
my #matches = <$in>;
close $in;
open my $ptn, '<', 'pattern.txt' or die $!;
for my $pattern (<$ptn>) {
chomp($pattern);
#matches = grep(/$pattern/, #matches);
}
close $ptn;
print #matches;
output
acb
bac
Not grep and not a one liner...
MFILE=file.txt
PFILE=patterns
i=0
while read line; do
let i++
pattern=$(head -$i $PFILE | tail -1)
if [[ $line =~ $pattern ]]; then
echo $line
fi
# (or use sed instead of bash regex:
# echo $line | sed -n "/$pattern/p"
done < $MFILE
A bash(Linux) based solution
#!/bin/sh
INPUTFILE=input.txt #Your input file
PATTERNFILE=patterns.txt # file with patterns
# replace new line with '|' using awk
PATTERN=`awk 'NR==1{x=$0;next}NF{x=x"|"$0}END{print x}' "$PATTERNFILE"`
PATTERNCOUNT=`wc -l <"$PATTERNFILE"`
# build regex of style :(a|b|c){3,}
PATTERN="($PATTERN){$PATTERNCOUNT,}"
egrep "${PATTERN}" "${INPUTFILE}"
Here's a grep-only solution:
#!/bin/sh
foo ()
{
FIRST=1
cat pattern.txt | while read line; do
if [ $FIRST -eq 1 ]; then
FIRST=0
echo -n "grep \"$line\""
else
echo -n "$STRING | grep \"$line\""
fi
done
}
STRING=`foo`
eval "cat input.txt | $STRING"
How could I convert:
awk '{print $2 >> $1}' file
in a short Perl one-liner?
"file" could look like this:
fruit banana
vegetable beetroot
vegetable carrot
mushroom chanterelle
fruit apple
there may some other ways, but here's what i can think of
perl -ane 'open(FILE,">>",$F[0]); print FILE $F[1];close(FILE);' file
I guess awk has to be better at some things :-)
This is right at the limit of what I'd do on the command line, but it avoids reopening filehandles.
$ perl -lane '$fh{$F[0]} || open $fh{$F[0]}, ">>", $F[0]; print {$fh{$F[0]}} $F[1]' file
Not pure Perl, but you can do:
perl -nae '`echo $F[1] >> $F[0]`' input_file
This is what a2p <<< '{print $2 >> $1}' produces
#!/usr/bin/perl
eval 'exec /usr/bin/perl -S $0 ${1+"$#"}'
if $running_under_some_shell;
# this emulates #! processing on NIH machines.
# (remove #! line above if indigestible)
eval '$'.$1.'$2;' while $ARGV[0] =~ /^([A-Za-z_0-9]+=)(.*)/ && shift;
# process any FOO=bar switches
$, = ' '; # set output field separator
$\ = "\n"; # set output record separator
while (<>) {
($Fld1,$Fld2) = split(' ', $_, -1);
&Pick('>>', $Fld1) &&
(print $fh $Fld2);
}
sub Pick {
local($mode,$name,$pipe) = #_;
$fh = $name;
open($name,$mode.$name.$pipe) unless $opened{$name}++;
}