What I love about awk is you can fetch all the lines from a file that satisfies the condition on some aritrary field you specify.
For example,
awk '$3~/hi/' < test.txt # print all lines where the third field matches the pattern "hi"
or
awk '$2>=2' < test.txt # print all lines where the second field is greater or equal to 2
As a beginner who's learning about the power of unix, I am absolutely fascinated about this.
Now I am wondering if there is an easy way to perform regex substitutions only on some arbitrary fields you specify? For example, I want to do regex substitution on the third field only.
my current method is to "cut" the field I want and perform substitution on that using perl or sed, which then I "paste" to the original file. But I am wondering if there is more efficient way to achieve this.
Thanks
Since you tagged this question with 'perl' (in addition to 'sed', 'awk', 'unix', and 'command-line'), I'll assume you're interested in answers that incorporate any of the above tools.
Perl has an auto-split command-line switch (-a):
perl -lane 'print if $F[2] =~ /some pattern/' filename
...or...
perl -lane 'print if $F[1] >= 42' filename
-a causes an auto-split into the #F array. -n causes Perl to iterate over the lines of the file you feed it. The rest is programming. ;)
Now for substitution:
perl -i.bak -lane '$F[2] =~ s/match/subst/; print join q/ /, #F' filename
Or, a little shorter using the -p switch, which tells Perl to print each line as it appears in $_. That means if you alter #F, you'll have to copy it back into $_:
perl -i.bak -pale '$F[2] =~ s/match/subst/ && $_="#F"' filename
This might work for you:
echo -e 'Fred barney Wilma\nfoo bar baz' |
awk '$2 == "barney"{sub(/b/,"B",$2)};1'
Fred Barney Wilma
foo bar baz
You can use the sub, gsub commands or this this case:
echo -e 'Fred barney Wilma\nfoo bar baz'|
awk '$2 == "barney"{$2="Barney"};1'
Fred Barney Wilma
foo bar baz
Just substitute the second field completely.
N.B. The 1 at the end of the line is shorthand for {print}.
Consider a simple example:
awk -F "," '{ OFS=","; sub ("1", "x", $3); print $0 }' file.txt > newfile.txt
newfile.txt will now contain:
1,2,3,4,5,6,7
8,9,x0,11,12,13,14
15,16,x7,18,19,20,21
Here, 1 was replaced with an x in the third column $3.
-F "," sets the delimiter of the input file.
OFS="," adds a comma to the output.
If you would like to make the substitution globally, consider using gsub instead of sub.
HTH
Related
I have mulitple lines
QQQQl123
hsdhjhksd
QQQQl234
ajkdkjsdh
QQQQl564
i want to print all matching QQQQl[0-9]+
like
QQQQl123
QQQQl234
QQQQl564
how to do this using perl
I tried:
$ perl -0777pe '/QQQQl[0-9]+/' filename
it shows nothing
perl -we 'while(<>){ next unless $_=~/QQQQl[0-9]+/; print $_; }' < filename
perl -ne 'print if /QQQQl[0-9]+/' filename
Or, if, for some reason, you insist on using -0777, you could do
perl -0777nE 'say for /QQQQl[0-9]+/g' filename
(or print "$_\n" instead of say)
Your code doesn't work because /QQQQl[0-9]+/ returns true because $_ indeed contains that pattern, but you never asked Perl to do anything based on that return value.
-n is preferable to -p in that case, since you don't want to print every line but only some (-p automatically prints every line, and there is very little you can do about it).
I'm writing a simple Perl script which is meant to output the second column of an external text file (columns one and two are separated by a comma).
I'm using AWK because I'm familiar with it.
This is my script:
use v5.10;
use File::Copy;
use POSIX;
$s = `awk -F ',' '\$1==500 {print \$2}' STD`;
say $s;
The contents of the local file "STD" is:
CIR,BS
60,90
70,100
80,120
90,130
100,175
150,120
200,260
300,500
400,600
500,850
600,900
My output is very strange and it prints out the desired "850" but it also prints a trailer of the line and a new line too!
ka#man01:$ ./test.pl
850
ka#man01:$
The problem isn't just printing. I need to use the variable generated by awk "i.e. the $s variable) but the variable is also being reserved with a long string and a new line!
Could you guys help?
Thank you.
I'd suggest that you're going down a dirty road by trying to inline awk into perl in the first place. Why not instead:
open ( my $input, '<', 'STD' ) or die $!;
while ( <$input> ) {
s/\s+\z//;
my #fields = split /,/;
print $fields[1], "\n" if $fields[0] == 500;
}
But the likely problem is that you're not handling linefeeds, and say is adding an extra one. Try using print instead, or chomp on the resultant string.
perl can do many of the things that awk can do. Here's something similar that replaces your entire Perl program:
$ perl -naF, -le 'chomp; print $F[1] if $F[0]==500' STD
850
The -n creates a while loop around your argument to -e.
The -a splits up each line into #F and -F lets you specify the separator. Since you want to separate the fields on a comma you use -F,.
The -l adds a newline each time you call print.
The -e argument is the program to run (with the added while from -n). The chomp removes the newline from the output. You get a newline in your output because you happen to use the last field in the line. The -l adds a newline when you print; that's important when you want to extract a field in the middle of the line.
The reason you get 2 newlines:
the backtick operator does not remove the trailing newline from the awk output. $s contains "850\n"
the say function appends a newline to the string. You have say "850\n" which is the same as print "850\n\n"
I've got a file with various wildcards in it that I want to be able to substitute from a (Bash) shell script. I've got the following which works great until one of the variables contains characters that are special to regexes:
VERSION="1.0"
perl -i -pe "s/VERSION/${VERSION}/g" txtfile.txt # No problems here
APP_NAME="../../path/to/myapp"
perl -i -pe "s/APP_NAME/${APP_NAME}/g" txtfile.txt # Error!
So instead I want something that just performs a literal text replacement rather than a regex. Are there any simple one-line invocations with Perl or another tool that will do this?
The 'proper' way to do this is to escape the contents of the shell variables so that they aren't seen as special regex characters. You can do this in Perl with \Q, as in
s/APP_NAME/\Q${APP_NAME}/g
but when called from a shell script the backslash must be doubled to avoid it being lost, like so
perl -i -pe "s/APP_NAME/\\Q${APP_NAME}/g" txtfile.txt
But I suggest that it would be far easier to write the entire script in Perl
Use the following:
perl -i -pe "s|APP_NAME|\\Q${APP_NAME}|g" txtfile.txt
Since a vertical bar is not a legal character as part of a path, you are good to go.
I don't particularly like this answer because there should be a better way to do a literal replace in Perl. \Q is cryptic. Using quotemeta adds extra lines of code.
But... You can use substr to replace a portion of a string.
#!/usr/bin/perl
my $name = "Jess.*";
my $sentence = "Hi, my name is Jess.*, dude.\n";
my $new_name = "Prince//";
my $name_idx = index $sentence, $name;
if ($name_idx >= 0) {
substr($sentence, $name_idx, length($name), $new_name);
}
print $sentence;
Output:
Hi, my name is Prince//, dude.
You don't have to use a regular expression for this (using substr(), index(), and length()):
perl -pe '
foreach $var ("VERSION", "APP_NAME") {
while (($i = index($_, $var)) != -1) {
substr($_, $i, length($var)) = $ENV{$var};
}
}
'
Make sure you export your variables.
You can use a regex but escape any special characters.
Something like this may work.
APP_NAME="../../path/to/myapp"
APP_NAME=`echo "$APP_NAME" | sed -e '{s:/:\/:}'`
perl -i -pe "s/APP_NAME/${APP_NAME}/g" txtfile.txt
Use:
perl -i -pe "\$r = qq/\Q${APP_NAME}\E/; s/APP_NAME/\$r/go"
Rationale: Escape sequences
I managed to get a working solution, partly based on bits and pieces from other peoples' answers:
app_name='../../path/to/myapp'
perl -pe "\$r = q/${app_name//\//\\/}/; s/APP_NAME/\$r/g" <<<'APP_NAME'
This creates a Perl variable, $r, from the result of the shell parameter expansion:
${app_name//\//\\/}
${ # Open parameter expansion
app_name # Variable name
// # Start global substitution
\/ # Match / (backslash-escaped to avoid being interpreted as delimiter)
/ # Delimiter
\\/ # Replace with \/ (literal backslash needs to be escaped)
} # Close parameter expansion
All that work is needed to prevent forward slashes inside the variable from being treated as Perl syntax, which would otherwise close the q// quotes around the string.
In the replacement part, use the variable $r (the $ is escaped, to prevent it from being treated as a shell variable within double quotes).
Testing it out:
$ app_name='../../path/to/myapp'
$ perl -pe "\$r = q/${app_name//\//\\/}/; s/APP_NAME/\$r/g" <<<'APP_NAME'
../../path/to/myapp
I have a file, someFile, like this:
$cat someFile
hdisk1 active
hdisk2 active
I use this shell script to check:
$cat a.sh
#!/usr/bin/ksh
for d in 1 2
do
grep -q "hdisk$d" someFile && echo "$d : ok"
done
I am trying to convert it to Perl:
$cat b.sh
#!/usr/bin/ksh
export d
for d in 1 2
do
cat someFile | perl -lane 'BEGIN{$d=$ENV{'d'};} print "$d: OK" if /hdisk$d\s+/'
done
I export the variable d in the shell script and get the value using %ENV in Perl. Is there a better way of passing this value to the Perl one-liner?
You can enable rudimentary command line argument with the "s" switch. A variable gets defined for each argument starting with a dash. The -- tells where your command line arguments start.
for d in 1 2 ; do
cat someFile | perl -slane ' print "$someParameter: OK" if /hdisk$someParameter\s+/' -- -someParameter=$d;
done
See: perlrun
Sometimes breaking the Perl enclosure is a good trick for these one-liners:
for d in 1 2 ; do cat kk2 | perl -lne ' print "'"${d}"': OK" if /hdisk'"${d}"'\s+/';done
Pass it on the command line, and it will be available in #ARGV:
for d in 1 2
do
perl -lne 'BEGIN {$d=shift} print "$d: OK" if /hdisk$d\s+/' $d someFile
done
Note that the shift operator in this context removes the first element of #ARGV, which is $d in this case.
Combining some of the earlier suggestions and adding my own sugar to it, I'd do it this way:
perl -se '/hdisk([$d])/ && print "$1: ok\n" for <>' -- -d='[value]' [file]
[value] can be a number (i.e. 2), a range (i.e. 2-4), a list of different numbers (i.e. 2|3|4) (or almost anything else, that's a valid pattern) or even a bash variable containing one of those, example:
d='2-3'
perl -se '/hdisk([$d])/ && print "$1: ok\n" for <>' -- -d=$d someFile
and [file] is your filename (that is, someFile).
If you are having trouble writing a one-liner, maybe it is a bit hard for one line (just my opinion). I would agree with #FM's suggestion and do the whole thing in Perl. Read the whole file in and then test it:
use strict;
local $/ = '' ; # Read in the whole file
my $file = <> ;
for my $d ( 1 .. 2 )
{
print "$d: OK\n" if $file =~ /hdisk$d\s+/
}
You could do it looping, but that would be longer. Of course it somewhat depends on the size of the file.
Note that all the Perl examples so far will print a message for each match - can you be sure there are no duplicates?
My solution is a little different. I came to your question with a Google search the title of your question, but I'm trying to execute something different. Here it is in case it helps someone:
FYI, I was using tcsh on Solaris.
I had the following one-liner:
perl -e 'use POSIX qw(strftime); print strftime("%Y-%m-%d", localtime(time()-3600*24*2));'
which outputs the value:
2013-05-06
I was trying to place this into a shell script so I could create a file with a date in the filename, of X numbers of days in the past. I tried:
set dateVariable=`perl -e 'use POSIX qw(strftime); print strftime("%Y-%m-%d", localtime(time()-3600*24*$numberOfDaysPrior));'`
But this didn't work due to variable substitution. I had to mess around with the quoting, to get it to interpret it properly. I tried enclosing the whole lot in double quotes, but this made the Perl command not syntactically correct, as it messed with the double quotes around date format. I finished up with:
set dateVariable=`perl -e "use POSIX qw(strftime); print strftime('%Y-%m-%d', localtime(time()-3600*24*$numberOfDaysPrior));"`
Which worked great for me, without having to resort to any fancy variable exporting.
I realise this doesn't exactly answer your specific question, but it answered the title and might help someone else!
That looks good, but I'd use:
for d in $(seq 1 2); do perl -nle 'print "hdisk$ENV{d} OK" if $_ =~ /hdisk$ENV{d}/' someFile; done
It's already written on the top in one long paragraph but I am also writing for lazy developers who don't read those lines.
Double quotes and single quote has big different meaning for the bash.
So please take care
Doesn't WORK perl '$VAR' $FILEPATH
WORKS perl "$VAR" $FILEPATH
Looking for an awk (or sed) one-liner to remove lines from the output if the first field is a duplicate.
An example for removing duplicate lines I've seen is:
awk 'a !~ $0; {a=$0}'
Tried using it for a basis with no luck (I thought changing the $0's to $1's would do the trick, but didn't seem to work).
awk '{ if (a[$1]++ == 0) print $0; }' "$#"
This is a standard (very simple) use for associative arrays.
this is how to remove duplicates
awk '!_[$1]++' file
If you're open to using Perl:
perl -ane 'print if ! $a{$F[0]}++' file
-a autosplits the line into the #F array, which is indexed starting at 0
The %a hash remembers if the first field has already been seen
This related solution assumes your field separator is a comma, rather than whitespace
perl -F, -ane 'print if ! $a{$F[0]}++' file
it print the unique as well as single value of the duplicates
awk '!a[$1]++' file_name