I have a file with thousands of lines in it. I would like to search each line for a non-ascii character, and if found, delete that entire line.
I found this bit of code in perl:
perl -i.bak -ne 'print unless(/[^[:ascii:]]/)' file
But I get this error when I run it with my file:
Can't find string terminator "'" anywhere before EOF at -e line 1.
Does anyone have any code for an actual perl script instead of a one liner like the above?
That's a shell error, most likely because you're on a windows machine.
Use double quotes instead of single quotes:
perl -i.bak -ne "print unless(/[^[:ascii:]]/)" file
Related
Problem Background
We have several thousand large (10M<lines) text files of tabular data produced by a windows machine which we need to prepare for upload to a database.
We need to change the file encoding of these files from cp1252 to utf-8, replace any bare Unix LF sequences (i.e. \n) with spaces, then replace the DOS line end sequences ("CR-LF", i.e \r\n) with Unix line end sequences (i.e. \n).
The dos2unix utility is not available for this task.
We initially had a bash function that packaged these operations together using iconv and sed, with iconv doing the encoding and sed dealing with the LF/CRLF sequences. I'm trying to replace part of this bash function with a perl command.
Example Code
Based on some helpful code review, I want to change this function to a perl script.
The author of the code review suggested the following perl to replace CRLF (i.e. "\r\n") with LF ("\n").
perl -g -pe 's/(?<!\r)\n/ /g; s/\r\n/\n/g;'
The explanation for why this is better than what we had previously makes perfect sense, but this line fails for me with:
Unrecognized switch: -g (-h will show valid options).
More interestingly, the author of the code review also suggests it is possible to perform the decode/recode in a perl script, too, but I am completely unsure where to start.
Questions
Please can someone explain why the suggested answer fails with Unrecognized switch: -g (-h will show valid options).?
If it helps, the line is supposed to receive piped input from incov as follows (though I am interested in learning how to use perl to do the redcoding/recoding step, too):
iconv --from-code=CP1252 --to-code=UTF-8 $1$ | \
perl -g -pe 's/(?<!\r)\n/ /g; s/\r\n/\n/g;'
> "$2"
(Highly simplified) example input for testing:
apple|orange|\n|lemon\r\nrasperry|strawberry|mango|\n\r\n
Desired output:
apple|orange| |lemon\nrasperry|strawberry|mango| \n
Perl recently added the command line switch -g as an alias for 'gulp mode' in Perl v5.36.0.
This works in Perl version v5.36.0:
s=$(printf "Line 1\nStill Line 1\r\nLine 2\r\nLine 3\r\n")
perl -g -pe 's/(?<!\r)\n/ /g; s/\r\n/\n/g;' <<<"$s"
Prints:
Line 1 Still Line 1
Line 2
Line 3
But any version of perl earlier than v5.36.0, you would do:
perl -0777 -pe 's/(?<!\r)\n/ /g; s/\r\n/\n/g;' <<<"$s"
# same
BTW, the conversion you are looking for a way easier in this case with awk since it is close to the defaults.
Just do this:
awk -v RS="\r\n" '{gsub(/\n/," ")} 1' <<<"$s"
Line 1 Still Line 1
Line 2
Line 3
Or, if you have a file:
awk -v RS="\r\n" '{gsub(/\n/," ")} 1' file
This is superior to the posted perl solution since the file is processed record be record (each block of text separated by \r\n) versus having the read the entire file into memory.
(On Windows you may need to do awk -v RS="\r\n" -v ORS="\n" '...')
Another note:
You can get similar behavior from Perl by:
Setting the input record separator to the fixed string $/="\r\n" in a BEGIN block;
Use the -l switch so every line has the input record separator removed;
Use tr for speedy replacement of \n with ' ';
Possible set the output record separator, $/="\n", on Windows.
Full command:
perl -lpE 'BEGIN{$/="\r\n"} tr/\n/ /' file
The error message is about the command line switch -g you use in perl -g -pe .... This is not about the switch at the regex - which is valid (but useless since there is only a single \n in a line anyway, and -p reads line by line).
This switch simply does not exist with the perl version you are using. It was only added with perl 5.36, so you are likely using an older version. Try -0777 instead.
Could command lines parameters been saved to a file and then pass the file to perl to parse out the options? Like response file (prefix the name with #) for some Microsoft tools.
I am trying to pass expression to perl via command line, like perl -e 'print "\n"', and Windows command prompt makes using double quotes a little hard.
There are several solutions, from most to least preferable.
Write your program to a file
If your one liner is too big or complicated, write it to a file and run it. This avoids messing with shell escapes. You can reuse it and debug it and work in a real editor.
perl path\to\some_program
Command line options to perl can be put on the otherwise useless on Windows #! line. Here's an example.
#!/usr/bin/perl -i.bak -p
# -i.bak Backs up the file.
# -p Puts each line into $_ and writes out the new value of $_.
# So this changes all instances in a file of " with '.
s{"}{'}g;
Use alternative quote delimiters
Perl has a slew of alternative ways to write quotes. Use them instead. This is good for both one liners as well as things like q[<tag key='value'>].
perl -e "print qq[\n]"
Escape the quote
^ is the cmd.exe escape character. So ^" is treated as a literal quote.
perl -e "print ^"\n^""
Pretty yucky. I'd prefer using qq[] and reserve ^" for when you need to print a literal quote.
perl -e "print qq[^"\n]"
Use the ASCII code
The ASCII and UTF-8 hex code for " is 22. You can supply this to Perl with qq[\x22].
perl -e "print qq[\x22\n]"
You can read the file into a string and then use
use Getopt::Long qw(GetOptionsFromString);
$ret = GetOptionsFromString($string, ...);
to parse the options from that.
I am calling a perl one-liner inside a Perl script.
The intention of the one-liner is to remove the trailing space from a file.
Inside the main perl script:
`perl -pi -e 's/\s+$//' tape.txt`;
Now it is throwing me an error Substitution replacement not terminated at -e line 2.
Where is it going wrong?
It's because of the $/ (special variable) inside your main perl script. Note that variables are interpolated inside `` strings just like inside "" strings, and the fact that there are some single quotes in there doesn't change that. You need to escape that $:
`perl -pi -e 's/\s+\$//' tape.txt;`
The backtick syntax invokes a shell and when invoked, the shell assumes it should interpolate the string passed.
A cleaner syntax might be:
system('perl -pli -e "s/\s*$//" tape.txt');
Since you aren't capturing the output of the command, using backticks or qx in lieu of system isn't an issue.
Too, adding the -l switch autochomps each line read and then adds a newline back --- probably what you want.
\s matches [ \t\n\r\f] and do not want to match \n.
Notice use of {} for subst delimiters:
$ echo -e 'hi \nbye'| perl -pe 's{[\t\040]+$}{};' | cat -A
hi$
bye$
I got a LDAP schema but ldifde wraps long lines so after googled I found this command to fix the file, but I'm getting the following error:
c:\Perl64\bin>perl -p -e 'BEGIN {$/ = undef} s/\n(?=[a-z])/ /g' test.ldf
Can't find string terminator "'" anywhere before EOF at -e line 1.
c:\Perl64\bin>
Even replacing the content with a single line I get the same error so I assume the sentence is wrong.
Could you give some clue about that, Im newbie on perl.
Thanks,
m0dest0.
You seem to be on windows. Windows does not recognize single quote ', you need to use double quote ":
c:\Perl64\bin>perl -pe "BEGIN {$/ = undef} s/\n(?=[a-z])/ /g" test.ldf
You should be aware that this does not change the input file, it just prints to standard output. If you want to alter the file, you can either add the in-place edit switch, e.g. -i.bak (saves backup in test.ldf.bak) or use redirection:
c:\Perl64\bin>perl -pe "BEGIN {$/ = undef} s/\n(?=[a-z])/ /g" test.ldf > out.ldf
Wild guess, but could it be that quoting is messing you up, in the command prompt? Try putting that one line in a file, and see if perl -p file.pl test.ldf would run.
I've got windows bat file, (using activeperl)
#echo off
perl -p -e 's/DIV\[/div\[/g' orginal.txt > output.txt
perl -p -e 'rename("output.txt", "orginal.txt")';
...
Running a .bat file, and I just cant get it to run properly..
Can't open ;: No such file or directory, <> line 12248.
Can't find string terminator "'" anywhere before EOF at -e line 1.
Not sure what I'm doing wrong..
You can't use single-quotes to enclose the Perl code in Windows. As a result, you need to escape the double-quotes or find other alternatives, such as qq(...).
perl -pe "s/DIV\[/div\[/g" original.txt > output.txt
perl -pe "rename(qq(output.txt), qq/original.txt/)"
Note that in this case, the arguments to rename can simply be rename('a.txt', 'b.txt') since they are literals and no variable interpolation is required.
You ought to use double quotes to quote the program text under Windows cmd. In your example, you can just swiztch double and single quotes. In cases where you really need double quotes in the perl text, use qq{ .... } instead.
The other posters are correct: windows requires double quotes for -e scripts to perl, which often screws things up. There is one more thing you can do, though: Use the -i switch, like this:
#echo off
perl -pi.bak -we "s/DIV\[/div\[/g" original.txt
The -i.bak switch will edit the file in place - no rename required - AND it will store a backup of the file in "original.txt.bak". If you do not want a backup, remove the ".bak" part and just use -pi.