I'm trying to use a perl one-liner to turn print0 output into quoted shell parameters, kind of like the trick that's something like .. | xargs -0 printf "%q" {} but I didn't want to require bash (whose printf implements %p). I was kind of amazed to, well, not find an easy way to do this in perl. For all of perl's quoting mechanisms, there's no way I saw for producing quoted strings. Surely I just haven't looked hard enough.
Hopefully the answer isn't a regular expression. Quoting an elaborate regular expression to put into a shell command-line is not my idea of fun (if only a simple perl program could quote it for me, oh back to the same problem).
You can roll your own quoting for POSIX-like shells fairly simply - no complicated regexes needed (just straightforward string substitution using literals):
$ echo "I'm \$HOME. 3\" of rain." | perl -lne "s/'/'\\\''/g; print q{'} . \$_ . q{'}"
'I'\''m $HOME. 3" of rain.'
The approach is modeled after AppleScript's quoted form of command:
The input string is broken into substrings by ', each substring is itself '-enclosed, with the original ' chars. spliced between the substrings as \' (an individually quoted ').
When passed to the shell, the shell rebuilds these parts into a single, literal string.
This multi-part string-concatenation approach is necessary, because POSIX-like shells categorically do not allow embedding ' itself inside single-quoted strings (there's not even an escape sequence).
Alternatively, you can install a CPAN module such as ShellQuote.
Optional background information
While it would be handy for Perl itself to support such a quoting mechanism for piecing together shell commands stored in a single string to pass to qx// (`...`), such a mechanism would have to operate platform-specifically.
Notably, quoting rules for Windows are very different from rules for Unix platforms, and except for simple cases shell commands as a whole will be incompatible too.
From inside Perl, you may be able to bypass the need for quoting altogether, by using the list forms of system() and open(), which allow you to pass the command arguments individually, as-is, but note that this is only an option if your command doesn't use any shell features; for a "shell-less" qx// (`...`) alternative, see this answer of mine, which also covers shell-quoting on Windows.
Related
I am trying to run a bsub command from a perl script in the following way:
system ("bsub -select "testid::1" -q normal");
but I think perl is getting confused because of the double quotes in "testid::1". What is the proper way to implement this?
You can escape the inner quotes:
system ("bsub -select \"testid::1\" -q normal");
or replace the outer quotes with single quotes, or in fact any character at all, thanks to the qq generalized quote operator in Perl which exists precisely for this sort of scenario;
system (qq{bsub -select "testid::1" -q normal});
There is a companion generalized single quote operator q.
Rather than fitting the entire command in a single quoted string (although using the generalized quoting operators makes that fairly simply), you can use the multi-argument version of system to avoid needing to quote the entire command line.
system 'bsub', 'select[type==LINUX64&&clearcase]', '-select', 'testid::1', '-q' 'normal';
I am using following kind of script in my perl script and expecting entry at line 1. I am getting some error as below; any help?
plz ignore perl variable....
Error Messaage -
sed: -e expression #1, char 22: extra characters after command
# Empty file will not work for Sed line number 1
`"Security Concerns Report" > $outputFile`;
`sed '1i\
\
DATE :- $CDate \
Utility accounts with super access:- $LNumOfSupUserUtil \
Users not found in LDAP with super access: - $LNumOfSupUserNonLdap\
' $outputFile > $$`;
`mv $$ $outputFile`;
}
Your immediate problem is that the backslash character is interpreted by Perl inside the backtick operator, as is the dollar character. So your backslash-newline sequence turns into a newline in the shell command that is executed. If you replace these backslashes by \\, you'll go over this hurdle, but you'll still have a very brittle program.
Perl is calling a shell which calls sed. This requires an extra level of quoting for the shell which you are not performing. If your file names and data contain no special characters, you may be able to get away with this, until someone uses a date format containing a ' (among many things that would break your code).
Rather than fix this, it is a lot simpler to do everything in Perl. Everything sed and shells can do, Perl can do almost as easily or easier. It's not very clear from your question what you're trying to do. I'll focus on the sed call, but this may not be the best way to write your program.
If you really need to prepend some text to an existing file, there's a widely-used module on CPAN that already does this well. Use existing libraries in preference to reinventing the wheel. File::Slurp has a prepend_file method just for that. In the code below I use a here-document operator for the multiline string.
use File::Slurp; # at the top of the script with the other use directives
File::Slurp->prepend_file($outputFile, <<EOF);
DATE :- $CDate
Utility accounts with super access:- $LNumOfSupUserUtil
Users not found in LDAP with super access: - $LNumOfSupUserNonLdap
EOF
I want to run some commands using the system() command, I do this way:
execute_command_error("trash-put '/home/$filename'");
Where execute_command_error will report if there was an error with whatever system command it ran. I know I could just unlink the file using Perl commands, but I want to delete stuff using trash-put as it's a type of recycling program.
My problem is that $filename will sometimes have apostrophes, quotes, and other weird characters in it that mess up the system command or Perl itself.
Generate the command name and arguments as an array, and pass that to system:
my(#command) = ("trash-put", 'home/$filename');
system #command;
This means that Perl does not invoke the shell to do any metacharacter expansion (or I/O redirection, or command piping, or ...). It does mean it does exactly what you told it to do.
sub execute_command_error
{
system #_;
}
Borrowing information from the copious collection of comments:
Which is clearly documented in perldoc -f system or at perldoc.perl.org/functions/system.html (#Ether).
(See also the discussion of 'exec' below which is closely related.)
Did you mean to put $filename in single quotes? (#mobrule).
I did intend to use single quotes - I'm demonstrating that the $filename does not get expanded by Perl or Shell...In my test script, I used 'my.$file', and that gave me a file with a $ in the name - as I intended.
I think the desired quoting if you do want to invoke the shell (for example if you want some piping) is $command_line = "\"$command\" \"$arg1\" \"$arg2\"...". (#Jefromi).
Adding double quotes around the arguments won't help with embedded $, backtick1, '$(...)' and related notations. You more nearly need single quotes around things, but then you need to rewrite embedded single quotes as "'\''" which generates a single quote to terminate the current single-quoted argument, a backslash-quote combination to represent a single quote, and another single quote to resume the single-quoted argument.
This would be a good solution if I used system command directly; however I am using webmin's execute_command function, which is a bit over my head so I wouldn't know how to edit it to allow for arrays. Could you expand on the rewrite of embedded single quotes as "'\''"...This is what I will use, for now. (#Brian)
Roughly speaking, the way the (Unix) shells treat single quotes is "everything from the first single quote up to the next is literal text, no metacharacters". So, to get the shell to treat something as literal text, enclose it in single characters. That deals with everything except single quotes themselves. As my comment says, you have to use the 4-character replacement string to get a single quote embedded into the middle of a single quoted argument.
There is probably a neater way to do it than this (using one or two map operations, perhaps), but this should work:
for (my $i = 0; $i < scalar(#command); $i++)
{
$command[$i] =~ s/'/'\\''/g; # Replace single quotes by the magic sequence
$command[$i] = "'$command[$i]'"; # Wrap value in single quotes
}
You can then join the array to make a single string for transmission to execute_command.
It's better to write that as system { $command[0] } #command to handle the case where #command has one element. This is one of the things I talk about in the "Secure Programming Techniques" chapter of Mastering Perl. (#briandfoy).
As a general rule, I'll accept this correction. I'm not sure it is crucial in this instance, though, where the command name is provided by the program and it is only the arguments that are possibly provided the user. The command name 'trash-put' is safe from shell expansions (IFS is reset to default by the shell when it starts, so that avenue of attack is not available).
This issue is discussed in the 'perldoc -f exec' man page:
If you don't really want to execute the first argument, but want to lie to the program you are executing about its own name, you can specify the program you actually want to run as an "indirect object" (without a comma) in front of the LIST. (This always forces interpretation of the LIST as a multivalued list, even if there is only a single scalar in the list.)
Example:
$shell = '/bin/csh';
exec $shell '-sh'; # pretend it's a login shell
or, more directly,
exec {'/bin/csh'} '-sh'; # pretend it's a login shell
When the arguments get executed via the system shell, results are subject to its quirks and capabilities. See "STRING" in perlop for details.
Using an indirect object with exec or system is also more secure. This usage (which also works fine with system()) forces interpretation of the arguments as a multivalued list, even if the list had just one argument. That way you're safe from the shell expanding wildcards or splitting up words with whitespace in them.
#args = ( "echo surprise" );
exec #args; # subject to shell escapes
# if #args == 1
exec { $args[0] } #args; # safe even with one-arg list
The first version, the one without the indirect object, ran the echo program, passing it "surprise" an argument. The second version didn't; it tried to run a program named "echo surprise", didn't find it, and set $? to a non-zero value indicating failure.
1 How do you get a back-tick to display in Markdown?
As stated in perldoc -f system:
If there is more than one argument in LIST, or if LIST is an array
with more than one value, starts the program given by the first element of the list with arguments given by the rest of the list. If there is
only one scalar argument, the argument is checked for shell metacharacters, and if there are any, the entire argument is passed to the system's
command shell for parsing (this is "/bin/sh -c" on Unix platforms, but varies on other platforms). If there are no shell metacharacters in the
argument, it is split into words and passed directly to "execvp", which is more efficient.
I like to use IPC::System::Simple's version of system(), which gives more control, such as being able to capture various exceptions and handle certain error codes as "bad" and others as "good":
use IPC::System::Simple qw(system);
system("cat *.txt"); # will die on failure
I'm working on a Perl script that I was hoping to capture a string entered on the command line without having to enter the quotes (similiar to bash's "$#" ability). I'll be using this command quite a bit so I was hoping that this is possible. If I have:
if ($ARGV) {
I have to put the command line string in quotes. I'd rather do the command something like this:
htmlencode some & HTML <> entities
Without the quotes. Is there a way to do this in Perl?
The #ARGV array contains the arguments to the Perl script - no quotes needed.
That said, the question asks about:
I have to put the command line string in quotes. I'd rather do the command something like this:
htmlencode some & HTML <> entities
Without the quotes. Is there a way to do this in perl?
Well, if the command shown is written at the shell command line, you have to obey shell conventions - which means escaping the '&' and '<>' to prevent the shell from interpreting them. Likewise, within a Perl script, that sequence would need to be protected from Perl. Maybe you'd write:
system "htmlencode", "some", "&", "HTML", "<>", "entities";
That is, everything would have to be in quotes - but that notation would avoid executing the shell and having the shell interpret the commands.
Alternatively again, if you put the arguments into an array (with quotes at the time the array is loaded), then you could pass that array to system and not use any quotes:
my #args = ( "htmlencode", "some", "&", "HTML", "<>", "entities" );
system #args;
But I think the question is confused.
You put quotes around $# in bash so that it expands to have each element in the array quoted. The reason to do that is so that each element of the array continues to be treated as a single argument when you pass them all to the next command.
The analogue to that in Perl is when you want to pass those parameters to another external command. If you're running the external program with the backtick operators, then you'd need to quote each parameter, but if you use system, then Perl will take care of keeping all the parameters separate for you.
In fact, separate parameters are the way programs are executed on Unix anyway. The single-string command-line format is there because we need to be able to type things at the command prompt. Like all shells, bash has special rules about how to split that single string into multiple arguments. But if you already have them separated in a Perl array, don't put them back into a single string. Keep them separate.
I'm sure this question may seem foolish to some of you, but I'm here to learn.
Are these assumptions true for most of the languages ?
EDIT : OK, let's assume I'm talking about Perl/Bash scripting.
'Single quotes'
=> No interpretation at all (e.g. '$' or any metacharacter will be considered as a character and will be printed on screen)
"Double quotes"
=> Variable interpretation
To be more precise about my concerns, I'm writing some shell scripts (in which quotes can sometimes be a big hassle), and wrote this line :
CODIR=`pwd | sed -e "s/$MODNAME//"`
If I had used single quotes in my sed, my pattern would have been '$MODNAME', right ? (and not the actual value of $MODNAME, which is `alpha' in this particular case)
Another problem I had, with an awk inside an echo :
USAGE=`echo -ne "\
Usage : ./\`basename $0\` [-hnvV]\n\
\`ls -l ${MODPATH}/reference/ | awk -F " " '$8 ~ /\w+/{print "> ",$8}'\`"`
I spent some time debugging that one. I came to the conclusion that backticks were escaped so that the interpreter doesn't "split" the command (and stop right before «basename»). In the awk commmand, '$8' is successfully interpreted by awk, thus not by shell. What if I wanted to use a shell variable ? Would I write awk -F "\"$MY_SHELL_VAR\"" ? Because $MY_SHELL_VAR as is, will be interpreted by awk, won't it ?
Don't hesitate to add any information about quoting or backticks !
Thank you ! :)
It varies massively by language. For example, in the C/Java/C++/C# etc family, you can't use single quotes for a string at all - they're only for single characters.
I think it's far better to learn the rules properly for the languages you're actually interested in than to try to generalise.
Are these assumptions true for most of the languages ?
Answer: No
In bash scripting, backticks are deprecated in favor of $() in part because it is non-obvious how nested quotes and escaping are supposed to work. You may also want to take a look at Bash Pitfalls.
It's definitely not the same for all languages. In Python, for example, single and double quotes are interchangeable. The only difference is that you can include single quotes within a double-quoted string without escaping them and vice versa ("How's it going?").
Also, there are triple-quoted strings that can span multiple lines.
In Perl, you also have q() and qq() to help you in nested quoting situations:
my $x = q(a string with 'single quotes');
my $y = qq(an $interpreted string with "double quotes");
These certainly will help you avoid "\"needlessly\"" '\'escaping\'' internal quotes.
Yes, something like awk -F "\"$MY_SHELL_VAR\"" will work, however in this case you wouldn't be able to use variables in awk, since they will be interpreted by shell, so the way to go is something like this (I will use command simpler than yours, if you don't mind :) ):
awk -F " " '$8 ~ /\w+/{print "> ",$8, '$SOME_SHELL_VAR'}'
Note the single quotes terminating and restarting.
The trickiest part, usually, is to pass a quote in the argument to the command. In this case you need to terminate single quote, add escaped quote character, start quote again, like this:
awk '$1 ~ '\''{print}'
Note, that single quote can't be escaped inside single quotes, since the "\" won't be treated as an escape character.
This is probably not related directly to your quiestion, but still useful.
I don't know about perl, but for bash you don't need to backslash the newline.
As for quotes, I have a (very personal) pattern that I call the "five quotes" pattern. It helps to put one quote in a string enclosed by the same kind of quotes
For instance:
doublequoted="some things "'"'"quoted"'"'" and some not"
simplequoted='again '"'"'quote this'"'"' but not that'
Note that you can freely append strings with different kinds of quotes, which is useful when you want the shell to interprete some vars but not some others:
awk -F " " '$8 ~ /\w+/{print "> ",$8, '"$SOME_SHELL_VAR"'}'
Also, I don't use the backtick anymore but the $(...)pattern which is more legible and can be nested.
USAGE=$(echo -ne "
Usage : ./$(basename $0) [-hnvV]\n
$(ls -l ${MODPATH}/reference/ | awk -F " " '$8 ~ /\w+/{print "> ",$8}')")
In perl, double quoted strings will have their variables expanded.
If you write that for instance:
my $email = "foo#bar.com" ;
perl will try to expand #bar. If you use strict, you'll see an complain about the array bar not existing. If you don't, you'll just see a weird behavior.
So it's better to write:
my $email = 'foo#bar.com' ;
For these types of reason, my advice is to always use single quote for strings, unless you know that you need variable expansion.