I want to call from perl the command "touch" on a large list of files.
The command I am using fails:
$cmd = "touch $list_of_files";
$ret=system("sudo -u user bash -c \"$cmd\"");
print $ret;
Is there a limitation of command length in bash or is it because I am using 'system'?
Beside splitting the list of files in smaller lists is there a solution to my problem?
Thanks
Is there a limitation of command length in bash
I believe so. But why are you suing bash at all?
system("sudo", "-u", "user", "touch", #list_of_files);
Related
I have a script where I need to start a command, then pass some additional commands as commands to that command. I tried
su
echo I should be root now:
who am I
exit
echo done.
... but it doesn't work: The su succeeds, but then the command prompt is just staring at me. If I type exit at the prompt, the echo and who am i etc start executing! And the echo done. doesn't get executed at all.
Similarly, I need for this to work over ssh:
ssh remotehost
# this should run under my account on remotehost
su
## this should run as root on remotehost
whoami
exit
## back
exit
# back
How do I solve this?
I am looking for answers which solve this in a general fashion, and which are not specific to su or ssh in particular. The intent is for this question to become a canonical for this particular pattern.
Adding to tripleee's answer:
It is important to remember that the section of the script formatted as a here-document for another shell is executed in a different shell with its own environment (and maybe even on a different machine).
If that block of your script contains parameter expansion, command substitution, and/or arithmetic expansion, then you must use the here-document facility of the shell slightly differently, depending on where you want those expansions to be performed.
1. All expansions must be performed within the scope of the parent shell.
Then the delimiter of the here document must be unquoted.
command <<DELIMITER
...
DELIMITER
Example:
#!/bin/bash
a=0
mylogin=$(whoami)
sudo sh <<END
a=1
mylogin=$(whoami)
echo a=$a
echo mylogin=$mylogin
END
echo a=$a
echo mylogin=$mylogin
Output:
a=0
mylogin=leon
a=0
mylogin=leon
2. All expansions must be performed within the scope of the child shell.
Then the delimiter of the here document must be quoted.
command <<'DELIMITER'
...
DELIMITER
Example:
#!/bin/bash
a=0
mylogin=$(whoami)
sudo sh <<'END'
a=1
mylogin=$(whoami)
echo a=$a
echo mylogin=$mylogin
END
echo a=$a
echo mylogin=$mylogin
Output:
a=1
mylogin=root
a=0
mylogin=leon
3. Some expansions must be performed in the child shell, some - in the parent.
Then the delimiter of the here document must be unquoted and you must escape those expansion expressions that must be performed in the child shell.
Example:
#!/bin/bash
a=0
mylogin=$(whoami)
sudo sh <<END
a=1
mylogin=\$(whoami)
echo a=$a
echo mylogin=\$mylogin
END
echo a=$a
echo mylogin=$mylogin
Output:
a=0
mylogin=root
a=0
mylogin=leon
A shell script is a sequence of commands. The shell will read the script file, and execute those commands one after the other.
In the usual case, there are no surprises here; but a frequent beginner error is assuming that some commands will take over from the shell, and start executing the following commands in the script file instead of the shell which is currently running this script. But that's not how it works.
Basically, scripts work exactly like interactive commands, but how exactly they work needs to be properly understood. Interactively, the shell reads a command (from standard input), runs that command (with input from standard input), and when it's done, it reads another command (from standard input).
Now, when executing a script, standard input is still the terminal (unless you used a redirection) but the commands are read from the script file, not from standard input. (The opposite would be very cumbersome indeed - any read would consume the next line of the script, cat would slurp all the rest of the script, and there would be no way to interact with it!) The script file only contains commands for the shell instance which executes it (though you can of course still use a here document etc to embed inputs as command arguments).
In other words, these "misunderstood" commands (su, ssh, sh, sudo, bash etc) when run alone (without arguments) will start an interactive shell, and in an interactive session, that's obviously fine; but when run from a script, that's very often not what you want.
All of these commands have ways to accept commands by ways other than in an interactive terminal session. Typically, each command supports a way to pass it commands as options or arguments:
su root -c 'who am i'
ssh user#remote uname -a
sh -c 'who am i; echo success'
Many of these commands will also accept commands on standard input:
printf 'uname -a; who am i; uptime' | su
printf 'uname -a; who am i; uptime' | ssh user#remote
printf 'uname -a; who am i; uptime' | sh
which also conveniently allows you to use here documents:
ssh user#remote <<'____HERE'
uname -a
who am i
uptime
____HERE
sh <<'____HERE'
uname -a
who am i
uptime
____HERE
For commands which accept a single command argument, that command can be sh or bash with multiple commands:
sudo sh -c 'uname -a; who am i; uptime'
As an aside, you generally don't need an explicit exit because the command will terminate anyway when it has executed the script (sequence of commands) you passed in for execution.
If you want a generic solution which will work for any kind of program, you can use the expect command.
Extract from the manual page:
Expect is a program that "talks" to other interactive programs according to a script. Following the script, Expect knows what can be expected from a program and what the correct response should be. An interpreted language provides branching and high-level control structures to direct the dialogue. In addition, the user can take control and interact directly when desired, afterward returning control to the script.
Here is a working example using expect:
set timeout 60
spawn sudo su -
expect "*?assword" { send "*secretpassword*\r" }
send_user "I should be root now:"
expect "#" { send "whoami\r" }
expect "#" { send "exit\r" }
send_user "Done.\n"
exit
The script can then be launched with a simple command:
$ expect -f custom.script
You can view a full example in the following page: http://www.journaldev.com/1405/expect-script-example-for-ssh-and-su-login-and-running-commands
Note: The answer proposed by #tripleee would only work if standard input could be read once at the start of the command, or if a tty had been allocated, and won't work for any interactive program.
Example of errors if you use a pipe
echo "su whoami" |ssh remotehost
--> su: must be run from a terminal
echo "sudo whoami" |ssh remotehost
--> sudo: no tty present and no askpass program specified
In SSH, you might force a TTY allocation with multiple -t parameters, but when sudo will ask for the password, it will fail.
Without the use of a program like expect any call to a function/program which might get information from stdin will make the next command fail:
ssh use#host <<'____HERE'
echo "Enter your name:"
read name
echo "ok."
____HERE
--> The `echo "ok."` string will be passed to the "read" command
I am writing a perl script, and the objective is to kick start an xterm, follow by setenv, follow by invoke a tool that pre-installed in the system.
Here is my system call command in my perl script
system("xterm","-hold", "-e", "setenv ZI_LIBERTY_IGNORE_CONSTRUCT_FILES $RAW_RF_DIR/lib/lib2v/qcdc_ignore", "setenv HOME_0IN /p/hdk/rtl/cad/x86-64_linux26/mentor/questaCDC/V10.4g_5/linux_x86_64", "modpath -n 1 \$HOME_0IN/bin", "modpath -n 1 \$HOME_0IN/modeltech/bin", "/p/hdk/rtl/cad/x86-64_linux30/mentor/questaCDC/V10.4f_5/linux_x86_64/bin/qcdc -c -licq -do run.tcl");
xterm was able to start, however it stopped when executing the setenv, after that i tried with the new command by replacing the setenv with $ENV
system("xterm","-hold", "-e", "\$ENV{ZI_LIBERTY_IGNORE_CONSTRUCT_FILES} = \"$RAW_RF_DIR/lib/lib2v/qcdc_ignore\"", "setenv HOME_0IN /p/hdk/rtl/cad/x86-64_linux26/mentor/questaCDC/V10.4g_5/linux_x86_64", "modpath -n 1 \$HOME_0IN/bin", "modpath -n 1 \$HOME_0IN/modeltech/bin", "/p/hdk/rtl/cad/x86-64_linux30/mentor/questaCDC/V10.4f_5/linux_x86_64/bin/qcdc -c -licq -do run.tcl");
Here is the error message showing up in the new xterm (same for both approach)
Can't execvp $ENV{ZI_LIBERTY_IGNORE_CONSTRUCT_FILES} = "/nfs/fm/stod/stod4003/w.eew.100/rf_uprev_model_2020ww14p1//subIP/hip/MTLM_SA/RF.1//lib/lib2v/qcdc_ignore": No such file or directory
Please advise that how to make the series of operation works in the new xterm? Thanks!
-Eric-
The program run with -e must be a program, not a shell built-in like setenv. Off the top of my head, I can think of two solutions:
Set the environment variables before starting xterm. They would then be inherited by xterm.
Let the program run by xterm be a shell, and use that shell to set the environment variables and launch the tool. Something like this (untested):
system("xterm", "-e", "/bin/sh", "-c", "FOO=bar; FIE=fum; /run/my/program");
According to the man page I read, the following is the syntax of the -e option:
-e program [ arguments ... ]
It takes a path to a program, and optionally arguments to pass to that program. Specifically, it doesn't take a shell command. (It would be bad to accept a shell command without having the user specify for which shell!) That doesn't preclude one from running a shell command, though. This simply requires launching a shell, as the following does:
xterm -e sh -c shell_cmd
Solution:
my $script = <<'__EOS__';
export ZI_LIBERTY_IGNORE_CONSTRUCT_FILES="$RAW_RF_DIR/lib/lib2v/qcdc_ignore"
export HOME_0IN=/p/hdk/rtl/cad/x86-64_linux26/mentor/questaCDC/V10.4g_5/linux_x86_64
modpath -n 1 "$HOME_0IN/bin"
modpath -n 1 "$HOME_0IN/modeltech/bin"
/p/hdk/rtl/cad/x86-64_linux30/mentor/questaCDC/V10.4f_5/linux_x86_64/bin/qcdc -c -licq -do run.tcl
__EOS__
system("xterm", "-hold", "-e", "sh", "-c", $script)
Since a process normally passes a copy of its env vars to process it creates, you could also write the above as follows:
local $ENV{ZI_LIBERTY_IGNORE_CONSTRUCT_FILES} = "$ENV{RAW_RF_DIR}/lib/lib2v/qcdc_ignore";
local $ENV{HOME_0IN} = "/p/hdk/rtl/cad/x86-64_linux26/mentor/questaCDC/V10.4g_5/linux_x86_64";
my $script = <<'__EOS__';
modpath -n 1 "$HOME_0IN/bin"
modpath -n 1 "$HOME_0IN/modeltech/bin"
/p/hdk/rtl/cad/x86-64_linux30/mentor/questaCDC/V10.4f_5/linux_x86_64/bin/qcdc -c -licq -do run.tcl
__EOS__
system("xterm", "-hold", "-e", "sh", "-c", $script)
You can use csh similarly; I'm simply more familiar with sh.
#!/bin/bash
jobname="job_201312161447_0003"
jobname_pre=${jobname:0:16}
jobname_post=${jobname:17}
This bash script gives me Bad substitution error on ubuntu. Any help will be highly appreciated.
The default shell (/bin/sh) under Ubuntu points to dash, not bash.
me#pc:~$ readlink -f $(which sh)
/bin/dash
So if you chmod +x your_script_file.sh and then run it with ./your_script_file.sh, or if you run it with bash your_script_file.sh, it should work fine.
Running it with sh your_script_file.sh will not work because the hashbang line will be ignored and the script will be interpreted by dash, which does not support that string substitution syntax.
I had the same problem. Make sure your script didnt have
#!/bin/sh
at the top of your script. Instead, you should add
#!/bin/bash
For others that arrive here, this exact message will also appear when using the env variable syntax for commands, for example ${which sh} instead of the correct $(which sh)
Your script syntax is valid bash and good.
Possible causes for the failure:
Your bash is not really bash but ksh or some other shell which doesn't understand bash's parameter substitution. Because your script looks fine and works with bash.
Do ls -l /bin/bash and check it's really bash and not sym-linked to some other shell.
If you do have bash on your system, then you may be executing your script the wrong way like: ksh script.sh or sh script.sh (and your default shell is not bash). Since you have proper shebang, if you have bash ./script.sh or bash ./script.sh should be fine.
Try running the script explicitly using bash command rather than just executing it as executable.
Also, make sure you don't have an empty string for the first line of your script.
i.e. make sure #!/bin/bash is the very first line of your script.
Not relevant to your example, but you can also get the Bad substitution error in Bash for any substitution syntax that Bash does not recognize. This could be:
Stray whitespace. E.g. bash -c '${x }'
A typo. E.g. bash -c '${x;-}'
A feature that was added in a later Bash version. E.g. bash -c '${x#Q}' before Bash 4.4.
If you have multiple substitutions in the same expression, Bash may not be very helpful in pinpointing the problematic expression. E.g.:
$ bash -c '"${x } multiline string
$y"'
bash: line 1: ${x } multiline string
$y: bad substitution
Both - bash or dash - work, but the syntax needs to be:
FILENAME=/my/complex/path/name.ext
NEWNAME=${FILENAME%ext}new
I was adding a dollar sign twice in an expression with curly braces in bash:
cp -r $PROJECT_NAME ${$PROJECT_NAME}2
instead of
cp -r $PROJECT_NAME ${PROJECT_NAME}2
I have found that this issue is either caused by the marked answer or you have a line or space before the bash declaration
Looks like "+x" causes problems:
root#raspi1:~# cat > /tmp/btest
#!/bin/bash
jobname="job_201312161447_0003"
jobname_pre=${jobname:0:16}
jobname_post=${jobname:17}
root#raspi1:~# chmod +x /tmp/btest
root#raspi1:~# /tmp/btest
root#raspi1:~# sh -x /tmp/btest
+ jobname=job_201312161447_0003
/tmp/btest: 4: /tmp/btest: Bad substitution
in my case (under ubuntu 18.04), I have mixed $( ${} ) that works fine:
BACKUPED_NB=$(ls ${HOST_BACKUP_DIR}*${CONTAINER_NAME}.backup.sql.gz | wc --lines)
full example here.
I used #!bin/bash as well tried all approaches like no line before or after #!bin/bash.
Then also tried using +x but still didn't work.
Finally i tried running the script ./script.sh it worked fine.
#!/bin/bash
jobname="job_201312161447_0003"
jobname_post=${jobname:17}
root#ip-10-2-250-36:/home/bitnami/python-module/workflow_scripts# sh jaru.sh
jaru.sh: 3: jaru.sh: Bad substitution
root#ip-10-2-250-36:/home/bitnami/python-module/workflow_scripts# ./jaru.sh
root#ip-10-2-250-36:/home/bitnami/python-module/workflow_scripts#
I know of system() and qx(), but I need to execute ~15 bash commands. E.g.
mkdir, chown, edquota -p user1 -u user2, cp -r, su - username, git, rm, ln -s
Question
Is there an efficient way to execute many Bash commands in Perl?
I don't care in this case about the output.
First, I'd use the equivalent Perl function for as many of those bash command as I could, which is most of the ones you included in your post. Then, for the rest of them I'd either use system() or qx() or backticks or one of the IPC:: modules (such as IPC::Run or IPC::Open3).
Use bash syntax for many commands. Separate them with ; or && or whatever takes your fancy (man bash).
$ perl -E 'system qq{date; date}'
In Linux, I like POE framework's POE::Wheel::Run module for running system commands (and code blocks) asynchronously. You say you do not care about the output, but if you need it in the future POE::Wheel::Run has an elegant interface allowing us to interact with the process.
my $s = <<END;
echo "1"
echo "2"
echo "3"
END
system("$s");
This works fine
system("perl -c C:/Users/mytest/scripts/file_name.pm")
This command gives many lines of output in cygwin and a single syntak ok line in centos. since ill be using cygwin, what am trying to do is to get this output into a variable and use it later in my program. How can i do it?
Thanks in advance for your time.
Instead of system, use backticks:
my $output = `perl -c C:/Users/mytest/scripts/file_name.pm`;
if you want to also include STDERR output, use:
my $output = `perl -c C:/Users/mytest/scripts/file_name.pm 2>&1`;