I am attempting to automate the following series of commands which work correctly into a BASH script:
kubectl exec -it mongo-pod -- bash
mongo DBNAME
db.auth("theUser", "thePw")
db.theCollection.find()
The script I am using is as follows:
#!/bin/bash
kubectl exec -it mongo-pod -- bash -c "mongo DBNAME && /
db.auth("theUser", "thePw") && /
db.theCollection.find()"
I have tried the following:
Executing multiple commands( or from a shell script) in a kubernetes pod
but any commands that are added after the first using & or && are not executed. For example just using "mongo DBNAME" correctly opens the prompt and sets it to the correct db, but adding any other command with && causes all commands to fail with the following:
bash: -c line 0: syntax error near unexpected token 'theUser'
All of the comments are spot on, but at least two things I have the highest confidence I have the answer to:
First, you have the line continuation character wrong; it should be \ and not /. It actually wouldn't even be required if you switched bash into "exit on error" mode, with
kubectl exec -it mongo-pod -- bash -ec "mongo DBNAME
echo 'this command only runs if mongo exits a-ok'
exit 1
and this never will run
"
However, the other mistake is around the quoting characters used: if you have bash -c " then you must either use the single-quote for the interior string literals, or escape them with \". You can actually see what I'm talking about by looking at the syntax highlighting of the shell snippet in your question. Observe that the string literal is red, but then the text theUser as well as thePw are both black -- that's because they are outside the string literal since the string stopped at the first " it encountered -- the one present in db.auth("
It is almost always the case that you'll want to use single quotes when invoking bash remotely like that, for several reasons but the most relevant is that you can then use db.auth("something") without having to unnecessarily escape the double quotes.
Since mongo (like many interpreters such a node and python) wants you to either type in it interactively, provide the input on its "standard input", or give it a local file containing commands, you will want to change the invocation to one of those strategies depending on your needs.
A very convenient way of redirecting standard input without having to use echo or printf and its associated quoting hell is to use what are called "here documents" (abbreviated "heredocs") in bash:
kubectl exec -it mongo-pod -- bash -ec 'mongo DBNAME<<"FOO"
db.auth("theUser", "thePw")
printjson(db.theCollection.find())
FOO
'
That causes bash to transmit almost all characters between the two "heredoc delimiters" to the standard input of the command. If you quote the delimiter, as I have with the [arbitrary] word FOO, then the contents are not subject to variable expansion, command interpolation, etc, which can be one more mechanism to avoid backtick and dollarsign weirdness.
Related
I have a script where I need to start a command, then pass some additional commands as commands to that command. I tried
su
echo I should be root now:
who am I
exit
echo done.
... but it doesn't work: The su succeeds, but then the command prompt is just staring at me. If I type exit at the prompt, the echo and who am i etc start executing! And the echo done. doesn't get executed at all.
Similarly, I need for this to work over ssh:
ssh remotehost
# this should run under my account on remotehost
su
## this should run as root on remotehost
whoami
exit
## back
exit
# back
How do I solve this?
I am looking for answers which solve this in a general fashion, and which are not specific to su or ssh in particular. The intent is for this question to become a canonical for this particular pattern.
Adding to tripleee's answer:
It is important to remember that the section of the script formatted as a here-document for another shell is executed in a different shell with its own environment (and maybe even on a different machine).
If that block of your script contains parameter expansion, command substitution, and/or arithmetic expansion, then you must use the here-document facility of the shell slightly differently, depending on where you want those expansions to be performed.
1. All expansions must be performed within the scope of the parent shell.
Then the delimiter of the here document must be unquoted.
command <<DELIMITER
...
DELIMITER
Example:
#!/bin/bash
a=0
mylogin=$(whoami)
sudo sh <<END
a=1
mylogin=$(whoami)
echo a=$a
echo mylogin=$mylogin
END
echo a=$a
echo mylogin=$mylogin
Output:
a=0
mylogin=leon
a=0
mylogin=leon
2. All expansions must be performed within the scope of the child shell.
Then the delimiter of the here document must be quoted.
command <<'DELIMITER'
...
DELIMITER
Example:
#!/bin/bash
a=0
mylogin=$(whoami)
sudo sh <<'END'
a=1
mylogin=$(whoami)
echo a=$a
echo mylogin=$mylogin
END
echo a=$a
echo mylogin=$mylogin
Output:
a=1
mylogin=root
a=0
mylogin=leon
3. Some expansions must be performed in the child shell, some - in the parent.
Then the delimiter of the here document must be unquoted and you must escape those expansion expressions that must be performed in the child shell.
Example:
#!/bin/bash
a=0
mylogin=$(whoami)
sudo sh <<END
a=1
mylogin=\$(whoami)
echo a=$a
echo mylogin=\$mylogin
END
echo a=$a
echo mylogin=$mylogin
Output:
a=0
mylogin=root
a=0
mylogin=leon
A shell script is a sequence of commands. The shell will read the script file, and execute those commands one after the other.
In the usual case, there are no surprises here; but a frequent beginner error is assuming that some commands will take over from the shell, and start executing the following commands in the script file instead of the shell which is currently running this script. But that's not how it works.
Basically, scripts work exactly like interactive commands, but how exactly they work needs to be properly understood. Interactively, the shell reads a command (from standard input), runs that command (with input from standard input), and when it's done, it reads another command (from standard input).
Now, when executing a script, standard input is still the terminal (unless you used a redirection) but the commands are read from the script file, not from standard input. (The opposite would be very cumbersome indeed - any read would consume the next line of the script, cat would slurp all the rest of the script, and there would be no way to interact with it!) The script file only contains commands for the shell instance which executes it (though you can of course still use a here document etc to embed inputs as command arguments).
In other words, these "misunderstood" commands (su, ssh, sh, sudo, bash etc) when run alone (without arguments) will start an interactive shell, and in an interactive session, that's obviously fine; but when run from a script, that's very often not what you want.
All of these commands have ways to accept commands by ways other than in an interactive terminal session. Typically, each command supports a way to pass it commands as options or arguments:
su root -c 'who am i'
ssh user#remote uname -a
sh -c 'who am i; echo success'
Many of these commands will also accept commands on standard input:
printf 'uname -a; who am i; uptime' | su
printf 'uname -a; who am i; uptime' | ssh user#remote
printf 'uname -a; who am i; uptime' | sh
which also conveniently allows you to use here documents:
ssh user#remote <<'____HERE'
uname -a
who am i
uptime
____HERE
sh <<'____HERE'
uname -a
who am i
uptime
____HERE
For commands which accept a single command argument, that command can be sh or bash with multiple commands:
sudo sh -c 'uname -a; who am i; uptime'
As an aside, you generally don't need an explicit exit because the command will terminate anyway when it has executed the script (sequence of commands) you passed in for execution.
If you want a generic solution which will work for any kind of program, you can use the expect command.
Extract from the manual page:
Expect is a program that "talks" to other interactive programs according to a script. Following the script, Expect knows what can be expected from a program and what the correct response should be. An interpreted language provides branching and high-level control structures to direct the dialogue. In addition, the user can take control and interact directly when desired, afterward returning control to the script.
Here is a working example using expect:
set timeout 60
spawn sudo su -
expect "*?assword" { send "*secretpassword*\r" }
send_user "I should be root now:"
expect "#" { send "whoami\r" }
expect "#" { send "exit\r" }
send_user "Done.\n"
exit
The script can then be launched with a simple command:
$ expect -f custom.script
You can view a full example in the following page: http://www.journaldev.com/1405/expect-script-example-for-ssh-and-su-login-and-running-commands
Note: The answer proposed by #tripleee would only work if standard input could be read once at the start of the command, or if a tty had been allocated, and won't work for any interactive program.
Example of errors if you use a pipe
echo "su whoami" |ssh remotehost
--> su: must be run from a terminal
echo "sudo whoami" |ssh remotehost
--> sudo: no tty present and no askpass program specified
In SSH, you might force a TTY allocation with multiple -t parameters, but when sudo will ask for the password, it will fail.
Without the use of a program like expect any call to a function/program which might get information from stdin will make the next command fail:
ssh use#host <<'____HERE'
echo "Enter your name:"
read name
echo "ok."
____HERE
--> The `echo "ok."` string will be passed to the "read" command
I am following https://docs.docker.com/get-started/06_bind_mounts/#start-a-dev-mode-container on a Windows PC and am stuck here:
Run the following command. We’ll explain what’s going on afterwards:
docker run -dp 3000:3000 \
-w /app -v "$(pwd):/app" \
node:12-alpine \
sh -c "yarn install && yarn run dev"
If you are using PowerShell then use this command:
docker run -dp 3000:3000 `
-w /app -v "$(pwd):/app" `
node:12-alpine `
sh -c "yarn install && yarn run dev"
When using Command Prompt, I get errors (tried multiple variations as shown below), and when using PowerShell, I don't appear to get errors but am not running anything as showed when executing docker ps.
Note that I would rather use Command Prompt and not PowerShell as I could use Linux commands with ComandPrompt on my PC.
What is the significance of backslashes when using Dockers with Command Prompt (and tick marks with PowerShell for that matter)?
I have since found that docker run -dp 3000:3000 -w /app -v "%cd%:/app" node:12-alpine sh -c "yarn install && yarn run dev" works without errors (got rid of backslashes, put on one line, and used %cd% instead of $(pwd)), but would still like to know why using the exact script in the example results in errors.
Using Command Prompt
C:\Users\michael\Documents\Docker\app>docker run -dp 3000:3000 \
docker: invalid reference format.
See 'docker run --help'.
C:\Users\michael\Documents\Docker\app> -w /app -v "$(pwd):/app" \
'-w' is not recognized as an internal or external command,
operable program or batch file.
C:\Users\michael\Documents\Docker\app> node:12-alpine \
The filename, directory name, or volume label syntax is incorrect.
C:\Users\michael\Documents\Docker\app> sh -c "yarn install && yarn run dev"
sh: yarn: command not found
C:\Users\michael\Documents\Docker\app>docker run -dp 3000:3000 \ -w /app -v "$(pwd):/app" \ node:12-alpine \ sh -c "yarn install && yarn run dev"
docker: invalid reference format.
See 'docker run --help'.
C:\Users\michael\Documents\Docker\app>docker run -dp 3000:3000 -w /app -v "$(pwd):/app" node:12-alpine sh -c "yarn install && yarn run dev"
docker: Error response from daemon: create $(pwd): "$(pwd)" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path.
See 'docker run --help'.
C:\Users\michael\Documents\Docker\app>
Using PowerShell
PS C:\Users\michael\Documents\Docker> docker run -dp 3000:3000 `
>> -w /app -v "$(pwd):/app" `
>> node:12-alpine `
>> sh -c "yarn install && yarn run dev"
849af42e78d4ab09242fdd6c3d03bcf1b6b58de984c4485a441a2e2c88603767
PS C:\Users\michael\Documents\Docker> docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
PS C:\Users\michael\Documents\Docker>
would still like to know why using the exact script in the example results in errors.
Because the command with the line-ending \ characters is meant for POSIX-compatible shells such as bash, not for cmd.exe
POSIX-compatible shells (sh, bash, dash, ksh, zsh):
use \ for line-continuation (continuing a command on the following line) and escaping in general.
use $varName to reference both environment and shell-only variables.
support $(...) for embedding the output from a command (...) in command lines (command substitution).
support both double-quoted ("...", interpolating) and single-quoted ('...', verbatim) strings; use '\'' to - in effect - include a ' inside '...'.
(Additionally, in bash, ksh, and zsh, there are the rarely used ANSI C-quoted strings, $'...', and, in bash and ksh, perhaps even more rarely, localizable strings, $"...").
cmd.exe:
uses ^ for line-continuation and escaping in general (in unquoted arguments only).
uses %varName% to reference environment variables (the only variable type supported).
doesn't support command substitutions at all.
supports only "..." strings (interpolating).
PowerShell:
uses ` (the backtick) for line-continuation and escaping in general.
uses $env:varName to reference environment variables, $varName to reference shell-only variables.
supports $(...), called subexpressions, the equivalent of command substitutions (outside of double-quoted strings, (...) is usually sufficient).
supports both double-quoted ("...", interpolating) and single-quoted ('...', verbatim) strings; use '' to embed a ' inside '...'.
Note: A common pitfall is that PowerShell has more metacharacters compared to both POSIX-compatible shells and cmd.exe, notably including # { } , ;, which therefore require individual `-escaping in unquoted arguments or embedding in quoted strings - see this answer.
Potential line-continuation pitfall: in all of the shells discussed, the escape character must be the very last character on the line - not even trailing (intra-line) whitespace is allowed (because the escape character would then apply to it rather than to the newline).
The information above is summarized in the following table:
Feature
POSIX shells _
cmd.exe _
PowerShell _
Line-continuation / escape character
Backslash (\)
Caret (^)
Backtick (`)
Double-quoted strings (interpolating)
✅
✅
✅
Single-quoted strings (verbatim)
✅
❌
✅
Get / set environment variables
$varName /export varName=...
%varName% /set varName=...
$env:varName /$env:varName = ...
Get / set shell-only variables
$varName/varName=...
❌ (no such variables exist, but you can limit the scope of env. vars. with setlocal)
$varName/$varName = ...
Command substitutions, subexpressions
$(...)
❌
(...) / $(...), esp. in strings
Note re setting variables with respect to whitespace on either side of the = symbol:
In POSIX-like shells, there must not be whitespace around =.
In cmd.exe, such whitespace is significant and becomes part of the variable / value name, and is therefore usually to be avoided.
In PowerShell, such whitespace is optional - you may use it to enhance readability; any string value to be assigned requires quoting (e.g., $var = 'hi!')
See also:
https://hyperpolyglot.org/shell for a much more comprehensive juxtaposition of these shells, though note that - as of this writing - the information about PowerShell is incomplete.
Sage Pourpre's helpful answer for links to the line-continuation documentation of the respective shells.
This is character escaping.
The X Character (\ for Bash, backtick for Powershell and ^ for Windows terminal )are used to remove any specific meanings to the next characters.
When used at the end of a line, this mean that the next character (The newline character) is completely ignored.
This keep the command essentially a one-line command from the point of view of the interpreter, but allow you to break it on multiple lines for better readability.
References
Powershell - About special characters
Escape sequences begin with the backtick character [`], known as the grave
accent (ASCII 96), and are case-sensitive. The backtick character can
also be referred to as the escape character.
Bash manual
3.1.2.1 Escape Character
A non-quoted backslash \ is the Bash escape character. It preserves the literal value of the next character that
follows, with the exception of newline. If a \newline pair appears,
and the backslash itself is not quoted, the \newline is treated as a
line continuation (that is, it is removed from the input stream and
effectively ignored).
How-to: Escape Characters, Delimiters and Quotes at the Windows command line
Escaping CR/LF line endings. The ^ escape character can be used to
make long commands more readable by splitting them into multiple lines
and escaping the Carriage Return + Line Feed (CR/LF) at the end of a
line:
ROBOCOPY \\FileServ1\e$\users ^ \\FileServ2\e$\BackupUsers ^ /COPYALL /B /SEC /MIR ^ /R:0 /W:0 /LOG:MyLogfile.txt /NFL /NDL
[...]
A couple of things to be aware of:
A stray space at the end of a line (after the ^) will break the
command, this can be hard to spot unless you have a text editor that
displays spaces and tab characters. If you want comment something out
with REM, then EVERY line needs to be prefixed with REM. Alternatively
if you use a double colon :: as a REM comment, that will still parse
the caret at the end of a line, so in the example above changing the
first line to :: ROBOCOPY… will comment out the whole multi-line
command.
I would like to pass LSB_JOBINDEX to as an argument to my script instead of using an environment variable.
This makes my script more LSF agnostic and avoids creating a helper script that uses the environment variable.
However, I was not able to use LSB_JOBINDEX in arguments: it only works as part of the initial command string.
For example, from a bash shell, I use the test command:
bsub -J 'myjobname[1-4]' -o bsub%I.log \
'echo $LSB_JOBINDEX' \
'$LSB_JOBINDEX' \
\$LSB_JOBINDEX \
'$LSB_JOBINDEX' \
"\$LSB_JOBINDEX"
and the output of say bsub2.log is:
2 $LSB_JOBINDEX $LSB_JOBINDEX $LSB_JOBINDEX $LSB_JOBINDEX
So in this case, only the first $LSB_JOBINDEX got expanded, but not any of the following ones.
But I would rather not pass the entire command as a single huge string as the 'echo $LSB_JOBINDEX' in this example. I would prefer to just use separate arguments as in a regular bash command.
I've also tried to play around with %I but it only works for -o and related bsub options, not for the command itself.
Related: Referencing job index in LSF job array
Tested in LSF 10.1.0. Related documentation: https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_admin/job_array_cl_args.html
bsub will add single quotes around the arguments if the argument starts with $. For example. If the bsub command line is
bsub command -a $ARG1 -b $ARG2
Then bsub will add quotes to the arguments to the 2nd and 4th parameters. The command is stored like this
command -a '$ARG1' -b '$ARG2'
One way to prevent this is to put the commands in a script. Like this:
$ cat cmd
echo $LSB_JOBINDEX
echo "line 2"
echo $LSB_JOBINDEX
Then run your job like this:
$ bsub -I < cmd
Job <2669> is submitted to default queue <normal>.
<<Waiting for dispatch ...>>
<<Starting on hostA>>
0
line 2
0
Note that the -I is not needed. Its just so you can see the job output on the bsub's stdout.
EDIT
OK. Looks like this works. But its not really a serious answer since it's so ugly. The thing is that bsub will surround the argument with single quotes if the argument starts with $. So the strategy is to find some way to make sure that the first character in the argument isn't a $. One way is to put any character other than $ as the first character of the argument. Follow it by a backspace literal, followed by the $. Note that it needs to be the actual backspace character, not ^ followed by H. Use ctrl-v followed by a ctrl-h to get the literal appended to the command line.
$ bsub -I echo "x^H\$LSB_JOBINDEX" "x^H\$LSB_JOBINDEX"
Job <2686> is submitted to default queue <normal>.
<<Waiting for dispatch ...>>
<<Starting on hostA>>
0 0
EDIT2
A tab literal also works. Not that its much better.
$ bsub -I echo " \$LSB_JOBINDEX" " \$LSB_JOBINDEX"
Job <2687> is submitted to default queue <normal>.
<<Waiting for dispatch ...>>
<<Starting on hostA>>
0 0
I'm having an issue trying to capture the output of a sed command in a makefile variable.
JS_SRC:=$(shell sed -n 's#.*src="\([^"]*\.js\).*#\1#p' index.html)
Which gives me
sed: -e expression #1, char 34: unknown option tos'
`
I've been trying to escape things and the like, but am always given that error.
All variations of escaping I have run, run fine from the terminal.
How does a makefile call the shell command?. /usr/bin/sh -c "cmd?" or something different?.
Somethings being interpolated but I have no idea what.
JS_SRC:=$(shell sed -n "s/.*src=\"\\([^\"]*\\.js\\).*/\\1/p" index.html)
Appears to work. I figured this out via running make -d and seeing the process it was creating.
What was baffling is that it did different things with ' vs " in the sed argument. " is run with /bin/sh -c "args" so I was able to tweak the escaping to get what I needed to appear there. Using ' seems to invoke sed directly.
There is a whole heap of escaping, that i imagine is unnecessary (I don't need to interpolate variables in the sed expression, but it sends it to a shell I understand. So it will have to do ! :)
cat "${pos}" | /usr/bin/iconv -f CP1251 -t UTF-8 | uniq | sed -En "/^CLIENT_ID.*/!p" | while read line
do
.....
......
cat >> "$TMPFILE" << EOF
INSERT INTO ......;
EOF
done
As you can see each iteration writes a SQL statement to a tmp-file.
I launched this script from a regular interactive shell and got the expected output. Launched from a cron job - nothing.
After investigating I found a problem. When I use "$TMPFILE" without "" the script works ok. Why does this happen?
OS: FreeBSD, bourne shell.
IIRC, cron doesn't source all the files that a login shell does, so you will end up with different settings for environment variables. Could be the path $TMPFILE is pointing to contains spaces when run from cron for example.
Also, on some systems (depending on setup), cron uses a different shell. So if you start your script from command line, for example /usr/bin/sh might be used, whereas when started by cron, /bin/sh is used. (I have no experience with *BSD, but I have observed this on linux.)