I'm trying to send a HTML request via cURL in PowerShell. The code in question:
$command = 'curl -d #request.txt -o testoutput.xml -X "POST" -H #header.txt -u "username:password" "URL"'
Invoke-Expression $command
The HTML Body (-d) and Header (-H) need to be read from files, hence the # before the filenames.
In order for this to work, I need to escape the # so PowerShell doesn't interpret it as a splat operator - and that's my problem, nothing I've tried so far worked:
putting it in single quotes
here document (#" "#)
here string (#' '#)
using string templates with placeholders ("{0}request.txt" -f "#")
Unicode escape ("`u{0040}request.txt")
How do I escape the # correctly? Or am I on the wrong path entirely? (I'm new to PowerShell sorry)
Invoke-Expression (iex) should generally be avoided; definitely don't use it to invoke an external program or PowerShell script.
Therefore:
invoke curl directly
and quote PowerShell metacharacters such as # (see this answer for the full list) - either individually, with ` (e.g., `#), or by enclosing the entire argument in quotes ('...' or "...", as appropriate).
curl -d `#request.txt -o testoutput.xml -X "POST" -H #header.txt -u "username:password" "URL"
As for what you tried:
The primary concern about Invoke-Expression use is one of security: unwanted execution of (additional) commands potentially injected into the string passed to it.
In terms of functionality, Invoke-Expression is not only not needed for regular calls to external programs, it can create quoting headaches.
However, given that you're passing a '...' string to Invoke-Expression, quoting the # as `# would have worked, as shown in this simple example:
# OK, but generally do NOT do this.
Invoke-Expression 'Write-Output `#'
Had you used a "..." string, the ` would have been "eaten" by the up-front parsing of such an expandable string, based on the escaping rules explained in the conceptual about_Quoting_Rules help topic:
# BREAKS, because the ` is stripped before Invoke-Expression sees it.
Invoke-Expression "Write-Output `#"
Related
I am following https://docs.docker.com/get-started/06_bind_mounts/#start-a-dev-mode-container on a Windows PC and am stuck here:
Run the following command. We’ll explain what’s going on afterwards:
docker run -dp 3000:3000 \
-w /app -v "$(pwd):/app" \
node:12-alpine \
sh -c "yarn install && yarn run dev"
If you are using PowerShell then use this command:
docker run -dp 3000:3000 `
-w /app -v "$(pwd):/app" `
node:12-alpine `
sh -c "yarn install && yarn run dev"
When using Command Prompt, I get errors (tried multiple variations as shown below), and when using PowerShell, I don't appear to get errors but am not running anything as showed when executing docker ps.
Note that I would rather use Command Prompt and not PowerShell as I could use Linux commands with ComandPrompt on my PC.
What is the significance of backslashes when using Dockers with Command Prompt (and tick marks with PowerShell for that matter)?
I have since found that docker run -dp 3000:3000 -w /app -v "%cd%:/app" node:12-alpine sh -c "yarn install && yarn run dev" works without errors (got rid of backslashes, put on one line, and used %cd% instead of $(pwd)), but would still like to know why using the exact script in the example results in errors.
Using Command Prompt
C:\Users\michael\Documents\Docker\app>docker run -dp 3000:3000 \
docker: invalid reference format.
See 'docker run --help'.
C:\Users\michael\Documents\Docker\app> -w /app -v "$(pwd):/app" \
'-w' is not recognized as an internal or external command,
operable program or batch file.
C:\Users\michael\Documents\Docker\app> node:12-alpine \
The filename, directory name, or volume label syntax is incorrect.
C:\Users\michael\Documents\Docker\app> sh -c "yarn install && yarn run dev"
sh: yarn: command not found
C:\Users\michael\Documents\Docker\app>docker run -dp 3000:3000 \ -w /app -v "$(pwd):/app" \ node:12-alpine \ sh -c "yarn install && yarn run dev"
docker: invalid reference format.
See 'docker run --help'.
C:\Users\michael\Documents\Docker\app>docker run -dp 3000:3000 -w /app -v "$(pwd):/app" node:12-alpine sh -c "yarn install && yarn run dev"
docker: Error response from daemon: create $(pwd): "$(pwd)" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path.
See 'docker run --help'.
C:\Users\michael\Documents\Docker\app>
Using PowerShell
PS C:\Users\michael\Documents\Docker> docker run -dp 3000:3000 `
>> -w /app -v "$(pwd):/app" `
>> node:12-alpine `
>> sh -c "yarn install && yarn run dev"
849af42e78d4ab09242fdd6c3d03bcf1b6b58de984c4485a441a2e2c88603767
PS C:\Users\michael\Documents\Docker> docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
PS C:\Users\michael\Documents\Docker>
would still like to know why using the exact script in the example results in errors.
Because the command with the line-ending \ characters is meant for POSIX-compatible shells such as bash, not for cmd.exe
POSIX-compatible shells (sh, bash, dash, ksh, zsh):
use \ for line-continuation (continuing a command on the following line) and escaping in general.
use $varName to reference both environment and shell-only variables.
support $(...) for embedding the output from a command (...) in command lines (command substitution).
support both double-quoted ("...", interpolating) and single-quoted ('...', verbatim) strings; use '\'' to - in effect - include a ' inside '...'.
(Additionally, in bash, ksh, and zsh, there are the rarely used ANSI C-quoted strings, $'...', and, in bash and ksh, perhaps even more rarely, localizable strings, $"...").
cmd.exe:
uses ^ for line-continuation and escaping in general (in unquoted arguments only).
uses %varName% to reference environment variables (the only variable type supported).
doesn't support command substitutions at all.
supports only "..." strings (interpolating).
PowerShell:
uses ` (the backtick) for line-continuation and escaping in general.
uses $env:varName to reference environment variables, $varName to reference shell-only variables.
supports $(...), called subexpressions, the equivalent of command substitutions (outside of double-quoted strings, (...) is usually sufficient).
supports both double-quoted ("...", interpolating) and single-quoted ('...', verbatim) strings; use '' to embed a ' inside '...'.
Note: A common pitfall is that PowerShell has more metacharacters compared to both POSIX-compatible shells and cmd.exe, notably including # { } , ;, which therefore require individual `-escaping in unquoted arguments or embedding in quoted strings - see this answer.
Potential line-continuation pitfall: in all of the shells discussed, the escape character must be the very last character on the line - not even trailing (intra-line) whitespace is allowed (because the escape character would then apply to it rather than to the newline).
The information above is summarized in the following table:
Feature
POSIX shells _
cmd.exe _
PowerShell _
Line-continuation / escape character
Backslash (\)
Caret (^)
Backtick (`)
Double-quoted strings (interpolating)
✅
✅
✅
Single-quoted strings (verbatim)
✅
❌
✅
Get / set environment variables
$varName /export varName=...
%varName% /set varName=...
$env:varName /$env:varName = ...
Get / set shell-only variables
$varName/varName=...
❌ (no such variables exist, but you can limit the scope of env. vars. with setlocal)
$varName/$varName = ...
Command substitutions, subexpressions
$(...)
❌
(...) / $(...), esp. in strings
Note re setting variables with respect to whitespace on either side of the = symbol:
In POSIX-like shells, there must not be whitespace around =.
In cmd.exe, such whitespace is significant and becomes part of the variable / value name, and is therefore usually to be avoided.
In PowerShell, such whitespace is optional - you may use it to enhance readability; any string value to be assigned requires quoting (e.g., $var = 'hi!')
See also:
https://hyperpolyglot.org/shell for a much more comprehensive juxtaposition of these shells, though note that - as of this writing - the information about PowerShell is incomplete.
Sage Pourpre's helpful answer for links to the line-continuation documentation of the respective shells.
This is character escaping.
The X Character (\ for Bash, backtick for Powershell and ^ for Windows terminal )are used to remove any specific meanings to the next characters.
When used at the end of a line, this mean that the next character (The newline character) is completely ignored.
This keep the command essentially a one-line command from the point of view of the interpreter, but allow you to break it on multiple lines for better readability.
References
Powershell - About special characters
Escape sequences begin with the backtick character [`], known as the grave
accent (ASCII 96), and are case-sensitive. The backtick character can
also be referred to as the escape character.
Bash manual
3.1.2.1 Escape Character
A non-quoted backslash \ is the Bash escape character. It preserves the literal value of the next character that
follows, with the exception of newline. If a \newline pair appears,
and the backslash itself is not quoted, the \newline is treated as a
line continuation (that is, it is removed from the input stream and
effectively ignored).
How-to: Escape Characters, Delimiters and Quotes at the Windows command line
Escaping CR/LF line endings. The ^ escape character can be used to
make long commands more readable by splitting them into multiple lines
and escaping the Carriage Return + Line Feed (CR/LF) at the end of a
line:
ROBOCOPY \\FileServ1\e$\users ^ \\FileServ2\e$\BackupUsers ^ /COPYALL /B /SEC /MIR ^ /R:0 /W:0 /LOG:MyLogfile.txt /NFL /NDL
[...]
A couple of things to be aware of:
A stray space at the end of a line (after the ^) will break the
command, this can be hard to spot unless you have a text editor that
displays spaces and tab characters. If you want comment something out
with REM, then EVERY line needs to be prefixed with REM. Alternatively
if you use a double colon :: as a REM comment, that will still parse
the caret at the end of a line, so in the example above changing the
first line to :: ROBOCOPY… will comment out the whole multi-line
command.
I want to write a PowerShell script that includes a single command that is split over new lines, such as (the actual command I am trying to run is longer):
ssh \
-L 8080:localhost:8080 \
foo#bar.com
However the \ doesn't escape the new line (neither does /).
Use the backtick ( ` ), like this:
Write-Output `
"Hello world!"
I am having some difficulty with a Perl script that is invoked from cron.
One of the arguments for the script is a GPG passphrase. This is later interpolated into a string that is sent to the shell.
This particular passphrase contains an open parentheses, and the script fails with a "syntax error near unexpected token `(" error.
This is the offending part of the phrase:
m3(ÃÝ4úŤ;:q!
I have tried single and double quoting it before the value is used in the script, but that does not have an effect.
The phrase works correctly when invoking GPG directly from the shell, just not when it gets interpolated into the following:
`gpg --passphrase $gpgpp --batch -o $gpgofile -d $file`;
Where $gpgpp is the passphrase variable.
What is the correct way to escape this, and other potentially problematic characters?
The \Q and \E escape sequences are used to escape all "special" characters between them.
`gpg --passphrase \Q$gpgpp\E --batch -o $gpgofile -d $file`;
This should be done any time you have a variable that may contain characters which need to be escaped.
http://perldoc.perl.org/functions/quotemeta.html
Enclose the variables in quotes:
gpg --passphrase "$gpgpp" --batch -o "$gpgofile" -d "$file"
Otherwise, the shell will try to interpret contents of the variables as expressions.
I followed a few examples here, but I still can not get my sqlcmd work after runas.
I can make it work with two steps:
Step one: Use runas to login into a new login and open a command prompt:
runas.exe /savecred /user:DOMAIN_NAME\login_name cmd.exe
Step two: execute the sqlcmd in a script
sqlcmd -S server_name -E /Q"exit(SELECT ##version )"
But I want to make it one step to get the results. I tried to add " " after the runas command as listed below, but it did not work:
runas.exe /savecred /user:DOMAIN_NAME\login_name "sqlcmd -S server_name -E /Q"exit(SELECT ##version )""
Any ideas?
Take a look at this article describing RunAs. Toward the end of the article, he specifically addresses needing to use quotes inside of quotes:
Fortunately, it’s fairly easy to make RunAs happy. All you need to do is use the \ character to “escape” any double quote marks that must to be embedded within the process path.
So, it looks like your command should be:
runas.exe /savecred /user:DOMAIN_NAME\login_name "sqlcmd -S server_name -E /Q\"exit(SELECT ##version )\""
Notice the \ before both of the nested double quotes.
I have a file named check.txt which has the below contents:
$ cat check.txt
~/bin/tibemsadmin -server $URL-user $USER -password $PASWRD
$
I have a main script where the values of $URL, $USER, $PASWRD are obtained from the main script. I want to use the SED utility to replace the $URL, $USER, $PASWRD to the actual values in the check.txt.
I am trying like this but it fails.
emsurl=tcp://myserver:3243
emsuser=test
emspasswd=new
sed s/$URL/${emsurl}/g check.txt >> check_new.txt
sed s/$USER/${emsuser}/g check.txt_new.txt >> check_new_1.txt
sed s/PASWRD/${emspasswd}/g check_new_1.txt >> final.txt
My final.txt output is desired as below:
~/bin/tibemsadmin -server tcp://myserver:3243 -user test -password new
Could you please help me?
You have to be rather careful with your use of quotes. You also need to learn how to do multiple operations in a single pass, and/or how to use pipes.
emsurl=tcp://myserver:3243
emsuser=test
emspasswd=new
sed -e "s%\$URL%${emsurl}%g" \
-e "s%\$USER%${emsuser}%g" \
-e "s%\$PASWRD%${emspasswd}%g" check.txt >final.txt
Your problem is that the shell expanded the '$URL' in your command line (probably to nothing), meaning that sed got to see something other than what you intended. By escaping the $ with the \, sed gets to see what you intended.
Note that I initially used / as the separator in the substitute operations; however, as DarkDust rightly points out, that won't work since there are slashes in the URLs. My normal fallback character is % - as now shown - but that can appear in some URLs and might not be appropriate. I'd probably use a control character, such as control-A, if I needed to worry about that - or I'd use Perl which would be able to play without getting confused.
You can also combine the three separate -e expressions into one with semi-colons replacing them. However, I prefer the clarity of the three operations clearly separated.
You could take a slightly different approach by modifying your main script as follows :-
export URL="tcp://myserver:3243"
export USER=test
export PASWRD=new
. ./check.txt
This sets up the variables and then runs check.txt within the context of your main script
Although you don't say what's failing I guess I see the problems.
I suggest you do this:
sed "s|\$URL|${emsurl}|g"
That is, the first $ needs to be escaped because you want it literally. Then, instead of / I suggest you use | (pipe) as delimiter since it's not used in your strings. Finally, use " to ensure the content is interpreted as string by the shell.
You can then pipe everything together to not need any temporary files:
sed "s|\$URL|${emsurl}|g" | sed "s|\$USER|${emsuser}|g" | sed "s|\$PASSWRD|${emspasswd}|g"
Variable substitution should be outside sed expression and '$' should be escaped; in your case something like this:
sed -e 's/\$URL/'$emsurl'/g' -e 's/\$USER/'$emsuser'/g' -e 's/\$PASSWORD/'$emaspasswd'/g'
Anyway in your place I would avoid using $ to match placeholders in a template file, because it's causing confusion with BASH variables, use a different pattern instead (for instance #URL#).