I'm writing a tool that receives an argument that is a file pattern as follows:
mytool arg1 *
Here is the problem: The character * is being resolved to the contents of the current directory, almost like the ls command.
If I escape the wildcard using grave (`) it is also received by the command.
How can I solve that problem?
Use a verbatim (single-quoted) string ('...')[1] to prevent the globbing (pathname extension) that PowerShell performs on Unix-like platforms (only) when calling external programs:
mytool arg1 '*'
I have no explanation for why character-individual escaping (`*) doesn't work (as of PowerShell 7.2.6) - arguably, it should; the problem has been reported in GitHub issue #18038.
On Windows, PowerShell performs no globbing, and *, '*', and `* are all passed as verbatim * to an external program.
As you state, given that you're on Windows, it must be the Python-based tool you're calling that performs globbing.
[1] An expandable (double-quoted) string ("...") string works too, but there's no need for one unless you truly need expansion (string interpolation).
Related
I'm aware that, in PowerShell, backticks(`) are used to escape characters instead of the backslash(\). Hence, when using sed, this:
echo '"where am I"' | sed "s/\"[^\"]*\"/\033[96m&\033[0m/g;"
will not work and I would have to write it like this:
echo '"where am I"' | sed "s/`"[^\`"]*`"/`e[96m&`e[0m/g;"
Hence, I would like to do the same with this code:
echo '[where am I]' | sed "s/\[\([^09]\)/\033[94m[\033[0m\1/g;"
The expected output should be:
However, when I tried to replace all the backslashes(\) with backticks(`), I can't seem to get the expected output as you can see below:
Any help would be appreciated. Thanks in advance.
As far as I'm aware, GNU sed doesn't support octal escape sequences - such as \033 to represent an ESC char. - only hexadecimal ones - such as \x1b.
The following command, which uses hex. escape sequences, should therefore work in both POSIX-compatible shells and in PowerShell:
# Note: Requires *GNU* sed
(echo '[where am I]' | sed 's/\[\([^09]\)/\x1b[94m[\x1b[0m\1/g')
In other words:
Because the entire, verbatim string ('...') is interpreted by sed itself, there is no need for PowerShell's string interpolation (via "...").
Note:
The enclosing (...), which on Windows is required for Windows PowerShell to properly render the ANSI escape codes / VT (Virtual Terminal) sequences in the terminal (console) - see this answer. It isn't needed if you capture the output in a variable or redirect it.
In PowerShell, use of echo - a built in alias of the Write-Output cmdlet - isn't necessary and can be omitted, due to PowerShell's implicit output behavior.
If your sed implementation doesn't support escape sequences such as \x1b - such as on macOS - you must indeed resort to using PowerShell's escape sequences inside an expandable (double-quoted) string ("..."); that is, you must replace \x1b with escape sequence `e in order to embed a verbatim ESC character into the string, up front (in Windows PowerShell, which doesn't support `e, use subexpression $([char] 0x1b) instead):
# Note: `e only works in PowerShell (Core) 7+
# In Windows PowerShell, use $([char] 0x1b)
(echo '[where am I]' | sed "s/\[\([^09]\)/`e[94m[`e[0m\1/g")
Important: The above commands contain no embedded " characters, which avoids a long-standing PowerShell bug that is still present as of PowerShell (Core) 7.2.2:
The need to unconditionally, manually \-escape embedded " chars. in arguments passed to external programs - see this answer for details.
A few quick examples that print verbatim 3" of snow:
# Up to at least PowerShell 7.2.2:
# Without escaping the embedded " with \ (too),
# they would in effect be removed from the arguments.
# Note that pipeline input is *not* affected.
'3" of snow' | findstr '\"' # Windows
/bin/echo '3\" of snow' # Unix
# In expandable (double-quoted) strings, you therefore must escape *twice*:
# once for PowerShell, then for the external-program call.
'3" of snow' | findstr "\`"" # Windows
/bin/echo "3\`" of snow" # Unix
You can pass non-interpolated expression strings to sed by using single quotes, '; e.g., 'a non-interpolated string'. instead of double quotes around the text. sed would take what it normally does for input. When you use double quotes, that's when powershell escaping comes into play.
Anyway, an alternative to sed would be leveraging the regular expression substitution built into powershell. For instance,
Write-Output ('[where am i]' -replace '\[(.*?)\]', "<<`e[5;36m`$1`e[0m>>")
Will capture the text between square brackets and place it between "chevrons" and make it blink.
Here's an article on inserting ANSI in text using powershell.
I know that both are used in PowerShell but for different contexts.
On the internet there is a paucity of information on this topic and the only site that talks about it (without making me understand the concept) is:
https://www.rlmueller.net/PowerShellEscape.htm
I am a beginner with PowerShell, I am recently approaching it.
A use case of the \ escape came up in the answer to this other topic of mine:
PowerShell removes multiple consecutive whitespaces when I pass arguments to a nested Start-Process command
Is there anyone who can explain to me in detail the difference between the escape backtick ` and backslash \ in PowerShell, with examples and use cases?
At least one source is welcome, but it is not mandatory.
vonPryz's helpful answer covers the PowerShell-internal angle well; let me attempt a systematic summary that includes the PowerShell CLI angle, as well as passing arguments with embedded " to external programs:
Inside a PowerShell session, the only escape character is ` (the so-called backtick), in the following contexts:
Inside an expandable string ("...", double quoted), but not inside a verbatim string ('...', single-quoted); for the supported escape sequences, see the conceptual about_Special_Characters help topic:
# " must be escaped; escape sequence `n expands to a newline.
"3`" of`nrain"
In unquoted command arguments:
# > must be escaped to write verbatim 'a>b',
# since outside a quoted string an unescaped > is a redirection.
Write-Output a`>b
For line-continuations:
# Spread the Get-Date call across two lines.
# Important: ` must be the *very last char* on the line.
Get-Date `
-Year 2525
Note: Various subsystems, whether PowerShell-specific or not, may have their own escaping rules, such as \ in regular expressions and ` in wildcard expressions. Since arguments to those subsystems are delivered via PowerShell strings, it's best to use verbatim string literals, so as to avoid confusion between PowerShell's own string interpolation and what the target subsystem ends up seeing; e.g. 'A $50 fine.' -match '\$50' (\ is needed to treat regex metacharacter $ literally).
When PowerShell is called from the outside, via its CLI, different rules apply, possibly in addition:
In order to adhere to widely used convention for CLIs (command-line interfaces, programs that accept arguments via a command line) on Windows:
In calls to powershell.exe, the Windows PowerShell CLI, " characters must be escaped with a backslash - i.e. as \" - in order to be preserved during parsing of the original command line.
pwsh.exe, the CLI of the cross-platform, install-on-demand PowerShell (Core) 7+ edition, now commendably alternatively accepts ""[1] in lieu of \", which makes calls from cmd.exe more robust. To get the same robustness in Windows PowerShell - from cmd.exe only - use "^"" (sic).
Note that - unlike \" - these escape sequences only work inside an (unescaped) "..." string (e.g., pwsh.exe -c " ""you & I"" " or powershell.exe -c " "^""you & I"^"" "
By contrast, unescaped " have syntactic function on the command line and tell PowerShell where the boundaries between arguments are; these " instances are removed during command-line parsing.
This ensures that outside callers that merely want to invoke a PowerShell script file (.ps1) with arguments, using the -File parameter, can use the conventional syntax and needn't special-case calls to PowerShell's CLI.
However, if you pass a string containing PowerShell code to the CLI, using the -Command parameter, what PowerShell ends up interpreting obviously must be syntactically valid PowerShell code.
Caveat: If you specify neither -Command nor -File:
powershell.exe defaults to -Command
pwsh.exe now defaults to -File
For the differences between -File and -Command calls and when to use which, see this answer.
If you use -Command, there are two, sequential parsing stages:
The command-line parsing stage, where syntactic (unescaped) " are removed, and escaped \" (or "") turn into literal ".
The result of this stage is then parsed as PowerShell code, as it would be from inside a PowerShell session.
Therefore, you may situationally have to combine \ and `-escaping; e.g. (call from cmd.exe):
C:>powershell.exe -Command " \"3`\" of snow as of $(Get-Date)\" "
3" of snow as of 11/04/2021 14:13:41
Note the use of `\" in order to make PowerShell see `", i.e. a properly escaped " inside a "..." string, after command-line parsing.
Alternatively, depending on the specifics of the command(s) you pass to -Command, using embedded '...' quoting may be an option, which simplifies matters, because ' chars. don't require escaping:
C:>powershell.exe -Command " '3\" of snow as of today.' "
3" of snow as of today.
Given that '...' strings in PowerShell are verbatim strings, use of '...' is only an option if you don't require string interpolation (such as the $(Get-Date) subexpression in the prior example).
Escaping " when calling external programs from PowerShell:
As a shell, it is PowerShell's job to pass the arguments that were passed based on its syntax rules to a target executable, so that the verbatim values that result from PowerShell's parsing are passed in a way that makes the target executable see them as such. In other words: PowerShell should perform any required escaping automatically, behind the scenes. (Unlike cmd.exe, PowerShell cannot just pass its own argument syntax through as-is, because external CLIs cannot be expected to understand '...' strings (single-quoting) or `-escaping).
To use a simply example: Passing '3" of snow' should be passed as "3\" of snow" behind the scenes, based on the most widely used escaping convention.
Sadly, up to PowerShell 7.2.x, this is not the case, and embedded " characters in arguments for external programs must additionally, manually be \-escaped in order to be passed through properly.
This has been fixed in PowerShell 7.3, with selective exceptions on Windows - see the new $PSNativeCommandArgumentPassing preference variable
# Broken behavior up to PS v7.2.x
PS> cmd /c echo '{ "foo": "bar" }'
"{ "foo": "bar" }" # !! Embedded " aren't escaped.
PS> choice.exe /d Y /t 0 /m '{ "foo": "bar" }'
{ foo: bar } [Y,N]?Y # !! When parsed by an executable,
# !! embedded " are effectively LOST.
# Manual escaping required.
PS> choice.exe /d Y /t 0 /m '{ \"foo\": \"bar\" }'
{ "foo": "bar" } [Y,N]?Y # OK
This bug has existed since v1, and has never been fixed so as to avoid breaking existing workarounds.
[1] Inside PowerShell, in "..." strings only, you may also use "" to escape an embedded ", as an alternative to `"
The backtick ` is used as the escape character in Powershell. That is, escaping quotes, tabs and whatnot. Unlike in many other environments, Powershell's newline is `n, not \n. This has the benefit of simplifying paths, since those use backslash in Windows.
As for a practical example, in many programming languages one needs to escape the backslashes in paths. If you have an application located at
c:\program files\newApplication\myApp.exe
Its path must be written as
c:\\program files\\newApplication\\myApp.exe
The double-backslash notation means that there is actaully a backslash, not a metacharacter such as tab or newline. Note that the path contains \newApplication. Now, \n usually means a newline, but it clearly isn't the case. The file does not reside in
c:\program files\
ewApplication\myApp.exe
after all, doesn't it?
The escaping is not needed in Powershell, as the backslash doesn't have special meaning by itself. When Powershell sees c:\program files\newApplication\myApp.exe, it doesn't assign any special meaning to the forementioned \n part, it is just a string literal.
The backslash \ is used as the escape character in regular expressions and to signify metacharacters. That is, to match a literal * in regex, it must be escaped, lest it means the Kleene star (zero or more matches).
Let's say I have the variable $password = Get-Passwd ACME\bob where I get the password of a given user. The password contains all sorts of special characters including $, that is used by PowerShell.
I need to dynamically use the password in the following command:
cmdkey.exe /add:$hostname /user:"$user" /pass:"`"$password`""
I need to escape the " character on both sides so the command is parsed as pass:"123qwe" and not pass:123qwe. The issue is that it breaks when the password includes a $ character. How can I pass the password to this command without breaking it?
Running Executables in PowerShell
For most executables, it's not necessary to manually quote the parameters in PowerShell. Just write the command using any variables, and the PowerShell parser will automatically quote the arguments for the program's command line. This usually "just works."
cmdkey.exe /add:hostname /user:username /pass:$pass
You can inspect the actual command line that PowerShell passes to an executable by using a small program I wrote called showargs.exe, available at the following page:
https://www.itprotoday.com/management-mobility/running-executables-powershell
Example:
showargs cmdkey.exe /add:hostname /user:username /pass:$pass
The showargs.exe program simply echoes its command line to standard output so you can see the literal command line that PowerShell actually passes to the executable.
An embedded $ character shouldn't be a problem. If that character is in a PowerShell string variable, PowerShell will pass it along as a part of the command line string. Example:
$pass = 'my$pass'
cmdkey.exe /add:hostname /user:username /password:$pass
No matter how you pass the command line to the program, it's important to note that the interpretation of that command line is up to that individual program. (For example, if an executable's parser doesn't support certain characters, no amount of quoting or parsing will allow you to work around the limitation.)
Cmdkey.exe Uses a Non-Standard Parser
In my prefunctory testing of of cmdkey.exe, it seems it does not have a way of "escaping" the " character on its command line. Since this seems to be the case, you will not be able to use cmdkey.exe to store a credential that contains an embedded " character.
Embedding a Space in a Cmdkey.exe Command Line Argument
Because cmdkey.exe uses a non-standard command-line parser, you can't use a variable on its command line that contains embedded spaces. For example:
PS C:\> $pass = "my pass"
PS C:\> showargs cmdkey.exe /add:hostname /user:username /password:$pass
cmdkey.exe /add:hostname /user:username "/password:my pass"
The "/password:my pass" evidently confuses the cmdkey.exe parser, so we have to work around this behavior by bypassing PowerShell's default parsing behavior. The simplest way to do this is by escaping the quotes around the argument containing the space:
PS C:\> showargs.exe cmdkey.exe /add:hostname /user:username /password:`"$pass`"
cmdkey.exe /add:hostname /user:username /password:"my pass"
In any case, you can use showargs.exe to diagnose the trouble and work out a solution appropriate to the executable you need to run.
tl;dr
Your command should work - except if $password contains " chars.
Embedded $ chars., by contrast, should not be a problem.
Your /pass:"`"$password`"" technique - i.e., explicit, embedded double-quoting - also handles values with embedded spaces correctly, unlike the /pass:$password technique (also) suggested in Bill Stewart's helpful answer.
You can simplify the command by omitting the outer "..." quoting, as also suggested in Bill's answer:
/pass:`"$password`"
Caveat: If PowerShell's argument-passing worked correctly, these techniques shouldn't work and if it ever gets fixed, such techniques will stop working - see this answer for background.
As for supporting " chars. in values: even \-escaping them doesn't always work, namely when spaces are also involved - see details below.
The issue is that it breaks when the password includes a $ character.
Your command passes any $ characters embedded in the value of $password through to the target program as-is.
Therefore, if there is a problem, the implication is that your target program - cmdkey.exe - interprets $ characters, but note that the docs make no mention of that.
If it indeed does, however, you would have to escape $ characters as required by the target program in order to pass them literally.
Caveat: There is a fundamental limitation, however:
Commands typically break if the argument value contains spaces AND embedded " chars, whether the latter are properly escaped for the target program or not.
Normally, you'd escape value-internal " as \" so as not to break the enclosing double-quoting of the value:
# !! Only works if:
# !! * $password contains NO "
# !! * $password contains " but NOT ALSO SPACES
# !! * $password contains PAIRS of " and what is between each
# !! pair does not contain spaces
cmdkey.exe /add:$hostname /user:"$user" /pass:`"$($password -replace '"', '\"')`"
Note:
Escaping embedded " as \" is a not a standard per se, but it is recognized by most external programs; the only notable exceptions are batch files - see this answer for details.
Arguably, PowerShell should handle this escaping automatically - see this GitHub issue for details.
If Windows PowerShell thinks it cannot pass the resulting token as-is as a single argument to the target program, it blindly applies double-quoting around the entire token[1], which, in combination with the escaped ", can result in invalid syntax:
E.g., if $password literally contains a " b and is manually escaped as a \" b by the command above, PowerShell ends up passing the following behind the scenes:
... "/pass:"a \" b""
That is, the resulting literal token that PowerShell saw - /pass:"a \" b" - was blindly enclosed in double quotes, as a whole, even though most target programs would parse /pass:"a \" b" correctly as-is.
As a result, the explicitly provided double-quoting is invalidated (as it would then require another level of escaping) - and short of using --%, the stop-parsing symbol, which then limits you to literals (or %...%-style environment-variable references), there is no way around that.
If the target program recognizes an argument if it is double-quoted as a whole (e.g.,
"/pass:a b" instead of /pass:"a b"), you can omit the explicit double-quoting around the argument value.
Do note, however, that some target programs - including cmdkey.exe and notably msiexec - do not recognize the argument if it is double-quoted as a whole.
... /pass:$($password -replace '"', '\"')
With $password literally containing a " b, PowerShell then passes behind the scenes:
... "/pass:a \" b"
This is syntactically valid - for target programs that recognize \" as an escaped ", which is the norm - but, as stated, the fact that the entire argument is enclosed in `"...", and not just the value may not be supported by the target program.
[1] Windows PowerShell ignores any \-escaping in the resulting token and considers all embedded " to have syntactic function: From that perspective, if the token is not composed of any mix of directly concatenated unquoted and double-quoted strings, enclosing double-quoting is blindly applied - which may break the command.
This behavior is not only obscure, it prevents robust, predictable parameter passing.
PowerShell Core now recognizes \" as as escaped; however, quotes that aren't pre-escaped as \" still result in broken quoting; e.g., 'a "b c" d' is passed as "a "b c" d", which target programs parse as 2 arguments, a b and c d (after quote removal).
I know there are lots of posts regarding this, but nothing worked for me.
I am trying to run this command line in PowerShell:
C:/Program Files (x86)/ClamWin/bin/clamd.exe --install
I have this in PowerShell:
&"C:/Program Files (x86)/ClamWin/bin/clamd.exe --install"
But all this does is execute clamd.exe, ignoring the --install parameter
How can I get the full command line to run?
Josef Z's comment on the question provides the solution:
& "C:/Program Files (x86)/ClamWin/bin/clamd.exe" --install # double-quoted exe path
or, given that the executable path is a literal (contains no variable references or subexpressions), using a verbatim (single-quoted) string ('...'):
& 'C:/Program Files (x86)/ClamWin/bin/clamd.exe' --install # single-quoted exe path
As for why your own solution attempt failed: The call operator, &, expects only a command name/path as an argument, not a full command line.
Invoke-Expression accepts an entire command line, but that complicates things further and can be a security risk.
As for why this is the solution:
The need for quoting stands to reason: you need to tell PowerShell that C:/Program Files (x86)/ClamWin/bin/clamd.exe is a single token (path), despite containing embedded spaces.
&, the so-called call operator, is needed, because PowerShell has two fundamental parsing modes:
argument mode, which works like a traditional shell, where the first token is a command name, with subsequent tokens representing the arguments, which only require quoting if they contain shell metacharacters (chars. with special meaning to PowerShell, such as spaces to separate tokens);
that is why --install need not, but can be quoted (PowerShell will simply remove the quotes for you before passing the argument to the target executable.)
expression mode, which works like expressions in programming languages.
PowerShell decides based on a statement's first token what parsing mode to apply:
If the first token is a quoted string - which we need here due to the embedded spaces in the executable path - or a variable reference (e.g., $var ...), PowerShell parses in expression mode by default.
A quoted string or a variable reference as an expression would simply output the string / variable value.
However, given that we want to execute the executable whose path is stored in a quoted string, we need to force argument mode, which is what the & operator ensures.
Generally, it's important to understand that PowerShell performs nontrivial pre-processing of the command line before the target executable is invoked, so what the command line looks like in PowerShell code is generally not directly what the target executable sees.
If you reference a PowerShell variable on the command line and that variable contains embedded spaces, PowerShell will implicitly enclose the variable's value in double quotes before passing it on - this is discussed in this answer to the linked question.
PowerShell's metacharacters differ from that of cmd.exe and are more numerous (notably, , has special meaning in PowerShell (array constructor), but not cmd.exe - see this answer).
To simplify reuse of existing, cmd.exe-based command lines, PowerShell v3 introduced the special stop-parsing symbol, --%, which turns off PowerShell's normal parsing of the remainder of the command line and only interpolates cmd.exe-style environment-variable references (e.g., %USERNAME%).
If I execute the following command on a Windows 8.1 machine:
robocopy "C:\Temp\A\" "C:\Temp\B\"
Robocopy fails due to the following problem:
Source : C:\Temp\A" C:\Temp\B"\
Dest -
...
ERROR : No Destination Directory Specified.
It looks like \ is used as some kind of escape character (which is not normal behavior in the windows command line) The final \" is even transformed to "\ which I do not understand at all. Why's that so?
Note: this is not the default behavior of the command line, if they would have used argv[1] and argv[2] within robocopy, they would've retrieved the correct arguments.
Why are they using their own command line parsing? It really confused me for the last hour...
You should omit the trailing backslashes.
From http://ss64.com/nt/robocopy.html :
If either the source or destination are a "quoted long foldername" do
not include a trailing backslash as this will be treated as an escape
character, i.e. "C:\some path\" will fail but "C:\some path\\" or
"C:\some path." or "C:\some path" will work.
robocopy is not an exception. Any executable uses its own line parser to determine the arguments that were sent to it. The OS just uses the API to create the process and pass to it a string to be handled as arguments. The process can handle the string as it wants.
In the case of robocopy, the parser used is the standard Microsoft C startup code. This parser follow the rules described here, and in the full list you can found
A double quotation mark preceded by a backslash, \", is interpreted as
a literal double quotation mark (").