Here's my args.bat file:
#echo off
python -c "import sys; print(sys.argv)" %*
I'm invoking python here because I know exactly how to interpret output, rather than guessing which quotes are in the string
Here's what happens if I invoke it from cmd:
>.\args.bat ^^^^^
More?
More?
['-c', '^']
>.\args.bat "^^^^^"
['-c', '^^^^^']
But here's what powershell does:
PS> .\args.bat ^^^^^
['-c', '^']
PS> .\args.bat "^^^^^"
['-c', '^']
PS> .\args.bat '^^^^^'
['-c', '^']
PS> .\args.bat "'^^^^^'"
['-c', '^']
PS> .\args.bat '"^^^^^"'
['-c', '^^^^^']
Why do I have to doubly-quote the argument when calling a batch file from powershell?
The same problem doesn't exist calling python from powershell:
PS> python -c "import sys; print(sys.argv)" "^^^^^"
['-c', '^^^^^']
PS> python -c "import sys; print(sys.argv)" '^^^^^'
['-c', '^^^^^']
PS> python -c "import sys; print(sys.argv)" ^^^^^
['-c', '^^^^^']
But persists when calling batch from python from powershell!
PS> python -c "import subprocess; subprocess.call(['args.bat', '^^^^^'])"
['-c', '^']
And batch from python from cmd:
> python -c "import subprocess; subprocess.call(['args.bat', '^^^^^'])"
['-c', '^']
For the larger question of whether batch files can robustly relay arbitrary arguments (spoiler alert: no), see Eric's follow-up question.
tl;dr:
PS> .\args.bat --% "^^^^^"
['-c', '^^^^^']
Using --%, the stop-parsing symbol, makes PowerShell pass the remaining arguments through as-is, except for expanding cmd.exe-style environment variable references such as %OS%.
Conversely, you won't able to reference PowerShell variables or expressions.
PowerShell, as a shell in its own right, performs its own parsing first and then - an unfortunate necessity on Windows - reapplies quoting when composing a command line for an external program.
The way it does that has changed over time and has been the source of many problems in the past.
These problems led to the introduction of --%, as described above, which has its own challenges, however.
The re-quoting is generally a necessity, however, given that PowerShell's syntax isn't necessarily understood by external programs.
Most notably, '...' quoting cannot be assumed to be understood by all external programs and therefore requires transforming to "...".
However, PowerShell re-quotes conditionally, based on whether it thinks it is necessary.
In the case at hand, because the content of the PowerShell "^^^^^" string contains no whitespace, PowerShell decides not to quote it when passing it through to the batch file; i.e.:
.\args.bat "^^^^^"
effectively becomes
.\args.bat ^^^^^
This is fine for most external programs, where ^ has no syntactic meaning, but in the case of a batch file (interpreted by cmd.exe) it matters, because ^ function as the escape char. in unquoted strings there.
To still force the use double-quoting in this scenario, as an alternative to the --% approach above, you can use embedded double-quoting, as you've already demonstrated in your question:
PS> .\args.bat '"^^^^^"' # '...' quoting with embedded "..." quoting
['-c', '^^^^^']
The same problem doesn't exist calling python from powershell:
PS> python -c "import sys; print(sys.argv)" "^^^^^"
That's because cmd.exe isn't involved in this scenario, and even though the enclosing double quotes are lost when calling from PowerShell, Python treats the ^ instances are as literals.
But persists when calling batch from python from powershell
PS> python -c "import subprocess; subprocess.call(['args.bat', '^^^^^'])"
And batch from python from cmd:
> python -c "import subprocess; subprocess.call(['args.bat', '^^^^^'])"
That's because you're again calling a batch file - with unquoted ^^^^^ in this case even from cmd.exe, because the enclosing '....' only have syntactical meaning to Python and aren't passed through - which reintroduces the special interpretation of ^.
Note that even "shell-less" methods of invoking a batch file (such as subprocess.call() from Python) still invariably invoke cmd.exe, because cmd.exe is the interpreter required to execute batch files.
When passing an unquoted string ^^^^^ to a batch file, there seem to be 2 rounds of interpretation: first, as part of argument parsing, each ^^ pair becomes a single ^, with any non-paired ^ getting discarded. Inside the batch file, you therefore see ^^, which, when used unquoted, becomes a single ^ - if you double-quote %* or %1 inside the batch file you'll see that ^^ was passed.
Related
Windows 11 & Powershell 7.2.5: I'm trying to pass a regular expression in a parameter to my node.js app. The expression, ^#Test, contains a ^ character, which is apparently also used to denote escape sequences in Powershell. Let's say this is my app:
const param = process.argv[2];
console.log(param);
When I run it this way:
node index.js "^#Test"
it works correctly and prints ^#Test
But if I configure a script in package.json like this:
"scripts": {
"start": "node index.js"
}
And I run:
npm run start "^#Test"
the leading ^ gets trimmed and it prints only #Test
Curiously, it does not get trimmed if ^ character is surrounded by a space on either side. So the following work correctly:
npm run start "^ #Test" // prints "^ #Test", correct
npm run start " ^#Test" // prints " ^#Test", correct
npm run start " ^ #Test" // prints " ^ #Test", correct
I tried escaping with ^^, `^ or ^, but neither works:
npm run start "^^#Test" // prints "#Test"
npm run start "`^#Test" // prints "#Test"
npm run start "\^#Test" // prints "\\#Test"
This also does not make a difference:
npm run start -- "^#Test" // prints "#Test"
With different quotes and without quotes it also does not work:
npm run start ^#Test // prints "#Test"
npm run start '^#Test' // prints "#Test"
Is this a bug or am I escaping it wrong?
tl;dr
Note: The following workarounds are required only if your argument does not contain spaces.
For literal arguments, use --%, the stop-parsing token:
npm run start --% "^#Test"
For variable-based arguments, use embedded quoting:
$var = 'Test'
npm run start "`"^#$var`""
You're seeing the confluence of two surprising behaviors, one by cmd.exe and the other by PowerShell:
On Windows, npm's entry point is a batch file, npm.cmd, and is therefore subject to cmd.exe's parsing rules.
When a batch file is called from outside cmd.exe, cmd.exe still - inappropriately - parses the command line as if it had been submitted from inside a cmd.exe session. (Unfortunately, it has always worked this way and is unlikely to change).
Therefore, any ^ characters in an unquoted argument are interpreted as cmd.exe escape character; if ^ precedes a character with no special meaning (to cmd.exe), it is simply removed (e.g., ^# turns into just #).
PowerShell, which has its own quoting syntax and escape character, of necessity needs to rebuild command lines behind the scenes, so as to use only "..."-based quoting (double-quoting), as that is the only form of quoting CLIs are generally expected to support.
In this rebuilding process, double-quoting is employed on demand, namely based on whether a given argument contains spaces.
Thus, despite you having specified "^#test" on the original command line, as parsed by PowerShell, on the rebuilt command line it is unquoted ^#test that is passed, which causes npm, due to being a batch file, to effectively drop the ^, as explained above.
Note:
PowerShell's behavior is defensible, as CLIs shouldn't parse their command lines as if they were shell command lines, which is what cmd.exe unfortunately does when batch files are called. Specifically, ^ should not be special when a batch file is called from outside a cmd.exe sessions.
GitHub issue #15143 proposes that PowerShell implement accommodations for cmd.exe and other high-profile CLIs on Windows, so as to minimize such edge cases - unfortunately, it looks like that won't happen.
As I have learned, when invoking PowerShell from cmd.exe, with the -Command option, escaping rules are essentially those used in Linux. So you use a backslash (\), rather than the usual backtick (`).
This is to say that you don't write:
C:\> powershell -c "echo `"That's a single quote: ' `""
but
C:\> powershell -c "echo \"That's a single quote: ' \""
to get:
That's a single quote: '
And this is the exact equivalent of what you would type in a Linux terminal:
~ >>> bash -c "echo \"That's a single quote: ' \""
If I am not wrong, this feature is named PSNativeCommandArgumentPassing.
But comparisons work up to a certain point. In fact, under Linux you write:
~ >>> bash -c "echo \"That's a double quote: \\\" \""
to get:
That's a double quote: "
while the equivalent PowerShell line:
C:\> powershell -c "echo \"That's a double quote: \\\" \""
gives
The string is missing the terminator: ".
By trial and error, I realised that:
C:\> powershell -c "echo \"That's a double-quote: `"" \""
works as expected.
Can you explain to me what is the logic behind: `""?
Also, what are the equivalent commands when calling PowerShell from powershell.exe, rather than cmd.exe?
tl;dr
On PowerShell's command line only, use \" to pass a " through to the code that -c (-Command) should execute.
Only in the resulting code does PowerShell's usual escaping of " as `" apply, so you may have to combine the two escaping techniques: `\".
Your attempt, `"", had the same effect, but it doesn't work reliably.[1]
\" works robustly with respect to PowerShell's own parsing of its command line, but can situationally break calls on the cmd.exe side - see the bottom section for workarounds.
When calling CLIs (external console applications) from PowerShell, not only do PowerShell's own, different quoting rules apply first (support for '...' strings, embedded " inside "..." escaped as `"), a long-standing bug up to PowerShell 7.2.x additionally requires escaping " chars. with \ when embedded in external-program arguments (only); see this answer.
this feature is named PSNativeCommandArgumentPassing
No; this feature - which became official in PowerShell 7.3 (see this answer) - does not come into play, because:
it generally isn't (and won't be) available in the legacy Windows PowerShell edition that you're invoking via its CLI, powershell.exe (whereas the modern, cross-platform PowerShell (Core) edition's CLI is pwsh.exe)
it only applies to calls from inside a PowerShell session.
it is designed to address a long-standing problem when calling external programs with embedded " or empty-string string arguments from PowerShell - see this answer.
Therefore, the linked answer addresses your last question:
Also, what are the equivalent commands when calling PowerShell from powershell.exe, rather than cmd.exe?
In short: unfortunately, up to at least PowerShell 7.2.x you'll have to additionally, manually escape " chars. with " embedded in external-program arguments (only).
Why the following works from cmd.exe:
powershell -c "echo \"That's a single quote: ' \""
PowerShell only recognizes \ as an escape character on its command line, for consistency with other CLIs.
Inside a PowerShell session, only ` (backtick) serves as the escape character.
Caveat: While \" to escape " works consistently in both PowerShell editions on the PowerShell (command-line parsing) side, situationally it can break cmd.exe's own parsing - see the bottom section.
When the PowerShell CLI is invoke via -c (-Command) with a piece of PowerShell source code to execute, that code (the argument(s) following -c is parsed in two stages:
First, all unescaped " chars. are stripped, wheres escaped ones (\") are kept and unescaped.
Only then is the result parsed and execute as PowerShell code.
Therefore, what PowerShell ends up executing is verbatim:
echo "That's a single quote: ' "
From the above follows why this does not work:
:: SYNTAX ERROR
powershell -c "echo \"That's a double quote: \\\" \""
PowerShell ends up trying to execute verbatim
echo "That's a double quote: \" "
which is a syntax error, because inside a PowerShell session \ doesn't escape " -only `" or - inside "...", alternatively - "" do.
From the above follows why this (mostly) works:
:: Works, but not robustly
powershell -nop -c "echo \"That's a double-quote: `"" \""
"" sometimes, but not always works as an alternative to \";[1] here it does - see also the bottom section.
As a result, PowerShell ends up executing the following verbatim, which works, because the escaped " that was passed through is now `-escaped, as it needs to be inside a PowerShell "..." string:
echo "That's a double-quote: `" "
To avoid the brittle "" escaping, it is better to formulate this command by combining the required `-escaping with the command-line \-escaping - i.e. `\" - but see the bottom section for a fully robust solution:
powershell -nop -c "echo \"That's a double-quote: `\" \""
Avoiding parsing problems on the cmd.exe side, a safe alternative to \":
While \" to escape " works consistently in both PowerShell editions on the PowerShell (command-line parsing) side, situationally it can break cmd.exe's own parsing. By contrast, using \" is safe when calling from no-shell environments such as Task Scheduler.
While there are workarounds, they are, unfortunately PowerShell edition-specific:
In Windows PowerShell (powershell.exe)
Use "^"" (sic) instead of \"
In PowerShell (Core) (v6+, pwsh.exe)
Use "" instead of \"
Important:
These workarounds require that the whole code to pass to -c (-Command) be passed as a single, "..."-enclosed argument.
-c (-Command) also accepts multiple arguments - which may individually be double-quoted or not - in which case it simply concatenates these arguments, after having stripped unescaped ", to form the code to execute. This technique situationally allows you to get away with \"-escaping (e.g., powershell -c \"Between 2 & 3`\"\", but (a) it requires you to pay close attention to which parts of the command cmd.exe will see as unquoted, (b) would require you to ^-escape any cmd.exe metacharacters such as & in those parts, and (c) invariably performs whitespace normalization, i.e. folds runs of multiple spaces into a single one each.
The following calls, designed to print verbatim Between 2 & 3" , demonstrate this:
:: BREAKS, because cmd.exe sees `&` as *outside a double-quoted string*
powershell -c " \" Between 2 & 3`\" \" "
:: PowerShell (Core) workaround
pwsh -c " "" Between 2 & 3`"" "" "
:: Windows PowerShell workaround
powershell -c " "^"" Between 2 & 3`"^"" "^"" "
[1] An example of where "" inside "..." doesn't work is powershell -c "echo \" Nat `""King`"" Cole \"": instead of Nat "King" Cole, it prints Nat "King Cole, i.e. the second escaped " is missing (it would work fine in pwsh.exe, however, as discussed in the bottom section). It's ultimately not worth speculating how, precisely, embedded "" sequences are parsed by powershell.exe -c, given that it demonstrably isn't reliable and that reliable alternatives do exist (\" or, from cmd.exe, also "^"").
I am trying to run a Bash command where an alias exists in PowerShell Core.
I want to clear the bash history. Example code below:
# Launch PowerShell core on Linux
pwsh
# Attempt 1
history -c
Get-History: Missing an argument for parameter 'Count'. Specify a parameter of type 'System.Int32' and try again.
# Attempt 2
bash history -c
/usr/bin/bash: history: No such file or directory
# Attempt 3
& "history -c"
&: The term 'history -c' is not recognized as the name of a cmdlet, function, script file, or operable program.
It seems the issue is related to history being an alias for Get-History - is there a way to run Bash commands in PowerShell core with an alias?
history is a Bash builtin, i.e. an internal command that can only be invoked from inside a Bash session; thus, by definition you cannot invoke it directly from PowerShell.
In PowerShell history is an alias of PowerShell's own Get-History cmdlet, where -c references the -Count parameter, which requires an argument (the number of history entries to retrieve).
Unfortunately, Clear-History is not enough to clear PowerShell's session history as of PowerShell 7.2, because it only clear's one history (PowerShell's own), not also the one provided by the PSReadLine module used for command-line editing by default - see this answer.
Your attempt to call bash explicitly with your command - bash history -c - is syntactically flawed (see bottom section).
However, even fixing the syntax problem - bash -c 'history -c' - does not clear Bash's history - it seemingly has no effect (and adding the -i option doesn't help) - I don't know why.
The workaround is to remove the file that underlies Bash's (persisted) command history directly:
if (Test-Path $HOME\.bash_history) { Remove-Item -Force $HOME\.bash_history }
To answer the general question implied by the post's title:
To pass a command with arguments to bash for execution, pass it to bash -c, as a single string; e.g.:
bash -c 'date +%s'
Without -c, the first argument would be interpreted as the name or path of a script file.
Note that any additional arguments following the first -c argument would become the arguments to the first argument; that is, the first argument acts as a mini-script that can receive arguments the way scripts usually do, via $1, ...:
# Note: the second argument, "-", becomes $0 in Bash terms,
# i.e. the name of the script
PS> bash -c 'echo $0; echo arg count: $#' self one two
self
arg count: 2
I suspect there is no good solution, but perhaps I'm overlooking something:
What I'm after is a way to:
(a) call a batch file from PowerShell in a way that robustly reflects its - implicit or explicit - exit code in PowerShell's automatic $LASTEXITCODE variable.
Notably, calling a batch file that exits with, say, whoami -nosuch || exit /b, should result in $LASTEXITCODE reflecting whoami's exit code, i.e. 1. This is not the case when you invoke a batch file (by name or path) from PowerShell: the exit code is 0 (by contrast, inside a cmd.exe session %ERRORLEVEL% is set to 1).
Also note that the invocation should remain integrated with PowerShell's output streams, so I am not looking for solutions based on System.Diagnostics.Process.
Furthermore, I have no knowledge of or control over the batch files getting invoked - I'm looking for a generic solution.
(b) without double-quoted arguments passed to the batch file getting altered in any way, and without cmd.exe's behavior getting modified in any way; notably:
^ characters should not be doubled (see below).
Enabling delayed expansion with /V:ON is not an option.
The only way I know how to solve (a) is to invoke the batch file via cmd /c call.
Unfortunately, this violates requirement (b), because the use of call seemingly invariably doubles ^ characters in arguments. (And, conversely, not using call then doesn't report the exit code reliably).
Is there a way to satisfy both requirements?
Note that PowerShell is only the messenger here: The problem lies with cmd.exe, and anyone calling a batch file from outside a cmd.exe session is faced with the same problem.
Example (PowerShell code):
# Create a (temporary) batch file that echoes its arguments,
# provokes an error, and exits with `exit /b` *without an explicit argument*.
'#echo off & echo [%*] & whoami -nosuch 2>NUL || exit /b' | Set-Content test.cmd
# Invoke the batch file and report the exit code.
.\test.cmd "a ^ 2"; $LASTEXITCODE
The output should be:
["a ^ 2"]
1
However, in reality the exit code is not reported:
["a ^ 2"]
0 # !! BROKEN
If I call with cmd /c call .\test.cmd instead, the exit code is correct, but the ^ characters are doubled:
PS> cmd /c call .\test.cmd "a ^ 2"; $LASTEXITCODE
["a ^^ 2"] # !! BROKEN
1 # OK
I've no idea why this works, but it does:
cmd /c '.\test.cmd "a ^ 2" & exit'
$LASTEXITCODE
Output:
["a ^ 2"]
1
Kudos to beatcracker for finding an effective workaround in his answer; let me add some background information and guidance:
First, to be clear, no workaround should be necessary; cmd.exe's behavior is clearly a bug.
cmd /c '.\test.cmd "a ^ 2" || exit' - i.e. || rather than & - is what one would expect to be an effective workaround too. The fact that only &, which unconditionally sequences commands, works, indicates that even cmd.exe-internally the failure status of the batch file isn't yet known as part of the same statement - only afterwards - which appears to be another manifestation of the bug.
Why an explicit exit call following the batch-file call as part of the same statement does relay the batch file's (zero or nonzero) exit code correctly is anyone's guess, but it seems to work.
Fortunately, the workaround is also effective for solving related exit-code problems in batch files that do not contain explicit exit /b / exit calls - see this answer.
Syntax considerations:
From PowerShell, the alternative to passing a single command-string is to pass individual arguments and escape the & character as `& (using `, the "backtick", PowerShell's escape character) so as to prevent PowerShell from interpreting it (quoting it as '&' would work too):
cmd /c .\test.cmd "a ^ 2" `& exit
From an environment that doesn't involve a shell, such as when launching from Task Scheduler, the `-escaping of & is not needed (and mustn't be used).
Not having to enclose the entire for-cmd.exe command in quotes makes it easier to pass arguments that (a) individually require double quotes and (b) involve references to PowerShell variables and/or expressions, given that the latter requires use of "..." rather than '...':
# Passing *individual* arguments makes double-quoting easier.
PS> cmd /c .\test.cmd "Version = $($PSVersionTable.PSVersion)" `& exit; $LASTEXITCODE
["Version = 7.2.0-preview.4"]
1
Using quoting of the entire for-cmd.exe command would be awkward in this case, due to the need to escape the argument-specific " chars.:
# Embedded double quotes must now be `-escaped.
PS> cmd /c ".\test.cmd `"Version = $($PSVersionTable.PSVersion)`" & exit"
["Version = 7.2.0-preview.4"]
1
The Native module (authored by me; install it from the PowerShell Gallery with Install-Module Native) comes with function ie, which:
automatically applies the above workaround.
generally compensates for problems arising from PowerShell's broken argument-passing to external programs (see this answer).
# After having run Install-Module Native:
# Use of function `ie` applies the workaround behind the scenes.
PS> ie .\test.cmd "Version = $($PSVersionTable.PSVersion)"; $LASTEXITCODE
["Version = 7.2.0-preview.4"]
1
The hope is that what function ie does will become a part of PowerShell itself, as part of the upcoming (in PowerShell v7.2) PSNativeCommandArgumentPassing experimental feature that is intended as an opt-in fix to the broken argument-passing - see GitHub issue #15143
In Powershell(pwsh)
Is it possible to "bypass" Powershell to not do anything and let the underlying shell do the work?
Sometime I need to call a command, that I copy from a site or other, that includes "\" and not "´" for line break - or other parameters, that Powershell try to interpolate.
Like:
<Powershell Function> kubectl get rolebindings,clusterrolebindings \
--all-namespaces \
-o custom-columns='KIND:kind,NAMESPACE:metadata.namespace,NAME:metadata.name,SERVICE_ACCOUNTS:subjects[?(#.kind=="ServiceAccount")].name' | grep name
and let the underlying shell do the work?
PowerShell is a shell, so there is no underlying shell as such (in the sense that PowerShell needn't be launched from another shell).
However, there are platform-specific default shells: cmd.exe on Windows, and /bin/sh on Unix-like platforms.
I need to call a command, that I copy from a site or other
Before PowerShell went cross-platform, --%, the stop-parsing token was introduced in Windows PowerShell v3, precisely to address your scenario.
Given that Windows was the only supported platform at the time, the behavior is focused on cmd.exe - and even there it doesn't support cmd.exe's line continuation via line-ending ^ chars.
More generally, while it tries to emulate cmd.exe's behavior (e.g., by expanding %-enclosed tokens such as %USERNAME% as environment-variable references) without actually delegating to the latter, it doesn't do so fully.
In PowerShell Core (v6+) --% is even supported on Unix-like platforms, but is of little utility there, given that cmd.exe semantics are applied even there; notably, '...'-quoting is not supported.
(Note that PowerShell emulates one /bin/sh feature on Unix-like platforms, but only without use of --%: globbing, i.e. the expansion of unquoted tokens such as *.txt to matching file paths).
However, you can call the platform-native shell explicitly, e.g, by passing the copied command string to sh -c on Unix-like platforms, using a (verbatim) here-string for the command string:
# Call the platform-native shell with a command crafted for it:
sh -c #'
echo This line\
keeps going\
with --all-namespaces and\
with -o custom-columns='KIND:kind,NAMESPACE'
'#
Unfortunately, there's a catch:
Up to at least v7.2.x, PowerShell's handling of embedded " chars. in arguments passed to external programs (such as sh) is fundamentally broken, requiring embedded " to be manually \-escaped:
# !! BROKEN
sh -c #'
echo "high noon"
'#
To make this work, you have to \-escape manually:
# WORKAROUND
sh -c (#'
echo "high noon"
'# -replace '"', '\"')
See this answer and this GitHub issue for background information.
In short: Backward-compatibility concerns prevent fixing the default behavior, but discussions are underway to at least allow opting into sane default behavior.
Not that I'm aware of. I also don't know of any shell where this is possible, either, save for Ruby where you have the string literal operators making escapes much easier.
I would suggest that before sending strings to external commands, that you convert the PowerShell escape sequences to the target runtime escape sequences. This isn't a problem unique to PowerShell, but is easier to work with in other shells because there is more of an overlap of the special characters between other runtimes.