In Powershell(pwsh)
Is it possible to "bypass" Powershell to not do anything and let the underlying shell do the work?
Sometime I need to call a command, that I copy from a site or other, that includes "\" and not "´" for line break - or other parameters, that Powershell try to interpolate.
Like:
<Powershell Function> kubectl get rolebindings,clusterrolebindings \
--all-namespaces \
-o custom-columns='KIND:kind,NAMESPACE:metadata.namespace,NAME:metadata.name,SERVICE_ACCOUNTS:subjects[?(#.kind=="ServiceAccount")].name' | grep name
and let the underlying shell do the work?
PowerShell is a shell, so there is no underlying shell as such (in the sense that PowerShell needn't be launched from another shell).
However, there are platform-specific default shells: cmd.exe on Windows, and /bin/sh on Unix-like platforms.
I need to call a command, that I copy from a site or other
Before PowerShell went cross-platform, --%, the stop-parsing token was introduced in Windows PowerShell v3, precisely to address your scenario.
Given that Windows was the only supported platform at the time, the behavior is focused on cmd.exe - and even there it doesn't support cmd.exe's line continuation via line-ending ^ chars.
More generally, while it tries to emulate cmd.exe's behavior (e.g., by expanding %-enclosed tokens such as %USERNAME% as environment-variable references) without actually delegating to the latter, it doesn't do so fully.
In PowerShell Core (v6+) --% is even supported on Unix-like platforms, but is of little utility there, given that cmd.exe semantics are applied even there; notably, '...'-quoting is not supported.
(Note that PowerShell emulates one /bin/sh feature on Unix-like platforms, but only without use of --%: globbing, i.e. the expansion of unquoted tokens such as *.txt to matching file paths).
However, you can call the platform-native shell explicitly, e.g, by passing the copied command string to sh -c on Unix-like platforms, using a (verbatim) here-string for the command string:
# Call the platform-native shell with a command crafted for it:
sh -c #'
echo This line\
keeps going\
with --all-namespaces and\
with -o custom-columns='KIND:kind,NAMESPACE'
'#
Unfortunately, there's a catch:
Up to at least v7.2.x, PowerShell's handling of embedded " chars. in arguments passed to external programs (such as sh) is fundamentally broken, requiring embedded " to be manually \-escaped:
# !! BROKEN
sh -c #'
echo "high noon"
'#
To make this work, you have to \-escape manually:
# WORKAROUND
sh -c (#'
echo "high noon"
'# -replace '"', '\"')
See this answer and this GitHub issue for background information.
In short: Backward-compatibility concerns prevent fixing the default behavior, but discussions are underway to at least allow opting into sane default behavior.
Not that I'm aware of. I also don't know of any shell where this is possible, either, save for Ruby where you have the string literal operators making escapes much easier.
I would suggest that before sending strings to external commands, that you convert the PowerShell escape sequences to the target runtime escape sequences. This isn't a problem unique to PowerShell, but is easier to work with in other shells because there is more of an overlap of the special characters between other runtimes.
Related
I use WinSCP within a Powershell script. It suddenly stopped working. After a while I could figure out that the problem appeared from a more recent version of PowerShell:
Reduced code:
& winscp `
/log `
/command `
'echo Connecting...' `
"open sftp://kjhgk:jkgh#lkjhlk.com/ -hostkey=`"`"ssh-ed25519 includes spaces`"`""
Error message using v7.2.7
Host "lkjhlk.com" does not exist.
Errror message using v7.3.0
Too many parameters for command 'open'.
As you can see with v7.3.0 WinSCP receives different input depending on the version of PS. I found out that the difference has something to do with the spaces in the hostkey. If they are omitted v7.3.0 outputs the same error.
What change to PowerShell caused this, and how can I fix it?
(How can I debug such issues? I played a bit around with escaping, but the strings look the same no matter the version, no obvious breaking change that could be responsible)
Version 7.3.0 of PowerShell (Core) introduced a breaking change with respect to how arguments with embedded " characters are passed to external programs, such as winscp:
While this change is mostly beneficial, because it fixes behavior that was fundamentally broken since v1 (this answer discusses the old, broken behavior), it also invariably breaks existing workarounds that build on the broken behavior, except those for calls to batch files and the WSH CLIs (wscript.exe and cscript.exe) and their associated script files (with file-name extensions such as .vbs and .js).
CAVEAT:
GitHub issue #18694 implies that some version after 7.3.1 and all later versions will make this breaking change opt-in, but on Windows only, for the sake of backward compatibility; that is:
On Windows:
Old workarounds will continue to work by default.
Getting the new, correct behavior requires (temporarily) setting
$PSNativeCommandArgumentPassing = 'Standard'
On Unix-like platforms:
The new, correct behavior ($PSNativeCommandArgumentPassing = 'Standard') will remain the default
Old workarounds will require (temporarily) setting $PSNativeCommandArgumentPassing = 'Legacy' in order to continue to work, as is already the case in 7.3.0
To make existing workarounds continue to work, set the $PSNativeCommandArgumentPassing preference variable (temporarily) to 'Legacy':
# Note: Enclosing the call in & { ... } makes it execute in a *child scope*
# limiting the change to $PSNativeCommandArgumentPassing to that scope.
& {
$PSNativeCommandArgumentPassing = 'Legacy'
& winscp `
/log `
/command `
'echo Connecting...' `
"open sftp://kjhgk:jkgh#lkjhlk.com/ -hostkey=`"`"ssh-ed25519 includes spaces`"`""
}
Unfortunately, because winscp.exe only accepts
"open sftp://kjhgk:jkgh#lkjhlk.com/ -hostkey=""ssh-ed25519 includes spaces""" on its process command line (i.e., embedded " escaped as ""), and not also the most widely used form
"open sftp://kjhgk:jkgh#lkjhlk.com/ -hostkey=\"ssh-ed25519 includes spaces\"" (embedded " escaped as \"), which the fixed behavior now employs, for winscp.exe, specifically, a workaround will continue to be required.
If you don't want to rely on having to modify $PSNativeCommandArgumentPassing for the workaround, here are workarounds that function in both v7.2- and v7.3+ :
Use --%, the stop-parsing token, which, however, comes with pitfalls and severe limitations, notably the inability to (directly) use PowerShell variables or subexpressions in the arguments that follow it - see this answer for details; however, you can bypass these limitations if you use --% as part of an array that you construct and assign to a variable first and then pass via splatting:
# Note: Must be single-line; note the --% and the
# unescaped use of "" in the argument that follows it.
# Only "..." quoting must be used after --%
# and the only variables that can be used are cmd-style
# *environment variables* such as %OS%.
winscp /log /command 'echo Connecting...' --% "open sftp://kjhgk:jkgh#lkjhlk.com/ -hostkey=""ssh-ed25519 includes spaces"""
# Superior alternative, using splatting:
$argList = '/log', '/command', 'echo Connecting...',
'--%', "open sftp://kjhgk:jkgh#lkjhlk.com/ -hostkey=""ssh-ed25519 includes spaces"""
winscp #argList
Alternatively, call via cmd /c:
# Note: Pass-through command must be single-line,
# Only "..." quoting supported,
# and the embedded command must obey cmd.exe's syntax rules.
cmd /c #"
winscp /log /command "echo Connecting..." "open sftp://kjhgk:jkgh#lkjhlk.com/ -hostkey=""ssh-ed25519 includes spaces"""
"#
Note: You don't strictly need to use a here-string (#"<newline>...<newline>"# or #'<newline>...<newline>'#), but it helps readability and simplifies using embedded quoting.
Both workarounds allow you to pass arguments directly as quoted, but unfortunately also require formulating the entire (pass-through) command on a single line - except if --% is combined with splatting.
Background information:
The v7.3 default $PSNativeCommandArgumentPassing value on Windows, 'Windows':
regrettably retains the old, broken behavior for calls to batch files and the WSH CLIs (wscript.exe and cscript.exe) and their associated script files (with file-name extensions such as .vbs and .js).
While, for these programs only, this allows existing workarounds to continue to function, future code that only needs to run in v7.3+ will continue to be burdened by the need for these obscure workarounds, which build on broken behavior.
The alternative, which was not implemented, would have been to build accommodations for these programs as well as some program-agnostic accommodations into PowerShell, so that in the vast majority of case there won't even be a need for workarounds in the future: see GitHub issue #15143.
There are also troublesome signs that this list of exceptions will be appended to, piecemeal, which all but guarantees confusion for a given PowerShell version as to which programs require workarounds and which don't.
commendably, for all other programs, makes PowerShell encode the arguments when it - of necessity - rebuilds the command line behind the scenes as follows with respect to ":
It encodes the arguments for programs that follow the C++ command-line parsing rules (as used by C / C++ / .NET applications) / the parsing rules of the CommandLineToArgv WinAPI function, which are the most widely observed convention for parsing a process' command line.
In a nutshell, this means that embedded " characters embedded in an argument, to be seen as a verbatim part of it by the target program, are escaped as \", with \ itself requiring escaping only (as \\) if it precedes a " but is meant to be interpreted verbatim.
Note that if you set $PSNativeCommandArgumentPassing value to 'Standard' (which is the default on Unix-like platforms, where this mode fixes all problems and makes v7.3+ code never require workarounds), this behavior applies to all external programs, i.e. the above exceptions no longer apply).
For a summary of the impact of the breaking v7.3 change, see this comment on GitHub.
If you have / need to write cross-edition, cross-version PowerShell code: The Native module (Install-Module Native; authored by me), has an ie function (short for: Invoke Executable), which is a polyfill that provides workaround-free cross-edition (v3+), cross-platform, and cross-version behavior in the vast majority of cases - simply prepend ie to your external-program calls.
Caveat: In the specific case at hand it will not work, because it isn't aware that winscp.exe requires ""-escaping.
I download a zip file from my cloud using a PowerShell command.
The command works correctly in PowerShell AND in the command line. However, if I insert the command from the command line into my batch script, only the html is downloaded.
Why does the command work correctly in the command line but not in the batch file?
Im out of ideas :D
powershell Invoke-WebRequest """https://sync.luckycloud.de/d/fb56e4a8239a4c6cac7a/files/?p=%2FValheimServer%20Buddelkiste%20Modpack%20v3.4%20-%20Standart.zip&dl=1""" -OutFile """C:\Users\Anonymos\Downloads\servermodpack.zip"""
Its works complete fine in cmd and load the ~40 Mb. But in Batch it loads only 9kb (its the Html)
In a batch file - as opposed to the interactive cmd.exe command prompt[1] - you need to to escape % chars. as %% in order to pass them through literally:
powershell -c "Invoke-WebRequest 'https://sync.luckycloud.de/d/fb56e4a8239a4c6cac7a/files/?p=%%2FValheimServer%%20Buddelkiste%%20Modpack%%20v3.4%%20-%%20Standart.zip&dl=1' -OutFile C:\Users\Anonymos\Downloads\servermodpack.zip"
Note:
I've used -c (-Command), the positionally implied Windows PowerShell CLI (powershell.exe) parameter explicitly for conceptual clarity (in PowerShell (Core) 7+, whose CLI is pwsh, the default is now -f (-File)).
Also, enclosing the entire command line to pass to PowerShell in "..." is generally preferable in order to prevent cmd.exe metacharacters other than % (such as &) from causing problems.
In case the command line requires use of embedded " characters (the solution above avoids this by using embedded '...' quoting), the safest way to escape them is to use "^"" (sic) with powershell.exe, and "" with pwsh.exe - see this answer for details.
[1] In interactive use, % characters cannot technically be escaped, but they're retained as-is unless they're part of a cmd.exe-style environment-variable reference that refers to an existing environment variable, such as %OS%. However, there are ways to treat % literally even in such cases, as discussed in this answer. These techniques are important for invoking cmd.exe command lines programmatically from outside cmd.exe, such as from Node.Js or Python, because such programmatic invocations - perhaps surprisingly - use the rules of interactive cmd.exe sessions, not batch files.
As I have learned, when invoking PowerShell from cmd.exe, with the -Command option, escaping rules are essentially those used in Linux. So you use a backslash (\), rather than the usual backtick (`).
This is to say that you don't write:
C:\> powershell -c "echo `"That's a single quote: ' `""
but
C:\> powershell -c "echo \"That's a single quote: ' \""
to get:
That's a single quote: '
And this is the exact equivalent of what you would type in a Linux terminal:
~ >>> bash -c "echo \"That's a single quote: ' \""
If I am not wrong, this feature is named PSNativeCommandArgumentPassing.
But comparisons work up to a certain point. In fact, under Linux you write:
~ >>> bash -c "echo \"That's a double quote: \\\" \""
to get:
That's a double quote: "
while the equivalent PowerShell line:
C:\> powershell -c "echo \"That's a double quote: \\\" \""
gives
The string is missing the terminator: ".
By trial and error, I realised that:
C:\> powershell -c "echo \"That's a double-quote: `"" \""
works as expected.
Can you explain to me what is the logic behind: `""?
Also, what are the equivalent commands when calling PowerShell from powershell.exe, rather than cmd.exe?
tl;dr
On PowerShell's command line only, use \" to pass a " through to the code that -c (-Command) should execute.
Only in the resulting code does PowerShell's usual escaping of " as `" apply, so you may have to combine the two escaping techniques: `\".
Your attempt, `"", had the same effect, but it doesn't work reliably.[1]
\" works robustly with respect to PowerShell's own parsing of its command line, but can situationally break calls on the cmd.exe side - see the bottom section for workarounds.
When calling CLIs (external console applications) from PowerShell, not only do PowerShell's own, different quoting rules apply first (support for '...' strings, embedded " inside "..." escaped as `"), a long-standing bug up to PowerShell 7.2.x additionally requires escaping " chars. with \ when embedded in external-program arguments (only); see this answer.
this feature is named PSNativeCommandArgumentPassing
No; this feature - which became official in PowerShell 7.3 (see this answer) - does not come into play, because:
it generally isn't (and won't be) available in the legacy Windows PowerShell edition that you're invoking via its CLI, powershell.exe (whereas the modern, cross-platform PowerShell (Core) edition's CLI is pwsh.exe)
it only applies to calls from inside a PowerShell session.
it is designed to address a long-standing problem when calling external programs with embedded " or empty-string string arguments from PowerShell - see this answer.
Therefore, the linked answer addresses your last question:
Also, what are the equivalent commands when calling PowerShell from powershell.exe, rather than cmd.exe?
In short: unfortunately, up to at least PowerShell 7.2.x you'll have to additionally, manually escape " chars. with " embedded in external-program arguments (only).
Why the following works from cmd.exe:
powershell -c "echo \"That's a single quote: ' \""
PowerShell only recognizes \ as an escape character on its command line, for consistency with other CLIs.
Inside a PowerShell session, only ` (backtick) serves as the escape character.
Caveat: While \" to escape " works consistently in both PowerShell editions on the PowerShell (command-line parsing) side, situationally it can break cmd.exe's own parsing - see the bottom section.
When the PowerShell CLI is invoke via -c (-Command) with a piece of PowerShell source code to execute, that code (the argument(s) following -c is parsed in two stages:
First, all unescaped " chars. are stripped, wheres escaped ones (\") are kept and unescaped.
Only then is the result parsed and execute as PowerShell code.
Therefore, what PowerShell ends up executing is verbatim:
echo "That's a single quote: ' "
From the above follows why this does not work:
:: SYNTAX ERROR
powershell -c "echo \"That's a double quote: \\\" \""
PowerShell ends up trying to execute verbatim
echo "That's a double quote: \" "
which is a syntax error, because inside a PowerShell session \ doesn't escape " -only `" or - inside "...", alternatively - "" do.
From the above follows why this (mostly) works:
:: Works, but not robustly
powershell -nop -c "echo \"That's a double-quote: `"" \""
"" sometimes, but not always works as an alternative to \";[1] here it does - see also the bottom section.
As a result, PowerShell ends up executing the following verbatim, which works, because the escaped " that was passed through is now `-escaped, as it needs to be inside a PowerShell "..." string:
echo "That's a double-quote: `" "
To avoid the brittle "" escaping, it is better to formulate this command by combining the required `-escaping with the command-line \-escaping - i.e. `\" - but see the bottom section for a fully robust solution:
powershell -nop -c "echo \"That's a double-quote: `\" \""
Avoiding parsing problems on the cmd.exe side, a safe alternative to \":
While \" to escape " works consistently in both PowerShell editions on the PowerShell (command-line parsing) side, situationally it can break cmd.exe's own parsing. By contrast, using \" is safe when calling from no-shell environments such as Task Scheduler.
While there are workarounds, they are, unfortunately PowerShell edition-specific:
In Windows PowerShell (powershell.exe)
Use "^"" (sic) instead of \"
In PowerShell (Core) (v6+, pwsh.exe)
Use "" instead of \"
Important:
These workarounds require that the whole code to pass to -c (-Command) be passed as a single, "..."-enclosed argument.
-c (-Command) also accepts multiple arguments - which may individually be double-quoted or not - in which case it simply concatenates these arguments, after having stripped unescaped ", to form the code to execute. This technique situationally allows you to get away with \"-escaping (e.g., powershell -c \"Between 2 & 3`\"\", but (a) it requires you to pay close attention to which parts of the command cmd.exe will see as unquoted, (b) would require you to ^-escape any cmd.exe metacharacters such as & in those parts, and (c) invariably performs whitespace normalization, i.e. folds runs of multiple spaces into a single one each.
The following calls, designed to print verbatim Between 2 & 3" , demonstrate this:
:: BREAKS, because cmd.exe sees `&` as *outside a double-quoted string*
powershell -c " \" Between 2 & 3`\" \" "
:: PowerShell (Core) workaround
pwsh -c " "" Between 2 & 3`"" "" "
:: Windows PowerShell workaround
powershell -c " "^"" Between 2 & 3`"^"" "^"" "
[1] An example of where "" inside "..." doesn't work is powershell -c "echo \" Nat `""King`"" Cole \"": instead of Nat "King" Cole, it prints Nat "King Cole, i.e. the second escaped " is missing (it would work fine in pwsh.exe, however, as discussed in the bottom section). It's ultimately not worth speculating how, precisely, embedded "" sequences are parsed by powershell.exe -c, given that it demonstrably isn't reliable and that reliable alternatives do exist (\" or, from cmd.exe, also "^"").
Ex:
cmd /C start C:\Users\Bob Builder\Desktop\New Folder\test.exe
I'm trying to use cmd to start a file but since there are spaces in the path, cmd is throwing an error after Bob.
Error:
"Windows cannot find C:\Users\Bob. Make sure you typed the name
correctly, then try again."
The system cannot find the file C:\Users\Bob.
Its simply failing to accept the spaces. It's driving me crazy because I'm spoiled with C# working out of the box. I don't know much about this, I have been spending way too much time trying to figure this out. Some help would be greatly appreciated.
In order for a path that contains spaces to be recognized as a single path (argument), it must be quoted.
In order for an executable to execute in the current console window, synchronously, with its streams connected to the calling shell, it must be invoked directly, not via start.
Direct invocation from cmd.exe (only "..." quoting supported):
"C:\Users\Bob Builder\Desktop\New Folder\test.exe"
From PowerShell:
& 'C:\Users\Bob Builder\Desktop\New Folder\test.exe'
Note:
PowerShell also supports '...' strings (single-quoted), which are verbatim strings that are preferable to "..." (double-quoted) ones if you do not require expansion of variables (string interpolation) - see the conceptual about_Quoting_Rules help topic.
For syntactic reasons, PowerShell requires the use of &, the call operator to invoke commands that are quoted and/or contain variable references - see this answer for details.
By contrast, use start in cmd.exe / Start-Process in PowerShell (whose built-in alias is also start) to launch an executable in a new window (on Windows), asynchronously, with no (direct) ability to capture the launched executable's output:
From cmd.exe:
start "title" "C:\Users\Bob Builder\Desktop\New Folder\test.exe"
Note:
Specifying "title" - i.e. a self-chosen (console) window title - is required for syntactic reasons in this case: without it, the double-quoted path itself would be interpreted as the window title, and the - implied - executable to launch would be another cmd.exe instance.
Note that if you launch a GUI application this way, the title argument is irrelevant, because no new console window is created.
Conversely, if you launch a console application specified by double-quoted path and therefore must use a title argument, note that "" will result in the new window having no title.
From PowerShell (parameter -FilePath is positionally implied):
Start-Process 'C:\Users\Bob Builder\Desktop\New Folder\test.exe'
Note:
Start-Process does not support specifying a window title, so you may want to call cmd.exe's internal start command for that (or other features not supported by Start-Process, such as specifying the process priority).
To work around quoting problems, invoke cmd.exe's start from PowerShell by passing the entire start command as a single string to cmd /c:
cmd /c 'start "title" "C:\Users\Bob Builder\Desktop\New Folder\test.exe"'
cmd /C start "C:\Users\Bob Builder\Desktop\New Folder\test.exe"
Quotes are your friend. Sometimes even double quotes are too!
Seems like cmd won't work for me. Powershell worked with this script:
$env:Path += ";C:\Users\Bob Builder\Desktop\New Folder\"
test.exe
When I used the command below [1] to set my configuration variable MONGODB_URI, it gives an error [2].
I am using Windows PowerShell.
[1] >> heroku config:set MONGODB_URI='mongodb+srv://myprojectname:<mypassword>#cluster0.rkitj.mongodb.net/<myusername>?retryWrites=true&w=majority'
[2] The system cannot find the file specified.
'w' is not recognized as an internal or external command,
operable program or batch file.
Note: myprojectname, mypassword and myusername are placeholders for the actual value.
It looks like the heroku CLI entry point is a batch file, as implied by the wording of the error messages, which are cmd.exe's, not PowerShell's.
PowerShell doesn't take the special parsing needs of batch files (cmd.exe) into account when it synthesizes the actual command line to use behind the scenes, which involves re-quoting, using double quotes only, and only when PowerShell thinks quoting is needed.
In this case PowerShell does not double-quote (because the value contains no spaces), which breaks the batch-file invocation.
You have the following options:
You can use embedded quoting so as to ensure that the value part of your MONGODB_URI=... key-value pair is passed in double quotes; note the '"..."' quoting:
heroku config:set MONGODB_URI='"mongodb+srv://myprojectname:<mypassword>#cluster0.rkitj.mongodb.net/<myusername>?retryWrites=true&w=majority"'
Caveat: This shouldn't work, and currently only works because PowerShell's passing of arguments to external program is fundamentally broken as of PowerShell 7.1 - see this answer. Should this ever get fixed, the above will break.
If your command line doesn't involve any PowerShell variables and expressions, you can use --%, the stop-parsing symbol, which, however, in general, has many limitations (see this answer); essentially, everything after --% is copied verbatim to the target command line, except for expanding cmd.exe-style environment-variable references (e.g., %USERNAME%):
heroku config:set --% MONGODB_URI="mongodb+srv://myprojectname:<mypassword>#cluster0.rkitj.mongodb.net/<myusername>?retryWrites=true&w=majority"
If you're willing to install a module, you can use the ie function from the PSv3+ Native module (install with Install-Module Native from the PowerShell Gallery in PSv5+), which internally compensates for all of PowerShell's argument-passing and cmd.exe's argument-parsing quirks (it is implemented in a forward-compatible manner so that should PowerShell itself ever get fixed, the function will simply defer to PowerShell); that way, you can simply focus on meeting PowerShell's syntax requirements, and let ie handle the rest:
# 'ie' prepended to an invocation that uses only PowerShell syntax
ie heroku config:set MONGODB_URI='mongodb+srv://myprojectname:<mypassword>#cluster0.rkitj.mongodb.net/<myusername>?retryWrites=true&w=majority'