I use WinSCP within a Powershell script. It suddenly stopped working. After a while I could figure out that the problem appeared from a more recent version of PowerShell:
Reduced code:
& winscp `
/log `
/command `
'echo Connecting...' `
"open sftp://kjhgk:jkgh#lkjhlk.com/ -hostkey=`"`"ssh-ed25519 includes spaces`"`""
Error message using v7.2.7
Host "lkjhlk.com" does not exist.
Errror message using v7.3.0
Too many parameters for command 'open'.
As you can see with v7.3.0 WinSCP receives different input depending on the version of PS. I found out that the difference has something to do with the spaces in the hostkey. If they are omitted v7.3.0 outputs the same error.
What change to PowerShell caused this, and how can I fix it?
(How can I debug such issues? I played a bit around with escaping, but the strings look the same no matter the version, no obvious breaking change that could be responsible)
Version 7.3.0 of PowerShell (Core) introduced a breaking change with respect to how arguments with embedded " characters are passed to external programs, such as winscp:
While this change is mostly beneficial, because it fixes behavior that was fundamentally broken since v1 (this answer discusses the old, broken behavior), it also invariably breaks existing workarounds that build on the broken behavior, except those for calls to batch files and the WSH CLIs (wscript.exe and cscript.exe) and their associated script files (with file-name extensions such as .vbs and .js).
CAVEAT:
GitHub issue #18694 implies that some version after 7.3.1 and all later versions will make this breaking change opt-in, but on Windows only, for the sake of backward compatibility; that is:
On Windows:
Old workarounds will continue to work by default.
Getting the new, correct behavior requires (temporarily) setting
$PSNativeCommandArgumentPassing = 'Standard'
On Unix-like platforms:
The new, correct behavior ($PSNativeCommandArgumentPassing = 'Standard') will remain the default
Old workarounds will require (temporarily) setting $PSNativeCommandArgumentPassing = 'Legacy' in order to continue to work, as is already the case in 7.3.0
To make existing workarounds continue to work, set the $PSNativeCommandArgumentPassing preference variable (temporarily) to 'Legacy':
# Note: Enclosing the call in & { ... } makes it execute in a *child scope*
# limiting the change to $PSNativeCommandArgumentPassing to that scope.
& {
$PSNativeCommandArgumentPassing = 'Legacy'
& winscp `
/log `
/command `
'echo Connecting...' `
"open sftp://kjhgk:jkgh#lkjhlk.com/ -hostkey=`"`"ssh-ed25519 includes spaces`"`""
}
Unfortunately, because winscp.exe only accepts
"open sftp://kjhgk:jkgh#lkjhlk.com/ -hostkey=""ssh-ed25519 includes spaces""" on its process command line (i.e., embedded " escaped as ""), and not also the most widely used form
"open sftp://kjhgk:jkgh#lkjhlk.com/ -hostkey=\"ssh-ed25519 includes spaces\"" (embedded " escaped as \"), which the fixed behavior now employs, for winscp.exe, specifically, a workaround will continue to be required.
If you don't want to rely on having to modify $PSNativeCommandArgumentPassing for the workaround, here are workarounds that function in both v7.2- and v7.3+ :
Use --%, the stop-parsing token, which, however, comes with pitfalls and severe limitations, notably the inability to (directly) use PowerShell variables or subexpressions in the arguments that follow it - see this answer for details; however, you can bypass these limitations if you use --% as part of an array that you construct and assign to a variable first and then pass via splatting:
# Note: Must be single-line; note the --% and the
# unescaped use of "" in the argument that follows it.
# Only "..." quoting must be used after --%
# and the only variables that can be used are cmd-style
# *environment variables* such as %OS%.
winscp /log /command 'echo Connecting...' --% "open sftp://kjhgk:jkgh#lkjhlk.com/ -hostkey=""ssh-ed25519 includes spaces"""
# Superior alternative, using splatting:
$argList = '/log', '/command', 'echo Connecting...',
'--%', "open sftp://kjhgk:jkgh#lkjhlk.com/ -hostkey=""ssh-ed25519 includes spaces"""
winscp #argList
Alternatively, call via cmd /c:
# Note: Pass-through command must be single-line,
# Only "..." quoting supported,
# and the embedded command must obey cmd.exe's syntax rules.
cmd /c #"
winscp /log /command "echo Connecting..." "open sftp://kjhgk:jkgh#lkjhlk.com/ -hostkey=""ssh-ed25519 includes spaces"""
"#
Note: You don't strictly need to use a here-string (#"<newline>...<newline>"# or #'<newline>...<newline>'#), but it helps readability and simplifies using embedded quoting.
Both workarounds allow you to pass arguments directly as quoted, but unfortunately also require formulating the entire (pass-through) command on a single line - except if --% is combined with splatting.
Background information:
The v7.3 default $PSNativeCommandArgumentPassing value on Windows, 'Windows':
regrettably retains the old, broken behavior for calls to batch files and the WSH CLIs (wscript.exe and cscript.exe) and their associated script files (with file-name extensions such as .vbs and .js).
While, for these programs only, this allows existing workarounds to continue to function, future code that only needs to run in v7.3+ will continue to be burdened by the need for these obscure workarounds, which build on broken behavior.
The alternative, which was not implemented, would have been to build accommodations for these programs as well as some program-agnostic accommodations into PowerShell, so that in the vast majority of case there won't even be a need for workarounds in the future: see GitHub issue #15143.
There are also troublesome signs that this list of exceptions will be appended to, piecemeal, which all but guarantees confusion for a given PowerShell version as to which programs require workarounds and which don't.
commendably, for all other programs, makes PowerShell encode the arguments when it - of necessity - rebuilds the command line behind the scenes as follows with respect to ":
It encodes the arguments for programs that follow the C++ command-line parsing rules (as used by C / C++ / .NET applications) / the parsing rules of the CommandLineToArgv WinAPI function, which are the most widely observed convention for parsing a process' command line.
In a nutshell, this means that embedded " characters embedded in an argument, to be seen as a verbatim part of it by the target program, are escaped as \", with \ itself requiring escaping only (as \\) if it precedes a " but is meant to be interpreted verbatim.
Note that if you set $PSNativeCommandArgumentPassing value to 'Standard' (which is the default on Unix-like platforms, where this mode fixes all problems and makes v7.3+ code never require workarounds), this behavior applies to all external programs, i.e. the above exceptions no longer apply).
For a summary of the impact of the breaking v7.3 change, see this comment on GitHub.
If you have / need to write cross-edition, cross-version PowerShell code: The Native module (Install-Module Native; authored by me), has an ie function (short for: Invoke Executable), which is a polyfill that provides workaround-free cross-edition (v3+), cross-platform, and cross-version behavior in the vast majority of cases - simply prepend ie to your external-program calls.
Caveat: In the specific case at hand it will not work, because it isn't aware that winscp.exe requires ""-escaping.
Related
I am experiencing a rather puzzling error while trying to perform a diff on two files using Visual Studio Code from the command line. I have a text file in the cloud where I save some work related notes. I need to resolve conflicts with other clients editing the file. Usually this only happens during a loss of connection though somehow I find myself having to resolve a lot of them so between this and other uses of diff I will use the usual syntax. It looks something like this:
code --diff "R&D (cloud conflict 2-5-23).txt" "R&D.txt"
My filename happens to have a '&' in it and this command launches the usual 2-way diff in VS Code and reads through the first file name with no problem but doesn't read past the second '&' and the resulting diff tab in VS Code looks something like:
R&D (cloud conflict 2-25-23).txt <-> R
Where the right side "R" doesn't exist. So it would seem '&' needs to be processed literally.
No problem, let's see if backslash \ is an accepted escape parameter...
code --diff "R\&D (cloud conflict 2-5-23).txt" "R\&D.txt"
Nope. Same problem. 🤔 In fact this outputs something even stranger:
Code diff tab:
&D (cloud conflict 2-25-23).txt <-> R
with shell output:
'D.txt' is not recognized as an internal or external command, operable program or batch file.
I also tried the carrot symbol '^' as an escape parameter to a similar effect. I just includes it in the first file and the editor still thinks the second file name is just "R".
The help file for the VS Code command line integration didn't have a lot to say about the --diff parameter other than a short description and I was hoping to get something about processing strings literally or escape characters. Perhaps another parameter that I need or maybe this has more to do with the shell in general.
I find it really strange that it can read the first full file name but breaks at the second '&'. Weirder still that if a supposed escape character is included in the second file name, it will omit that as well. 😵
For now all I can do is rename the file which is a bummer. 🤷♂️ I have VS Code version 1.75.0 on Windows 10 Home latest version/build and I'm using PowerShell version 5.1.19041.2364.
Edit: The issue definitely appears to be PowerShell related as it turns out. I was finally able to run this command successfully in a regular command prompt. (Simply typing "cmd" and Enter into the PowerShell window before running the diff command). Unfortunately, I happen to be running this command as part of PowerShell script. I may have to figure out how to run a CMD command from inside my PowerShell script if that is at all possible. I'm not sure. 🤔 If not, I need to figure out what exactly PowerShell is doing to my command when it reaches the '&' character.
tl;dr
You need a workaround:
cmd /c 'code --diff "R&D (cloud conflict 2-5-23).txt" "R&D.txt"'
Alternatively, using --%, the stop-parsing token:
code --diff "R&D (cloud conflict 2-5-23).txt" --% "R&D.txt"
Note: --% comes with fundamental limitations, notably the inability to reference PowerShell variables - see this answer.
Background information:
The root cause is that code is implemented as a batch file (code.cmd) and that cmd.exe, the interpreter that executes batch file inappropriately parses its list of arguments as if they had been submitted from INSIDE a cmd.exe session.
PowerShell, which - of necessity - has to rebuild the process command line behind the scenes on Windows after having performed argument parsing based on its rules, and - justifiably - places "R&D.txt" as verbatim R&D.txt on the process command line, given that the argument value contains no spaces.
The result is that cmd.exe interprets the unquoted R&D.txt argument on its command line as containing metacharacter &, which is its command-sequencing operator, causing the call to break.
Given that cmd.exe, the legacy Windows shell, is unlikely to receive fixes, the actively maintained PowerShell (Core) 7+ edition could as a courtesy compensate for cmd.exe's inappropriate behavior.
Doing so has been proposed in GitHub issue #15143, but, alas, it looks like these accommodations will not be implemented.
My windows Powershell gives the expected results when interpreting commands such as "date" and "echo", while problems occur when it interprets such commands as "which" and "tail".
I thought it might be that I haven't add the address of these commands to the target directories, but where can I probably find these commands.
What happens when I apply "which", with the same thing happening when applying "tail"
which and tail are the names of external programs, which are unrelated to PowerShell. These programs can only be expected to be present on Unix-like platforms, not (natively) on Windows. (However, they would be present in Unix-like subsystems on Windows, such as WSL.)
By contrast, date and echo are (potentially) commands that are built into PowerShell, irrespective of what platforms it runs on:
echo is an alias of PowerShell's Write-Output cmdlet on all platforms.
date - unless an external program by that name is present in a directory listed in $env:PATH on a given platform - refers to the built in Get-Date cmdlet.
Note: This relies on PowerShell's ill-conceived default verb feature, which falls back on prefixing a command name with Get- if no command form is found by the given name. This shouldn't be relied upon, both in the interest of conceptual clarity and to avoid unnecessary overhead. Also, this fallback is (unexpectedly) not reported by the Get-Command discussed below, and also not by Get-Help - see GitHub issue #3987.
Use Get-Command to determine what command form, if any, a given name refers to (add -All to see if potentially multiple forms exist, with the effective one being listed first).
I wrote a program which prints a string, which contains ANSI escape sequences to make the text colored. But it doesn't work as expected in the default Windows 10 console, as you can see in the screenshot.
The program output appears with the escape sequences as printed characters.
If I feed that string to PowerShell via a variable or piping, the output appears as intended (red text).
How can I achieve that the program prints colored text without any workarounds?
This is my program source (Haskell) - but the language is not relevant, just so you can see how the escape sequences are written.
main = do
let red = "\ESC[31m"
let reset = "\ESC[39m"
putStrLn $ red ++ "RED" ++ reset
Note:
The following applies to regular (legacy) console windows on Windows (provided by conhost.exe), which are used by default, including when a console application is launched from a GUI application.
By contrast, the console windows (terminals) provided by Windows Terminal as well as Visual Studio Code's integrated terminal provide support for VT / ANSI escape sequences by default, for all console applications.
While console windows in Windows 10 do support VT (Virtual Terminal) / ANSI escape sequences in principle, support is turned OFF by default.
You have three options:
(a) Activate support globally by default, persistently, via the registry, as detailed in this SU answer.
In short: In registry key [HKEY_CURRENT_USER\Console], create or set the VirtualTerminalLevel DWORD value to 1
From PowerShell, you can do this programmatically as follows:
Set-ItemProperty HKCU:\Console VirtualTerminalLevel -Type DWORD 1
From cmd.exe (also works from PowerShell):
reg add HKCU\Console /v VirtualTerminalLevel /t REG_DWORD /d 1
Open a new console window for changes to take effect.
See caveats below.
(b) Activate support from inside your program, for that program (process) only, with a call to the SetConsoleMode() Windows API function.
See details below.
(c) Ad-hoc workaround, from PowerShell:
PowerShell (Core) 7+: Enclose external-program calls in (...) (invariably collects all output first before printing):
(.\test.exe)
Streaming Windows PowerShell-only alternative: Pipe output from external programs to Write-Host
.\test.exe | Out-Host
See details below.
Re (a):
The registry-based approach invariably activates VT support globally, i.e., for all console windows, irrespective of what shell / program runs in them:
Individual executables / shells can still deactivate support for themselves, if desired, using method (b).
Conversely, however, this means that the output of any program that doesn't explicitly control VT support will be subject to interpretation of VT sequences; while this is generally desirable, hypothetically this could lead to misinterpretation of output from programs that accidentally produce output with VT-like sequences.
Note:
While there is a mechanism that allows console-window settings to be scoped by startup executable / window title, via subkeys of [HKEY_CURRENT_USR\Console], the VirtualTerminalLevel value seems not to be supported there.
Even if it were, however, it wouldn't be a robust solution, because opening a console window via a shortcut file (*.lnk) (e.g. from the Start Menu or Task Bar) wouldn't respect these settings, because *.lnk files have settings built into them; while you can modify these built-in settings via the Properties GUI dialog, as of this writing the VirtualTerminalLevel setting is not surfaced in that GUI.
Re (b):
Calling the SetConsoleMode() Windows API function from inside the program (process), as shown here, is cumbersome even in C# (due to requiring P/Invoke declarations), and may not be an option:
for programs written in languages from which calling the Windows API is not supported.
if you have a preexisting executable that you cannot modify.
In that event, option (c) (from PowerShell), discussed next, may work for you.
Re (c):
PowerShell automatically activates VT (virtual terminal) support for itself when it starts (in recent releases of Windows 10 this applies to both Windows PowerShell and PowerShell (Core) 7+) - but that does not extend to external programs called from PowerShell, in either edition, as of v7.3.2.
Separately, in v7.2+ there is the $PSStyle.OutputRendering preference variable, which controls whether PowerShell commands produce colored output via the formatting system, such as the colored headers of Get-ChildItem output. However, this setting has no effect on (direct) output from external programs. $PSStyle.OutputRendering defaults to Host, meaning that only formatted output that prints to the terminal (console) is colored. $PSStyle.OutputRendering = 'PlainText' disables coloring, and $PSStyle.OutputRendering = 'Ansi' makes it unconditional; see this answer for more information.
However, as a workaround you can relay an external program's (stdout) output via PowerShell's (success) output stream, in which case VT sequences are recognized:
As of PowerShell (Core) 7.3.2, this only works either by enclosing the call in (...) or by using Out-String, but note that all output is invariably collected first before it is printed.[1]
(.\test.exe)
In Windows PowerShell, in addition to the above, streaming the relayed output is possible too, by piping to Write-Host (Out-Host, Write-Output or Out-String -Stream would work too)
.\test.exe | Write-Host
Note: You need these techniques only if you want to print to the console. If, by contrast, you want to capture the external program's output (including the escape sequences), use $capturedOutput = .\test.exe
Character-encoding caveat: Windows PowerShell by default expects output from external programs to use the OEM code page, as defined by the legacy system locale (e.g., 437 on US-English systems) and as reflected in [console]::OutputEncoding.
.NET console programs respect that setting automatically, but for non-.NET programs (e.g., Python scripts) that use a different encoding (and produce not just pure ASCII output (in the 7-bit range)), you must (at least temporarily) specify that encoding by assigning to [console]::OutputEncoding; e.g., for UTF-8:
[console]::OutputEncoding = [Text.Encoding]::Utf8.
Note that this is not only necessary for the VT-sequences workaround, but generally necessary for PowerShell to interpret non-ASCII characters correctly.
PowerShell Core (v6+), unfortunately, as of v7.3.2, still defaults to the OEM code page too, but that should be considered a bug (see GitHub issue #7233), given that it otherwise defaults to UTF-8 without BOM.
[1] Using Out-String -Stream or its built-in wrapper function, oss, is tempting in order to achieve streaming output, but this no longer works as of PowerShell 7.3.2, possibly due to the optimization implemented in GitHub PR #16612.
Thanks to user mklement0 for making me aware that VT support is not enabled automatically. This made me look in the right direction and I found this helpful post.
So to answer my question: Add or set the registry key
HKCU:\Console - [DWORD] VirtualTerminalLevel = 1
Restart the console and it works.
For Haskell specifically, you need to call the hSupportsANSIWithoutEmulation function to enable ANSI escape sequences in your program on Windows 10.
When I used the command below [1] to set my configuration variable MONGODB_URI, it gives an error [2].
I am using Windows PowerShell.
[1] >> heroku config:set MONGODB_URI='mongodb+srv://myprojectname:<mypassword>#cluster0.rkitj.mongodb.net/<myusername>?retryWrites=true&w=majority'
[2] The system cannot find the file specified.
'w' is not recognized as an internal or external command,
operable program or batch file.
Note: myprojectname, mypassword and myusername are placeholders for the actual value.
It looks like the heroku CLI entry point is a batch file, as implied by the wording of the error messages, which are cmd.exe's, not PowerShell's.
PowerShell doesn't take the special parsing needs of batch files (cmd.exe) into account when it synthesizes the actual command line to use behind the scenes, which involves re-quoting, using double quotes only, and only when PowerShell thinks quoting is needed.
In this case PowerShell does not double-quote (because the value contains no spaces), which breaks the batch-file invocation.
You have the following options:
You can use embedded quoting so as to ensure that the value part of your MONGODB_URI=... key-value pair is passed in double quotes; note the '"..."' quoting:
heroku config:set MONGODB_URI='"mongodb+srv://myprojectname:<mypassword>#cluster0.rkitj.mongodb.net/<myusername>?retryWrites=true&w=majority"'
Caveat: This shouldn't work, and currently only works because PowerShell's passing of arguments to external program is fundamentally broken as of PowerShell 7.1 - see this answer. Should this ever get fixed, the above will break.
If your command line doesn't involve any PowerShell variables and expressions, you can use --%, the stop-parsing symbol, which, however, in general, has many limitations (see this answer); essentially, everything after --% is copied verbatim to the target command line, except for expanding cmd.exe-style environment-variable references (e.g., %USERNAME%):
heroku config:set --% MONGODB_URI="mongodb+srv://myprojectname:<mypassword>#cluster0.rkitj.mongodb.net/<myusername>?retryWrites=true&w=majority"
If you're willing to install a module, you can use the ie function from the PSv3+ Native module (install with Install-Module Native from the PowerShell Gallery in PSv5+), which internally compensates for all of PowerShell's argument-passing and cmd.exe's argument-parsing quirks (it is implemented in a forward-compatible manner so that should PowerShell itself ever get fixed, the function will simply defer to PowerShell); that way, you can simply focus on meeting PowerShell's syntax requirements, and let ie handle the rest:
# 'ie' prepended to an invocation that uses only PowerShell syntax
ie heroku config:set MONGODB_URI='mongodb+srv://myprojectname:<mypassword>#cluster0.rkitj.mongodb.net/<myusername>?retryWrites=true&w=majority'
In Powershell(pwsh)
Is it possible to "bypass" Powershell to not do anything and let the underlying shell do the work?
Sometime I need to call a command, that I copy from a site or other, that includes "\" and not "´" for line break - or other parameters, that Powershell try to interpolate.
Like:
<Powershell Function> kubectl get rolebindings,clusterrolebindings \
--all-namespaces \
-o custom-columns='KIND:kind,NAMESPACE:metadata.namespace,NAME:metadata.name,SERVICE_ACCOUNTS:subjects[?(#.kind=="ServiceAccount")].name' | grep name
and let the underlying shell do the work?
PowerShell is a shell, so there is no underlying shell as such (in the sense that PowerShell needn't be launched from another shell).
However, there are platform-specific default shells: cmd.exe on Windows, and /bin/sh on Unix-like platforms.
I need to call a command, that I copy from a site or other
Before PowerShell went cross-platform, --%, the stop-parsing token was introduced in Windows PowerShell v3, precisely to address your scenario.
Given that Windows was the only supported platform at the time, the behavior is focused on cmd.exe - and even there it doesn't support cmd.exe's line continuation via line-ending ^ chars.
More generally, while it tries to emulate cmd.exe's behavior (e.g., by expanding %-enclosed tokens such as %USERNAME% as environment-variable references) without actually delegating to the latter, it doesn't do so fully.
In PowerShell Core (v6+) --% is even supported on Unix-like platforms, but is of little utility there, given that cmd.exe semantics are applied even there; notably, '...'-quoting is not supported.
(Note that PowerShell emulates one /bin/sh feature on Unix-like platforms, but only without use of --%: globbing, i.e. the expansion of unquoted tokens such as *.txt to matching file paths).
However, you can call the platform-native shell explicitly, e.g, by passing the copied command string to sh -c on Unix-like platforms, using a (verbatim) here-string for the command string:
# Call the platform-native shell with a command crafted for it:
sh -c #'
echo This line\
keeps going\
with --all-namespaces and\
with -o custom-columns='KIND:kind,NAMESPACE'
'#
Unfortunately, there's a catch:
Up to at least v7.2.x, PowerShell's handling of embedded " chars. in arguments passed to external programs (such as sh) is fundamentally broken, requiring embedded " to be manually \-escaped:
# !! BROKEN
sh -c #'
echo "high noon"
'#
To make this work, you have to \-escape manually:
# WORKAROUND
sh -c (#'
echo "high noon"
'# -replace '"', '\"')
See this answer and this GitHub issue for background information.
In short: Backward-compatibility concerns prevent fixing the default behavior, but discussions are underway to at least allow opting into sane default behavior.
Not that I'm aware of. I also don't know of any shell where this is possible, either, save for Ruby where you have the string literal operators making escapes much easier.
I would suggest that before sending strings to external commands, that you convert the PowerShell escape sequences to the target runtime escape sequences. This isn't a problem unique to PowerShell, but is easier to work with in other shells because there is more of an overlap of the special characters between other runtimes.