I'm trying to pass a JSON string from within a powershell script to the build.phonegap.com api, using curl.
According to phonegap's forum, when running on a Windows machine, the JSON data has to be formatted as:
curl.exe -ku user#email:mypass -X PUT -d "data={\"password\":\"keypass\"}" https://build.phonegap.com/api/v1/key
Indeed, this does run fine when invoked from the command line.
However, when I try to invoke this from within a powershell script, the double quotes seem to be stripped.
So far, I have tried:
Putting the JSON in single quoted string:
curl.exe -ku user#email:mypass -X PUT -d '"data={\"password\":\"keypass\"}"' https://build.phonegap.com/api/v1/key
Putting the JSON in single quoted string, without the DOS escape backslashes:
curl.exe -ku user#email:mypass -X PUT -d '"data={"password":"keypass"}"' https://build.phonegap.com/api/v1/key
Putting the JSON in single quoted string, escaping the double quotes and backslashes (DOS style with a backslash):
curl.exe -ku user#email:mypass -X PUT -d '\"data={\\\"password\\\":\\\"keypass\\\"}\"' https://build.phonegap.com/api/v1/key
Putting the JSON in a double quoted string, escaping the double quotes with the powershell backtick character (`):
curl.exe -ku user#email:mypass -X PUT -d "`"data={\`"password\`":\`"build*2014`\`"}`"" https://build.phonegap.com/api/v1/key
Any idea how to achieve this?
Thanks for your time,
Koen
Try using the --% operator to put PowerShell into simple (dumb) argument parsing mode:
curl.exe --% -ku user#email:mypass -X PUT -d "data={\"password\":\"keypass\"}" https://build.phonegap.com/api/v1/key
This is quite often useful for invoking exes with argument syntax that runs afoul of PowerShell's argument syntax. This does require PowerShell V3 or higher.
Update:
PowerShell 7.3.0 mostly fixed the problem, with selective exceptions on Windows, though a future version may requires opt-in - see this answer for details.
For cross-version, cross-edition code, the Native module discussed below may still be of interest.
tl;dr:
Up to at least PowerShell 7.2.x, manual \-escaping of " characters inside PowerShell strings is - unfortunately - required.
# From inside PowerShell:
# Note the outer '...' quoting and the unexpected need to escape
# the embedded " chars. as \" (unexpected, because PowerShell itself doesn't
# require " inside '...' to be escaped; also, PowerShell's escape char. is `).
# If outer "..." quoting must be used, use \`" (sic) to escape the embeded "
curl.exe -ku user#email:mypass -X PUT -d 'data={\"password\":\"keypass\"}' https://build.phonegap.com/api/v1/key
Read on for why that is necessary, and about a potential future fix.
PowerShell's escape character is ` (the so-called backtick), so in order to embed " characters in a "..." (double-quoted, interpolating) string, use `" (or "") rather than \"; by contrast, inside a '...' (single-quoted, verbatim) string, " need not be escaped.
In your attempt, PowerShell didn't see \" as an escaped " and therefore saw multiple "..." strings, which ultimately - when PowerShell of necessity applied its on demand re-quoting behind the scenes, passed two separate string arguments that individually didn't need double-quoting, due to not containing spaces, namely: verbatim data={\ and password\:\keypass\}
Using PowerShell's quoting rules, you should have used:
either:
"data={`"password`":`"keypass`"}"
or, more simply, given that no string interpolation is needed, via a verbatim, single-quoted string, inside of which " chars. don't require escaping:
'data={"password":"keypass"}'
Unfortunately, however, as of PowerShell 7.2.x this is NOT enough, though the experimental PSNativeCommandArgumentPassing feature available since PowerShell Core 7.2.0-preview.5 may fix this at least for some external programs, including curl.exe; read on for details.
As of PowerShell 7.2.x, an unexpected extra layer of escaping of embedded " characters is needed, using \-escaping when calling (most) external programs:
In Windows PowerShell there are edge cases where this approach doesn't work, in which case use of --% is required (see below). Notably, escaping '"foo bar"' as '\"foo bar\"' doesn't work, due to the enclosing \" being at the very start and end of the string - see this answer for details.
Also, some external programs on Windows understand ""-escaping only (e.g. msiexec); for them, use -replace '"', '""' in order to programmatically perform the extra escaping, assuming the value contains at least one space. Do the same for programs that do not support embedded " chars. at all (WSH), so that the embedded " at least do not break argument boundaries (but they will be stripped).
For programs that expect \"-escaping, use the following -replace operation to robustly perform the extra escaping programmatically:
'...' -replace '([\\]*)"', '$1$1\"'
If the input string contains no preexisting verbatim \" sequences, you can simplify to '...' -replace '"', '\"'
# Note: Escaping the embedded " chars. as `" is enough for PowerShell itself,
# but, unfortunately, not when calling *external programs*.
# The `-replace` operation performs the necessary additional \-escaping.
$passwd = 'foo'
curl.exe -ku user#email:mypass -X PUT -d (
"data={`"password`": `"$passwd`"}" -replace '([\\]*)"', '$1$1\"'
) https://build.phonegap.com/api/v1/key
This shouldn't be required, but is due to a bug since v1 that hasn't been fixed for fear of breaking backward compatibility - see this answer.
A - presumably - opt-in fix is now being considered for some future version, post v7.2 - see GitHub issue #14747 - and using it, once available, would obviate the need for the manual escaping.
Since PowerShell Core 7.2.0-preview.5, experimental feature PSNativeCommandArgumentPassing with an attempted fix is available, but, unfortunately, it looks like it will lack important accommodations for high-profile CLIs on Windows (though curl.exe wouldn't be affected) - see this summary from GitHub issue #15143.
A backward- and forward-compatible helper function is the ie function from the Native module (Install-Module Native), which obviates the need for the extra escaping, contains important accommodations for high-profile CLIs on Windows, and will continue to work as expected even with the opt-in fix in place:
ie curl.exe ... -d "data={`"password`": `"$passwd`"}" ... )
Using --%, the stop-parsing symbol, as in Keith Hill's answer is a suboptimal workaround that also doesn't require the extra \-escaping, however:
--% has inherent limitations - see GitHub docs issue #6149 - and is virtually useless on Unix-like platforms - see GitHub docs issue #4963.
The only - awkward and side effect-producing - way to embed PowerShell variable values in the arguments following --% is to (a) define them as environment variables (e.g., $env:passwd = 'foo') and (b) to reference these variables cmd.exe-style, even on Unix (e.g., %passwd%).
An alternative workaround - especially if you need to include the values of PowerShell variables or expressions in your calls - is to call via cmd /c with a single argument containing the entire command line; for quoting convenience, the following example uses a here-string (see the bottom section of this answer for an overview of PowerShell's string literals):
# Use #"<newline>...<newline>"# if you need to embed PowerShell variables / expressions.
cmd /c #'
curl.exe -ku user#email:mypass -X PUT -d "data={\"password\":"\keypass\"}" https://build.phonegap.com/api/v1/key
'#
Here's how I did manage to send a json request in PowerShell 7.2
Finally found the solution. Istead of using one " use 3 of them ("""), and thats it. So it would be:
data={"""password""":"""keypass"""}
Set the content type:
curl -H "Content-Type: application/json" -d '{"password":"keypass"}' https://build.phonegap.com/api/v1/key
Related
I know that both are used in PowerShell but for different contexts.
On the internet there is a paucity of information on this topic and the only site that talks about it (without making me understand the concept) is:
https://www.rlmueller.net/PowerShellEscape.htm
I am a beginner with PowerShell, I am recently approaching it.
A use case of the \ escape came up in the answer to this other topic of mine:
PowerShell removes multiple consecutive whitespaces when I pass arguments to a nested Start-Process command
Is there anyone who can explain to me in detail the difference between the escape backtick ` and backslash \ in PowerShell, with examples and use cases?
At least one source is welcome, but it is not mandatory.
vonPryz's helpful answer covers the PowerShell-internal angle well; let me attempt a systematic summary that includes the PowerShell CLI angle, as well as passing arguments with embedded " to external programs:
Inside a PowerShell session, the only escape character is ` (the so-called backtick), in the following contexts:
Inside an expandable string ("...", double quoted), but not inside a verbatim string ('...', single-quoted); for the supported escape sequences, see the conceptual about_Special_Characters help topic:
# " must be escaped; escape sequence `n expands to a newline.
"3`" of`nrain"
In unquoted command arguments:
# > must be escaped to write verbatim 'a>b',
# since outside a quoted string an unescaped > is a redirection.
Write-Output a`>b
For line-continuations:
# Spread the Get-Date call across two lines.
# Important: ` must be the *very last char* on the line.
Get-Date `
-Year 2525
Note: Various subsystems, whether PowerShell-specific or not, may have their own escaping rules, such as \ in regular expressions and ` in wildcard expressions. Since arguments to those subsystems are delivered via PowerShell strings, it's best to use verbatim string literals, so as to avoid confusion between PowerShell's own string interpolation and what the target subsystem ends up seeing; e.g. 'A $50 fine.' -match '\$50' (\ is needed to treat regex metacharacter $ literally).
When PowerShell is called from the outside, via its CLI, different rules apply, possibly in addition:
In order to adhere to widely used convention for CLIs (command-line interfaces, programs that accept arguments via a command line) on Windows:
In calls to powershell.exe, the Windows PowerShell CLI, " characters must be escaped with a backslash - i.e. as \" - in order to be preserved during parsing of the original command line.
pwsh.exe, the CLI of the cross-platform, install-on-demand PowerShell (Core) 7+ edition, now commendably alternatively accepts ""[1] in lieu of \", which makes calls from cmd.exe more robust. To get the same robustness in Windows PowerShell - from cmd.exe only - use "^"" (sic).
Note that - unlike \" - these escape sequences only work inside an (unescaped) "..." string (e.g., pwsh.exe -c " ""you & I"" " or powershell.exe -c " "^""you & I"^"" "
By contrast, unescaped " have syntactic function on the command line and tell PowerShell where the boundaries between arguments are; these " instances are removed during command-line parsing.
This ensures that outside callers that merely want to invoke a PowerShell script file (.ps1) with arguments, using the -File parameter, can use the conventional syntax and needn't special-case calls to PowerShell's CLI.
However, if you pass a string containing PowerShell code to the CLI, using the -Command parameter, what PowerShell ends up interpreting obviously must be syntactically valid PowerShell code.
Caveat: If you specify neither -Command nor -File:
powershell.exe defaults to -Command
pwsh.exe now defaults to -File
For the differences between -File and -Command calls and when to use which, see this answer.
If you use -Command, there are two, sequential parsing stages:
The command-line parsing stage, where syntactic (unescaped) " are removed, and escaped \" (or "") turn into literal ".
The result of this stage is then parsed as PowerShell code, as it would be from inside a PowerShell session.
Therefore, you may situationally have to combine \ and `-escaping; e.g. (call from cmd.exe):
C:>powershell.exe -Command " \"3`\" of snow as of $(Get-Date)\" "
3" of snow as of 11/04/2021 14:13:41
Note the use of `\" in order to make PowerShell see `", i.e. a properly escaped " inside a "..." string, after command-line parsing.
Alternatively, depending on the specifics of the command(s) you pass to -Command, using embedded '...' quoting may be an option, which simplifies matters, because ' chars. don't require escaping:
C:>powershell.exe -Command " '3\" of snow as of today.' "
3" of snow as of today.
Given that '...' strings in PowerShell are verbatim strings, use of '...' is only an option if you don't require string interpolation (such as the $(Get-Date) subexpression in the prior example).
Escaping " when calling external programs from PowerShell:
As a shell, it is PowerShell's job to pass the arguments that were passed based on its syntax rules to a target executable, so that the verbatim values that result from PowerShell's parsing are passed in a way that makes the target executable see them as such. In other words: PowerShell should perform any required escaping automatically, behind the scenes. (Unlike cmd.exe, PowerShell cannot just pass its own argument syntax through as-is, because external CLIs cannot be expected to understand '...' strings (single-quoting) or `-escaping).
To use a simply example: Passing '3" of snow' should be passed as "3\" of snow" behind the scenes, based on the most widely used escaping convention.
Sadly, up to PowerShell 7.2.x, this is not the case, and embedded " characters in arguments for external programs must additionally, manually be \-escaped in order to be passed through properly.
This has been fixed in PowerShell 7.3, with selective exceptions on Windows - see the new $PSNativeCommandArgumentPassing preference variable
# Broken behavior up to PS v7.2.x
PS> cmd /c echo '{ "foo": "bar" }'
"{ "foo": "bar" }" # !! Embedded " aren't escaped.
PS> choice.exe /d Y /t 0 /m '{ "foo": "bar" }'
{ foo: bar } [Y,N]?Y # !! When parsed by an executable,
# !! embedded " are effectively LOST.
# Manual escaping required.
PS> choice.exe /d Y /t 0 /m '{ \"foo\": \"bar\" }'
{ "foo": "bar" } [Y,N]?Y # OK
This bug has existed since v1, and has never been fixed so as to avoid breaking existing workarounds.
[1] Inside PowerShell, in "..." strings only, you may also use "" to escape an embedded ", as an alternative to `"
The backtick ` is used as the escape character in Powershell. That is, escaping quotes, tabs and whatnot. Unlike in many other environments, Powershell's newline is `n, not \n. This has the benefit of simplifying paths, since those use backslash in Windows.
As for a practical example, in many programming languages one needs to escape the backslashes in paths. If you have an application located at
c:\program files\newApplication\myApp.exe
Its path must be written as
c:\\program files\\newApplication\\myApp.exe
The double-backslash notation means that there is actaully a backslash, not a metacharacter such as tab or newline. Note that the path contains \newApplication. Now, \n usually means a newline, but it clearly isn't the case. The file does not reside in
c:\program files\
ewApplication\myApp.exe
after all, doesn't it?
The escaping is not needed in Powershell, as the backslash doesn't have special meaning by itself. When Powershell sees c:\program files\newApplication\myApp.exe, it doesn't assign any special meaning to the forementioned \n part, it is just a string literal.
The backslash \ is used as the escape character in regular expressions and to signify metacharacters. That is, to match a literal * in regex, it must be escaped, lest it means the Kleene star (zero or more matches).
In pwsh call the following:
Write-Host '{"drop_attr": "name"}'
Result ok:
{"drop_attr": "name"}
Now do the same via pwsh:
pwsh -Command Write-Host '{"drop_attr": "name"}'
Result is missing quotation marks and square brackets?
drop_attr: name
Update:
PowerShell 7.3.0 mostly fixed the problem, with selective exceptions on Windows, and it seems that in some version after 7.3.1 the fix will require opt-in - see this answer for details.
For cross-version, cross-edition code, the Native module discussed at the bottom may still be of interest.
Unfortunately, PowerShell's handling of passing arguments with embedded " chars. to external programs - which includes PowerShell's own CLI (pwsh) - is fundamentally broken (and always has been), up to at least PowerShell 7.2.x:
You need to manually \-escape " instances embedded in your arguments in order for them to be correctly passed through to external programs (which happens to be PowerShell in this case as well):
# Note: The embedded '' sequences are the normal and expected
# way to escape ' chars. inside a PowerShell '...' string.
# What is *unexpected* is the need to escape " as \"
# even though " can normally be used *as-is* inside a '...' string.
pwsh -Command ' ''{\"drop_attr\": \"name\"}'' '
Note that I'm assuming your intent is to pass a JSON string, hence the inner '' ... '' quoting (escaped single quotes), which ensures that pwsh ultimately sees a single-quoted string ('...'). (No need for an explicit output command; PowerShell implicitly prints command and expression output).
Another way to demonstrate this on Windows is via the standard choice.exe utility, repurposed to simply print its /m (message) argument (followed by verbatim [Y,N]?Y):
# This *should* preserve the ", but doesn't as of v7.2
PS> choice /d Y /t 0 /m '{"drop_attr": "name"}'
{drop_attr: name} [Y,N]?Y # !! " were REMOVED
# Only the extra \-escaping preserves the "
PS> choice /d Y /t 0 /m '{\"drop_attr\": \"name\"}'
{"drop_attr": "name"} [Y,N]?Y # OK
Note that from inside PowerShell, you can avoid the need for \-escaping, if you call pwsh with a script block ({ ... }) - but that only works when calling PowerShell itself, not other external programs:
# NOTE: Works from PowerShell only.
pwsh -Command { '{"drop_attr": "name"}' }
Background info on PowerShell's broken handling of arguments with embedded " in external-program calls, as of PowerShell 7.2.1:
This GitHub docs issue contains background information.
GitHub issue #1995 discusses the problem and the details of the broken behavior as well as manual workarounds are summarized in this comment; the state of the discussion as of PowerShell [Core] 7 seems to be:
A fix is being considered as an experimental feature, which may become an official feature, in v7.3 at the earliest. Whether it will become a regular feature - i.e whether the default behavior will be fixed or whether the fix will require opt-in or even if the feature will become official at all - remains to be seen.
Fixing the default behavior would substantially break backward compatibility; as of this writing, this has never been allowed, but a discussion as to whether to allow breaking changes in the future and how to manage them has begun: see GitHub issue #13129.
See GitHub PR #14692 for the relevant experimental feature, which, however, as of this writing is missing vital accommodations for batch files and msiexec-style executables on Windows - see GitHub issue #15143.
In the meantime, you can use the PSv3+ ie helper function from the Native module (in PSv5+, install with Install-Module Native from the PowerShell Gallery), which internally compensates for all broken behavior and allows passing arguments as expected; e.g.,
ie pwsh -Command ' ''{"drop_attr": "name"}'' ' would then work properly.
Another way. Are you in Windows or Unix?
pwsh -c "[pscustomobject]#{drop_attr='name'} | convertto-json -compress"
{"drop_attr":"name"}
Another way is to use "encoded commands".
> $cmd1 = "Write-Host '{ ""description"": ""Test program"" }'"
> pwsh -encoded ([Convert]::ToBase64String([Text.Encoding]::Unicode.GetBytes($cmd1)))
{ "description": "Test program" }
Let's say I have the variable $password = Get-Passwd ACME\bob where I get the password of a given user. The password contains all sorts of special characters including $, that is used by PowerShell.
I need to dynamically use the password in the following command:
cmdkey.exe /add:$hostname /user:"$user" /pass:"`"$password`""
I need to escape the " character on both sides so the command is parsed as pass:"123qwe" and not pass:123qwe. The issue is that it breaks when the password includes a $ character. How can I pass the password to this command without breaking it?
Running Executables in PowerShell
For most executables, it's not necessary to manually quote the parameters in PowerShell. Just write the command using any variables, and the PowerShell parser will automatically quote the arguments for the program's command line. This usually "just works."
cmdkey.exe /add:hostname /user:username /pass:$pass
You can inspect the actual command line that PowerShell passes to an executable by using a small program I wrote called showargs.exe, available at the following page:
https://www.itprotoday.com/management-mobility/running-executables-powershell
Example:
showargs cmdkey.exe /add:hostname /user:username /pass:$pass
The showargs.exe program simply echoes its command line to standard output so you can see the literal command line that PowerShell actually passes to the executable.
An embedded $ character shouldn't be a problem. If that character is in a PowerShell string variable, PowerShell will pass it along as a part of the command line string. Example:
$pass = 'my$pass'
cmdkey.exe /add:hostname /user:username /password:$pass
No matter how you pass the command line to the program, it's important to note that the interpretation of that command line is up to that individual program. (For example, if an executable's parser doesn't support certain characters, no amount of quoting or parsing will allow you to work around the limitation.)
Cmdkey.exe Uses a Non-Standard Parser
In my prefunctory testing of of cmdkey.exe, it seems it does not have a way of "escaping" the " character on its command line. Since this seems to be the case, you will not be able to use cmdkey.exe to store a credential that contains an embedded " character.
Embedding a Space in a Cmdkey.exe Command Line Argument
Because cmdkey.exe uses a non-standard command-line parser, you can't use a variable on its command line that contains embedded spaces. For example:
PS C:\> $pass = "my pass"
PS C:\> showargs cmdkey.exe /add:hostname /user:username /password:$pass
cmdkey.exe /add:hostname /user:username "/password:my pass"
The "/password:my pass" evidently confuses the cmdkey.exe parser, so we have to work around this behavior by bypassing PowerShell's default parsing behavior. The simplest way to do this is by escaping the quotes around the argument containing the space:
PS C:\> showargs.exe cmdkey.exe /add:hostname /user:username /password:`"$pass`"
cmdkey.exe /add:hostname /user:username /password:"my pass"
In any case, you can use showargs.exe to diagnose the trouble and work out a solution appropriate to the executable you need to run.
tl;dr
Your command should work - except if $password contains " chars.
Embedded $ chars., by contrast, should not be a problem.
Your /pass:"`"$password`"" technique - i.e., explicit, embedded double-quoting - also handles values with embedded spaces correctly, unlike the /pass:$password technique (also) suggested in Bill Stewart's helpful answer.
You can simplify the command by omitting the outer "..." quoting, as also suggested in Bill's answer:
/pass:`"$password`"
Caveat: If PowerShell's argument-passing worked correctly, these techniques shouldn't work and if it ever gets fixed, such techniques will stop working - see this answer for background.
As for supporting " chars. in values: even \-escaping them doesn't always work, namely when spaces are also involved - see details below.
The issue is that it breaks when the password includes a $ character.
Your command passes any $ characters embedded in the value of $password through to the target program as-is.
Therefore, if there is a problem, the implication is that your target program - cmdkey.exe - interprets $ characters, but note that the docs make no mention of that.
If it indeed does, however, you would have to escape $ characters as required by the target program in order to pass them literally.
Caveat: There is a fundamental limitation, however:
Commands typically break if the argument value contains spaces AND embedded " chars, whether the latter are properly escaped for the target program or not.
Normally, you'd escape value-internal " as \" so as not to break the enclosing double-quoting of the value:
# !! Only works if:
# !! * $password contains NO "
# !! * $password contains " but NOT ALSO SPACES
# !! * $password contains PAIRS of " and what is between each
# !! pair does not contain spaces
cmdkey.exe /add:$hostname /user:"$user" /pass:`"$($password -replace '"', '\"')`"
Note:
Escaping embedded " as \" is a not a standard per se, but it is recognized by most external programs; the only notable exceptions are batch files - see this answer for details.
Arguably, PowerShell should handle this escaping automatically - see this GitHub issue for details.
If Windows PowerShell thinks it cannot pass the resulting token as-is as a single argument to the target program, it blindly applies double-quoting around the entire token[1], which, in combination with the escaped ", can result in invalid syntax:
E.g., if $password literally contains a " b and is manually escaped as a \" b by the command above, PowerShell ends up passing the following behind the scenes:
... "/pass:"a \" b""
That is, the resulting literal token that PowerShell saw - /pass:"a \" b" - was blindly enclosed in double quotes, as a whole, even though most target programs would parse /pass:"a \" b" correctly as-is.
As a result, the explicitly provided double-quoting is invalidated (as it would then require another level of escaping) - and short of using --%, the stop-parsing symbol, which then limits you to literals (or %...%-style environment-variable references), there is no way around that.
If the target program recognizes an argument if it is double-quoted as a whole (e.g.,
"/pass:a b" instead of /pass:"a b"), you can omit the explicit double-quoting around the argument value.
Do note, however, that some target programs - including cmdkey.exe and notably msiexec - do not recognize the argument if it is double-quoted as a whole.
... /pass:$($password -replace '"', '\"')
With $password literally containing a " b, PowerShell then passes behind the scenes:
... "/pass:a \" b"
This is syntactically valid - for target programs that recognize \" as an escaped ", which is the norm - but, as stated, the fact that the entire argument is enclosed in `"...", and not just the value may not be supported by the target program.
[1] Windows PowerShell ignores any \-escaping in the resulting token and considers all embedded " to have syntactic function: From that perspective, if the token is not composed of any mix of directly concatenated unquoted and double-quoted strings, enclosing double-quoting is blindly applied - which may break the command.
This behavior is not only obscure, it prevents robust, predictable parameter passing.
PowerShell Core now recognizes \" as as escaped; however, quotes that aren't pre-escaped as \" still result in broken quoting; e.g., 'a "b c" d' is passed as "a "b c" d", which target programs parse as 2 arguments, a b and c d (after quote removal).
I need to use \r\n inside of ffmpeg --header parameter. This works in Unix, but not in Windows command prompt. So I was wondering to use powershell
powershell c:\ffmpeg -headers 'User-Agent: user-agent'"`r`n"'Cookies: my-cookie' ...
I understand I have to use 'string' when special characters are used and
"`r`n"
as my \r\n delimiter.
I have also tested that I can mix them together, like 'this '"is "'text' to get this is text
However if my string (cookie or user-agent) contains & character, it fails.
Example: powershell c:\ffmpeg -headers 'Cookies: param=a=1&b=2; session=123'
How can I "escape" & character in one-liner?
These examples (some parts are masked) are accepted by CMD, but they are not working
powershell -c "c:\ffplay -user_agent ""Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0"" -headers ""Cookie: nlqptid=h=676886edeea5bae904b0cf53daec8038.1519940538670&hdnea=exp=1519940598~acl=/*~hmac=C0E019502B060D23AB02BB157FCFFC72404500770A2CE5B00789A84AAEFBD77F&nltid=xxxxxxxxxx&nltdt=0&uid=744826&csid=xxxxxxxxxx; hdntl=exp=1520026938~acl=%2f*~hmac=952689e6de57a2a201ddc1d4c0794962fdc886ea48cf41494a42e787ef923bf9`r`n"" -i ""https://xxxxxxxxxx.net/nlds_vod/xxxxxxxxxx/vod/2018/02/28/cd264e10-dd54-3de2-df90-4c1cac104dcb/v1/stream/cd264e10-dd54-3de2-df90-4c1cac104dcb_1_4500_pc.mp4.m3u8"""
or
powershell -c "c:\ffplay -headers ""User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0`r`nCookie: nlqptid=h=676886edeea5bae904b0cf53daec8038.1519940538670&hdnea=exp=1519940598~acl=/*~hmac=C0E019502B060D23AB02BB157FCFFC72404500770A2CE5B00789A84AAEFBD77F&nltid=xxxxxxxxxx&nltdt=0&uid=744826&csid=xxxxxxxxxx; hdntl=exp=1520026938~acl=%2f*~hmac=952689e6de57a2a201ddc1d4c0794962fdc886ea48cf41494a42e787ef923bf9`r`n"" -i ""https://xxxxxxxxxx.net/nlds_vod/xxxxxxxxxx/vod/2018/02/28/cd264e10-dd54-3de2-df90-4c1cac104dcb/v1/stream/cd264e10-dd54-3de2-df90-4c1cac104dcb_1_4500_pc.mp4.m3u8"""
ffplay says
An input file must be specified
But the input file is specified after -i parameter.
What is wrong with this command?
Note that the implementation is a part of the existing batch script, therefore it is required to use a syntax working inside of batch file.
When calling from cmd.exe, things get tricky; here's a simplified example:
powershell -c "ffmpeg.exe -h "^""User-Agent: usra`r`nCookies: parm=a=1&b=2; session=1"^"""
In short: From cmd.exe / batch files, to pass what PowerShell's CLI should see as a double-quoted argument inside an overall "..." string passed to -Command (-c)[1] (which is powershell.exe's default parameter):
\"...\" is normally the preferred form of escaping, because it works in both PowerShell editions and also aligns with how most CLIs expect " to be escaped; however, it doesn't work in all cases when calling from cmd.exe, such as the case at hand.
By contrast, \"...\" is always safe to use from no-shell contexts, such as Task Scheduler and the Windows Run dialog (WinKey-R).
To also handle all edge cases of calling the PowerShell CLI from cmd.exe / batch files - namely when the string to be quoted contains cmd.exe metacharacters such as &, such as in your case - use the following in lieu of \"...\":
When calling Windows PowerShell (powershell.exe): use "^""..."^"" (sic), as shown above.
Caveat: This doesn't work for capturing the output in a variable via a for /f loop: If whitespace normalization (runs of two or more spaces becoming a single space each) isn't a concern (it rarely is) use \""...\"" (sic) inside a for /f loop. In the rare event that it is a concern, use " ^^^"\"...\"" (sic) - see the bottom section of this answer.
When calling PowerShell (Core) v6+ (pwsh.exe): use ""...""
As a general requirement, literal % chars. must be escaped as %% to prevent them from being interpreted as part of a cmd.exe variable reference in batch files. From the command prompt, things are unfortunately more complicated.
The command line for PowerShell is best passed as a single, "..."-enclosed string, via parameter -c (short for -Command, which in Windows PowerShell is the default, but that has changed in PowerShell (Core) v6+, which now defaults to -File).
Since the PowerShell CLI strips unescaped " characters during command-line parsing, before interpreting the result as PowerShell code, any " instances to be retained as part of the command to ultimately execute must be escaped (note that PowerShell-internally `" is used to escape a "; alternatively, in the context of "..." strings only, "" may be used).
"^""..."^"" (Windows PowerShell) and ""..."" (PowerShell (Core) v6+) inside an overall "..." -c argument ensure that cmd.exe itself interprets ... as being inside a double-quoted string, which is what makes these escaping forms robust.
If \"...\" were used inside "..." from cmd.exe (which only recognizes "" as an escaped "), it would in effect see ... as being outside a double-quoted string, which would cause values that contain cmd.exe metacharacters such as & and | to break the command. Compare the following invocations from cmd.exe:
# OK - prints verbatim: The king of "Rock & Roll"
C:\>powershell.exe -c " 'The king of "^""Rock & Roll"^""' "
C:\>pwsh.exe -c " 'The king of ""Rock & Roll""' "
# !! BROKEN - cmd.exe interprets "&" as a metachar.
C:\>powershell.exe -c " 'The king of \"Rock & Roll\"' "
C:\>pwsh.exe -c " 'The king of \"Rock & Roll\"' "
Note: You could fix the use of \" by ^-escaping such cmd.exe characters (e.g., ^&), but that requires you to analyze the string carefully and determine which specific parts cmd.exe sees as unquoted. If you make a mistake, the ^ will be retained as a literal part of the string. The advantage of the "^"" / "" approach is that you needn't worry about such pitfalls.
As an aside: 'this '"is "'text' does not create a single string the way it does in Bash; in PowerShell:
As a stand-alone expression, it causes a syntax error (try executing it by itself); you'd have to use ('this ' + "is " + 'text') or use a single quoted string ('this is text').
As an argument passed to a command (program), it is interpreted as 3 distinct arguments - see this answer for an explanation of this - surprising - behavior.
[1] Using overall "..." quoting around what then becomes a single -Command argument is conceptually preferable, but not strictly necessary. If overall "..." quoting is not used, \"-escaping of " characters that are to be treated as part of the command is the only supported form - see this answer.
I am using Ack version 2.04, in PowerShell. I want to search texts like "jsonClass":"Page" (quotes included) inside text files.
I can't seem to get the quoting and escaping correct.
ack -c --match '"jsonClass":"Page"'
It doesn't work in PowerShell. I guess ack's picking up the single quotes as well.
Escaping the double quotes gives an invalid regex error:
ack -c --match "\"jsonClass\":\"Page\""
Invalid regex '\':
Trailing \ in regex m/\/ at C:\CHOCOL~1\lib\ACK2~1.04\content\ack.pl line 315
I tried the literal option as well, but I think ack's interpreting the colon as file parameters.
ack -c -Q --match "jsonClass":"Page"
ack.pl: :Page: No such file or directory
What am I missing?
I am using PowerShell v2.
To complement JPBlanc's effective answer with a PowerShell v3+ solution:
When invoking external programs such as ack
, use of the so-called stop-parsing symbol, --%, makes PowerShell pass the remaining arguments through as-is, with the exception of expanding cmd.exe-style environment-variable references such as %PATH%:
ack --% -c --match "\"jsonClass\":\"Page\""
This allows you to focus on the escaping rules of the target program only, without worrying about the complex interplay with PowerShell's own parsing and escaping.
Thus, in PowerShell v3 or higher, the OP's own 2nd solution attempt would have worked by passing --% as the first parameter.
See Get-Help about_Parsing.
Note that the exact equivalent of the above command without the use of --% (that also works in PSv2 and is generally helpful if you want to include PowerShell-expanded variables / expressions in other arguments) would look like this:
ack -c --match '"\"jsonClass\":\"Page\""'
That is, the entire argument to be passed as-is is enclosed in single quotes, which ensures that PowerShell doesn't interpret it.
Note the inner enclosing " that aren't present in JPBlanc's answer (as of this writing). They guarantee that the argument is ultimately seen as a single argument by ack, even if it contains whitespace.
If ack is a function it could happen that -c is interpreted by PowerShell. Can you test the following?
ack '-c' '--match' '"jsonClass":"Page"'
-c --match should be interpreted as --match is the value of parameter c in function ask.
If it's an EXE file, try:
ack -c --match '\"jsonClass\":\"Page\"'
In Powershell you can always use ' quote to handle inner " without problems.
But if you need to use " for a string (to interpolate a variable, for instance), you can escape them with `
$myvar = 'test'
$formatted = "{`"value`":`"$myvar`"}"
Write-Host $formatted
# output: {"value":"test"}