Using backslashes to pass a JSON string as a parameter in PowerShell is cumbersome.
executable --json-input '{ \"name\": \"Bob\" }' output.txt
Is there a way to avoid using these backslashes? I tried using single quotes, and doubles quotes in and out without any success. In Python I use triple quotes print(""" here is an example "" """) to avoid character escaping.
Is there a similar solution in PowerShell? One where we never need to worry about reformating a JSON string?
PowerShell has here-strings, similar to the multiline literals from Perl (<<<) or Python (""").
The starting quote must be preceded by # and immediately followed by a line break, whereas the closing quote must follow a newline and be followed by another # sign:
command --json-input #"
{ "name": "Bob" }
"# output.txt
The unfortunate need for manual \-escaping of " chars. embedded in string arguments passed to external programs is due to a long-standing PowerShell bug that may finally get fixed in PowerShell 7.3, though possibly on an opt-in basis - see this answer for details.
That is to say, you should be able to just pass '{ "name": "Bob" }' - no \-escaping, but the bug prevents that.
To automate this escaping for now, without having to modify a given JSON string, you can apply a regex-based -replace operation, namely $json -replace '([\\]*)"', '$1$1\"'
# Note: only needed for *external executables*, up to at least PowerShell 7.2.x
executable --json-input ('{ "name": "Bob" }' -replace '([\\]*)"', '$1$1\"') output.txt
Note:
The above replacement operation also handles escaped embedded " characters correctly.
E.g., { "name": "Nat \"King\" Cole" } becomes { \"name\": \"Nat \\\"King\\\" Cole\" }, with the \ before " properly escaped as \\
See this regex101.com page for an explanation of the regex and replacement operation (for technical reasons, the linked page uses C#'s string format, which requires escaping \ and " in the regex and substitution expression too, but the solution is equivalent to what is shown here).
If you know your JSON input not to contain them, you can simplify to -replace '"', '\"'
Related
Both of these commands doesn't create multiple line text:
Set-Content .\test.md 'Hello`r`nWorld'
Set-Content .\test.md 'Hello\r\nWorld'
Only this can
Set-Content .\test.md #("Hello`nWorld")
Do you know why is that?
Escape sequences such as `r`n only work inside "...", i.e, expandable (interpolating) strings.
By contrast, '...' strings are verbatim strings that do not interpret their contents - even ` instances are used as verbatim (literally).
Only ` (the so-called backtick) serves as the escape character in PowerShell, not \.
That is, in both "..." and '...' strings a \ is a literal.
(However, \ is the escape character in the context of regexes (regular expressions), but it is then the .NET regex engine that interprets them, not PowerShell; e.g.,
"`r" -match '\r' is $true: the (interpolated) literal CR char. matched its escaped regex representation).
As for what you tried:
It is the fact that "Hello`nWorld" in your last command is a "..." string that made it work.
By contrast, enclosing the string in #(...), the array-subexpression operator, is incidental to the solution. (Set-Content's (positionally implied) -Value parameter is array-valued anyway (System.Object[]), so even a single string getting passed is coerced to an array).
Finally, note that Set-Content by default adds a trailing, platform-native newline to the output file; use -NoNewLine to suppress that, but note that doing so also places no newline between the (string representations of) multiple input objects, if applicable (in your case there's only one).
Therefore (note the -NoNewLine and the trailing `n):
Set-Content -NoNewLine .\test.md "Hello`nWorld`n"
Optional reading: design rationale for PowerShell's behavior:
Why doesn't PowerShell use the backslash (\) as the escape character, like other languages?
Because PowerShell must (also) function on Windows (it started out as Windows-only), use of \ as the escape character - as known from Unix (POSIX-compatible) shells such as Bash - is not an option, given that \ is used as the path separator on Windows.
If \ were the escape character, you'd have to use Get-ChildItem C:\\Windows\\System32 instead of Get-ChildItem C:\Windows\System32, which is obviously impractical in a shell, where dealing with file-system paths is very common.
Thus, a different character had to be chosen, which turned out to be `, the so-called backtick: At least on US keyboards, it is easy to type (just like \), and it has the benefit of occurring rarely (as itself) in real-world strings, so that the need to escape it rarely arises.
Note that the much older legacy shell on Windows, cmd.exe, too had to pick a different character: it chose ^, the so-called caret.
Why doesn't it use single quote and double quote interchangeably, like other languages?
Different languages made different design choices, but in making "..." strings interpolating, but '...' strings not, PowerShell did follow existing languages here, namely that of POSIX-compatible shells such as Bash.
As an improvement on the latter PowerShell also supports embedding verbatim ' inside '...', escaped as '' (e.g., '6'' tall')
Given PowerShell's commitment to backward compatibility, this behavior won't change, especially given how fundamental it is to the language.
Conceptually speaking, you could argue that the aspect of what quoting character a string uses should be separate from whether it is interpolating, so that you'd be free to situationally choose one or the other quoting style for syntactic convenience, while separately controlling whether interpolation should occur.
Thus, hypothetically, PowerShell could have used a separate sigil to make a string interpolating, say $"..." and $'...' (similar to what C# now offers, though it notably only has one string-quoting style).
(As an aside: Bash and Ksh do have this syntax form, but it serves a different purpose (localization of strings) and is rarely used in pratice).
In practice, however, once you know how "..." and '...' work in PowerShell, it isn't hard to make them work as intended.
See this answer for a juxtaposition of PowerShell, cmd.exe, and POSIX-compatible shells with respect to fundamental features.
I've found plenty of post explaining how to literally escape both single and double quotation marks using either """" for one double quotation mark, or '''' for a single quotation mark (or just doing `"). I find myself in a situation where I need to search through a list of names that is input in a different query:
Foreach($Username in $AllMyUsers.Username){
$Filter = "User = '$Username'"
# do some more code here using $Filter
}
The problem occurs when I reach a username like O'toole or O'brian which contains quotation marks. If the string is literal I could escape it with
O`'toole
O''brian
etc.
But, since it's in a loop I need to escape the quotation mark for each user.
I tried to use [regex]::Escape() but that doesn't escape quotation marks.
I could probably do something like $Username.Replace("'","''") but it feels like there should be a more generic solution than having to manually escape the quotation marks. In other circumstances I might need to escape both single and double, and just tacking on .Replace like so $VariableName.Replace("'","''").Replace('"','""') doesn't feel like it's the most efficient way to code.
Any help is appreciated!
EDIT: This feels exactly like a "how can I avoid SQL injection?" question but for handling strings in Powershell. I guess I'm looking for something like mysqli_real_escape_string but for Powershell.
I could probably do something like $Username.Replace("'","''") but it feels like there should be a more generic solution than having to manually escape the quotation marks
As implied by Mathias R. Jessen's comment, there is no generic solution, given that you're dealing with an embedded '...' string, and the escaping requirements entirely depend on the ultimate target API or utility - which is unlikely to be PowerShell itself (where ' inside a '...' string must be doubled, i.e. represented as '').
In the case at hand, where you're passing the filter string to a System.Data.DataTable's .DataView's .RowFilter property, '' is required as well.
The conceptually cleanest way to handle the required escaping is to use -f, the format operator, combined with a separate string-replacement operation:
$Filter = "User = '{0}'" -f ($UserName -replace "'", "''")
Note how PowerShell's -replace operator - rather than the .NET [string] type's .Replace() method - is used to perform the replacement.
Aside from being more PowerShell-idiomatic (with respect to: syntax, being case-insensitive by default, accepting arrays as input, converting to strings on demand), -replace is regex-based, which also makes performing multiple replacements easier.
To demonstrate with your hypothetical .Replace("'","''").Replace('"','""') example:
#'
I'm "fine".
'# -replace '[''"]', '$0$0'
Output:
I''m ""fine"".
In Bash, what are the differences between single quotes ('') and double quotes ("")?
Single quotes won't interpolate anything, but double quotes will. For example: variables, backticks, certain \ escapes, etc.
Example:
$ echo "$(echo "upg")"
upg
$ echo '$(echo "upg")'
$(echo "upg")
The Bash manual has this to say:
3.1.2.2 Single Quotes
Enclosing characters in single quotes (') preserves the literal value of each character within the quotes. A single quote may not occur between single quotes, even when preceded by a backslash.
3.1.2.3 Double Quotes
Enclosing characters in double quotes (") preserves the literal value of all characters within the quotes, with the exception of $, `, \, and, when history expansion is enabled, !. The characters $ and ` retain their special meaning within double quotes (see Shell Expansions). The backslash retains its special meaning only when followed by one of the following characters: $, `, ", \, or newline. Within double quotes, backslashes that are followed by one of these characters are removed. Backslashes preceding characters without a special meaning are left unmodified. A double quote may be quoted within double quotes by preceding it with a backslash. If enabled, history expansion will be performed unless an ! appearing in double quotes is escaped using a backslash. The backslash preceding the ! is not removed.
The special parameters * and # have special meaning when in double quotes (see Shell Parameter Expansion).
The accepted answer is great. I am making a table that helps in quick comprehension of the topic. The explanation involves a simple variable a as well as an indexed array arr.
If we set
a=apple # a simple variable
arr=(apple) # an indexed array with a single element
and then echo the expression in the second column, we would get the result / behavior shown in the third column. The fourth column explains the behavior.
#
Expression
Result
Comments
1
"$a"
apple
variables are expanded inside ""
2
'$a'
$a
variables are not expanded inside ''
3
"'$a'"
'apple'
'' has no special meaning inside ""
4
'"$a"'
"$a"
"" is treated literally inside ''
5
'\''
invalid
can not escape a ' within ''; use "'" or $'\'' (ANSI-C quoting)
6
"red$arocks"
red
$arocks does not expand $a; use ${a}rocks to preserve $a
7
"redapple$"
redapple$
$ followed by no variable name evaluates to $
8
'\"'
\"
\ has no special meaning inside ''
9
"\'"
\'
\' is interpreted inside "" but has no significance for '
10
"\""
"
\" is interpreted inside ""
11
"*"
*
glob does not work inside "" or ''
12
"\t\n"
\t\n
\t and \n have no special meaning inside "" or ''; use ANSI-C quoting
13
"`echo hi`"
hi
`` and $() are evaluated inside "" (backquotes are retained in actual output)
14
'`echo hi`'
`echo hi`
`` and $() are not evaluated inside '' (backquotes are retained in actual output)
15
'${arr[0]}'
${arr[0]}
array access not possible inside ''
16
"${arr[0]}"
apple
array access works inside ""
17
$'$a\''
$a'
single quotes can be escaped inside ANSI-C quoting
18
"$'\t'"
$'\t'
ANSI-C quoting is not interpreted inside ""
19
'!cmd'
!cmd
history expansion character '!' is ignored inside ''
20
"!cmd"
cmd args
expands to the most recent command matching "cmd"
21
$'!cmd'
!cmd
history expansion character '!' is ignored inside ANSI-C quotes
See also:
ANSI-C quoting with $'' - GNU Bash Manual
Locale translation with $"" - GNU Bash Manual
A three-point formula for quotes
If you're referring to what happens when you echo something, the single quotes will literally echo what you have between them, while the double quotes will evaluate variables between them and output the value of the variable.
For example, this
#!/bin/sh
MYVAR=sometext
echo "double quotes gives you $MYVAR"
echo 'single quotes gives you $MYVAR'
will give this:
double quotes gives you sometext
single quotes gives you $MYVAR
Others explained it very well, and I just want to give something with simple examples.
Single quotes can be used around text to prevent the shell from interpreting any special characters. Dollar signs, spaces, ampersands, asterisks and other special characters are all ignored when enclosed within single quotes.
echo 'All sorts of things are ignored in single quotes, like $ & * ; |.'
It will give this:
All sorts of things are ignored in single quotes, like $ & * ; |.
The only thing that cannot be put within single quotes is a single quote.
Double quotes act similarly to single quotes, except double quotes still allow the shell to interpret dollar signs, back quotes and backslashes. It is already known that backslashes prevent a single special character from being interpreted. This can be useful within double quotes if a dollar sign needs to be used as text instead of for a variable. It also allows double quotes to be escaped so they are not interpreted as the end of a quoted string.
echo "Here's how we can use single ' and double \" quotes within double quotes"
It will give this:
Here's how we can use single ' and double " quotes within double quotes
It may also be noticed that the apostrophe, which would otherwise be interpreted as the beginning of a quoted string, is ignored within double quotes. Variables, however, are interpreted and substituted with their values within double quotes.
echo "The current Oracle SID is $ORACLE_SID"
It will give this:
The current Oracle SID is test
Back quotes are wholly unlike single or double quotes. Instead of being used to prevent the interpretation of special characters, back quotes actually force the execution of the commands they enclose. After the enclosed commands are executed, their output is substituted in place of the back quotes in the original line. This will be clearer with an example.
today=`date '+%A, %B %d, %Y'`
echo $today
It will give this:
Monday, September 28, 2015
Since this is the de facto answer when dealing with quotes in Bash, I'll add upon one more point missed in the answers above, when dealing with the arithmetic operators in the shell.
The Bash shell supports two ways to do arithmetic operation, one defined by the built-in let command and the other the $((..)) operator. The former evaluates an arithmetic expression while the latter is more of a compound statement.
It is important to understand that the arithmetic expression used with let undergoes word-splitting, pathname expansion just like any other shell commands. So proper quoting and escaping need to be done.
See this example when using let:
let 'foo = 2 + 1'
echo $foo
3
Using single quotes here is absolutely fine here, as there isn't any need for variable expansions here. Consider a case of
bar=1
let 'foo = $bar + 1'
It would fail miserably, as the $bar under single quotes would not expand and needs to be double-quoted as
let 'foo = '"$bar"' + 1'
This should be one of the reasons, the $((..)) should always be considered over using let. Because inside it, the contents aren't subject to word-splitting. The previous example using let can be simply written as
(( bar=1, foo = bar + 1 ))
Always remember to use $((..)) without single quotes
Though the $((..)) can be used with double quotes, there isn't any purpose to it as the result of it cannot contain content that would need the double quote. Just ensure it is not single quoted.
printf '%d\n' '$((1+1))'
-bash: printf: $((1+1)): invalid number
printf '%d\n' $((1+1))
2
printf '%d\n' "$((1+1))"
2
Maybe in some special cases of using the $((..)) operator inside a single quoted string, you need to interpolate quotes in a way that the operator either is left unquoted or under double quotes. E.g., consider a case, when you are tying to use the operator inside a curl statement to pass a counter every time a request is made, do
curl http://myurl.com --data-binary '{"requestCounter":'"$((reqcnt++))"'}'
Notice the use of nested double quotes inside, without which the literal string $((reqcnt++)) is passed to the requestCounter field.
There is a clear distinction between the usage of ' ' and " ".
When ' ' is used around anything, there is no "transformation or translation" done. It is printed as it is.
With " ", whatever it surrounds, is "translated or transformed" into its value.
By translation/ transformation I mean the following:
Anything within the single quotes will not be "translated" to their values. They will be taken as they are inside quotes. Example: a=23, then echo '$a' will produce $a on standard output. Whereas echo "$a" will produce 23 on standard output.
A minimal answer is needed for people to get going without spending a lot of time as I had to.
The following is, surprisingly (to those looking for an answer), a complete command:
$ echo '\'
whose output is:
\
Backslashes, surprisingly to even long-time users of bash, do not have any meaning inside single quotes. Nor does anything else.
I am trying to execute a powershell command to copy text to the windows clipboard including carriage returns and ALL special characters. I can execute the command ok using:
powershell.exe -command Set-Clipboard 'TEXT'
This is not in the powershell console directly so syntax differs.
I was using double quotes around text, substituting carriage returns in original text with `r`n and escaping all other special characters with `
It worked up until I got to a single ' which I understand is used by powershell to mean a literal text string.
So I changed approaches and wrapped un-escaped text in single quotes (except substituting 1 ' for 2 ''). Of course `r`n within the single quoted text are interpreted literally so doesn't work. I have tried stringing them together outside single quoted text like:
'some text here' "`r`n" 'more text here'
This works in the console but not in the command. Tried adding + either side but still does not work.
User "TessellatingHeckler" suggested -EncodedCommand but unfortunately I am unable to produce any version of base 64 encoded strings (to include in the command) which match the same string encoded via the PS console. So that is not going to work.
I have been attempting to simply substitute carriage returns in the original text with an obscure string, wrap the text in single quotes (literal) and then substitute it back to `r`n within PS. I have gotten the substitution to work in the console directly but cannot figure out how to actually send it as a command.
powershell.exe -command Set-Clipboard $Str = 'this is a test--INSERT_CRLF_HERE--1234'; $Car = '--INSERT_CRLF_HERE--'; $clr = "`r`n"; $Str = $Str -replace $Car, $clr
Can the above command be modified to work? Is it possible to achieve the intended outcome without writing to a temp file? It is preferable to be able to use single quoted text blocks as it is more robust and lightweight than trying to escape everything (even spaces) in the original text.
I was informed about a rather tidy solution by Rob Simmers on another forum, which I am currently employing.
Original Text:
Test text
!##$%^&*()_+-=[]\{}|;':",./<>?
String with 5x characters substituted ({ = {{, } = }}, ' = '', crlf = {0}, " = {1}):
Test text{0}!##$%^&*()_+-=[]\{{}}|;'':{1},./<>?
Powershell.exe command:
powershell.exe -command "$str = 'Test text{0}!##$%^&*()_+-=[]\{{}}|;'':{1},./<>?' -f [System.Environment]::NewLine, [Char] 0x22 ; Set-Clipboard $str"
Output (literal text placed on the clipboard) - same as the input, as desired:
Test text
!##$%^&*()_+-=[]\{}|;':",./<>?
As an aside: A fully robust solution that works in any invocation scenario would indeed require use of -EncodedCommand with a string that is the Base64 encoding of the command string's UTF16-LE byte representation - but you've stated that creating such a string is not an option for you.
If you were to call from PowerShell, you could more simply use a script block (see bottom).
Update: The OP's own answer now contains a robust, if nontrivial, solution based on careful string substitutions.
The answer below may still be of interest for simpler scenarios and for background information on quoting requirements and pitfalls.
Using the scenario from your question, this simpler solution should do (verified from cmd.exe - we still don't know where you're calling PowerShell from, but I expect it to work if there's no shell involved):
powershell.exe -command "Set-Clipboard \"this is`r`na test\""
As for other special characters:
' can be embedded as-is - no escaping needed
" requires escaping as `\"
$ as `$, but only if you want it to be treated as a literal (the same escaping that would apply in a regular double-quoted PowerShell string)
If your use case requires passing an arbitrary preexisting string, you'd have to employ string substitution to perform the above escaping - including replacing embedded newlines with literal `r`n.
If there is no shell involved (such as with subprocess.check_output() from Python), the above rules should suffice and make for a robust solution (assuming you only use printable characters and there are no character-encoding issues).
From cmd.exe, however, a fully robust solution that doesn't use -EncodedCommand requires extra, nontrivial work, due to its lack of proper parsing of embedded double quotes:
The following cmd.exe metacharacters typically require ^-escaping, but sometimes the mustn't be escaped:
& | < >
In the following example, & requires ^-escaping:
powershell.exe -command "Set-Clipboard \"this is`r`na ^& test\""
However, if your string also has embedded (and escaped) " chars., whether these characters require^-escaping depends on their placement relative to the embedded "; note how in the following example & need not be ^-escaped and indeed must not be, because the ^ would then become part of the string:
powershell.exe -command "Set-Clipboard \"this is`r`na 3`\" & test\""
Anticipating these variations algorithmically is a nontrivial undertaking.
Additionally, if your string had %...% tokens that look like cmd.exe-style environment variables - e.g., %FOO% - but you want to treat them as literals, the % cannot be escaped.
There is a workaround that may work, but it again depends on the presence and placement of embedded " chars.
In the following example, the "^ disrupter" trick can be used to prevent expansion of %OS%:
powershell.exe -command "Set-Clipboard \"do not expand: %^OS%\""
However, if an embedded " precedes it, the workaround becomes ineffective (the ^ is retained), and there's no direct fix that I know of in this scenario:
powershell.exe -command "Set-Clipboard \"do not expand: 3`\" %^OS%\""
You'd have to split the string into multiple pieces to break the %OS% token apart and concatenate the pieces via a PowerShell expression:
powershell.exe -command "Set-Clipboard (\"do not expand: 3`\" %\" + \"OS%\")"
Algorithmically, you could use placeholder chars. that you then replace as part of the PowerShell command, a technique you've used in your own answer.
As an aside:
Extra escaping would be required if you wanted to execute this from PowerShell:
powershell.exe -command "Set-Clipboard \`"this is`r`na test\`""
However, from PowerShell it's not hard to construct a Base64-encoded string to pass to
-EncodedCommand, but there's an even easier method: use a script block to pass your command; no extra quoting requirements apply in that case, because -EncodedCommand is then automatically used behind the scenes (along with -OutputFormat Xml).
Do note that this only works from within PowerShell.
powershell.exe -command { Set-Clipboard "this is`r`na test" }
I have a file containing some properties which value of some of them contains escape characters, for example some Urls and Regex patterns.
When reading the content and converting back to the json, with or without unescaping, the content is not correct. If I convert back to json with unescaping, some regular expression break, if I convert with unescaping, urls and some regular expressions will break.
How can I solve the problem?
Minimal Complete Verifiable Example
Here are some simple code blocks to allow you simply reproduce the problem:
Content
$fileContent =
#"
{
"something": "http://domain/?x=1&y=2",
"pattern": "^(?!(\\`|\\~|\\!|\\#|\\#|\\$|\\||\\\\|\\'|\\\")).*"
}
"#
With Unescape
If I read the content and then convert the content back to json using following command:
$fileContent | ConvertFrom-Json | ConvertTo-Json | %{[regex]::Unescape($_)}
The output (which is wrong) would be:
{
"something": "http://domain/?x=1&y=2",
"pattern": "^(?!(\|\~|\!|\#|\#|\$|\||\\|\'|\")).*"
}
Without Unescape
If I read the content and then convert the content back to json using following command:
$fileContent | ConvertFrom-Json | ConvertTo-Json
The output (which is wrong) would be:
{
"something": "http://domain/?x=1\u0026y=2",
"pattern": "^(?!(\\|\\~|\\!|\\#|\\#|\\$|\\||\\\\|\\\u0027|\\\")).*"
}
Expected Result
The expected result should be same as the input file content.
I decided to not use Unescape, instead replace the unicode \uxxxx characters with their string values and now it works properly:
$fileContent =
#"
{
"something": "http://domain/?x=1&y=2",
"pattern": "^(?!(\\`|\\~|\\!|\\#|\\#|\\$|\\||\\\\|\\'|\\\")).*"
}
"#
$fileContent | ConvertFrom-Json | ConvertTo-Json | %{
[Regex]::Replace($_,
"\\u(?<Value>[a-zA-Z0-9]{4})", {
param($m) ([char]([int]::Parse($m.Groups['Value'].Value,
[System.Globalization.NumberStyles]::HexNumber))).ToString() } )}
Which generates the expected output:
{
"something": "http://domain/?x=1&y=\\2",
"pattern": "^(?!(\\|\\~|\\!|\\#|\\#|\\$|\\||\\\\|\\'|\\\")).*"
}
If you don't want to rely on Regex (from #Reza Aghaei's answer), you could import the Newtonsoft JSON library. The benefit is the default StringEscapeHandling property which escapes control characters only. Another benefit is avoiding the potentially dangerous string replacements you would be doing with Regex.
This StringEscapeHandling is also the default handling of PowerShell Core (version 6 and up) because they started to use Newtonsoft internally since then. So another alternative would be to use ConvertFrom-Json and ConvertTo-Json from PowerShell Core.
Your code would look something like this if you import the Newtonsoft JSON library:
[Reflection.Assembly]::LoadFile("Newtonsoft.Json.dll")
$json = Get-Content -Raw -Path file.json -Encoding UTF8 # read file
$unescaped = [Newtonsoft.Json.Linq.JObject]::Parse($json) # similar to ConvertFrom-Json
$escapedElementValue = [Newtonsoft.Json.JsonConvert]::ToString($unescaped.apiName.Value) # similar to ConvertTo-Json
$escapedCompleteJson = [Newtonsoft.Json.JsonConvert]::SerializeObject($unescaped) # similar to ConvertTo-Json
Write-Output "Variable passed = $escapedElementValue"
Write-Output "Same JSON as Input = $escapedCompleteJson"
Note:
Applying [regex]::Unescape() isn't called for, as JSON's escaping is unrelated to regex escaping.
That is, $fileContent | ConvertFrom-Json | ConvertTo-Json should work as-is, but doesn't due to a quirk in Windows PowerShell, which caused the & in your input string to be represented as its equivalent escape sequence on re-conversion, \u0026; the quirk similarly affects ' (\u0026), < (\u003c) and > (\u003e).
tl;dr
The problem does not affect PowerShell (Core) 6+ (the install-on-demand, cross-platform PowerShell edition), which uses a different implementation of the ConvertTo-Json and ConvertFrom-Json cmdlets, namely, as of PowerShell 7.2.x, one based on Newtonsoft.JSON (whose direct use is shown in r3verse's answer). There, your sample roundtrip command works as expected.
Only ConvertTo-Json in Windows PowerShell is affected (the bundled-with-Windows PowerShell edition whose latest and final version is 5.1). But note that the JSON representation - while unexpected - is technically correct.
A simple, but robust solution focused only on unescaping those Unicode escape sequences that ConvertTo-Json unexpectedly creates - namely for & ' < > - while ruling out false positives:
# The following sample JSON with undesired Unicode escape sequences for `& < > '`
# was created with Windows PowerShell's ConvertTo-Json as follows:
# ConvertTo-Json "Ten o'clock at <night> & later. \u0027 \\u0027"
$json = '"Ten o\u0027clock at \u003cnight\u003e \u0026 later. \\u0027 \\\\u0027"'
[regex]::replace(
$json,
'(?<=(?:^|[^\\])(?:\\\\)*)\\u(00(?:26|27|3c|3e))',
{ param($match) [char] [int] ('0x' + $match.Groups[1].Value) },
'IgnoreCase'
)
The above outputs the desired JSON representation, without the unnecessary escaping of &, ', <, and >, and without having falsely replaced the escaped substrings \\u0027 and \\\\u0027:
"Ten o'clock at <night> & later. \\u0027 \\\\u0027"
Background information:
ConvertTo-Json in Windows PowerShell unexpectedly represents the following ASCII-range characters by their Unicode escape sequences in JSON strings:
& (Unicode escape sequence: \u0026)
' (\u0027)
< and > (\u003c and \u003e)
There's no good reason to do so (these characters only require escaping in HTML/XML text).
However, any compliant JSON parser - including ConvertFrom-Json - converts these escape sequences back to the characters they represent.
In other words: While the JSON text created by Windows PowerShell's ConvertTo-Json is unexpected and can impede readability, it is technically correct and - while not identical - equivalent to the original representation in terms of the data it represents.
Fixing the readability problem:
As an aside: While [regex]::Unescape(), whose purpose is to unescape regexes only, also converts Unicode escape sequences to the characters they represent, it is fundamentally
unsuited to selectively unescaping Unicode sequences JSON strings, given that all other \ escapes must be preserved in order for the JSON string to remain syntactically valid.
While your answer works well in general, it has limitations (aside from the easily corrected problem that a-zA-Z should be a-fA-F to limit matching to those letters that are valid hex. digits):
It doesn't rule out false positives, such as \\u0027 or \\\\u0027 (\\ escapes \, so that the u0027 part becomes a verbatim string and must not be treated as an escape sequence).
It converts all Unicode escape sequences, which presents two problems:
Escape sequences representing characters that require escaping would also be converted to the verbatim character representations, which would break the JSON representations with \u005c, for instance, given that the character it represents, \, requires escaping.
For non-BMP Unicode characters that must be represented as pairs of Unicode escape sequences (so-called surrogate pairs), your solution would mistakenly try to unescape each half of the pair separately.
For a robust solution that overcomes these limitations, see this answer
(surrogate pairs are left as Unicode escape sequences, Unicode escape sequences
whose characters require escaping are converted to \-based (C-style) escapes, such as \n, if possible).
However, if the only requirement is to unescape those Unicode escape sequences
that Windows PowerShell's ConvertTo-Json unexpectedly creates, the solution at the top is sufficient.