How do you properly escape quotes in powershell v2 (called from within a batch file)?
I have tried:
powershell -Command "(gc file1.txt) -join "`n" | Out-File file2.txt"
and
powershell -Command "(gc file1.txt) -join ""`n"" | Out-File file2.txt"
and
powershell -Command "(gc file1.txt) -join '"`n`" | Out-File file2.txt"
but they all fail.
Editor's note: The purpose of the command is to transform Windows CRLF line breaks to Unix LF-only ones, so as to create a file that will be processed on Linux.
From a batch file (cmd.exe), you must \-escape embedded " instances
(even though PowerShell-internally it is ` (the backtick) that serves as the escape character):
As wOxxOm points out in a comment on the question, in Windows PowerShell using """ to embed a single " is also an option.
However, given that most command-line utilities support \", \" is easier to remember. However, both """ and \" can break on the cmd.exe side, in which case "^"" (sic) is required ("" in PowerShell (Core) 7+).
powershell -Command "(gc file1.txt) -join \"`n\" | Set-Content -NoNewLine file2.txt"
Note:
Set-Content -NoNewline requires PSv5+.
Set-Content writes "ANSI"-encoded files by default (e.g., based on code page Windows-1252 on US-English systems); use the -Encoding parameter to change that.
Since you're only dealing with strings, Set-Content is preferable to Out-File, which is only needed if you have non-string objects that must have PowerShell's default formatting applied to them first.
Consider using powershell -NoProfile ... to suppress loading of PowerShell's profile files, both for faster execution and for a more predictable execution environement.
PSv2 solution:
Unfortunately, prior to PSv5, only the Write-Host cmdlet supports the -NoNewline parameter (introduced in v2), which is of no help here, so the .NET framework must be used:
powershell -Command "[IO.File]::WriteAllText(\"$PWD/file2.txt\", ((gc $PWD/file1.txt) -join \"`n\") + \"`n\")"
Note the need to use path prefix $PWD explicitly, because the .NET Framework's current directory typically differs from PowerShell's.
Also, the output file's encoding will be UTF-8, without a BOM, but you can pass a different encoding as the 3rd argument to [IO.File]::WriteAllText(), such as [System.Text.Encoding]::Default to match Set-Content's default behavior (as of Windows PowerShell v5.1).
Optional reading: platform-specific line breaks "`n" vs. "`r`n" vs. [Environment]::Newline
Implicit choice of newlines (line breaks):
when reading, PowerShell accepts LF-only (Unix-style) and CRLF (Windows-style) newlines (line breaks) interchangeably.
when writing (e.g., when sending an array of lines / objects to a file with > / Out-File / Set-Content), PowerShell uses the platform-appropriate newline sequence.
Note, however, that any newline sequences embedded in a given string input object are sent to the file as-is.
As for escape sequences / constants:
[Environment]::Newline contains the platform-appropriate newline.
"`n" is always just LF (\n)
as evidenced by "`n".Length returning 1 and [char] 10 -eq [char] "`n" returning $True (10 is the decimal Unicode/ASCII code point of LF).
The documentation isn't explicit about this: Get-Help about_Special_Characters mentions "new line" and "line break", without mentioning what specific character [sequence] that represents. (As mentioned, LF by itself is just as valid a line break as CRLF is in PowerShell).
Therefore, to create CRLF sequences, you must use "`r`n".
If you need to match newlines in either format, you can use regex '\r?\n', with operators such as -match, -replace, and -split
As for multi-line string literals in scripts (including here-documents):
They reflect their source file's newline style. That is, if your script uses CRLF newlines, so will the newlines embedded in the multi-line string.
Here's one way to do it from the PowerShell command line:
(Get-Content Input.txt -Raw) -replace "`r`n","`n" | Out-File Output.txt -Encoding ASCII -NoNewline
As others have noted this is a PowerShell v5 solution (-Raw appeared in v3, and -NoNewline appeared in v5).
Here's a PowerShell v2 version of the same thing:
$content = [IO.File]::ReadAllText("C:\Path\Input.txt") -replace "`r`n","`n"
[IO.File]::WriteAllText("C:\Path\Output.txt", $content)
(The paths are needed because the .NET methods don't use PowerShell's "current location".)
Related
I am looking for a search string inside a json file:
> type .\input.json
[
{"name": "moish"},
{"name": "oren"}
]
> type .\input.json | findstr /n /l "\`"name\`": \`"or"
2: {"name": "moish"},
3: {"name": "oren"}
How come moish entry is found? what am I missing?
Note: The quoted lines below, originally a direct part of the answer, turned out not to apply to the problem at hand, because the escaping in the question is correct (the only thing missing was placing /c: directly before the string to make findstr.exe search for it as a whole).
See this answer for a more comprehensive analysis of the problem.
Escape the literal quotation marks by doubling them:
type input.json |findstr /n /l """name"": ""or"
... or use single-quotes to qualify the search term:
type input.json |findstr /n /l '"name": "or'
.... or perhaps use the native PowerShell cmdlet Select-String instead of findstr:
Select-String -LiteralPath input.json -Pattern '"name": "or'
Prepend /c: to your search string in order to make findstr treat it as a single string to search for:
Get-Content .\input.json | findstr /n /l /c:"\`"name\`": \`"or" # Note the /c:
Note the use of the Get-Content cmdlet for reading a file line by line, which type is a built-in alias for in PowerShell.
Note:
By default, if a search string contains spaces, findstr searches for the occurrence of any of the space-separated words, i.e., "name" or "or, causing both lines to match. /c: signals that the string as a whole string should be searched for (either as a regular expression, by default, or as a literal string, with /l)
Except for the missing /c:, your search string was correct, but you could have simplified by using a verbatim (single-quoted) string ('...'):
... | findstr /n /l /c:'\"name\": \"or'
Sadly, the additional \-escaping of the embedded " chars. is a requirement either way, up to at least PowerShell 7.2.x, even though it shouldn't be necessary.
It is due to a long-standing bug in how PowerShell passes arguments with embedded double-quotes to external programs; a - possibly opt-in - fix may be coming - see this answer.
If you're using a 7.2.x version or a 7.3 preview version of PowerShell with the experimental feature named PSNativeCommandArgumentPassing enabled and the $PSNativeCommandArgumentPassing preference variable set to either 'Standard' or 'Windows', the \-escaping is no longer needed, because PowerShell then (finally) does it for you; that is, findstr /n /l /c:'"name": "or' would suffice.
PowerShell alternative: Select-String:
As shown in Mathias R. Jessen's answer, you may alternatively use the Select-String cmdlet, the more powerful PowerShell counterpart to findstr.exe, not least to avoid quoting headaches (see above) and potential character-encoding issues.
Like findstr.exe, Select-String uses regular expressions by default; use -SimpleMatch to opt for literal matching.
Unlike findstre.exe, Select-String is case-insensitive by default (as PowerShell generally is). Use -CaseSensitive to make matching case-sensitive.
Select-String wraps matching lines in objects that include metadata about each match; if you're interested in the line text only, use -Raw in PowerShell (Core) 7+, or pipe to ForEach-Object Line in Windows PowerShell.
While piping lines read from a file via Get-Content works, it is much slower than the passing the file path as an argument directly to Select-String, via its -LiteralPath parameter (you may also pipe file-info objects obtained with Get-ChildItem to it).
This has the added advantage that the display representation of the matching lines includes the file name an line number (see below).
Thus, the equivalent of your (corrected) findstr.exe call is:
Select-String -LiteralPath .\input.json -CaseSensitive -SimpleMatch -Pattern '"name": "or'
# Alternative:
Get-ChildItem .\input.json |
Select-String -CaseSensitive -SimpleMatch -Pattern '"name": "or'
You'll get the following output in the console (note the output-line prefix consisting of the file name and the line number):
input.json:3: {"name": "oren"}
Note that this is the for-display representation of the object of type [Microsoft.PowerShell.Commands.MatchInfo] that Select-String emitted for the matching line.
I am using a script in git bash, which performs few curl calls to HTTP endpoints expecting and producing protobuf.
The curl-output is piped to a custom proto2json.exe file and finally the result is saved to a JSON file:
#!/bin/bash
SCRIPT_DIR=$(dirname $0)
JSON2PROTO="$SCRIPT_DIR/json2proto.exe"
PROTO2JSON="$SCRIPT_DIR/proto2json.exe"
echo '{"key1":"value1","version":3}' | $JSON2PROTO -v 3 > request.dat
curl --insecure --data-binary #request.dat --output - https://localhost/protobuf | $PROTO2JSON -v 3 > response.json
The script works well and now I am trying to port it to Powershell:
$SCRIPT_DIR = Split-Path -parent $PSCommandPath
$JSON2PROTO = "$SCRIPT_DIR/json2proto.exe"
$PROTO2JSON = "$SCRIPT_DIR/proto2json.exe"
#{
key1 = value1;
version = 3;
} | ConvertTo-Json | &$JSON2PROTO -v 3 > request.dat
Unfortunately, when I compare the generated binary files in "git bash" and in Powershell, then I see that the latter file has additionaly zero bytes entered.
Is the GitHub issue #1908 related to my issue?
It looks like you're ultimately after this:
$SCRIPT_DIR = Split-Path -parent $PSCommandPath
$JSON2PROTO = "$SCRIPT_DIR/json2proto.exe"
$PROTO2JSON = "$SCRIPT_DIR/proto2json.exe"
# Make sure that the output from your $JSON2PROTO executable is correctly decoded
# as UTF-8.
# You may want to restore the original encoding later.
[Console]::OutputEncoding = [System.Text.Utf8Encoding]::new()
# Capture the output lines from calling the $JSON2PROTO executable.
# Note: PowerShell captures a *single* output line as-is, and
# *multiple* ones *as an array*.
[array] $output =
#{
key1 = value1;
version = 3;
} | ConvertTo-Json | & $JSON2PROTO -v 3
# Filter out empty lines to extract the one line of interest.
[string] $singleOutputLineOfInterest = $output -ne ''
# Write a BOM-less UTF-8 file with the given text as-is,
# without appending a newline.
[System.IO.File]::WriteAllText(
"$PWD/request.dat",
$singleOutputLineOfInterest
)
As for what you tried:
In PowerShell, > is an effective alias of the Out-File cmdlet, whose default output character encoding in Windows PowerShell is "Unicode" (UTF-16LE) - which is what you saw - and, in PowerShell (Core) 7+, BOM-less UTF8. To control the character encoding, call Out-File or, for text input, Set-Content with the -Encoding parameter.
Note that you may also have to ensure that an external program's output is first properly decoded, which happens based on the encoding stored in [Console]::OutputEncoding - see this answer for more information.
Note that you can't avoid these decoding + re-encoding steps in PowerShell as of v7.2.4, because the PowerShell pipeline currently cannot serve as a conduit for raw bytes, as discussed in this answer, which also links to the GitHub issue you mention.
Finally, note that both Out-File and Set-Content by default append a trailing, platform-native newline to the output file. While -NoNewLine suppresses that, it also suppresses newlines between multiple input objects, so you may have to use the -join operator to manually join the inputs with newlines in the desired format, e.g. (1, 2) -join "`n" | Set-Content -NoNewLine out.txt
If, in Windows PowerShell, you want to create UTF-8 files without a BOM, you can't use a file-writing cmdlet and must instead use .NET APIs directly (PowerShell (Core) 7+, by contrast, produces BOM-less UTF-8 files by default, consistently). .NET APIs do and always have created BOM-less UTF-8 files by default; e.g.:
[System.IO.File]::WriteAllLines() writes the elements of an array as lines to an output file, with each line terminated with a platform-native newline, i.e. CRLF (0xD 0xA) on Windows, and LF (0xA) on Unix-like platforms.
[System.IO.File]::WriteAllText() writes a single (potentially multi-line) string as-is to an output file.
Important: Always pass full paths to file-related .NET APIs, because PowerShell's current location (directory) usually differs from .NET's.
I'd like to escape \ to \\ in csv file to upload to Redshift.
Following simple PowerShell script can replace $TargetWord \ to $ReplaceWord \\ , as expected, but export utf-8 with bom and sometimes causes the Redshift copy error.
Any advice would be appreciated to improve it. Thank you in advance.
Exp_Escape.ps1
Param(
[string]$StrExpFile,
[string]$TargetWord,
[string]$ReplaceWord
)
# $(Get-Content "$StrExpFile").replace($TargetWord,$ReplaceWord) | Set-Content -Encoding UTF8 "$StrExpFile"
In PowerShell (Core) 7+, you would get BOM-less UTF-8 files by default; -Encoding utf8 and -Encoding utf8NoBom express that default explicitly; to use a BOM, -Encoding utf8BOM is needed.
In Windows PowerShell, unfortunately, you must use a workaround to get BOM-less UTF-8, because -Encoding utf8 only produces UTF-8 files with BOM (and no other utf8-related values are supported).
The workaround requires combining Out-String with New-Item, which (curiously) creates BOM-less UTF-8 files by default even in Windows PowerShell:
Param(
[string]$StrExpFile,
[string]$TargetWord,
[string]$ReplaceWord
)
$null =
New-Item -Force $StrExpFile -Value (
(Get-Content $StrExpFile).Replace($TargetWord, $ReplaceWord) | Out-String
)
Note:
$null = is needed to discard the output object that New-Item emits (which is a file-info object describing the newly created files.
-Force is needed in order to quietly overwrite an existing file by the same name (as Set-Content and Out-File do by default).
The -Value argument must be a single (multi-line) string to write to the file, which is what Out-String ensures.
Caveats:
For non-string input objects, Out-String creates the same rich for-display representations as Out-File and as you would see in the console by default.
New-Item itself does not append a trailing newline when it writes the string to the file, but Out-String curiously does; while this happens to be handy here, it is generally problematic, as discussed in GitHub issue #14444.
The alternative to using Out-String is to create the multi-line string manually, which is a bit more cumbersome ("`n" is used to create LF-only newlines, which PowerShell and most programs happily accept even on Windows; for platform-native newlines (CRLF) on Windows, use [Environment]::NewLine instead):
$null =
New-Item -Force $StrExpFile -Value (
((Get-Content $StrExpFile).Replace($TargetWord, $ReplaceWord) -join "`n`") + "`n"
)
Since the entire file content must be passed as an argument,[1] it must fit into memory as a whole; the convenience function discussed next avoids this problem.
For a convenience wrapper function around Out-File for use in Windows PowerShell that creates BOM-less UTF-8 files, see this answer.
Alternative, with direct use of .NET APIs:
.NET APIs produce BOM-less UTF-8 files by default.
However, because .NET's working directory usually differs from PowerShell's, full file paths must always be used, which requires more effort:
# In order for .NET API calls to work as expected,
# file paths must be expressed as *full, native* paths.
$OutDir = Split-Path -Parent $StrExpFile
if ($OutDir -eq '') { $OutDir = '.' }
$strExpFileFullPath = Join-Path (Convert-Path $OutDir) (Split-Path -Leaf $StrExpFile)
# Note: .NET APIs create BOM-less UTF-8 files *by default*
[IO.File]::WriteAllLines(
$strExpFileFullPath,
(Get-Content $StrExpFile).Replace($TargetWord, $ReplaceWord)
)
The above uses the System.IO.File.WriteAllLines method.
[1] Note that while New-Item technically supports receiving the content to write to the file via the pipeline, it unfortunately writes each to the target file, successively, with only the last one ending up in the file.
When you include something like [1] in a file name like File[1].txt used with the Invoke-WebRequest and the -OutFile parameter you get an error Cannot perform operation because the wildcard path File[1].txt did not resolve to a file.
This is caused by the behavior documented here.
With other cmdlets you would use -LiteralPath to force the path to be taken literally but in this case that is not an option.
I have tried escaping the [ and ] characters with ` or \ but it still gives the same error.
To simplify testing you can reproduce the same issue with Out-File, Test-Path, etc.
#Fails
Out-File -FilePath "file[1].txt"
Out-File -FilePath "file`[1`].txt"
Out-File -FilePath "file\[1\].txt"
#Succeeds
Out-File -LiteralPath "file[1].txt"
#Fails
Test-Path -Path "file[1].txt"
#Succeeds
Test-Path -LiteralPath "file[1].txt"
How can I escape characters that would be used to express wildcards in -Path, -FilePath, -OutFile, etc. so that they function like the string was specified with -LiteralPath since -LiteralPath isn't available with Invoke-WebRequest?
Update:
In PowerShell (Core) 7.1+, file paths passed to the -OutFile parameter of Invoke-WebRequest and Invoke-RestMethod are now interpreted literally:
That is, -OutFile now acts like -LiteralPath,[1] and there is no longer a need to escape [ and ] characters, so that the following example command works as-is:
# PowerShell 7.1+ only.
Invoke-WebRequest http://example.org -OutFile File[1].txt
Therefore, the following applies only to Windows PowerShell (and to now-obsolete PowerShell (Core) versions v7.0 and below):
Escaping the [ and ] characters as `[ and `] so that they are treated literally when interpreted as a wildcard expression with -Path (-FilePath) and -OutFile unfortunately only half works at the moment, due to a bug discussed in the bottom section:
Performing the escaping ensures that the target parameter accepts the path (the command doesn't break anymore) ...
... but on creation of the file it is mistakenly the escaped representation that is used as the literal filename - see bottom section.
Workaround for now:Tip of the hat to hashbrown for helping to simplify it.
Make Invoke-RestMethod / Invoke-WebRequest save to a temporary file...
... and then rename (move) the temporary file to the desired output file path.
# Literal output file path.
$outFile = '.\file[1].txt'
# Simulate a call to Invoke-RestMethod / Invoke-WebRequest -OutFile.
# Save to a *temporary file*, created on demand - such
# a temporary file path can be assumed to never contain '[' or ']'
'hi' | Out-File -FilePath ($tempFile = New-TemporaryFile)
# Rename (move) the temporary file to the desired target path.
Move-Item -Force -LiteralPath $tempFile -Destination $outFile
In Windows PowerShell v4-, use [IO.Path]::GetTempfileName() in lieu of New-TemporaryFile.
Escaping [literal] paths for use as wildcard patterns:
Use any of the following string-literal representations, which ultimately result in the same string with verbatim content file`[1`].txt, which, when interpreted as a wildcard expression, is the escaped equivalent of literal string file[1].txt:
'file`[1`].txt'
"file``[1``].txt"
file``[1``].txt
To create this escaping programmatically, use:
$literalName = 'file[1].txt'
$escapedName = [WildcardPattern]::Escape($literalName) # -> 'file`[1`].txt'
What matters is that the target cmdlet sees the [ and ] as `-escaped in the -Path (-FilePath) argument it is passed for them to be treated verbatim.
If you use "..." quoting or an unquoted argument (which mostly behaves as if it were enclosed in "..."), PowerShell's string parsing gets in the way: ` is also used as the escape character inside expandable strings ("..."), so in order to pass ` through, you must escape it itself, as ``.
Otherwise something like `[ inside "..." turns into just [ - the ` is "eaten" - because `[ is an escaped [ from "..."'s perspective, and escaping a character that doesn't need escaping turns into just that character; in short: both "file`[1`].txt" and file`[1`].txt turn into plain file[1].txt, as if you had never used `.
By contrast, ` characters are used verbatim inside '...'-quoted strings and need no escaping.
Flawed file-creation behavior of many cmdlets with -Path:
The bug mentioned above - that on file creation the escaped representation is mistakenly used as the literal filename - affects most cmdlets, unfortunately: That is, they unexpectedly retain the ` characters in the escaped pattern on creating a file, so that by specifying -Path 'file[1].txt' you'll end up with a file literally named file`[1`].txt.
Fortunately, most cmdlets do support -LiteralPath, so use of -LiteralPath file[1].txt is the better choice anyway and avoids this bug.
Some of the affected cmdlets:
Invoke-WebRequest and Invoke-RestMethod
Out-File and therefore also redirection operators > and >>, which effectively call Out-File behind the scenes.
Note that Set-Content and Add-Content do not exhibit this problem.
All(?) Export-* cmdlets.
Others?
The bug has been reported in GitHub issue #9475.
[1] This was technically a breaking change, but it was considered acceptable, due to the counterintuitive nature of the original behavior. Unfortunately, the counterintuitive behavior still surfaces in many other contexts - including still with Out-File unless -LiteralPath is explicitly used. See GitHub issue #17106 for a summary.
My original config file (web1.config) has no extra line and when viewed in notepad (showing all characters) looks as:
<?xml version="1.0"?>
<configuration>
<system.web>
<compilation debug="true" targetFramework="4.6" />
<httpRuntime targetFramework="4.6" />
</system.web>
<appSettings>
<add key="myConnectionString" value="server=localhost;database=myDb;uid=myUser;password=myPass;" />
</appSettings>
</configuration>
Now, I need to apply the script to change my database name to something else which looks like:
Move-Item "web1.config" "webtemp.config"
Get-Content "webtemp.config" | ForEach-Object {$_ -replace "database=myDb;", "database=newDb;"} |Set-Content "web1.config" -Force
Remove-Item "webtemp.config"
Write-Output('Settings Changed')
So, the new file (web1.config) generated looks as:
Notice the extra line added at the end of the file (which is completely not needed)
I tried all other options such as:
- using out-file api
- using .net IO method System.IO.StreamWriter
- using -nonewline flag (it converts all 10 lines into single line)
- using different encoding options
- tried replacing \r\n to \r (don't work as again set-content generates the crlf always)
I'm using PowerShell v5.1.
tl;dr (PSv5+; see bottom for older versions):
(Get-Content webtemp.config) -replace 'database=myDb;', 'database=newDb;' -join "`n" |
Set-Content -NoNewline -Force web1.config
Note: Replace "`n" with "`r`n" if you want Windows-style CRLF line endings rather than Unix-style LF-only line endings (PowerShell and many utilities can handle both).
In PSv5+, Set-Content supports the -NoNewline switch, which instructs Set-Content not to add a newline (line break) after each input object. The same applies analogously to the Add-Content and Out-File cmdlets.
In other words: Set-Content -NoNewline directly concatenates the string representations of all its input objects:
PS> 'one', 'two' | Set-Content -NoNewline tmp.txt; Get-Content tmp.txt
onetwo
If what you're passing to Set-Content -NoNewline is a single string that already has embedded newlines, you can use it as-is and get the desired result:
PS> "one`ntwo" | Set-Content -NoNewline tmp.txt; "$(Get-Content -Raw tmp.txt)?"
one
two?
Note that Get-Content -Raw reads the file as a whole, as-is (aside from character decoding) and the fact that the ? appears directly after two implies that the file has no trailing newline.
In your case, since you're processing input lines one by one (via Get-Content without -Raw) and therefore outputting an array of lines (strings), you must first join them with a newline as the separator - between lines only - and pass the result to Set-Content -NoNewline, as shown at the top; here's a simplified example:
PS> ('one', 'two') -join "`n" | Set-Content -NoNewline tmp.txt; "$(Get-Content -Raw tmp.txt)?"
one
two?
'one', 'two' is a two-element string array that is a stand-in for your line-by-line processing command.
Encoding note:
In Windows PowerShell, Set-Content produces "ANSI"-encoded files by default, based on your system's legacy, single-byte code page.
To control the encoding explicitly, use the -Encoding parameter.
In PSv4-, a solution that uses the .NET Framework is needed:
PS> [System.IO.File]::WriteAllText('tmp.txt', ('one', 'two') -join "`n"); "$(Get-Content -Raw tmp.txt)?"
one
two?
Note that [System.IO.File]::WriteAllText(), in the absence of an encoding argument, defaults to BOM-less UTF-8.
Pass the desired [System.Text.Encoding] encoding instance as the 3rd argument as needed.
I never noticed this, so i did a quick search and found:
set-content adds newlines by default
The suggested solution is to encode your content to bytes and then use Set-Content with the -Encoding parameter.
Set-Content test.txt ([byte[]][char[]] "test") -Encoding Byte
I tested it myself so i can confirm that this works.
I see that's an xml file. This way doesn't add a newline.
[xml]$xml = get-content web1.config
$xml.configuration.appSettings.add.value =
$xml.configuration.appSettings.add.value -replace 'database=myDb;',
'database=newDb;'
$xml.save("$pwd\web1.config") # in case of .net weirdness