Prevent trailing newline in PowerShell Out-File command - powershell

How do I prevent PowerShell's Out-File command from appending a newline after the text it outputs?
For example, running the following command produces a file with contents "TestTest\r\n" rather than just "TestTest".
"TestTest" | Out-File -encoding ascii test.txt

In PowerShell 5.0+, you would use:
"TestTest" | Out-File -encoding ascii test.txt -NoNewline
But in earlier versions you simply can't with that cmdlet.
Try this:
[System.IO.File]::WriteAllText($FilePath,"TestTest",[System.Text.Encoding]::ASCII)

To complement briantist's helpful answer re -NoNewline:
The following applies not just to Out-File, but analogously to Set-Content / Add-Content as well; as stated, -NoNewline requires PSv5+.
Note that -NoNewline means that with multiple objects to output, it is not just a trailing newline (line break) that is suppressed, but any newlines.
In other words: The string representations of the input objects are directly concatenated, without a separator (terminator).
Therefore, the following commands result in the same file contents (TestTest without a trailing newline):
# Single input string
"TestTest" | Out-File -encoding ascii test.txt -NoNewline
# Equivalent command: 2-element array of strings that are directly concatenated.
"Test", "Test" | Out-File -encoding ascii test.txt -NoNewline
In order to place newlines only between, but not also after the output objects, you must join the objects with newlines explicitly:
"Test", "Test" -join [Environment]::NewLine |
Out-File -encoding ascii test.txt -NoNewline
[Environment]::NewLine is the platform-appropriate newline sequence (CRLF on Windows, LF on Unix-like platforms); you can also produce either sequence explicitly, if needed, with "`r`n" and "`n"
Caveat:
The above -join solution implicitly converts the input objects to strings, if they aren't already and does so by calling the .NET .ToString() method on each object. This often yields a different representation than the one that Out-File would directly create, because Out-File uses PowerShell's default output formatter; for instance, compare the outputs of (Get-Date).ToString() and just Get-Date.
If your input comprises only strings and/or non-strings whose .ToString() representation is satisfactory, the above solution works, but note that it is then generally preferable to use the Set-Content cmdlet, which applies the same stringification implicitly.
For a complete discussion of the differences between Out-File and Set-Content, see this answer of mine.
If your input has non-strings that do you want to be formatted as they would print to the console, there is actually no simple solution: while you can use Out-String to create per-object string representations with the default formatter, Out-String's lack of -NoNewline (as of v5.1; this GitHub issue suggests introducing it) would invariably yield trailing newlines.

To complement briantist's and mklement0's helpful answers re -NoNewline:
I created this little function to replace the -NoNewLine parameter of Out-File in previous versions of powershell.
Note: In my case it was for a .csv file with 7 lines (Days of the week and some more values)
## Receive the value we want to add and "yes" or "no" depending on whether we want to
put the value on a new line or not.
function AddValueToLogFile ($value, $NewLine) {
## If the log file exists:
if (Test-path $Config.LogPath) {
## And we don't want to add a new line, the value is concatenated at the end.
if ($NewLine -eq "no") {
$file = Get-Content -Path $Config.LogPath
## If the file has more than one line
if ($file -is [array]) {
$file[-1]+= ";" + $value
}
## if the file only has one line
else {
$file += ";" + $value
}
$file | Out-File -FilePath $Config.LogPath
}
## If we want to insert a new line the append parameter is used.
elseif ($NewLine -eq "yes") {
$value | Out-File -Append -FilePath $Config.LogPath
}
}
## If the log file does not exist it is passed as a value
elseif (!(Test-path $Config.LogPath)) {
$value | Out-File -FilePath $Config.LogPath
}
}

Related

Why are all newlines gone after PowerShell's Get-Content, Regex, and Set-Content?

I want to load a file template into a variable, modify data within the variable and output the modified template to a new location from the variable.
The issue is that PowerShell is removing newlines from my template.
The input file (template file) has Unix line endings which are also required for output since the recipient of the modified version is a Unix-based system.
I have the following code which results into a concatted one-liner:
[String] $replacement = "Foo Bar"
[String] $template = Get-Content -Path "$pwd\template.sh" -Encoding UTF8
$template = $template -replace '<REPLACE_ME>', $replacement
$template | Set-Content -Path "$pwd\script.sh" -Encoding UTF8
Having the template input:
#!/bin/sh
myvar="<REPLACE_ME>"
echo "my variable: $myvar"
exit 0
Resulted into:
#!/bin/sh myvar="Foo Bar" echo "my variable: $myvar" exit 0
It appears to me that somewhere LF where replaced by one simple whitespace. Finally at the end of the script there is an added CR LF which was not present in the template file.
How do I preserve the line endings and prevent adding further (CR LF) wrong line endings to the final script?
For the $replacement variable, you don't really need to specify the type [string], PowerShell will infer that from the assignment.
For the $template variable, [string] is actually wrong. By default, Get-Content will give you an array of strings (i.e. lines) instead of one string.
But in fact you don't even want to split the input into lines in the first place. When Set-Content or Out-File see an array as their input, they will join it with spaces.
Using -Raw makes Get-Content return the entire file as one string, this way also the line endings (like LF for Linux files) will stay the way they are.
$replacement = "Foo Bar"
$template = Get-Content -Path "$pwd\template.sh" -Encoding UTF8 -Raw
$template = $template -replace '<REPLACE_ME>', $replacement
Set-Content -Path "$pwd\script.sh" -Value $template -Encoding UTF8
PowerShell will save all UTF-8 files with a BOM. If you don't want that, you must use a different utility to write the file:
$UTF8_NO_BOM = New-Object System.Text.UTF8Encoding $False
$replacement = "Foo Bar"
$template = Get-Content -Path "$pwd\template.sh" -Encoding UTF8 -Raw
$template = $template -replace '<REPLACE_ME>', $replacement
[System.IO.File]::WriteAllText("$pwd\script.sh", $template, $UTF8_NO_BOM)
Notes:
PowerShell operators (like -replace) silently operate on arrays. $x -replace "search", "replacement" will perform a replace operation on every member of $x, be that a single string or an array of them.
Recommended reading: PowerShell Set-Content and Out-File what is the difference?
Use the -delimiter "`n" option instead of -raw. The -raw option reads/returns the entire content as a single string, although it preserves the new-line characters but it is useless if you need to manipulate the content e.g. skip Header/1st row or skip blank lines etc.
Get-Content - background info:
By default, the Get-Content cmdlet reads & returns content line-by-line, which means if you pipe a Set-Content or Add-Content to instantly write each-line (being read) into the output file - the newline characters are preserved and written as expected, e.g.:
Get-Content $inputFile | Set-Content $outputFilePath
However, if you store the entire content (read) into a $variable, you will receive a single string-array without any separator/delimiter (by default), which means you lose the new-line characters, however, when reading file (using Get-Content) you can use the -delimiter option to specify a newline character, e.g.:
Get-Content -Delimiter "`n" $fileToRead
HTH.
I think you need to use the -Raw switch with Get-Content in order to load the file as a single string:
[String] $replacement = "Foo Bar"
[String] $template = Get-Content -Path "$pwd\template.sh" -Encoding UTF8 -Raw
$template = $template -replace '<REPLACE_ME>', $replacement
To stop the Windows line ending being added to the end of the script, I think you need to use this .NET method for writing the file:
[io.file]::WriteAllText("$pwd\template.sh",$template)
By default PowerShell attempts to convert your input in to an array of strings for each line in the file. I think because of the Unix line endings its not doing this successfully but is subsequently removing the new line characters.
In PowerShell 3.0 we now have a new dynamic parameter, Raw. When
specified, Get-Content ignores newline characters and returns the
entire contents of a file in one string. Raw is a dynamic parameter,
it is available only in file system drives.
https://social.technet.microsoft.com/Forums/windowsserver/en-US/6026b31a-2a0e-4e0a-90b5-355387dce9ac/preventing-newline-with-outfile-or-addcontent?forum=winserverpowershell
I was using Get-Content-Tail, which doesn't allow you to specify -Raw at the same time, but I did have luck with Out-String. So, in your case:
$template = Out-String -InputObject $( Get-Content -Path "$pwd\template.sh" -Encoding UTF8 -Raw)
Or perhaps, if you care about tail:
$template = Out-String -InputObject $(Get-Content -Path "$pwd\template.sh" -tail 4)

Out-File -Encoding problems using PowerShell replace command [duplicate]

Out-File seems to force the BOM when using UTF-8:
$MyFile = Get-Content $MyPath
$MyFile | Out-File -Encoding "UTF8" $MyPath
How can I write a file in UTF-8 with no BOM using PowerShell?
Update 2021
PowerShell has changed a bit since I wrote this question 10 years ago. Check multiple answers below, they have a lot of good information!
Using .NET's UTF8Encoding class and passing $False to the constructor seems to work:
$MyRawString = Get-Content -Raw $MyPath
$Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding $False
[System.IO.File]::WriteAllLines($MyPath, $MyRawString, $Utf8NoBomEncoding)
The proper way as of now is to use a solution recommended by #Roman Kuzmin in comments to #M. Dudley answer:
[IO.File]::WriteAllLines($filename, $content)
(I've also shortened it a bit by stripping unnecessary System namespace clarification - it will be substituted automatically by default.)
I figured this wouldn't be UTF, but I just found a pretty simple solution that seems to work...
Get-Content path/to/file.ext | out-file -encoding ASCII targetFile.ext
For me this results in a utf-8 without bom file regardless of the source format.
Note: This answer applies to Windows PowerShell; by contrast, in the cross-platform PowerShell Core edition (v6+), UTF-8 without BOM is the default encoding, across all cmdlets.
In other words: If you're using PowerShell [Core] version 6 or higher, you get BOM-less UTF-8 files by default (which you can also explicitly request with -Encoding utf8 / -Encoding utf8NoBOM, whereas you get with-BOM encoding with -utf8BOM).
If you're running Windows 10 and you're willing to switch to BOM-less UTF-8 encoding system-wide - which can have side effects - even Windows PowerShell can be made to use BOM-less UTF-8 consistently - see this answer.
To complement M. Dudley's own simple and pragmatic answer (and ForNeVeR's more concise reformulation):
A simple, (non-streaming) PowerShell-native alternative is to use New-Item, which (curiously) creates BOM-less UTF-8 files by default even in Windows PowerShell:
# Note the use of -Raw to read the file as a whole.
# Unlike with Set-Content / Out-File *no* trailing newline is appended.
$null = New-Item -Force $MyPath -Value (Get-Content -Raw $MyPath)
Note: To save the output from arbitrary commands in the same format as Out-File would, pipe to Out-String first; e.g.:
$null = New-Item -Force Out.txt -Value (Get-ChildItem | Out-String)
For convenience, below is advanced function Out-FileUtf8NoBom, a pipeline-based alternative that mimics Out-File, which means:
you can use it just like Out-File in a pipeline.
input objects that aren't strings are formatted as they would be if you sent them to the console, just like with Out-File.
an additional -UseLF switch allows you use Unix-format LF-only newlines ("`n") instead of the Windows-format CRLF newlines ("`r`n") you normally get.
Example:
(Get-Content $MyPath) | Out-FileUtf8NoBom $MyPath # Add -UseLF for Unix newlines
Note how (Get-Content $MyPath) is enclosed in (...), which ensures that the entire file is opened, read in full, and closed before sending the result through the pipeline. This is necessary in order to be able to write back to the same file (update it in place).
Generally, though, this technique is not advisable for 2 reasons: (a) the whole file must fit into memory and (b) if the command is interrupted, data will be lost.
A note on memory use:
M. Dudley's own answer
and the New-Item alternative above require that the entire file contents be built up in memory first, which can be problematic with large input sets.
The function below does not require this, because it is implemented as a proxy (wrapper) function (for a concise summary of how to define such functions, see this answer).
Source code of function Out-FileUtf8NoBom:
Note: The function is also available as an MIT-licensed Gist, and only it will be maintained going forward.
You can install it directly with the following command (while I can personally assure you that doing so is safe, you should always check the content of a script before directly executing it this way):
# Download and define the function.
irm https://gist.github.com/mklement0/8689b9b5123a9ba11df7214f82a673be/raw/Out-FileUtf8NoBom.ps1 | iex
function Out-FileUtf8NoBom {
<#
.SYNOPSIS
Outputs to a UTF-8-encoded file *without a BOM* (byte-order mark).
.DESCRIPTION
Mimics the most important aspects of Out-File:
* Input objects are sent to Out-String first.
* -Append allows you to append to an existing file, -NoClobber prevents
overwriting of an existing file.
* -Width allows you to specify the line width for the text representations
of input objects that aren't strings.
However, it is not a complete implementation of all Out-File parameters:
* Only a literal output path is supported, and only as a parameter.
* -Force is not supported.
* Conversely, an extra -UseLF switch is supported for using LF-only newlines.
.NOTES
The raison d'ĂȘtre for this advanced function is that Windows PowerShell
lacks the ability to write UTF-8 files without a BOM: using -Encoding UTF8
invariably prepends a BOM.
Copyright (c) 2017, 2022 Michael Klement <mklement0#gmail.com> (http://same2u.net),
released under the [MIT license](https://spdx.org/licenses/MIT#licenseText).
#>
[CmdletBinding(PositionalBinding=$false)]
param(
[Parameter(Mandatory, Position = 0)] [string] $LiteralPath,
[switch] $Append,
[switch] $NoClobber,
[AllowNull()] [int] $Width,
[switch] $UseLF,
[Parameter(ValueFromPipeline)] $InputObject
)
begin {
# Convert the input path to a full one, since .NET's working dir. usually
# differs from PowerShell's.
$dir = Split-Path -LiteralPath $LiteralPath
if ($dir) { $dir = Convert-Path -ErrorAction Stop -LiteralPath $dir } else { $dir = $pwd.ProviderPath }
$LiteralPath = [IO.Path]::Combine($dir, [IO.Path]::GetFileName($LiteralPath))
# If -NoClobber was specified, throw an exception if the target file already
# exists.
if ($NoClobber -and (Test-Path $LiteralPath)) {
Throw [IO.IOException] "The file '$LiteralPath' already exists."
}
# Create a StreamWriter object.
# Note that we take advantage of the fact that the StreamWriter class by default:
# - uses UTF-8 encoding
# - without a BOM.
$sw = New-Object System.IO.StreamWriter $LiteralPath, $Append
$htOutStringArgs = #{}
if ($Width) { $htOutStringArgs += #{ Width = $Width } }
try {
# Create the script block with the command to use in the steppable pipeline.
$scriptCmd = {
& Microsoft.PowerShell.Utility\Out-String -Stream #htOutStringArgs |
. { process { if ($UseLF) { $sw.Write(($_ + "`n")) } else { $sw.WriteLine($_) } } }
}
$steppablePipeline = $scriptCmd.GetSteppablePipeline($myInvocation.CommandOrigin)
$steppablePipeline.Begin($PSCmdlet)
}
catch { throw }
}
process
{
$steppablePipeline.Process($_)
}
end {
$steppablePipeline.End()
$sw.Dispose()
}
}
Starting from version 6 powershell supports the UTF8NoBOM encoding both for set-content and out-file and even uses this as default encoding.
So in the above example it should simply be like this:
$MyFile | Out-File -Encoding UTF8NoBOM $MyPath
When using Set-Content instead of Out-File, you can specify the encoding Byte, which can be used to write a byte array to a file. This in combination with a custom UTF8 encoding which does not emit the BOM gives the desired result:
# This variable can be reused
$utf8 = New-Object System.Text.UTF8Encoding $false
$MyFile = Get-Content $MyPath -Raw
Set-Content -Value $utf8.GetBytes($MyFile) -Encoding Byte -Path $MyPath
The difference to using [IO.File]::WriteAllLines() or similar is that it should work fine with any type of item and path, not only actual file paths.
This script will convert, to UTF-8 without BOM, all .txt files in DIRECTORY1 and output them to DIRECTORY2
foreach ($i in ls -name DIRECTORY1\*.txt)
{
$file_content = Get-Content "DIRECTORY1\$i";
[System.IO.File]::WriteAllLines("DIRECTORY2\$i", $file_content);
}
important!: this only works if an extra space or newline at the start is no problem for your use case of the file
(e.g. if it is an SQL file, Java file or human readable text file)
one could use a combination of creating an empty (non-UTF8 or ASCII (UTF8-compatible)) file and appending to it (replace $str with gc $src if the source is a file):
" " | out-file -encoding ASCII -noNewline $dest
$str | out-file -encoding UTF8 -append $dest
as one-liner
replace $dest and $str according to your use case:
$_ofdst = $dest ; " " | out-file -encoding ASCII -noNewline $_ofdst ; $src | out-file -encoding UTF8 -append $_ofdst
as simple function
function Out-File-UTF8-noBOM { param( $str, $dest )
" " | out-file -encoding ASCII -noNewline $dest
$str | out-file -encoding UTF8 -append $dest
}
using it with a source file:
Out-File-UTF8-noBOM (gc $src), $dest
using it with a string:
Out-File-UTF8-noBOM $str, $dest
optionally: continue appending with Out-File:
"more foo bar" | Out-File -encoding UTF8 -append $dest
Old question, new answer:
While the "old" powershell writes a BOM, the new platform-agnostic variant does behave differently: The default is "no BOM" and it can be configured via switch:
-Encoding
Specifies the type of encoding for the target file. The default value is utf8NoBOM.
The acceptable values for this parameter are as follows:
ascii: Uses the encoding for the ASCII (7-bit) character set.
bigendianunicode: Encodes in UTF-16 format using the big-endian byte order.
oem: Uses the default encoding for MS-DOS and console programs.
unicode: Encodes in UTF-16 format using the little-endian byte order.
utf7: Encodes in UTF-7 format.
utf8: Encodes in UTF-8 format.
utf8BOM: Encodes in UTF-8 format with Byte Order Mark (BOM)
utf8NoBOM: Encodes in UTF-8 format without Byte Order Mark (BOM)
utf32: Encodes in UTF-32 format.
Source: https://learn.microsoft.com/de-de/powershell/module/Microsoft.PowerShell.Utility/Out-File?view=powershell-7
Emphasis mine
For PowerShell 5.1, enable this setting:
Control Panel, Region, Administrative, Change system locale, Use Unicode UTF-8
for worldwide language support
Then enter this into PowerShell:
$PSDefaultParameterValues['*:Encoding'] = 'Default'
Alternatively, you can upgrade to PowerShell 6 or higher.
https://github.com/PowerShell/PowerShell
I would say to use just the Set-Content command, nothing else needed.
The powershell version in my system is :-
PS C:\Users\XXXXX> $PSVersionTable.PSVersion | fl
Major : 5
Minor : 1
Build : 19041
Revision : 1682
MajorRevision : 0
MinorRevision : 1682
PS C:\Users\XXXXX>
So you would need something like following.
PS C:\Users\XXXXX> Get-Content .\Downloads\finddate.txt
Thursday, June 23, 2022 5:57:59 PM
PS C:\Users\XXXXX> Get-Content .\Downloads\finddate.txt | Set-Content .\Downloads\anotherfile.txt
PS C:\Users\XXXXX> Get-Content .\Downloads\anotherfile.txt
Thursday, June 23, 2022 5:57:59 PM
PS C:\Users\XXXXX>
Now when we check the file as per the screenshot it is utf8.
anotherfile.txt
Change multiple files by extension to UTF-8 without BOM:
$Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding($False)
foreach($i in ls -recurse -filter "*.java") {
$MyFile = Get-Content $i.fullname
[System.IO.File]::WriteAllLines($i.fullname, $MyFile, $Utf8NoBomEncoding)
}
[System.IO.FileInfo] $file = Get-Item -Path $FilePath
$sequenceBOM = New-Object System.Byte[] 3
$reader = $file.OpenRead()
$bytesRead = $reader.Read($sequenceBOM, 0, 3)
$reader.Dispose()
#A UTF-8+BOM string will start with the three following bytes. Hex: 0xEF0xBB0xBF, Decimal: 239 187 191
if ($bytesRead -eq 3 -and $sequenceBOM[0] -eq 239 -and $sequenceBOM[1] -eq 187 -and $sequenceBOM[2] -eq 191)
{
$utf8NoBomEncoding = New-Object System.Text.UTF8Encoding($False)
[System.IO.File]::WriteAllLines($FilePath, (Get-Content $FilePath), $utf8NoBomEncoding)
Write-Host "Remove UTF-8 BOM successfully"
}
Else
{
Write-Warning "Not UTF-8 BOM file"
}
Source How to remove UTF8 Byte Order Mark (BOM) from a file using PowerShell
If you want to use [System.IO.File]::WriteAllLines(), you should cast second parameter to String[] (if the type of $MyFile is Object[]), and also specify absolute path with $ExecutionContext.SessionState.Path.GetUnresolvedProviderPathFromPSPath($MyPath), like:
$Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding $False
Get-ChildItem | ConvertTo-Csv | Set-Variable MyFile
[System.IO.File]::WriteAllLines($ExecutionContext.SessionState.Path.GetUnresolvedProviderPathFromPSPath($MyPath), [String[]]$MyFile, $Utf8NoBomEncoding)
If you want to use [System.IO.File]::WriteAllText(), sometimes you should pipe the second parameter into | Out-String | to add CRLFs to the end of each line explictly (Especially when you use them with ConvertTo-Csv):
$Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding $False
Get-ChildItem | ConvertTo-Csv | Out-String | Set-Variable tmp
[System.IO.File]::WriteAllText("/absolute/path/to/foobar.csv", $tmp, $Utf8NoBomEncoding)
Or you can use [Text.Encoding]::UTF8.GetBytes() with Set-Content -Encoding Byte:
$Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding $False
Get-ChildItem | ConvertTo-Csv | Out-String | % { [Text.Encoding]::UTF8.GetBytes($_) } | Set-Content -Encoding Byte -Path "/absolute/path/to/foobar.csv"
see: How to write result of ConvertTo-Csv to a file in UTF-8 without BOM
I have the same error in the PowerShell and used this isolation and fixed it
$PSDefaultParameterValues['*:Encoding'] = 'utf8'
One technique I utilize is to redirect output to an ASCII file using the Out-File cmdlet.
For example, I often run SQL scripts that create another SQL script to execute in Oracle. With simple redirection (">"), the output will be in UTF-16 which is not recognized by SQLPlus. To work around this:
sqlplus -s / as sysdba "#create_sql_script.sql" |
Out-File -FilePath new_script.sql -Encoding ASCII -Force
The generated script can then be executed via another SQLPlus session without any Unicode worries:
sqlplus / as sysdba "#new_script.sql" |
tee new_script.log
Update: As others have pointed out, this will drop non-ASCII characters. Since the user asked for a way to "force" conversion, I assume they do not care about that as perhaps their data does not contain such data.
If you care about the preservation of non-ASCII characters, this is not the answer for you.
Used this method to edit a UTF8-NoBOM file and generated a file with correct encoding-
$fileD = "file.xml"
(Get-Content $fileD) | ForEach-Object { $_ -replace 'replace text',"new text" } | out-file "file.xml" -encoding ASCII
I was skeptical at this method at first, but it surprised me and worked!
Tested with powershell version 5.1
Could use below to get UTF8 without BOM
$MyFile | Out-File -Encoding ASCII
This one works for me (use "Default" instead of "UTF8"):
$MyFile = Get-Content $MyPath
$MyFile | Out-File -Encoding "Default" $MyPath
The result is ASCII without BOM.

Why does Powershell file concatenation convert UTF8 to UTF16?

I am running the following Powershell script to concatenate a series of output files into a single CSV file. whidataXX.htm (where xx is a two digit sequential number) and the number of files created varies from run to run.
$metadataPath = "\\ServerPath\foo"
function concatenateMetadata {
$cFile = $metadataPath + "whiconcat.csv"
Clear-Content $cFile
$metadataFiles = gci $metadataPath
$iterations = $metadataFiles.Count
for ($i=0;$i -le $iterations-1;$i++) {
$iFile = "whidata"+$i+".htm"
$FileExists = (Test-Path $metadataPath$iFile -PathType Leaf)
if (!($FileExists))
{
break
}
elseif ($FileExists)
{
Write-Host "Adding " $metadataPath$iFile
Get-Content $metadataPath$iFile | Out-File $cFile -append
Write-Host "to" $cfile
}
}
}
The whidataXX.htm files are encoded UTF8, but my output file is encoded UTF16. When I view the file in Notepad, it appears correct, but when I view it in a Hex Editor, the Hex value 00 appears between each character, and when I pull the file into a Java program for processing, the file prints to the console with extra spaces between c h a r a c t e r s.
First, is this normal for PowerShell? or is there something in the source files that would cause this?
Second, how would I fix this encoding problem in the code noted above?
The Out-* cmdlets (like Out-File) format the data, and the default format is unicode.
You can add an -Encoding parameter to Out-file:
Get-Content $metadataPath$iFile | Out-File $cFile -Encoding UTF8 -append
or switch to Add-Content, which doesn't re-format
Get-Content $metadataPath$iFile | Add-Content $cFile
First, the fact that you get 2 bytes per character indicates that fixed length UTF16 is being used. More accurately, it is called UCS-2. This article explains that file redirection in Powershell causes the output to be in UCS-2. See http://www.kongsli.net/nblog/2012/04/20/powershell-gotchas-redirect-to-file-encodes-in-unicode/. That same article also provides a fix.

Export-CSV cmdlet rewriting entire CSV during each iteration of a FOREACH statement

I'm working with some code that is going to take a series of performance counters, and then put the counters in a .csv file that rolls over every time it hits 1MB.
$Folder="C:\Perflogs\BBCRMLogs" # Change the bit in the quotation marks to whatever directory you want the log file stored in
$Computer = $env:COMPUTERNAME
$1GBInBytes = 1GB
$p = LOTS OF COUNTERS;
# If you want to change the performance counters, change the above list. However, these are the recommended counters for a client machine.
$num = 0
$file = "$Folder\SQL_log_${num}.csv"
if( !(test-path $folder)) {New-Item $Folder -type directory}
Get-Counter -counter $p -SampleInterval 2 -Continuous | Foreach {
if ((Get-Item $file -ErrorAction SilentlyContinue ).Length -gt 1mb)
{
$num +=1
$file = "$Folder\SQL_log_${num}.csv"
}
$_
} | Foreach-Object { $_ | Export-Csv $file -Force -Append}
Right now, it's working quite well. The iteration works fine, and it does create a new file each time the .csv reaches 1MB. However, each .CSV after the first is being created after 2 minutes already at 1MB, causing a new file to be created. I'm not quite sure why this is occurring, although I believe it's because Powershell is just rewriting the entirety of the .csv each time it creates it.
[I'm posting this as a new answer rather than editing the original because it's completely different. Replacing or appending to the original answer would make the ensuing discussion confusing.]
What you need to do is use a regex to extract the values from the Readings property of the output of Get-Counter, and manually construct CSV output from the timestamp and those values. Change the last line to this (format according to your preferred style):
| %{'"' + (Get-Date $_.Timestamp -f 's') + '","' + (([regex]::matches($_.Readings, '(?<=\\\\.+?:\n)(.+?)(?=\n)') | select -ExpandProperty Value) -join '","') + '"'} | Out-File $file -Append -Encoding ASCII
To break that down:
(Get-Date $_.Timestamp -f 's') This part is not strictly necessary, though I think it will make your results easier to follow. The 's' format puts the date in an ISO 8601 sortable pattern. You could substitute 'u' for another sortable format, or use your favorite custom format string. Or just replace it with $_.Timestamp to retain the original format.
[regex]::matches($_.Readings, '(?<=\\\\.+?:\n)(.+?)(?=\n)') The regex matches the contents of any line that is preceded by a line that begins with \\ and ends with : (those pesky counter names you wanted to get rid of). Note that I'm using [regex]::matches, which performs a global match, as opposed to [regex]::match or -match, which will just give you the first match for each string (the Readings property is a single string, so only the first counter reading would be returned).
| select -ExpandProperty Value Produces an array of all the matches, which you can then join with "," and surround with "'s to produce CSV output.
Since you're not using a conversion function, you also need to construct a header row. Add this line right above the pipeline:
`'"Timestamp","' + ($p -join '","') + '"' | Out-File $file -Append -Encoding ASCII`
That's assuming that $p is an array (which it should be). If it's a string, then depending on the format you can either use it as-is, or -split it and rejoin it in CSV format.
Change the last line to this, to convert each line to CSV format and then append it to the output file:
} | Foreach-Object {($_ | ConvertTo-Csv -NoTypeInformation)[1] | Out-File $file -Append -Encoding ASCII}.
A few notes:
The -Encoding ASCII is not strictly necessary, but you might have trouble with a Unicode CSV file in some applications (Excel, for example, won't open it as a CSV file by default, and everything will be in Column A)
The reason for the index in ($_ | ConvertTo-Csv -NoTypeInformation)[1] is that ConvertTo-Csv -NoTypeInformation still outputs the header row each time, so you want to grab the second line of the two-line output (($_ | ConvertTo-Csv -NoTypeInformation)[0] is the header row)
Since you're not outputting a header row, you'll need to output one to $file before the loop

Using PowerShell to write a file in UTF-8 without the BOM

Out-File seems to force the BOM when using UTF-8:
$MyFile = Get-Content $MyPath
$MyFile | Out-File -Encoding "UTF8" $MyPath
How can I write a file in UTF-8 with no BOM using PowerShell?
Update 2021
PowerShell has changed a bit since I wrote this question 10 years ago. Check multiple answers below, they have a lot of good information!
Using .NET's UTF8Encoding class and passing $False to the constructor seems to work:
$MyRawString = Get-Content -Raw $MyPath
$Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding $False
[System.IO.File]::WriteAllLines($MyPath, $MyRawString, $Utf8NoBomEncoding)
The proper way as of now is to use a solution recommended by #Roman Kuzmin in comments to #M. Dudley answer:
[IO.File]::WriteAllLines($filename, $content)
(I've also shortened it a bit by stripping unnecessary System namespace clarification - it will be substituted automatically by default.)
I figured this wouldn't be UTF, but I just found a pretty simple solution that seems to work...
Get-Content path/to/file.ext | out-file -encoding ASCII targetFile.ext
For me this results in a utf-8 without bom file regardless of the source format.
Note: This answer applies to Windows PowerShell; by contrast, in the cross-platform PowerShell Core edition (v6+), UTF-8 without BOM is the default encoding, across all cmdlets.
In other words: If you're using PowerShell [Core] version 6 or higher, you get BOM-less UTF-8 files by default (which you can also explicitly request with -Encoding utf8 / -Encoding utf8NoBOM, whereas you get with-BOM encoding with -utf8BOM).
If you're running Windows 10 and you're willing to switch to BOM-less UTF-8 encoding system-wide - which can have side effects - even Windows PowerShell can be made to use BOM-less UTF-8 consistently - see this answer.
To complement M. Dudley's own simple and pragmatic answer (and ForNeVeR's more concise reformulation):
A simple, (non-streaming) PowerShell-native alternative is to use New-Item, which (curiously) creates BOM-less UTF-8 files by default even in Windows PowerShell:
# Note the use of -Raw to read the file as a whole.
# Unlike with Set-Content / Out-File *no* trailing newline is appended.
$null = New-Item -Force $MyPath -Value (Get-Content -Raw $MyPath)
Note: To save the output from arbitrary commands in the same format as Out-File would, pipe to Out-String first; e.g.:
$null = New-Item -Force Out.txt -Value (Get-ChildItem | Out-String)
For convenience, below is advanced function Out-FileUtf8NoBom, a pipeline-based alternative that mimics Out-File, which means:
you can use it just like Out-File in a pipeline.
input objects that aren't strings are formatted as they would be if you sent them to the console, just like with Out-File.
an additional -UseLF switch allows you use Unix-format LF-only newlines ("`n") instead of the Windows-format CRLF newlines ("`r`n") you normally get.
Example:
(Get-Content $MyPath) | Out-FileUtf8NoBom $MyPath # Add -UseLF for Unix newlines
Note how (Get-Content $MyPath) is enclosed in (...), which ensures that the entire file is opened, read in full, and closed before sending the result through the pipeline. This is necessary in order to be able to write back to the same file (update it in place).
Generally, though, this technique is not advisable for 2 reasons: (a) the whole file must fit into memory and (b) if the command is interrupted, data will be lost.
A note on memory use:
M. Dudley's own answer
and the New-Item alternative above require that the entire file contents be built up in memory first, which can be problematic with large input sets.
The function below does not require this, because it is implemented as a proxy (wrapper) function (for a concise summary of how to define such functions, see this answer).
Source code of function Out-FileUtf8NoBom:
Note: The function is also available as an MIT-licensed Gist, and only it will be maintained going forward.
You can install it directly with the following command (while I can personally assure you that doing so is safe, you should always check the content of a script before directly executing it this way):
# Download and define the function.
irm https://gist.github.com/mklement0/8689b9b5123a9ba11df7214f82a673be/raw/Out-FileUtf8NoBom.ps1 | iex
function Out-FileUtf8NoBom {
<#
.SYNOPSIS
Outputs to a UTF-8-encoded file *without a BOM* (byte-order mark).
.DESCRIPTION
Mimics the most important aspects of Out-File:
* Input objects are sent to Out-String first.
* -Append allows you to append to an existing file, -NoClobber prevents
overwriting of an existing file.
* -Width allows you to specify the line width for the text representations
of input objects that aren't strings.
However, it is not a complete implementation of all Out-File parameters:
* Only a literal output path is supported, and only as a parameter.
* -Force is not supported.
* Conversely, an extra -UseLF switch is supported for using LF-only newlines.
.NOTES
The raison d'ĂȘtre for this advanced function is that Windows PowerShell
lacks the ability to write UTF-8 files without a BOM: using -Encoding UTF8
invariably prepends a BOM.
Copyright (c) 2017, 2022 Michael Klement <mklement0#gmail.com> (http://same2u.net),
released under the [MIT license](https://spdx.org/licenses/MIT#licenseText).
#>
[CmdletBinding(PositionalBinding=$false)]
param(
[Parameter(Mandatory, Position = 0)] [string] $LiteralPath,
[switch] $Append,
[switch] $NoClobber,
[AllowNull()] [int] $Width,
[switch] $UseLF,
[Parameter(ValueFromPipeline)] $InputObject
)
begin {
# Convert the input path to a full one, since .NET's working dir. usually
# differs from PowerShell's.
$dir = Split-Path -LiteralPath $LiteralPath
if ($dir) { $dir = Convert-Path -ErrorAction Stop -LiteralPath $dir } else { $dir = $pwd.ProviderPath }
$LiteralPath = [IO.Path]::Combine($dir, [IO.Path]::GetFileName($LiteralPath))
# If -NoClobber was specified, throw an exception if the target file already
# exists.
if ($NoClobber -and (Test-Path $LiteralPath)) {
Throw [IO.IOException] "The file '$LiteralPath' already exists."
}
# Create a StreamWriter object.
# Note that we take advantage of the fact that the StreamWriter class by default:
# - uses UTF-8 encoding
# - without a BOM.
$sw = New-Object System.IO.StreamWriter $LiteralPath, $Append
$htOutStringArgs = #{}
if ($Width) { $htOutStringArgs += #{ Width = $Width } }
try {
# Create the script block with the command to use in the steppable pipeline.
$scriptCmd = {
& Microsoft.PowerShell.Utility\Out-String -Stream #htOutStringArgs |
. { process { if ($UseLF) { $sw.Write(($_ + "`n")) } else { $sw.WriteLine($_) } } }
}
$steppablePipeline = $scriptCmd.GetSteppablePipeline($myInvocation.CommandOrigin)
$steppablePipeline.Begin($PSCmdlet)
}
catch { throw }
}
process
{
$steppablePipeline.Process($_)
}
end {
$steppablePipeline.End()
$sw.Dispose()
}
}
Starting from version 6 powershell supports the UTF8NoBOM encoding both for set-content and out-file and even uses this as default encoding.
So in the above example it should simply be like this:
$MyFile | Out-File -Encoding UTF8NoBOM $MyPath
When using Set-Content instead of Out-File, you can specify the encoding Byte, which can be used to write a byte array to a file. This in combination with a custom UTF8 encoding which does not emit the BOM gives the desired result:
# This variable can be reused
$utf8 = New-Object System.Text.UTF8Encoding $false
$MyFile = Get-Content $MyPath -Raw
Set-Content -Value $utf8.GetBytes($MyFile) -Encoding Byte -Path $MyPath
The difference to using [IO.File]::WriteAllLines() or similar is that it should work fine with any type of item and path, not only actual file paths.
This script will convert, to UTF-8 without BOM, all .txt files in DIRECTORY1 and output them to DIRECTORY2
foreach ($i in ls -name DIRECTORY1\*.txt)
{
$file_content = Get-Content "DIRECTORY1\$i";
[System.IO.File]::WriteAllLines("DIRECTORY2\$i", $file_content);
}
important!: this only works if an extra space or newline at the start is no problem for your use case of the file
(e.g. if it is an SQL file, Java file or human readable text file)
one could use a combination of creating an empty (non-UTF8 or ASCII (UTF8-compatible)) file and appending to it (replace $str with gc $src if the source is a file):
" " | out-file -encoding ASCII -noNewline $dest
$str | out-file -encoding UTF8 -append $dest
as one-liner
replace $dest and $str according to your use case:
$_ofdst = $dest ; " " | out-file -encoding ASCII -noNewline $_ofdst ; $src | out-file -encoding UTF8 -append $_ofdst
as simple function
function Out-File-UTF8-noBOM { param( $str, $dest )
" " | out-file -encoding ASCII -noNewline $dest
$str | out-file -encoding UTF8 -append $dest
}
using it with a source file:
Out-File-UTF8-noBOM (gc $src), $dest
using it with a string:
Out-File-UTF8-noBOM $str, $dest
optionally: continue appending with Out-File:
"more foo bar" | Out-File -encoding UTF8 -append $dest
Old question, new answer:
While the "old" powershell writes a BOM, the new platform-agnostic variant does behave differently: The default is "no BOM" and it can be configured via switch:
-Encoding
Specifies the type of encoding for the target file. The default value is utf8NoBOM.
The acceptable values for this parameter are as follows:
ascii: Uses the encoding for the ASCII (7-bit) character set.
bigendianunicode: Encodes in UTF-16 format using the big-endian byte order.
oem: Uses the default encoding for MS-DOS and console programs.
unicode: Encodes in UTF-16 format using the little-endian byte order.
utf7: Encodes in UTF-7 format.
utf8: Encodes in UTF-8 format.
utf8BOM: Encodes in UTF-8 format with Byte Order Mark (BOM)
utf8NoBOM: Encodes in UTF-8 format without Byte Order Mark (BOM)
utf32: Encodes in UTF-32 format.
Source: https://learn.microsoft.com/de-de/powershell/module/Microsoft.PowerShell.Utility/Out-File?view=powershell-7
Emphasis mine
For PowerShell 5.1, enable this setting:
Control Panel, Region, Administrative, Change system locale, Use Unicode UTF-8
for worldwide language support
Then enter this into PowerShell:
$PSDefaultParameterValues['*:Encoding'] = 'Default'
Alternatively, you can upgrade to PowerShell 6 or higher.
https://github.com/PowerShell/PowerShell
I would say to use just the Set-Content command, nothing else needed.
The powershell version in my system is :-
PS C:\Users\XXXXX> $PSVersionTable.PSVersion | fl
Major : 5
Minor : 1
Build : 19041
Revision : 1682
MajorRevision : 0
MinorRevision : 1682
PS C:\Users\XXXXX>
So you would need something like following.
PS C:\Users\XXXXX> Get-Content .\Downloads\finddate.txt
Thursday, June 23, 2022 5:57:59 PM
PS C:\Users\XXXXX> Get-Content .\Downloads\finddate.txt | Set-Content .\Downloads\anotherfile.txt
PS C:\Users\XXXXX> Get-Content .\Downloads\anotherfile.txt
Thursday, June 23, 2022 5:57:59 PM
PS C:\Users\XXXXX>
Now when we check the file as per the screenshot it is utf8.
anotherfile.txt
Change multiple files by extension to UTF-8 without BOM:
$Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding($False)
foreach($i in ls -recurse -filter "*.java") {
$MyFile = Get-Content $i.fullname
[System.IO.File]::WriteAllLines($i.fullname, $MyFile, $Utf8NoBomEncoding)
}
[System.IO.FileInfo] $file = Get-Item -Path $FilePath
$sequenceBOM = New-Object System.Byte[] 3
$reader = $file.OpenRead()
$bytesRead = $reader.Read($sequenceBOM, 0, 3)
$reader.Dispose()
#A UTF-8+BOM string will start with the three following bytes. Hex: 0xEF0xBB0xBF, Decimal: 239 187 191
if ($bytesRead -eq 3 -and $sequenceBOM[0] -eq 239 -and $sequenceBOM[1] -eq 187 -and $sequenceBOM[2] -eq 191)
{
$utf8NoBomEncoding = New-Object System.Text.UTF8Encoding($False)
[System.IO.File]::WriteAllLines($FilePath, (Get-Content $FilePath), $utf8NoBomEncoding)
Write-Host "Remove UTF-8 BOM successfully"
}
Else
{
Write-Warning "Not UTF-8 BOM file"
}
Source How to remove UTF8 Byte Order Mark (BOM) from a file using PowerShell
If you want to use [System.IO.File]::WriteAllLines(), you should cast second parameter to String[] (if the type of $MyFile is Object[]), and also specify absolute path with $ExecutionContext.SessionState.Path.GetUnresolvedProviderPathFromPSPath($MyPath), like:
$Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding $False
Get-ChildItem | ConvertTo-Csv | Set-Variable MyFile
[System.IO.File]::WriteAllLines($ExecutionContext.SessionState.Path.GetUnresolvedProviderPathFromPSPath($MyPath), [String[]]$MyFile, $Utf8NoBomEncoding)
If you want to use [System.IO.File]::WriteAllText(), sometimes you should pipe the second parameter into | Out-String | to add CRLFs to the end of each line explictly (Especially when you use them with ConvertTo-Csv):
$Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding $False
Get-ChildItem | ConvertTo-Csv | Out-String | Set-Variable tmp
[System.IO.File]::WriteAllText("/absolute/path/to/foobar.csv", $tmp, $Utf8NoBomEncoding)
Or you can use [Text.Encoding]::UTF8.GetBytes() with Set-Content -Encoding Byte:
$Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding $False
Get-ChildItem | ConvertTo-Csv | Out-String | % { [Text.Encoding]::UTF8.GetBytes($_) } | Set-Content -Encoding Byte -Path "/absolute/path/to/foobar.csv"
see: How to write result of ConvertTo-Csv to a file in UTF-8 without BOM
I have the same error in the PowerShell and used this isolation and fixed it
$PSDefaultParameterValues['*:Encoding'] = 'utf8'
One technique I utilize is to redirect output to an ASCII file using the Out-File cmdlet.
For example, I often run SQL scripts that create another SQL script to execute in Oracle. With simple redirection (">"), the output will be in UTF-16 which is not recognized by SQLPlus. To work around this:
sqlplus -s / as sysdba "#create_sql_script.sql" |
Out-File -FilePath new_script.sql -Encoding ASCII -Force
The generated script can then be executed via another SQLPlus session without any Unicode worries:
sqlplus / as sysdba "#new_script.sql" |
tee new_script.log
Update: As others have pointed out, this will drop non-ASCII characters. Since the user asked for a way to "force" conversion, I assume they do not care about that as perhaps their data does not contain such data.
If you care about the preservation of non-ASCII characters, this is not the answer for you.
Used this method to edit a UTF8-NoBOM file and generated a file with correct encoding-
$fileD = "file.xml"
(Get-Content $fileD) | ForEach-Object { $_ -replace 'replace text',"new text" } | out-file "file.xml" -encoding ASCII
I was skeptical at this method at first, but it surprised me and worked!
Tested with powershell version 5.1
Could use below to get UTF8 without BOM
$MyFile | Out-File -Encoding ASCII
This one works for me (use "Default" instead of "UTF8"):
$MyFile = Get-Content $MyPath
$MyFile | Out-File -Encoding "Default" $MyPath
The result is ASCII without BOM.