I need to modify an existing UTF8 encoded JSON file with PowerShell. I tried with the following code:
$fileContent = ConvertFrom-Json "$(Get-Content $filePath -Encoding UTF8)"
$fileContent.someProperty = "someValue"
$fileContent | ConvertTo-Json -Depth 999 | Out-File $filePath
This adds a BOM to the file and also encodes it in UTF16. Is it possible to have ConvertFrom-Json and ConvertTo-Json do not do the encoding / BOM?
This has nothing to do with ConvertTo-Json or ConvertFrom-Json. The encoding is defined by the output cmdlet. Out-File defaults to Unicode, Set-Content to ASCII. With each of them the desired encoding can be defined explicitly:
... | Out-File $filePath -Encoding UTF8
or
... | Set-Content $filePath -Encoding UTF8
That will still write a (UTF8) BOM to the output file, but I wouldn't consider UTF-8 encoding without BOM a good practice anyway.
If you want ASCII-encoded output files (no BOM) replace UTF8 with Ascii:
... | Out-File $filePath -Encoding Ascii
or
... | Set-Content $filePath # Ascii is default encoding for Set-Content
Related
I am manipulating a file with the Get-Content command, doing a replace then piping it to Set-Content or Out-File. But the the file gets written with a CR before all my linefeeds. My original file has linefeeds, which is OK, but the CR breaks the file for what its being used for.
I tried this Powershell replace content in file adds redundent carriage return
But cannot get it to work. And I would like to avoid using .NET if possible
Here is my code:
(Get-Content "FILE.TEMPLATE") -replace('SEQUENCE', "$NEWSEQUENCE") | Set-Content ("FILE.NEW")
or
(Get-Content "FILE.TEMPLATE") -replace('SEQUENCE', "$NEWSEQUENCE") | Set-Content ("FILE.NEW") -Encoding UTF8NoBOM
I tried setting the encoding with '-Encoding' after the Set-Content pipe, but that does nothing. Tried UTF8, UTF8NoBOM. This out put file is destined to be used to on a UNIX box, which is why the CR breaks the file.
Any ideas?
It took some more digging around and here is what worked
(Get-Content "FILE.TEMPLATE" -Raw) -replace 'SEQUENCE', "$NEWSEQUENCE" |
Set-Content "FILE.NEW" -NoNewLine -Encoding UTF8NoBOM
Had to add -Raw to Get-Content, then add -NoNewLine and -Encoding UTF8NoBOM to Set-Content.
This will only work with PowerShell 6+, since anything below does not have UTF8NoBOM as an encoding option. Powershell 5 defaults to UTF8BOM.
We receive Cognos reports that are encoded as UTF16LE. I am trying to create a powershell script to convert the UTF16LE files to UTF8. My logic so far does loop through the directory (whichever directory I place the script in as hardcoding the directory names that contain date/numbers caused errors) and save the files as UTF-8; however, the delimiters seem to be removed.
I believe that it may be due to the way that I am reading the data, as I am not specifying UTF16LE; however, I am unsure of any way to do that. My script so far is:
$files = Get-ChildItem
$dt = get-date -Format yyyyMMdd
$extension = "_" + "$dt" + "_utf8.csv"
ForEach ($file in $files) {
$file_name = $file.basename
$new_file = "$file_name" + "$extension"
echo $new_file
#Get-Content $file | Set-Content -Encoding UTF8 $new_file
}
Read-Host -Prompt "Press Enter to Close Window"
Any and all insight into this issue would be greatly appreciated.
PowerShell's Import-CSV and Export-CSV cmdlets support the -Encoding parameter (links to Microsoft Docs), so you could replace your line
Get-Content $file | Set-Content -Encoding UTF8 $new_file
with
Import-CSV -Path $File -Encoding Unicode | Export-CSV -Path $New_File -Encoding UTF8
(UTF16LE encoding is what PowerShell calls "Unicode"; UTF16BE is "BigEndianUnicode". The default is UTF8NoBOM, UTF8 without Byte Order Mark.)
Since all you want to do is convert the character encoding, reading and writing as a string would be the most straightforward. As always, read a text file with the character encoding it was written with:
Get-Content -Encoding Unicode $file | Set-Content -Encoding UTF8 $new_file
Encoding "Unicode" for UTF-16 harkens back to the infancy of the Unicode character set when UCS-2 was going to be "it" for many environments. Then the explosion happened and UTF-16 was born from UCS-2. Systems invented since then quite reasonably use UTF16 or similar when they mean UTF-16 and "Unicode" for UTF-16 is esoteric and imponderable.
I'm using this script to upload a JOB on the Mainframe.
"open $FTPserver" | Out-File ftp.scr -Encoding ASCII
$FTPusername | Out-File ftp.scr -Encoding ASCII -Append
$FTPpassword | Out-File ftp.scr -Encoding ASCII -Append
"quote site filetype=jes" | Out-File ftp.scr -Encoding ASCII -Append
"put " + $FTPfile | Out-File ftp.scr -Encoding ASCII -Append
"quit" | Out-File ftp.scr -Encoding ASCII -Append
ftp.exe -s:ftp.scr
Remove-Item ftp.scr
Script works great. But I would like to save this response to variable so I can download in the next step this JOB for the response. This is my targeted response:
250-It is known to JES as JOB24503
Is there a way also to hide the output text from the FTP?
To redirect output to a file use the > operator.
ftp.exe -s:ftp.scr > ftp.out
I'm not a powershell guru but I do know that you can use the full .NET API which gives you access to a very sophiticated runtime. Check out FtpWebRequest which can be used to run FTP requests and pipe the output back to streams.
I have an specific scenario where I have to log a batch file using Invoke-Expression in Powershell but my logs are being saved with "UCS-2 Little Endian" Encoding and I would like to save it with UTF-8 or any other encoding.
This is a simple example of what I'm trying to do:
batch file (test.bat):
echo Test
Powershell file (test.ps1):
Invoke-Expression "c:\test.bat > log.txt"
Is there a way I could change the encoding on log.txt?
You can try this:
C:\test.bat | Out-File C:\log.txt -Encoding UTF8
Or if for whatever reason you really have to use Invoke-Expression:
Invoke-Expression "C:\test.bat" | Out-File C:\log.txt -Encoding UTF8
Note that this will overwrite log.txt everytime. If you want to append to the file do this:
Invoke-Expression "C:\test.bat" | Out-File C:\log.txt -Encoding UTF8 -append
or
Invoke-Expression "C:\test.bat" | Add-Content C:\log.txt -Encoding UTF8
I have a file saved as UCS-2 Little Endian I want to change the encoding so I ran the following code:
cat tmp.log -encoding UTF8 > new.log
The resulting file is still in UCS-2 Little Endian. Is this because the pipeline is always in that format? Is there an easy way to pipe this to a new file as UTF8?
As suggested here:
Get-Content tmp.log | Out-File -Encoding UTF8 new.log
I would do it like this:
get-content tmp.log -encoding Unicode | set-content new.log -encoding UTF8
My understanding is that the -encoding option selects the encdoing that the file should be read or written in.
load content from xml file with encoding.
(Get-Content -Encoding UTF8 $fileName)
If you are reading an XML file, here's an even better way that adapts to the encoding of your XML file:
$xml = New-Object -Typename XML
$xml.load('foo.xml')
PowerShell's get-content/set-content encoding flag doesn't handle all encoding types. You may need to use IO.File, for example to load a file using Windows-1252:
$myString = [IO.File]::ReadAllText($filePath, [Text.Encoding]::GetEncoding(1252))
Text.Encoding::GetEncoding
Text.Encoding::GetEncodings