Image Here For Reference
Specifically the text after "Similarity" and "->"
I have a small PowerShell script that uses ssh to connect to a server and run a command.
I believe it produces unreadable output due to some of the text having to be in colour.
I've tried adding the following, but still no luck:
$OutputEncoding = [console]::InputEncoding = [console]::OutputEncoding = New-Object System.Text.UTF8Encoding
To define console encoding you can use following expressions.
Choose encoding you need.
$ConsoleEncoding = [System.Text.Encoding]::UTF8
Apply your encoding to console.
[Console]::OutputEncoding = [System.Text.Encoding]::GetEncoding( "$( $ConsoleEncoding.BodyName )" )
Solved it, turns out that the command I needed was ssh -t host command to enable the colour display
More info here: https://explainshell.com/explain?cmd=ssh+-t+my_host+my_command
Related
I am creating a PowerShell script just to backup my WSL distros, but when I try to run the command with variables it's not working, it displays the usage text as though I provided the incorrect arguments.
$DistrosArray1 = (wsl -l --quiet) | where {$_ -ne ""}
$DistrosArray2 = 'Arch', 'Ubuntu-22.04', 'docker-desktop-data', 'docker-desktop'
$CheckArrayDifference = Compare-Object -ReferenceObject $DistrosArray1 -DifferenceObject $DistrosArray2 -PassThru
echo $CheckArrayDifference
# Does not return anything (there is no difference)
foreach ($Distro in $DistrosArray1) {
wsl --export $Distro "$Distro.tar"
# This method is not working
}
foreach ($Distro in $DistrosArray2) {
wsl --export $Distro "$Distro.tar"
# This method is working
}
It sounds like you are running into complications from issue #4607 -- The wsl.exe command outputs some oddly mangled UTF16 encoding that creates issues when attempting to process it from PowerShell (or even from inside WSL).
This is now fixed in the latest WSL Preview release 0.64.0, but you do have to "opt-in" to the fix so that older workarounds (like the ones #Bender and I provided) don't inadvertently break.
Simply set:
$env:WSL_UTF8=1
... before your code, and WSL will no longer spit out the "mangled UTF16."
Other examples in my answers to:
Why cannot I match for strings from wsl.exe output?
Powershell - Strange WSL output string encoding
How to ask WSL to check if the distribution exists, using Bash and wsl.exe?
Older solution:
Let's simplify the problem and make a "safe" example that doesn't attempt to export:
$DistrosArray1 = (wsl -l --quiet) | where {$_ -ne ""}
wsl -d $DistrosArray1[0]
Results in:
There is no distribution with the supplied name.
I've successfully used the method in this comment to handle it. For this particular example:
$console = ([console]::OutputEncoding)
[console]::OutputEncoding = New-Object System.Text.UnicodeEncoding
$DistrosArray1 = (wsl -l --quiet) | where {$_ -ne ""}
wsl -d $DistrosArray1[0]
This will run the first distribution in the list correctly.
Reset the encoding after with:
[console]::OutputEncoding = $console
That shouldn't be a problem for most non-interactive scripts since it will just be the final line of the "wrapper", but as #BendertheGreatest pointed out in the comments, it's a critical step.
This is part of a known issue with wsl.exe output. Here is what I put together from workarounds provided on that issue:
$DistrosArray1 = wsl -l --quiet | wsl iconv -c -f utf16 -t ascii
foreach ($Distro in $DistrosArray1) {
wsl --export $Distro "$Distro.tar"
}
Unfortunately I could not get this working with a conversion to UTF8 (changing ascii to utf8 produces additional garbage characters although they are consistent and detectable in my limited testing), so only any characters outside of the ASCII range will likely cause problems for you.
I have a client who requires a database export from a SQL Server 2016 database in UTF8 no BOM. I've used PowerShell to import the raw output from the database (which is in ANSI) and output the file in UTF-8.
Now I am hearing back if I could remove some 'special characters', and saw that PowerShell has changed it as shown in the picture.
Is there any way PowerShell could keep the character or remove it entirely?
This might also happen with other characters in the future, our sample dataset only contains this particular character.
EDIT: The Customer has a batch script which exports a select request from a MSSQL Server to a CSV File. Script as follows:
sqlcmd -S [SERVER]\[INSTANCE] -U sa -P [PASSWORD] -d [DATABASE] -I -i "C:\Path\To\Query.sql" -o "C:\Path\For\Output\Ouput.csv" -W -s"|"
The CSV is seperated by a Pipe.
The request was then to add double-quotes as text identifier as well as change the encoding to UTF-8 no BOM. The Database apparently exports the file in ANSI.
I've created a powershell script since I know it will automatically add the double quotes for me and I should be able to change encoding through it.
Script goes as follows:
$file = Import-Csv -Path "C:\Path\For\Output\Ouput.csv" -Encoding "UTF7" -Delimiter "|"
$file | Export-Csv -path "C:\Path\For\Output\Ouput.csv" -delimiter "|" -Encoding "UTF8noBOM" -NoTypeInformation
The reason for the -Encoding UTF7 flag in the input step was that without it, we had problems with special letters like ß and äöü (we're in Germany, those will be frequent).
After running the file through this script, it's mostly as it should be however the example in the screenshot is a problem for the people trying to import the file into their system afterwards.
Did this help? I'll gladly provide any further information, thank you in you advanced!
EDIT: Found a solution. I've edited the customers original script which creates the export from the database, I've added the -u flag making the output Unicode. It's not UTF8 yet but the powershell script can now convert the file properly, also no need to set import encoding to UTF7. Thanks to JosefZ for questioning my use of forced UTF7 encoding, made me realise I was looking at the wrong place to fix this.
I have a powershell.ps1 script that I performed base64 encoding upon as below
$Base64 = [System.Convert]::ToBase64String([System.IO.File]::ReadAllBytes('c:\path\to\powershell.ps1'));
Now I have stored this output to base64.txt file.
I tried to launch this script as below via CMD,
powershell.exe -EncodedCommand (Base64String)
But I ended up in the below error
Cannot process the command because the value specified with -EncodedCommand is not properly encoded. The value must be Base64 encoded.
I realized that the CMD is not taking the entire (Base64String). The full length of my (Base64String) is 11,133 characters. But CMD is accepting only 8160 characters.
Is there any way or workaround to run this base64 encoding?
Thanks in advance.
This worked for me (myscript.ps1 contains the base64 encoded command):
powershell -encodedcommand (Get-Content 'myscript.ps1' -Raw)
Which is very similar to what you would do in Bash:
$ powershell -encodedcommand `cat myscript.ps1`
Obs: Addressing some comments, this is sometimes indeed needed. My particular use case was to do a reverse shell while dodging an AV on a windows machine that was detecting my plaintext shell code.
I have the following problems with a powershell script that runs inside a TFS build. Both problems are unrelated to TFS and can be reproduced using an simple powershell command line window.
1) Completely unrelated to TFS. It seems Powershell does not like german umlauts when it comes to pipe.
1a) This line of code works fine and all umlauts are shown correctly
.\TF.exe hist "$/Test" /recursive /collection:https://TestTFS/tfs/TestCollection /noprompt /version:C1~T
1b) This line messes with umlauts
.\TF.exe hist "$/Test" /recursive /collection:https://TestTFS/tfs/TestCollection /noprompt /version:C1~T | Out-String
Initially I tried Out-File and changed encoding only to the that the umlauts are encoded wrong in every typeset (UTF8, unicode, UTF32,...)
I really do not know how to extract a string from standard output and get the umlauts right.
2) When using Out-File or Out-String each line in the output got truncated after 80 characters with seems to be the default screen buffer setting. How can I change that inside a powershell script and why does it even have an impact when redirecting the output.
Problem number 2 is not a Powershell problem. tfs documentation says following about default /format parameter (i.e. /format:brief)
Some of the data may be truncated.
/format:detailed does not have that warning, but it returns more information, which you can process with Powershell before doing Out-String or Out-File.
tl;dr
The following should solve both your problems, which stem from tf.exe using ANSI character encoding rather than the expected OEM encoding, and from truncating output by default.:
If you're using Windows PowerShell (the Windows-only legacy edition of PowerShell with versions up to v5.1):
$correctlyCapturedOutput =
& {
$prev = [Console]::OutputEncoding
[Console]::OutputEncoding = [System.Text.Encoding]::Default
# Note the addition of /format:detailed
.\tf.exe hist '$/Test' /recursive /collection:https://TestTFS/tfs/TestCollection /noprompt /format:detailed /version:C1~T
[Console]::OutputEncoding = $prev
}
If you're using the cross-platform, install-on-demand PowerShell (Core) 7+:
Note: [System.Text.Encoding]::Default, which reports the active ANSI code page's encoding in Windows PowerShell, reports (BOM-less) UTF-8 in PowerShell (Core) (reflecting .NET Core's / .NET 5+'s behavior). Therefore, the active ANSI code page must be determined explicitly, which is most robustly done via the registry.
$correctlyCapturedOutput =
& {
$prev = [Console]::OutputEncoding
[Console]::OutputEncoding = [System.Text.Encoding]::GetEncoding(
[int] ((Get-ItemProperty HKLM:\SYSTEM\CurrentControlSet\Control\Nls\CodePage ACP).ACP)
)
# Note the addition of /format:detailed
.\tf.exe hist '$/Test' /recursive /collection:https://TestTFS/tfs/TestCollection /noprompt /format:detailed /version:C1~T
[Console]::OutputEncoding = $prev
}
This Gist contains helper function Invoke-WithEncoding, which can simplify the above in both PowerShell edition as follows:
$correctlyCapturedOutput =
Invoke-WithEncoding -Encoding Ansi {
.\tf.exe hist '$/Test' /recursive /collection:https://TestTFS/tfs/TestCollection /noprompt /format:detailed /version:C1~T
}
You can directly download and define the function with the following command (while I can personally assure you that doing so is safe, it is advisable to check the source code first):
# Downloads and defines function Invoke-WithEncoding in the current session.
irm https://gist.github.com/mklement0/ef57aea441ea8bd43387a7d7edfc6c19/raw/Invoke-WithEncoding.ps1 | iex
Read on for a detailed discussion.
Re the umlaut (character encoding) problem:
While the output from external programs may print OK to the console, when it comes to capturing the output in a variable or redirecting it - such as sending it through the pipeline to Out-String in your case - PowerShell decodes the output into .NET strings, using the character encoding stored in [Console]::OutputEncoding.
If [Console]::OutputEncoding doesn't match the actual encoding used by the external program, PowerShell will misinterpret the output.
The solution is to (temporarily) set [Console]::OutputEncoding to the actual encoding used by the external program.
While the official tf.exe documentation doesn't discuss character encodings, this comment on GitHub suggests that tf.exe uses the system's active ANSI code page, such as Windows-1252 on US-English or Western European systems.
It should be noted that the use of the ANSI code page is nonstandard behavior for a console application, because console applications are expected to use the system's active OEM code page. As an aside: python too exhibits this nonstandard behavior by default, though its behavior is configurable.
The solutions at the top show how to temporarily switch [Console]::OutputEncoding to the active ANSI code page's encoding in order to ensure that PowerShell correctly decodes tf.exe's output.
Re output-line truncation with Out-String / Out-File (and therefore also > and >>):
As Mustafa Zengin's helpful answer points out, in your particular case - due to use of tf.exe - the truncation happens at the source, i.e. it is tf.exe itself that outputs truncated data per its default formatting (implied /format:brief when /noprompt is also specified).
In general, Out-String and Out-File / > / >> do situationally truncate or line-wrap their output lines based on the console-window width (with a default of 120 chars. in the absence of a console):
Truncation of line-wrapping applies only to output lines stemming from the representations of non-primitive, non-string objects generated by PowerShell's rich output-formatting system:
Strings themselves ([string] input) as well as the string representations of .NET primitive types (plus a few more singe-value-only types) are not subject to truncation / line-wrapping.
Since PowerShell only ever interprets output from external programs as text ([string] instances), truncation / line-wrapping do not occur.
It follows that there's usually no reason to use Out-String on external-program output - unless you need to join the stream (array) of output lines to form a single, multiline string for further in-memory processing.
However, note that Out-String invariably adds a trailing newline to the resulting string, which may be undesired; use (...) -join [Environment]::NewLine to avoid that; Out-String's problematic behavior is discussed in GitHub issue #14444.
I have curl command stored in variable ($command). The command looks like this:
curl -F "login=xxx" -F "password=xxx" "title=Some title" -F "img=#/path/to/image" https://example.com/api/import
Then i execute the command:
Invoke-Expression ($command)
Everything is fine unless title contains special characters like "č,š,ý..." because server expects UTF8 encoded parameters. In such case special characters are replaced with questionmarks on the website.
I tried setting [Console]::OutputEncoding and $OutputEncoding to UTF8 but it didn't solve the problem.When i run the command on linux (ubuntu) everything is fine because it uses UTF8 as default encoding, so i rewrote the script to bash to get the job done. But i'm still wondering if it's possible in powershell somehow. Any suggestions appreciated.
Settting [Console]::OutputEncoding works for me. Are you sure you're setting it correctly?
C:\PS> [console]::OutputEncoding = [text.encoding]::UTF8
C:\PS> echoargs "title=č,š,ý"
Arg 0 is <title=č,š,ý>
Command line:
"C:\Program Files (x86)\PowerShell Community Extensions\Pscx3\Pscx\Apps\EchoArgs.exe" title=č,š,ý
Echoargs is a tool from PSCX.