Special characters is encoded when run script inside a file Powershell - powershell

Hi I have a script to automate some tasks, running in powershell core v.7.+.
In one of these scripts when I run the command inside the ps1 file and the special characters returned is encoded and I can't decode to right format, below the command used to do this and the return:
// the variable $script is my command its a ask-cli command to work in alexa
$model = pwsh -Command $script
/*
* when I running the script from here the special characters returned is these:
* "nächste",
* "nächstgelegene"
*/
But when I run the same command directly in the terminal the strings returned are:
/*
* "nächste",
* "nächstgelegene"
*/
I would like to know how can I run the command inside the file without encode the return.
I already tried some things as:
$encoding = [System.Text.Encoding]::Unicode
$model = pwsh -Command $script
Write-Output $model
$model = $encoding.GetBytes($model)
$model = $encoding.GetString($model)
But don't work as expected, I don't know more how I can to this, if someone could help me with this I appreciate too much.

Try returning the string as bytes and then decode it from the place you are calling the function pwsh. This would preserve it from any changes. What you're doing is converting it into bytes after receiving it then returning it to string.

Below my script:
(Get-Content "$currentPath\skill-package\skill.json" -Raw | ConvertFrom-Json).manifest.publishingInformation.locales.PSObject.Properties | ForEach-Object {
$lang = $_.Name
Write-Output "Profile: $profile skill-id: $skillId language: $lang"
$script = "ask smapi get-interaction-model -l $lang -s $skillId -p $profile -g $env"
Write-Output 'Running script'
Write-Warning $script
# $encoding = [System.Text.Encoding]::ASCII
$model = pwsh -Command $script
Write-Output $model
$model = $model
| ConvertFrom-Json -Depth 100
| Select-Object * -ExcludeProperty version
| ConvertTo-Json -Depth 100
# out-file "$file$lang.json" -InputObject $model -Encoding ascii
Write-Output "New model saved locally $file$lang.json"
}
Write-Warning 'Finished!!!'

(Get-Content "$currentPath\skill-package\skill.json" -Raw | ConvertFrom-Json).manifest.publishingInformation.locales.PSObject.Properties | ForEach-Object {
$lang = $_.Name
Write-Output "Profile: $profile skill-id: $skillId language: $lang"
$script = "`$command = ask smapi get-interaction-model -l $lang -s $skillId -p $profile -g $env;`$command = [Convert]::ToBase64String([Text.Encoding]::Unicode.GetBytes(Invoke-Expression `$command));`$command"
Write-Output 'Running script'
Write-Warning $script
# $encoding = [System.Text.Encoding]::ASCII
$model = pwsh -Command $script
$model = Text.Encoding::Unicode.GetString([Convert]::FromBase64String($model))
Write-Output $model
$model = $model
| ConvertFrom-Json -Depth 100
| Select-Object * -ExcludeProperty version
| ConvertTo-Json -Depth 100
# out-file "$file$lang.json" -InputObject $model -Encoding ascii
Write-Output "New model saved locally $file$lang.json"
}
Write-Warning 'Finished!!!'

After many researches, I could find something more easiest to solve my problem.
Powershell has by default a different output encoder used in these cases, and the only thing I need to do it's change it.
I used the command:
$OutputEncoding = [console]::InputEncoding = [console]::OutputEncoding = New-Object System.Text.UTF8Encoding
I find this question explaining how this work and this help a lot, for more question please check this answer.

Related

How to suppress "keyboard-interactive" prompts in plink.exe

I have a PowerShell script that calls plink.exe regularly. Normally, the two output lines about keyboard-interactive prompts are simply annoying.
However, when run using Start-Job, they get output as error text as soon as I call Receive-Job.
Is there any way to suppress these? I'd rather not suppress all errors.
My test code:
$test_scriptblock = [scriptblock]::Create(#"
param(
`$argumentlist
)
`$pw = `$argumentlist.pw
& 'C:\Program Files\Putty\Plink.exe' -ssh `"admin#*.*.*.*" -pw `$pw -batch whoami
"#)
$testParm = #{
pw = Read-Host "password"
}
$testjob = Start-Job -scriptblock $test_scriptblock -Argumentlist $testParm
$i = 0
do {
$i++
sleep 2
$results = Receive-Job $testjob
ForEach ($result in $results) {
Write-Host $result
}
if ($testjob.State -eq "Completed") {
$jobcompleted = $true
}
If ($i -gt 10) {
Stop-job $testjob
$jobcompleted = $true
}
} until ($jobcompleted)
Just add the stderr redirect to your plink or pscp commandline, to an extra dummy file, like
pscp ... 2> stderr.txt
With a caveat that it may swallow other valid error msgs, at your own risk :)
There's no way to suppress keyboard-interactive prompts in Plink.
I suggest you use a native .NET SSH implementation, like SSH.NET instead of driving an external console application.
Ir was a bit cumbersome, but finally I managed to suppress the "keyboard-interactive" messages this way:
[String] $Plink = 'C:\Program Files\PuTTY\plink.exe'
[Array] $PlinkPar = #("-ssh", "-l", $usr, "-pw", $pwd, $hst) # Set plink parameters
[Boolean] $PlinkOK = $True
Write-Host "Accept possibly unknown host key"
"y" | & $Plink $PlinkPar "exit" 2>&1 | Tee-Object -Variable PlinkOut | Out-Null
$PlinkOut | Foreach-Object {
$PlinkStr = $_.ToString()
If ($_ -is [System.Management.Automation.ErrorRecord]) {
If (! $PlinkStr.Contains("eyboard-interactive")) {
Write-Host "Error: $PlinkStr"
$PlinkOK = $False
}
} else {
Write-Host "$PlinkStr"
}
}
If (! $PlinkOK) { exit }
$PlinkPar += "-batch
And the output is like this:
>powershell .\InstCR.ps1 -usr myuser -pwd mypassword -hst doesnotexist
Accept possibly unknown host key
Error: Unable to open connection:
Error: Host does not exist
This plink call is just to accept a possibly unknown host key (without "-batch" and piping the "y" to answer the prompt). Then "-batch" is added to the Plink parameters to be used on all subsequent plink calls.

PowerShell & cURL - Get JSON Response and Check HTTP Response Code [duplicate]

Is it possible to redirect stdout from an external program to a variable and stderr from external programs to another variable in one run?
For example:
$global:ERRORS = #();
$global:PROGERR = #();
function test() {
# Can we redirect errors to $PROGERR here, leaving stdout for $OUTPUT?
$OUTPUT = (& myprogram.exe 'argv[0]', 'argv[1]');
if ( $OUTPUT | select-string -Pattern "foo" ) {
# do stuff
} else {
$global:ERRORS += "test(): oh noes! 'foo' missing!";
}
}
test;
if ( #($global:ERRORS).length -gt 0 ) {
Write-Host "Script specific error occurred";
foreach ( $err in $global:ERRORS ) {
$host.ui.WriteErrorLine("err: $err");
}
} else {
Write-Host "Script ran fine!";
}
if ( #($global:PROGERR).length -gt 0 ) {
# do stuff
} else {
Write-Host "External program ran fine!";
}
A dull example however I am wondering if that is possible?
One option is to combine the output of stdout and stderr into a single stream, then filter.
Data from stdout will be strings, while stderr produces System.Management.Automation.ErrorRecord objects.
$allOutput = & myprogram.exe 2>&1
$stderr = $allOutput | ?{ $_ -is [System.Management.Automation.ErrorRecord] }
$stdout = $allOutput | ?{ $_ -isnot [System.Management.Automation.ErrorRecord] }
The easiest way to do this is to use a file for the stderr output, e.g.:
$output = & myprogram.exe 'argv[0]', 'argv[1]' 2>stderr.txt
$err = get-content stderr.txt
if ($LastExitCode -ne 0) { ... handle error ... }
I would also use $LastExitCode to check for errors from native console EXE files.
You should be using Start-Process with -RedirectStandardError -RedirectStandardOutput options. This other post has a great example of how to do this (sampled from that post below):
$pinfo = New-Object System.Diagnostics.ProcessStartInfo
$pinfo.FileName = "ping.exe"
$pinfo.RedirectStandardError = $true
$pinfo.RedirectStandardOutput = $true
$pinfo.UseShellExecute = $false
$pinfo.Arguments = "localhost"
$p = New-Object System.Diagnostics.Process
$p.StartInfo = $pinfo
$p.Start() | Out-Null
$p.WaitForExit()
$stdout = $p.StandardOutput.ReadToEnd()
$stderr = $p.StandardError.ReadToEnd()
Write-Host "stdout: $stdout"
Write-Host "stderr: $stderr"
Write-Host "exit code: " + $p.ExitCode
This is also an alternative that I have used to redirect stdout and stderr of a command line while still showing the output during PowerShell execution:
$command = "myexecutable.exe my command line params"
Invoke-Expression $command -OutVariable output -ErrorVariable errors
Write-Host "STDOUT"
Write-Host $output
Write-Host "STDERR"
Write-Host $errors
It is just another possibility to supplement what was already given.
Keep in mind this may not always work depending upon how the script is invoked. I have had problems with -OutVariable and -ErrorVariable when invoked from a standard command line rather than a PowerShell command line like this:
PowerShell -File ".\FileName.ps1"
An alternative that seems to work under most circumstances is this:
$stdOutAndError = Invoke-Expression "$command 2>&1"
Unfortunately, you will lose output to the command line during execution of the script and would have to Write-Host $stdOutAndError after the command returns to make it "a part of the record" (like a part of a Jenkins batch file run). And unfortunately it doesn't separate stdout and stderr.
In case you want to get any from a PowerShell script and to pass a function name followed by any arguments you can use dot sourcing to call the function name and its parameters.
Then using part of James answer to get the $output or the $errors.
The .ps1 file is called W:\Path With Spaces\Get-Something.ps1 with a function inside named Get-It and a parameter FilePath.
Both the paths are wrapped in quotes to prevent spaces in the paths breaking the command.
$command = '. "C:\Path Spaces\Get-Something.ps1"; Get-It -FilePath "W:\Apps\settings.json"'
Invoke-Expression $command -OutVariable output -ErrorVariable errors | Out-Null
# This will get its output.
$output
# This will output the errors.
$errors
Copied from my answer on how to capture both output and verbose information in different variables.
Using Where-Object(The alias is symbol ?) is an obvious method, but it's a bit too cumbersome. It needs a lot of code.
In this way, it will not only take longer time, but also increase the probability of error.
In fact, there is a more concise method that separate different streams to different variable in PowerShell(it came to me by accident).
# First, declare a method that outputs both streams at the same time.
function thisFunc {
[cmdletbinding()]
param()
Write-Output 'Output'
Write-Verbose 'Verbose'
}
# The separation is done in a single statement.Our goal has been achieved.
$VerboseStream = (thisFunc -Verbose | Tee-Object -Variable 'String' | Out-Null) 4>&1
Then we verify the contents of these two variables
$VerboseStream.getType().FullName
$String.getType().FullName
The following information should appear on the console:
PS> System.Management.Automation.VerboseRecord
System.String
'4>&1' means to redirect the verboseStream to the success stream, which can then be saved to a variable, of course you can change this number to any number between 2 and 5.
Separately, preserving formatting
cls
function GetAnsVal {
param([Parameter(Mandatory=$true, ValueFromPipeline=$true)][System.Object[]][AllowEmptyString()]$Output,
[Parameter(Mandatory=$false, ValueFromPipeline=$true)][System.String]$firstEncNew="UTF-8",
[Parameter(Mandatory=$false, ValueFromPipeline=$true)][System.String]$secondEncNew="CP866"
)
function ConvertTo-Encoding ([string]$From, [string]$To){#"UTF-8" "CP866" "ASCII" "windows-1251"
Begin{
$encFrom = [System.Text.Encoding]::GetEncoding($from)
$encTo = [System.Text.Encoding]::GetEncoding($to)
}
Process{
$Text=($_).ToString()
$bytes = $encTo.GetBytes($Text)
$bytes = [System.Text.Encoding]::Convert($encFrom, $encTo, $bytes)
$encTo.GetString($bytes)
}
}
$all = New-Object System.Collections.Generic.List[System.Object];
$exception = New-Object System.Collections.Generic.List[System.Object];
$stderr = New-Object System.Collections.Generic.List[System.Object];
$stdout = New-Object System.Collections.Generic.List[System.Object]
$i = 0;$Output | % {
if ($_ -ne $null){
if ($_.GetType().FullName -ne 'System.Management.Automation.ErrorRecord'){
if ($_.Exception.message -ne $null){$Temp=$_.Exception.message | ConvertTo-Encoding $firstEncNew $secondEncNew;$all.Add($Temp);$exception.Add($Temp)}
elseif ($_ -ne $null){$Temp=$_ | ConvertTo-Encoding $firstEncNew $secondEncNew;$all.Add($Temp);$stdout.Add($Temp)}
} else {
#if (MyNonTerminatingError.Exception is AccessDeniedException)
$Temp=$_.Exception.message | ConvertTo-Encoding $firstEncNew $secondEncNew;
$all.Add($Temp);$stderr.Add($Temp)
}
}
$i++
}
[hashtable]$return = #{}
$return.Meta0=$all;$return.Meta1=$exception;$return.Meta2=$stderr;$return.Meta3=$stdout;
return $return
}
Add-Type -AssemblyName System.Windows.Forms;
& C:\Windows\System32\curl.exe 'api.ipify.org/?format=plain' 2>&1 | set-variable Output;
$r = & GetAnsVal $Output
$Meta2=""
foreach ($el in $r.Meta2){
$Meta2+=$el
}
$Meta2=($Meta2 -split "[`r`n]") -join "`n"
$Meta2=($Meta2 -split "[`n]{2,}") -join "`n"
[Console]::Write("stderr:`n");
[Console]::Write($Meta2);
[Console]::Write("`n");
$Meta3=""
foreach ($el in $r.Meta3){
$Meta3+=$el
}
$Meta3=($Meta3 -split "[`r`n]") -join "`n"
$Meta3=($Meta3 -split "[`n]{2,}") -join "`n"
[Console]::Write("stdout:`n");
[Console]::Write($Meta3);
[Console]::Write("`n");

file not found in current directory

I have the following script:
function Export-sql-toExcel {
[CmdletBinding()]
Param (
[string]$scriptFile,
[string]$excelFile,
[string]$serverInstance,
[string]$database,
[string]$userName,
[string]$password
)
$tokens = ( [system.io.file]::ReadAllText( $scriptFile ) -split '(?:\bGO\b)' )
foreach ($token in $tokens) {
$token = $token.Trim()
if ($token -ne "") {
$lines = $token -split '\n'
$title = $lines[0]
if ($title.StartsWith("--")) {
$title = $title.Substring(2)
$title
}
Invoke-Sqlcmd -ServerInstance $serverInstance -Database $database -Username $userName -Password $password -Query $token |
Export-Excel -Path $excelFile -WorkSheetname $title -FreezeTopRow -ExcludeProperty RowError,RowState,Table,ItemArray,HasErrors
}
}
}
I have installed this function as a module. When I invoke the command from, for example, desktop folder, like this:
PS D:\Usuarios\mnieto\Desktop> Export-sql-toExcel -scriptFile .\EXPORT.txt -excelFile Excel.xlsx
I get the following error (the export.txt file is in the desktop folder):
Exception calling "ReadAllText" with "1" argument(s): "Can't find the file 'D:\Usuarios\<MyUserName>\EXPORT.txt'."
EDITED
if I debug and try [system.environment]::CurrentDirectory, it returns
'D:\Usuarios\<MyUserName>
That is because my script fails. NET functions and powershell don't share the 'current directory'
Any other way to get the content and parse the $scriptFile file?
I got the solution changing the NET call by a powershell command at this line
$content = Get-Content $scriptFile -Raw
$tokens = ( $content -split '(?:\bGO\b)' )
the trick was in the -Raw parameter, so the file is read as a single string
To my experience .dot NET functions don't like relative path's.
I'd use
$scriptfile = (Get-Item $Scriptfile).FullName
to resolve to a full path in the function just ahead :
$tokens = ( [system.io.file]::ReadAllText( $scriptFile ) -split '(?:\bGO\b)' )
I had the same situation, it was resolved when i added a dot "." to call the csv
$path2 = $path + ".\file.csv"
Or check for spaces at the $excelFile variable.

Start-Process using EncodedCommand outputs to Standard Error

The following script demonstrates an issue I'm experiencing after moving from PowerShell v4 to v5.1. I make extensive use of Start-Process redirecting StandardError to a file, if this file contains data after execution I can read the file to see what the issue was.
However, with v5.1 when the launched PowerShell process starts, it makes use of progress output which is on the error stream, so this is redirected to the error file. In previous versions of PowerShell this did not happen, if I use Command rather than EncodedCommand it works as expected.
I believe this is a bug but it would be useful if someone could confirm or suggest a workaround?
The output to the script below is as follows:
PowerShell v4
No Errors
hello world!
PowerShell v5.1
WARNING:
SourceId Record
1 parent = -1 id = 0 act = Preparing modules for first use. stat = cur = pct = -1 sec = -1 type = Completed
hello world!
hello world!
Demo Script
Note: this will create 2 temporary files, which are both deleted at the end.
$outfile = [System.IO.Path]::GetTempFileName()
$errorfile = [System.IO.Path]::GetTempFileName()
$command = "write-host 'hello world!'"
$encodedCommand = [Convert]::ToBase64String([Text.Encoding]::Unicode.GetBytes($command))
$argumentList = #(
"-NoLogo",
"-NonInteractive",
"-NoProfile",
"-EncodedCommand",
$encodedCommand
)
Start-Process powershell `
-argumentlist $argumentList `
-PassThru `
-wait `
-RedirectStandardOut $outfile `
-NoNewWindow `
-RedirectStandardError $errorfile | Out-Null
if ((Get-Item $errorfile).length -gt 0)
{
Import-Clixml $errorfile | Out-String | Write-Warning
}
else
{
Write-Host "No Errors"
}
Get-Content $outfile
$outfile, $errorfile | Remove-item

Tranistion bat file to powershell script - Invoke-RestMethod

I have a basic bat script that uses [curl]( https://curl.haxx.se/). The script takes values from a txt document named location_ids.txt (this file is found in the same folder as the script). I have set it up to check the location_id with 3 different urls. It works well. However, it is very slow! since batch files aren't threaded, and each command will block until it completes. I am aware this can be done with more ease and speed using a powershell script (windows environment) using Invoke-RestMethod. How can I replicate the below in ps? I would like to make the curl calls simultaneous.
#echo off
setlocal enabledelayedexpansion
set OUTPUT_FILE=%DATE:/=-%#%TIME::=-%
set OUTPUT_FILE=file_%OUTPUT_FILE: =%.html
for /f %%i in (location_ids.txt) do (
set LOCATION_ID=%%i
echo ^<h2 style='color:green;'^>!LOCATION_ID::=!^</h2^> >>%OUTPUT_FILE%
curl -k -H "application/x-www-form-urlencoded" -X GET -d "id=!LOCATION_ID::=!" http://localhost:5000/location_site1 >>%OUTPUT_FILE%
curl -k -H "application/x-www-form-urlencoded" -X GET -d "id=!LOCATION_ID::=!" http://localhost:5000/location_site2 >>%OUTPUT_FILE%
curl -k -H "application/x-www-form-urlencoded" -X GET -d "id=!LOCATION_ID::=!" http://localhost:5000/location_site3 >>%OUTPUT_FILE%
echo ^<br^>^<br^> >>%OUTPUT_FILE%
)
EDIT:
My attempt to run multiple web server calls to http://localhost:5000/location_site1 that run simultaneously using scriptblock. The below is not outputting any results.
$runRoutePost =
{ param([string]$id, [string]$fileLocation)
Write-Host "Accessing site for $id";
$ResponseData = New-Object PSObject;
$webclient = new-object System.Net.WebClient;
$apiParams = "id=$_";
$ResponseData = $webclient.UploadString("http://localhost:5000/location_site1",$apiParams) |Add-Content $fileLocation;
}
Get-Content location_ids.txt | ForEach-Object {
Start-Job -ScriptBlock $runRoutePost -ArgumentList $_, $LOG_FILE
}
To convert your example, you really just need to make a request to the url and specify the location id as a query string parameter. The example below uses string interpolation to set the value of the id parameter. The $_ variable is the current item that is being enumerated within the ForEach-Object script block.
$outputFile = # not sure of your date time format
Get-Content location_ids.txt | ForEach-Object {
Add-Content $outputFile "<h2 style=`"color:green`">$_</h2>"
Invoke-RestMethod -Uri "http://localhost:5000/location_site1?id=$_" | Add-Content $outputFile
Invoke-RestMethod -Uri "http://localhost:5000/location_site2?id=$_" | Add-Content $outputFile
Invoke-RestMethod -Uri "http://localhost:5000/location_site3?id=$_" | Add-Content $outputFile
Add-Content $outputFile "<br><br>"
}
For a GET request you do not need to specify the content-type or method. However, if you need to for other scripts you can use the -ContentType and/or -Method parameters.
Invoke-RestMethod -Method GET -ContentType application/x-www-form-urlencoded -Uri "http://localhost:5000/location_site3?id=$_"
More documentation can be found by running this:
get-help Invoke-RestMethod -full
Since you have a restriction of using PowerShell v2, you can use the .NET WebClient type.
$web = new-object System.Net.WebClient
Get-Content location_ids.txt | ForEach-Object {
Add-Content $outputFile "<h2 style=`"color:green`">$_</h2>"
$web.DownloadString("http://localhost:5000/location_site1?id=$_") | Add-Content $outputFile
$web.DownloadString("http://localhost:5000/location_site2?id=$_") | Add-Content $outputFile
$web.DownloadString("http://localhost:5000/location_site3?id=$_") | Add-Content $outputFile
Add-Content $outputFile "<br><br>"
}
If instead you want to send a POST request using WebClient, the UploadString method can be used. However, in this case I'm not sure of how to set the Content-Type header.
$web.UploadString("http://localhost:5000/location_site1","id=$_") | Add-Content $outputFile
Update in response to edit
To run these jobs in parallel and collect the results, you need to wait for all the jobs to complete using Wait-Job and then extract the results using Receive-Job.
$runRoutePost = {
param([string]$id)
Write-Host "Accessing site for $id"
$webclient = new-object System.Net.WebClient
$webclient.UploadString("http://localhost:5000/location_site1","id=$id")
}
$Jobs = Get-Content location_ids.txt | ForEach-Object {
Start-Job -ScriptBlock $runRoutePost -ArgumentList $_
}
Wait-Job -Job $Jobs
Receive-Job -Job $Jobs | Add-Content $LOG_FILE