How to escape non-ascii symbols in PowerShell 5.1? - powershell

I'm trying to send a file content to server:
$uri = ...
$headers = #{
...
"Content-Type" = "application/json"
}
[string] $content = Get-Content .\filename -Encoding utf8 -Raw
$body = #{
...
"content" = $content
} | ConvertTo-Json
$response = Invoke-WebRequest $uri -Method 'PUT' -Headers $headers -Body $body
But all of non-ascii symbols are changed to another similar symbols or question marks
How can I escape them?
I've read documentation and I know about parameter -EscapeHandling of cmdlet ConvertTo-Json, but it's available from PowerShell 6.2, I have only 5.1

As a result, I wrote a simple function:
function EscapeNonAscii([string] $s)
{
$sb = New-Object System.Text.StringBuilder;
for ([int] $i = 0; $i -lt $s.Length; $i++)
{
$c = $s[$i];
if ($c -gt 127)
{
$sb = $sb.Append("\u").Append(([int] $c).ToString("X").PadLeft(4, "0"));
}
else
{
$sb = $sb.Append($c);
}
}
return $sb.ToString();
}
And used it like this:
$updateFileResponse = Invoke-WebRequest $updateFileUri -Method 'PUT' -Headers $headers -Body (EscapeNonAscii $body)
It helped. For everybody who will google it in the future, it's a request to GitLab API Update existing file in repository
PS: I use PS as C# because I know it badly. If somebody knows how to rewrite this fragment better please let me know.
PPS: And also I know that StringBuilder.Append changes an existing object, but I add here assigning ($sb = $sb.Append($c) instead of simple $sb.Append($c)) because it prints every action to console. If you know how to fix it please let me know.

Related

Alter object value upon assignment to PSCustomObject

I'm creating a log of attempted posts to an API. The API key is stored in a simple hash table and passed via Invoke-WebRequest:
$headers = #{ 'x-api-key' = 'ABC123DEF456GHI789' }
Try {
[Net.ServicePointManager]::SecurityProtocol = 'tls12, tls11'
$apiResponse = Invoke-WebRequest -Uri $url -Method $method -Headers $headers -Body $body
$status = $apiResponse.StatusCode
$statusDescription = $apiResponse.StatusDescription
} Catch {
$status = $_.Exception.Response.StatusCode.value__
$statusDescription = $_.Exception.Response.StatusDescription
}
I want to obscure the header key in the log, so I created and modified a new variable.
$obscured = $headers | ConvertTo-Json -depth 100 | ConvertFrom-Json
$obscured.'x-api-key' = $obscured.'x-api-key'.Substring(0,2) + '...' + $obscured.'x-api-key'.Substring($obscured.'x-api-key'.Length-2,2)
$logresults += [PSCustomObject]#{
status = $status
statusDescription = $statusDescription
url = $url
method = $method
header = $obscured
body = ConvertFrom-JSON $body
}
I want to retain the header's structure as a key/value pair in the log. The extra steps prepping a new variable seem wasteful. Does PowerShell have a way to change the header key value upon assignment to the PSCustomObject?
AFAIK, there is no easy way to obscure strings (or objects) in PowerShell or even .Net, see my related purpose: #16921 Add [HiddenString] Class. The only thing that exists is the gone crazy SecureString Class with difficult methodes to convert from a string and reveal string (as that is not secure). Besides, the SecureString might get obsolete (as it appears less secure than intended) and possibly replaced by a shrouded buffer which is even more difficult to use for obscuring information (if even possible in PowerShell).
Anyways, in the HiddenString idea you might do something like this:
$ApiKey = [HiddenString](Read-Host -AsSecureString 'Enter Api key:')
See also: How to encrypt/hide ClearText password in PowerShell Transcript
$Headers = #{
'Accept' = 'application/json'
'X-My-Header' = 'Hello World'
'x-api-key' = $ApiKey
}
$apiResponse = Invoke-WebRequest -Uri $url -Method $method -Headers $($Headers.'x-api-key' = $ApiKey.Reveal(); $Headers) -Body $body
$logresults += [PSCustomObject]#{ # Avoid +=, see: https://stackoverflow.com/a/60708579/1701026
status = $status
statusDescription = $statusDescription
url = $url
method = $method
header = $Headers
body = ConvertFrom-JSON $body
}

Trying to use the "SetDeviceName" action with Microsoft Graph

Trying to do a bulk rename of device when they haven't been logged on for a long time.
I have managed to gather the information I need it and pipe it, so it comes out correctly.
But for some reason, I get Bad request and my formating is some how wrong, but can't figure out what. Tried every type of modification to the URI, but no luck.
This should work according to Microsoft's Doc about SetDevicename action
$date = (Get-date (Get-date).adddays(-316) -format "yyy-MM-ddTHH:mm:ssZ")
$devices | where {$_.lastSyncDateTime -le $date} | ForEach-Object {
$newname = "Test-$($_.Devicename)"
$deviceID = "$($_.ID)"
$URI = "https://graph.microsoft.com/beta/deviceManagement/managedDevices/$deviceID/setDeviceName"
$Body = #{ "deviceName" = "$NewName" } | ConvertTo-Json
$Method = "POST"
Invoke-RestMethod -Uri $URI -Method $Method -Headers $appauthToken -body $body -ContentType "application/json"
}
Any Idea what I'm doing wrong?
There is another method of renaming of bulk devices:
Document reference: https://learn.microsoft.com/en-us/mem/intune/remote-actions/device-rename#bulk-rename-devices.

Get-content not producing an array that Invoke-restmethod can process

As a follow-up to this question, instead of using a long array in the script I wanted to draw from a text file. So I replaced this:
$URLs = 'http://websiteone.com','http://websitetwo.com','http://websitethree.com'
with this
$URLs = Get-Content ./urlfile.txt
or (functionally the same as far I know) this
$URLs = #(Get-Content ./urlfile.txt)
But I end up with Invoke-RestMethod : The remote server returned an error: (400) Bad Request.
Incorporating the great response form my last question, my foreach loop looks like this:
foreach($URL in $URLs) {
$BODY = #([pscustomobject]#{"client" = #{"clientId" = "company"; "clientVersion" = "1.0"}; "threatInfo" = #{"threatTypes" = "MALWARE","SOCIAL_ENGINEERING","THREAT_TYPE_UNSPECIFIED","UNWANTED_SOFTWARE","POTENTIALLY_HARMFUL_APPLICATION"; "platformTypes" = "ANY_PLATFORM"; "threatEntryTypes" = "URL","EXECUTABLE","THREAT_ENTRY_TYPE_UNSPECIFIED"; "threatEntries" = #{"url" = $URL}}})
$JSONBODY = $BODY | ConvertTo-Json
$Result = Invoke-RestMethod -Method 'POST' -Uri $Uri -Body $JSONBODY -Headers $HEADERS
if ( ([string]::IsNullOrEmpty($Result)) ) {} else {write-host $URL "ALERT: Safe browsing match!"}
}
... but this doesn't work if I create the array with the Get-Content cmdlet. If I run the script either way, then type $URLs, I get the exact same data returned. What am I doing wrong with get-content?
The Invoke-RestMethod cmdlet is there to make one Rest request at a time and can't take an array.
You will need to add a forEach loop to step through your $urls one at a time, something like this:
foreach($url in $urls){
$result = Invoke-RestMethod -Uri $url
#do something with $result
}
So to integrate into your sample from the previous question, you should have a urls.txt file which looks like this:
http://google.com
http://small.com
https://fast.com/
And then your code would look like this:
$URLs = get-content .\urls.txt
$HEADERS = #{ 'Content-Type' = "application/json" }
$GOOGLE_API_KEY='[API Key]'
$Uri = 'https://safebrowsing.googleapis.com/v4/threatMatches:find?key='+ $GOOGLE_API_KEY
foreach($URL in $URLs) {
$BODY = #([pscustomobject]#{"client" = #{"clientId" = "company"; "clientVersion" = "1.0"}; "threatInfo" = #{"threatTypes" = "MALWARE","SOCIAL_ENGINEERING","THREAT_TYPE_UNSPECIFIED","UNWANTED_SOFTWARE","POTENTIALLY_HARMFUL_APPLICATION"; "platformTypes" = "ANY_PLATFORM"; "threatEntryTypes" = "URL"; "threatEntries" = #{"url" = $URL}}})
$JSONBODY = $BODY | ConvertTo-Json
$result = Invoke-RestMethod -Method 'POST' -Uri $Uri -Body $JSONBODY -Headers $HEADERS
[pscustomObject]#{SiteName=$url;ThreatInfo=$result.Matches}
}
This would load up the list of $urls from your text file, then run a Rest Request on each, storing the result in $result. Finally, it will make a new PowerShell Object with the site name and show you if there are any matches from the Google SafeBrowsing API.
You'll need to run the command interactively and see which properties from $result are meaningful to you, but you can see all of the expected properties in the Google API Docs.
Edit
Found the bug. It turns out when we use Get-Content the object returned back retains some of the document formatting information from the original file! We can see this by inspecting $JSONBODY. We also see that the conversion to Json from [PSCustomObject is leaving a lot of cruft behind too.
To fix this, we should cast $URL into a string using the ToString() method and also ditch casting to [psCustomObject] too as shown below.
$BODY = #{
"client" = #{
"clientId" = "company"; "clientVersion" = "1.0"
};
"threatInfo" = #{
"threatTypes" = "MALWARE",
"SOCIAL_ENGINEERING",
"THREAT_TYPE_UNSPECIFIED",
"UNWANTED_SOFTWARE",
"POTENTIALLY_HARMFUL_APPLICATION"; "platformTypes" = "ANY_PLATFORM"; "threatEntryTypes" = "URL"; "threatEntries" = #{
"url" = $URL.ToString()
}
}
}
$JSONBODY = $BODY | ConvertTo-Json

Getting 400 Bad Request Errors using Invoke-RestMethod for Microsoft Cognitive API

we're trying to test and evaluate the Text Analysis API from Microsoft's Cognitive service. We're trying to get the quick and dirty PowerShell script using Invoke-RestMethod to work. After some tweaks we're still getting 400 errors returned to us. We're unsure of what is wrong because the JSON appears to be corrected and the API key we've entered appears to be correct. We've made use of what we found on another person's blog about the use of the additional headers and tried some variants but still no dice. Can someone do a sanity check for us?
#html tag stripper function
function htmlStrip ($results)
{
#using .NET toString method to ensure PS doesn't interpret same var incorrectly
$results = $results.toString()
$results -replace '<[^>]*(>|$)'
}
Try
{
[string]$sourceUrl = Read-Host "Enter a URL such as https://foobar.com"
}
Catch
{
Write-Host "URL requires http:// or https:// prefix e.g. https://cnn.com"
}
$webClient = New-Object Net.WebClient
[string]$results = $webClient.DownloadString($sourceUrl)
[string]$cleanResults = htmlStrip $results
$body =
[ordered]#{"documents" =
#{ "language" = "en"; "id" = $sourceUrl; "text" = $cleanResults }
}
#>
$body = [ordered]#{
"documents" =
#(
#{ "language" = "en"; "id" = $sourceUrl; "text" = $cleanResults }
)
}
$jsonBody = $body | ConvertTo-Json
#Begin Text Analytics API Call with Invoke-RestMethod wrapper
#[string]$apiUrl = "https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/keyPhrases"
[string]$apiKey = "REDACTED"
$headers = #{ "Ocp-Apim-Subscription-Key" = $apiKey }
$analyticsResults = Invoke-RestMethod -Method Post -Uri $apiUrl -Headers $headers -Body $jsonBody -ContentType "application/json" -ErrorAction Stop
Write-Host $analyticsResults
Write-Host $jsonBody
The data you put into the text property of your request is probably not valid.
I tried your script with a fix URL to the README.md of the TypeScript repo on GitHub and it works.
Your script (slightly shortened)
$sourceUrl = 'https://raw.githubusercontent.com/Microsoft/TypeScript/master/README.md'
$webClient = New-Object Net.WebClient
$results = $webClient.DownloadString($sourceUrl)
$body = [ordered]#{
"documents" =
#(
#{ "language" = "en"; "id" = $sourceUrl; "text" = $results }
)
}
$jsonBody = $body | ConvertTo-Json
$apiUrl = "https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/keyPhrases"
$apiKey = "..."
$headers = #{ "Ocp-Apim-Subscription-Key" = $apiKey }
$analyticsResults = Invoke-RestMethod -Method Post -Uri $apiUrl -Headers $headers -Body $jsonBody -ContentType "application/json" -ErrorAction Stop
$analyticsResults.documents.keyPhrases
Result
TypeScript compiler
gulp tests
g typescript
TypeScript source
built compiler
TypeScript users
g gulp
TypeScript directory
cd TypeScript
gulp baseline
gulp lint
gulp local
gulp clean
gulp runtests-browser
gulp LKG
Install Gulp tools
...

How to send multipart/form-data with PowerShell Invoke-RestMethod

I'm trying to send a file via Invoke-RestMethod in a similar context as curl with the -F switch.
Curl Example
curl -F FileName=#"/path-to-file.name" "https://uri-to-post"
In powershell, I've tried something like this:
$uri = "https://uri-to-post"
$contentType = "multipart/form-data"
$body = #{
"FileName" = Get-Content($filePath) -Raw
}
Invoke-WebRequest -Uri $uri -Method Post -ContentType $contentType -Body $body
}
If I check fiddler I see that the body contains the raw binary data, but I get a 200 response back showing no payload has been sent.
I've also tried to use the -InFile parameter with no luck.
I've seen a number of examples using a .net class, but was trying to keep this simple with the newer Powershell 3 commands.
Does anyone have any guidance or experience making this work?
The accepted answer won't do a multipart/form-data request, but rather a application/x-www-form-urlencoded request forcing the Content-Type header to a value that the body does not contain.
One way to send a multipart/form-data formatted request with PowerShell is:
$ErrorActionPreference = 'Stop'
$fieldName = 'file'
$filePath = 'C:\Temp\test.pdf'
$url = 'http://posttestserver.com/post.php'
Try {
Add-Type -AssemblyName 'System.Net.Http'
$client = New-Object System.Net.Http.HttpClient
$content = New-Object System.Net.Http.MultipartFormDataContent
$fileStream = [System.IO.File]::OpenRead($filePath)
$fileName = [System.IO.Path]::GetFileName($filePath)
$fileContent = New-Object System.Net.Http.StreamContent($fileStream)
$content.Add($fileContent, $fieldName, $fileName)
$result = $client.PostAsync($url, $content).Result
$result.EnsureSuccessStatusCode()
}
Catch {
Write-Error $_
exit 1
}
Finally {
if ($client -ne $null) { $client.Dispose() }
if ($content -ne $null) { $content.Dispose() }
if ($fileStream -ne $null) { $fileStream.Dispose() }
if ($fileContent -ne $null) { $fileContent.Dispose() }
}
The problem here was what the API required some additional parameters. Initial request required some parameters to accept raw content and specify filename/size. After setting that and getting back proper link to submit, I was able to use:
Invoke-RestMethod -Uri $uri -Method Post -InFile $filePath -ContentType "multipart/form-data"
I found this post and changed it a bit
$fileName = "..."
$uri = "..."
$currentPath = Convert-Path .
$filePath="$currentPath\$fileName"
$fileBin = [System.IO.File]::ReadAlltext($filePath)
$boundary = [System.Guid]::NewGuid().ToString()
$LF = "`r`n"
$bodyLines = (
"--$boundary",
"Content-Disposition: form-data; name=`"file`"; filename=`"$fileName`"",
"Content-Type: application/octet-stream$LF",
$fileBin,
"--$boundary--$LF"
) -join $LF
Invoke-RestMethod -Uri $uri -Method Post -ContentType "multipart/form-data; boundary=`"$boundary`"" -Body $bodyLines
For anyone wondering (like Jelphy) whether David's answer can be used with cookies/credentials, the answer is yes.
First set the session with Invoke-WebRequest:
Invoke-WebRequest -Uri "$LoginUri" -Method Get -SessionVariable 'Session'
Then POST to the Login URL, which stores the authentication cookie in $Session:
$Response = Invoke-WebRequest -Uri "$Uri" -Method Post -Body $Body -WebSession $Session
The steps above are the standard way to deal with session in Powershell. But here is the important part. Before creating the HttpClient, create an HttpClientHandler and set it's CookieContainer property with the cookies from the session:
$ClientMessageHandler = New-Object System.Net.Http.HttpClientHandler
$ClientMessageHandler.CookieContainer = $Session.Cookies
Then pass this object to the HttpClient constructor
$Client = [System.Net.Http.HttpClient]::new($ClientMessageHandler)
Voila, you now have an HttpClient with session cookies set automatically via Invoke-WebRequest. The rest of David's example should work (copied here for completeness):
$MultipartFormData = New-Object System.Net.Http.MultipartFormDataContent
$FileStream = [System.IO.File]::OpenRead($FilePath)
$FileName = [System.IO.Path]::GetFileName($FilePath)
$FileContent = New-Object System.Net.Http.StreamContent($FileStream)
$MultipartFormData.Add($FileContent, $FieldName, $FileName)
$Result = $Client.PostAsync($url, $content).Result
$Result.EnsureSuccessStatusCode()
I had many files to upload with each request, so I factored out this last bit into a lambda function:
function Add-FormFile {
param ([string]$Path, [string]$Name)
if ($Path -ne "")
{
$FileStream = [System.IO.File]::OpenRead($Path)
$FileName = [System.IO.Path]::GetFileName($Path)
$FileContent = [System.Net.Http.StreamContent]::new($FileStream)
$MultipartFormData.Add($FileContent, $Name, $FileName)
}
}