Powershell encode JSON array in URL for RestMethod - powershell

I am using Knack to solve a business process issue, however to ensure it's success I need to run a daily script to sync the data between Knack and our HR system.
Knack uses a REST API and I want to apply a filter to the GET call so that I don't have to populate a local table in order to compare the data.
Knack requires a JSON array like the example below to be encoded in the URL in order to filter the results returned. The example shows how to do this in Javascript but I cannot work out how to do this in Powershell would anyone be able to assist?
// Request route
var api_url = 'https://api.knack.com/v1/objects/object_1/records';
// Prepare filters
var filters = {
'match': 'or',
'rules': [
{
'field':'field_1',
'operator':'is',
'value':'Dodgit'
},
{
'field':'field_1',
'operator':'is blank'
}
]
};
// Add filters to route
api_url += '?filters=' + encodeURIComponent(JSON.stringify(filters));

You can do the following:
$api_url = 'https://api.knack.com/v1/objects/object_1/records'
$filters = #'
{
'match': 'or',
'rules': [
{
'field':'field_1',
'operator':'is',
'value':'Dodgit'
},
{
'field':'field_1',
'operator':'is blank'
}
]
}
'#
$CompressedFilters = $filters | ConvertFrom-Json | ConvertTo-Json -Depth 10 -Compress
$encodedUri = "{0}?filters={1}" -f $api_url,[System.Web.HttpUtility]::UrlEncode($CompressedFilters)
Explanation:
$filters is defined as a here-string. $filters is piped into ConvertFrom-Json and then to ConvertTo-Json in order to compress the JSON data easily. However, that does swap single quotes for double quotes.
UrlEncode() from the System.Web.HttpUtility .NET class encodes special characters in URLs.
-f is the string format operator. This helps to build your URI string. The final URI is stored in $encodedUri.
NOTE: The [System.Web.HttpUtility]::UrlEncode encodes special characters with lowercase hex characters. If your system requires uppercase hex characters, then use [System.Net.WebUtility]::UrlEncode instead.

Related

Accidentally URL Encoding URI Twice, Invalid Signature Response

I'll try to explain this best I can. I'm trying to perform a simple GET against the NetSuite "employees" API (using PowerShell). As you can see in the $query below, this variable needs to be URL encoded (spaces in the query) which I am doing on line 20 of the below snippet. I'm then taking that encoded URL along with a couple other variables and building the $base_string. I use the $base_string to create the Base64 OAuth signature and, on line 36, URL encode the signature. My response from NetSuite is always Invalid Signature.
When I perform any sort of "standard" query (like the one immediately below, without spaces... meaning no changes to the URL after encoding) I do not get an Invalid Signature response. This leads me to believe the problem is entirely related to the more unique query I am attempting and, possibly, the fact that it is being "double-encoded."
I'd appreciate any feedback as I would really benefit from being able to perform a query against the "custentity" variable in the below snippet.
$query = "/services/rest/record/v1/employee/$($netsuite_id)" # This query will find a user via their NetSuite ID.
$url = "https://$($realm.ToLower().Replace("_","-")).suitetalk.api.netsuite.com"
$query = "/services/rest/record/v1/employee?q=custentity_coupa_emp_id IS $($employee_id)" # This query will find a user via a custom entity --- their Coupa ID.
$oauth_nonce = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes([System.DateTime]::Now.Ticks.ToString()))
$oauth_timestamp = [int64](([datetime]::UtcNow)-(Get-Date "1970-01-01")).TotalSeconds
# BUILD THE BASE STRING VARIABLE
$oAuthParamsForSigning = #{}
$oAuthParamsForSigning.Add("oauth_consumer_key",$oauth_consumer_key)
$oAuthParamsForSigning.Add("oauth_token",$oauth_token)
$oAuthParamsForSigning.Add("oauth_signature_method",$oauth_signature_method)
$oAuthParamsForSigning.Add("oauth_nonce",$oauth_nonce)
$oAuthParamsForSigning.Add("oauth_timestamp",$oauth_timestamp)
$oAuthParamsForSigning.Add("oauth_version",$oauth_version)
$oAuthParamsString = ($oAuthParamsForSigning.Keys | Sort-Object | % {
"$_=$($oAuthParamsForSigning[$_])"
}) -join "&"
$encodedOAuthParamsString = [uri]::EscapeDataString($oAuthParamsString)
# BUILD THE ENCODED FULL URL VARIABLE
$encodedUrl = [uri]::EscapeDataString($url+$query)
# BUILD THE OAUTH SIGNATURE VARIABLE: KEY (CONSUMER SECRET + TOKEN SECRET) + BASE STRING
$base_string = $HTTP_method + "&" + $encodedUrl + "&" + $encodedOAuthParamsString
$key = $oauth_consumer_secret + "&" + $oauth_token_secret
$hmacsha256 = New-Object System.Security.Cryptography.HMACSHA256
$hmacsha256.Key = [System.Text.Encoding]::ASCII.GetBytes($key)
$oauth_signature = [System.Convert]::ToBase64String($hmacsha256.ComputeHash([System.Text.Encoding]::ASCII.GetBytes($base_string)))
# BUILD THE HEADERS VARIABLE
$authHeaderString = ($oAuthParamsForSigning.Keys | Sort-Object | % {
"$_=`"$([uri]::EscapeDataString($oAuthParamsForSigning[$_]))`""
}) -join ","
$authHeaderString += ",realm=`"$([uri]::EscapeDataString($realm))`""
$authHeaderString += ",oauth_signature=`"$([uri]::EscapeDataString($oauth_signature))`""
$authHeaders = #{
"Content-Type"="application/json"
;"Prefer"="transient"
;"Authorization"="OAuth $authHeaderString"
;"Accept"="*/*"
;"Cache-Control"="no-cache"
;"Host"="3489459-sb1.suitetalk.api.netsuite.com"
;"Accept-Encoding"="gzip, deflate, br"
;"Cookie"="NS_ROUTING_VERSION=LAGGING"
}
I have no background with PowerShell but I've had this issue previously with Python and it's tricky without using libraries. I found out that:
Your Base URL cannot contain query params. Meaning your base url should be:
"https://$($realm.ToLower().Replace("_","-")).suitetalk.api.netsuite.com/services/rest/record/v1/employee"
Since you're only doing GET request, just throw your query into a payload (JSON formatted). At least with my SuiteQL this is how I did it. Moving the query to payload worked for me.
For POST Request, the parameters will need to be included in the Base String Creation, sorted alphabetically or by value if keys are the same name.
My query for example for SuiteQL was query = json.dumps({ q: SELECT FROM WHERE })

How can I Include brackets in a PowerShell hash table value?

I can successfully make POST requests in Postman using this body in my call, which uses nested key pair values:
{
"meta": null,
"data": {
"type": "spaces_schema"
"attributes": {
"spaces": ["space1", "space2"]
}
}
}
I am transposing the call into PowerShell, and creating the body using a nested hash table. The call returns an error regarding the body when done in PowerShell:
$Body = #{
"meta" = "null"
"data" = #{
"type" = "spaces_schema"
"attributes" = #{
"spaces" = ["space1", "space2"]
}
}
}
The problem I am running into is the "spaces" attribute. I am not sure how to pass the brackets as literal characters in the hash table without wrapping them in quotes, since doing so puts out an error as well and interferes with the quotes used inside of the brackets.
Here is the error:
Invoke-WebRequest : {"errors":[{"additional_data":null,"detail":"Unhandled exception from route request: Expecting value: line 1 column 1 (char 0)"}]}
At C:\Users\{user}\Desktop\{Folder}\{Scriptname}.ps1:26 char:1
+ Invoke-WebRequest -URI 'https://{URL} ...
You are trying too hard to make the object look like JSON syntax.
As said, to define an element's value as array, you need to enclose it in #(). In Json syntax that would display inside square brackets.
Writing "meta" = "null" is wrong too, because you then define it as string with a literal value of "null".
What you should do in the object is meta = $null which wil translate to JSON as "meta": null
In your case, the default -Depth setting of the ConvertTo-Json cmdlet is not deep enough, so you need to set that to a value of at least 3 (in the example you gave).
Try
$Body = #{
meta = $null
data = #{
type = "spaces_schema"
attributes = #{
spaces = #("space1", "space2")
}
}
}
$Body | ConvertTo-Json -Depth 3
Since you will be uploading this data using Invoke-WebRequest or Invoke-RestMethod, I would advice to add switch -Compress to the ConvertTo-Json cmdlet as well. This will return a less human readable json, but the payload will be smaller because all extra whitespace will be removed:
{"meta":null,"data":{"type":"spaces_schema","attributes":{"spaces":["space1","space2"]}}}

Convert Single Line to Multiple Lines

I am new to this Powershell.
I am trying to learn how to modified output.
When I run "Write-output $result | format-list" I have the following output
userDetails : #{id=AA:BB:CC:DD:11:22; connectionStatus=CONNECTED; hostType=WIRELESS;
authType=WPA2/WPA3+802.1x/FT-802.1x}
connectedDevice : {#{deviceDetails=}}
How do I rewrite this output to below using powershell 7.2 ? I would like to have
userDetails :
connectionStatus= CONNECTED
hostType = WIRELESS
authType = WPA2/WPA3+802.1x/FT-802.1x
connectedDevice :
Thank you for your help.
Note: I'm assuming that you're looking for a friendlier display representation of your data. For programmatic processing, Format-* cmdlets should be avoided, for the reasons explained in this answer.
What you're looking for is for Format-List to work recursively, i.e. to not only list the individual properties and their values for each input object itself, but also for nested objects contained in property values.
Format-List does not support this:
Nested objects are represented by their single-line .ToString() representations.
If they're part of a collection (enumerable), the individual elements' representations are joined with , on a single line, and are enclosed in {...}(!) as a whole. How many elements are shown at most is controlled by the $FormatEnumerationLimit preference variable, which defaults to 4.
However, you can approximate recursive listing behavior with Format-Custom; using a simplified example:
# Nested sample object to format.
[pscustomobject]#{
userDetails = [pscustomobject] #{
id = 'AA:BB:CC:DD:11:22'
connectionStatus= 'CONNECTED'
hostType = 'WIRELESS'
authType = 'WPA2/WPA3+802.1x/FT-802.1x'
}
connectedDevice = '...'
} |
Format-Custom -Depth 1 # use higher -Depth levels for multi-level expansion
Output:
class PSCustomObject
{
userDetails =
[
class PSCustomObject
{
id = AA:BB:CC:DD:11:22
connectionStatus = CONNECTED
hostType = WIRELESS
authType = WPA2/WPA3+802.1x/FT-802.1x
}
]
connectedDevice = ...
}
Note:
Caveat: If a custom view happens to be defined for a given input object's type via associated formatting data, it is that custom view that Format-Custom will invoke, not the structural representation shown above; however, this is rare ([datetime] is a rare example).
Apart from the output showing the structure recursively, the format differs from that of Format-List as follows:
Complex objects are enclosed in class <typeName> { ... }
Elements of collections (enumerables) each render on their own (group of) line(s), enclosed in [ ... ] overall. However, as with Format-List, the number of elements that are shown at most is limited by $FormatEnumerationLimit.
To prevent excessively nested output, Format-Custom stops recursing at a depth of 5 by default; you can control the recursion depth via the -Depth parameter, 1 meaning that only objects in immediate child properties are expanded.
When the recursion depth limit is reached, non-collection objects are represented by their .ToString() representations, as with Format-List.
Here is some code that produces output close to your desired output:
# Create sample data
$result = [pscustomobject] #{
userDetails = [pscustomobject]#{ id="AA:BB:CC:DD:11:22"; connectionStatus="CONNECTED"; hostType="WIRELESS"; authType="WPA2/WPA3+802.1x/FT-802.1x"}
connectedDevice = [pscustomobject]#{ deviceDetails=$null }
}
# Produce output
"userDetails :"
($result.userDetails |
Format-List -Property connectionStatus, hostType, authType |
Out-String).Trim() -replace '(?m)(?<=^[^:]+):', '='
"`nconnectedDevice :"
# TODO: add similar code as for .userDetails
Output:
userDetails :
connectionStatus = CONNECTED
hostType = WIRELESS
authType = WPA2/WPA3+802.1x/FT-802.1x
connectedDevice :
Using member access .userDetails to select a child object (similar to Select-Object -ExpandProperty userDetails).
Using Format-List -Property to output a list of the given properties
Using Out-String to create a string from the formatting data that is produced by Format-List. This string looks exactly like the output you normally see on the console.
Use String method .Trim() to remove whitespace (in this case newlines) from the beginning and end.
Use the -replace operator to replace the first : of each line by =. See this regex101 demo for more information.

How to list objects in Google Cloud Storage from PHP

I am trying to list objects in a folder within a Google Cloud Storage bucket. I can get a result with 1000 objects easily (or increase the number if I want) using the following code:
$names = [];
$bucket = $client->bucket('mybucketname');
$options = ['prefix' => 'myfoldername', 'fields' =>' items/name,nextPageToken'];
$objects = $bucket->objects($options);
foreach ($objects as $object) {
$names[] = $object->name();
}
So far so good, but now I want to get the next 1000 objects (or whatever limit I set using maxResults and resultLimit) using the fact that I specified the nextPageToken object. I know that I have to do this by specifying pageToken as an option - it's just that I have no idea how.
I expect my final code will look something like this - what I need is the line of code which retrieves the next page token.
$names = [];
$bucket = $client->bucket('mybucketname');
$options = ['prefix' => 'myfoldername', 'fields' =>' items/name,nextPageToken'];
while(true) {
$objects = $bucket->objects($options);
foreach ($objects as $object) {
$names[] = $object->name();
}
$nextPageToken = $objects->getNextPageTokenSomehowOrOther(); // #todo Need help here!!!!!!!
if (empty($objects) || empty($nextPageToken)){
break;
}
$options['pageToken'] = $nextPageToken;
}
Any ideas?
The nextPageToken is the name of the last object of the first request encoded in Base64.
Here we have an example from the documentation:
{
"kind": "storage#objects",
"nextPageToken": "CgtzaGliYS0yLmpwZw==",
"items": [
objects Resource
…
]
}
If you decode the value "CgtzaGliYS0yLmpwZw==" this will reveal the value "shiba-2.jpg"
Here we have the definition of PageToken based on API documentation:
The pageToken is an encoded field that marks the name and generation of the last
object in the returned list. In a subsequent request using the pageToken, items
that come after the pageToken are shown (up to maxResults).
References:
https://cloud.google.com/storage/docs/json_api/v1/objects/list#parameters
https://cloud.google.com/storage/docs/paginate-results#rest-paginate-results
See ya

How do I read values of a variable of type Map passed from terraform to powershell userdata script?

I need to pass the variable of type map from terraform to powershell userdata script and be able to access the key value pairs of the map in the powershell script. Thank you
userdata.tf
data "template_file" "user_data" {
template = "${file("${path.module}/init.ps1")}"
vars = {
environment = var.env
# I want to pass the values as shown below
hostnames = {"dev":"devhost","test":"testhost","prod":"prodhost"}
}
}
init.ps1
$hostnames = "${hostnames}"
$environment = "${environment}"
if ($environment -eq "dev"){
# print the value of the dev key in the hostname map here
}
The template_file data source is discouraged.
Note In Terraform 0.12 and later, the templatefile function offers a built-in mechanism for rendering a template from a file. Use that function instead, unless you are using Terraform 0.11 or earlier.
The templatefile function is preferred which is why my solution uses it instead.
In either case, only map(string) is supported for template vars. The values must be strings. JSON can encode arbitrary tree structures, including your map of hostnames as strings.
In your terraform code, encode your hostnames to JSON with jsonencode.
userdata.tf:
locals {
user_data = templatefile("${path.module}/init.ps1" ,{
environment = var.env
# I want to pass the values as shown below
hostnames = jsonencode({"dev":"devhost","test":"testhost","prod":"prodhost"})
})
}
In your PowerShell, decode your hostnames from JSON with the ConvertFrom-Json cmdlet.
init.ps1:
$hostnames = '${hostnames}' | ConvertFrom-Json
$environment = "${environment}"
if ($environment -eq "dev"){
# print the value of the dev key in the hostname map here
}
Update: As noted in the comments, -AsHashtable won't necessarily work as it was added in PowerShell 6.0. Windows 10 and Windows Server 2016 include PowerShell 5.1. If you have maps with case-only differences in keys ({"name" = "foo" ; "Name" = "bar"}) then you will need to install PowerShell 6.0 or later and use ConvertFrom-Json -AsHashtable.
In order to include a collection value in a template result you must decide how you want to represent it as a string, because template results are always strings.
PowerShell supports JSON encoding via the ConvertFrom-Json cmdlet, so a JSON string might be a good candidate, although it presents some challenges because you must ensure that the JSON string is written into the result as a valid PowerShell expression, which means we must also apply PowerShell escaping.
Putting that all together, you can adjust the template like this:
$hostnames = '${replace(jsonencode(hostnames), "'", "''")}' | ConvertFrom-Json
$environment = '${replace(environment, "'", "''")}'
if ($environment -eq "dev"){
Write-Output $hostnames["dev"]
}
The jsonencode function produces a JSON-encoded version of the given value. The above then passes that result to replace so that any ' characters in the result will be escaped as '', which then allows placing the entire result in single quotes ' to ensure valid PowerShell syntax.
The result of rendering the template would be something like this:
$hostnames = '{"dev":"devhost","test":"testhost","prod":"prodhost"}' | ConvertFrom-Json -AsHashtable
$environment = 'dev'
if ($environment -eq "dev"){
Write-Output $hostnames["dev"]
}
You seem to be using Terraform 0.12, so you should use the templatefile function instead of the template_file data source. The function is better because it can accept values of any type, whereas the data source can only accept string values (because it is designed for Terraform 0.11).
To use templatefile, find the place where you were previously referring to data.template_file.user_data and use the templatefile function there instead:
templatefile("${path.module}/init.ps1", {
environment = var.env
hostnames = {"dev":"devhost","test":"testhost","prod":"prodhost"}
})
You can then remove the data "template_file" "user_data" block, because this templatefile function call replaces it.