I am currently working on a use case for invoking ReST request in PowerShell. The body of POST request is created dynamically, reading data from a CSV file.
Here is how my final request body should be
{
"#type": "mtTaskParameter",
"name": "$src_sfdc$",
"type": "EXTENDED_SOURCE",
"sourceConnectionId":"00002E0B00000000000C"
},
{
"#type": "mtTaskParameter",
"name": "$tgt_db_del$",
"type": "TARGET",
"targetConnectionId":"00002E0B00000000000D"
},
{
"#type": "mtTaskParameter",
"name": "$tgt_db_ups$",
"type": "TARGET",
"targetConnectionId":"00002E0B00000000000D"
},
{
"#type": "mtTaskParameter",
"name": "$tgt_status$",
"type": "TARGET",
"targetConnectionId":"00002E0B00000000000D"
}
}
Currently I have implemented like below
if($connectionParameterized -eq "true"){
$str = #"
"#type": "mtTaskParameter",
"name": "$name",
"type": "$type"
"#
if($type -eq "SOURCE"){
$sourceConnectionId = <get source id>
$str = $str+
#"
,"sourceConnectionId":"$sourceConnectionId"
"#
}
if($type -eq "TARGET"){
$targetConnectionId = <get target id>
$str = $str+
#"
,"targetConnectionId":"$targetConnectionId"
"#
}
$finalstr = $finalstr+#"
{
$str
},
"#
}
This works fine, but the code becomes really messy and so difficult to scale. Also while printing, the format is not proper.
Is there a better way to handle this?
Note: As evident from the example, the request body contains several special characters like #,$ etc.
This would be easier if you included your CSV, but basically, you can import the CSV as an array of objects, then convert that to JSON.
You can customize the objects that are created from importing the CSV by adding custom members so that the translation to JSON gives you the output you want.
You can also group or filter the array of objects to make different ones depending on certain conditions.
Here's a some sample code that probably won't work directly but should somewhat demonstrate the concept:
$json = Import-Csv -Path C:\my\data.csv |
ForEach-Object -Process {
$row = $_
$propName = $row.Type.ToLower() + 'ConnectionId'
$row | Add-Member -NotePropertyName $propName -NotePropertyValue $out["$mapping_name"].$name -Force -PassThru
} |
ConvertTo-Json
Related
I am running the following command in AWS:
$ aws ec2 describe-images \
--owners self \
--query 'reverse(sort_by(Images,&CreationDate))[:5].{id:ImageId,date:CreationDate}'
[
{
"id": "ami-0a1b2c3d4e5f60001",
"date": "2018-11-28T17:16:38.000Z"
},
{
"id": "ami-0a1b2c3d4e5f60002",
"date": "2018-09-15T13:51:22.000Z"
},
{
"id": "ami-0a1b2c3d4e5f60003",
"date": "2018-08-19T10:22:45.000Z"
},
{
"id": "ami-0a1b2c3d4e5f60004",
"date": "2018-05-03T12:04:02.000Z"
},
{
"id": "ami-0a1b2c3d4e5f60005",
"date": "2017-12-13T17:16:38.000Z"
}
]
I would like to use a foreach loop for each individual id. I have attempted this with text output, however, the foreach loop will only grab the first id. I am new to programming and not sure how I to make this work. I am limited to Powershell.
If you capture this output, which is in JSON format in a variable say $json, you can do the following to iterate over the individual id properties:
# convert the JSON text to an array of PSObjects and loop through
($json | ConvertFrom-Json) | ForEach-Object {
# do something using the items id property
"Found ID: $($_.id)"
}
Or, if you like do it a bit differently you can use
$allItems = $json | ConvertFrom-Json
foreach ($item in $allItems) {
# do something using the items id property
"Found ID: $($item.id)"
}
The $() construct is needed to make sure PowerShell expands the value of $item.id into the string. You can get the same output string by doing "Found ID: {0}" -f $item.id
Given an example object (coverted from JSON):
{
"Id": 1,
"Name": "Pablo",
"UnwantedProperty1XOXO": true,
"UnwantedProperty2XOXO": false,
"Things": [
{
"Name": "Something",
"UnwantedProperty3XOXO": true
}
]
...
}
How can I remove all properties that match a pattern? In the example I want to remove the three properties that end in XOXO.
My current approach is to use -ExcludeProperty like this:
$myObject | Select-Object -Property * -ExcludeProperty *XOXO
That only removes the first two properties. I need to reach deeper into the collection of Things as well. The object will change as well so I can't hardcode a check for Things and there could be many collections.
Indeed, Select-Object -ExcludeProperty does not act recursively - it only acts on the immediate properties - so a custom solution is needed.
Defining function Remove-Property, printed below, should provide the desired recursive logic:
$sampleJson = #'
{
"Id": 1,
"Name": "Pablo",
"UnwantedProperty1XOXO": true,
"UnwantedProperty2XOXO": false,
"Things": [
{
"Name": "Something",
"UnwantedProperty3XOXO": true
}
]
}
'#
$sampleJson | ConvertFrom-Json |
Remove-Property -NamePattern *XOXO |
ConvertTo-Json
An important aside: ConvertFrom-Json limits parsing to depth of just 2 levels by default, so you may have to specify a greater depth with -Depth <n>.
This problematic default behavior is discussed in GitHub issue #8393.
The result is as follows - note how all properties ending in XOXO, across all levels of the hierarchy, were removed:
{
"Id": 1,
"Name": "Pablo",
"Things": [
{
"Name": "Something"
}
]
}
Remove-Property source code
Important: Remove-Property:
assumes that the input objects are custom objects ([pscustomobject]), such as created by ConvertFrom-Json.
it modifies these objects in place, in addition to outputting the modified object; this differs from Select-Object, which creates new objects from the input.
function Remove-Property {
param(
[Parameter(Mandatory, ValueFromPipeline, Position = 0)]
[object] $InputObject,
[Parameter(Mandatory, Position = 1)]
[string] $NamePattern
)
process {
foreach ($el in $InputObject) {
foreach ($propName in $el.psobject.Properties.Name) {
if ($propName -like $NamePattern) {
$el.psobject.Properties.Remove($propName)
}
else {
$null = Remove-Property -InputObject $el.$propName -NamePattern $NamePattern
}
}
}
$InputObject
}
}
I don't prefer this solution, but it does seem easier than recursively traversing an object's nested properties of unknown depths.
$json = #'
{
"Id": 1,
"Name": "Pablo",
"UnwantedProperty1XOXO": true,
"UnwantedProperty2XOXO": false,
"Things": [
{
"Name": "Something",
"UnwantedProperty3XOXO": true
}
]
}
'#
$filter = "XOXO"
$json -replace ".*$filter.*\r?\n" -replace ",(?=\r?\n\W+})" | ConvertFrom-Json
Maybe this will work.
filter Remove-Property ($Name) {
$queue = [Collections.Generic.Queue[object]]::new(#(Get-Variable _))
while ($queue.Count) {
foreach ($elem in $queue.Dequeue().Value) {
$props = $elem.psobject.Properties
foreach ($p in $props) {
if ($p.Name -like $Name) { $props.Remove($p.Name) } else { $queue.Enqueue($p) }
}
}
}
}
The usage is as follows.
$myObject | Remove-Property -Name "*XOXO"
The ConvertFrom-Json Cmdlet by default has a depth of 2. This is most likely causing your issue.
To fix, use this ConvertFrom-Json command:
ConvertFrom-Json $input -Depth 10
Reference: ConvertFrom-Json
I have a json file with the following content -
{
"IsEnabled": true,
"EngineConfiguration": {
"PollInterval": "00:00:15",
"Components": [{
"Id": "Logs",
"FullName": "AWS.EC2.Windows.CloudWatch.CustomLog.CustomLogInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"LogDirectoryPath": "C:\\log\\2018-05-25",
"TimestampFormat": "yyyy-MM-dd HH:mm:ss",
"Encoding": "UTF-8",
"Filter": "",
"CultureName": "en-US",
"TimeZoneKind": "UTC",
"LineCount": "1"
}
}]
}
}
I want to replace this date(mentioned in LogDirectoryPath) everyday using a managed task using powershell.
How can this be done using powershell?
This script will help you to update log directory.
Steps:
Get content of json file.
Update attribute value if exists. $JsonData.update | % {if(...)
Save content in same file.
Script:
$JsonData = Get-Content $JsonFilePath -raw | ConvertFrom-Json
$JsonData.update | % { if($JsonData.engineconfiguration.Components.Parameters.LogDirectoryPath)
{
$JsonData.engineconfiguration.Components.Parameters.LogDirectoryPath = "C:\log\$(Get-Date -Format 'yyyy-MM-dd')"
}
}
$JsonData | ConvertTo-Json -Depth 4 | set-content $JsonFilePath
The first step is converting the content to json:
$json = Convertfrom-json (get-content "\\path\To\Json\File.json")
Then editing the value as desired:
$json.engineconfiguration.Components.Parameters.LogDirectoryPath = "C:\log\$(Get-Date -Format 'yyyy-MM-dd')"
Then writing it back to file:
ConvertTo-Json $json -Depth 4 | Out-File C:\ProgramData\Temp\test.txt -Force
I have also faced the same kind of issue. I was looking to change the records of the below JSON file
{
"SQS_QUEUE_URL": "https://que-url.com/server1",
"SQS_EVENTS_QUEUE_URL": "https://events-server.com/server1/development_events",
"REGION": "region1",
"BUCKET": "test-bucket",
"AE_WORK_PATH": "C:\\workpath\\path1",
"ENV": "env"
}
Finally, I managed to find the easiest way to generate a JSON file from Powershell.
$json = Get-Content "c:\users\bharat.gadade\desktop\test.json" | ConvertFrom-Json
$json.SQS_QUEUE_URL = "https://que-url.com/server2"
$json.SQS_EVENTS_QUEUE_URL = "https://events-server.com/Server2/development_events"
$json.REGION = "region1 "
$json.BUCKET = "test-bucket"
$json.AE_WORK_PATH = "C:\workpath\path1"
$json.ENV = "env"
$json | ConvertTo-Json | Out-File "c:\users\bharat.gadade\desktop\test.json"
The Row Key will be passed in the query string. What is needed in the function to create the "connection string" to the Table Storage?
Assume that you already have an app setting in your Function App named AzureWebJobsStorage that has the connection string to your Table Storage, then to retrieve that value in your PowerShell script, you will add the following,
$connectionString = $env:AzureWebJobsStorage;
However, if you need to just write to Table Storage based on the row key, you could leverage the Table Storage binding that is already supported in Azure Functions.
Let's assume that there is a table named testtable is already created in your Table Storage and that is the table we will need to write to. Then, here's a sample setup that reads the row key from query string of an HTTP-trigger and writes an entry to Table Storage.
function.json:
{
"bindings": [
{
"name": "req",
"type": "httpTrigger",
"direction": "in",
"authLevel": "anonymous"
},
{
"type": "table",
"name": "outputTable",
"tableName": "testtable",
"connection": "AzureWebJobsStorage",
"direction": "out"
},
{
"name": "res",
"type": "http",
"direction": "out"
}
],
"disabled": false
}
run.ps1:
# POST method: $req
$requestBody = Get-Content $req -Raw | ConvertFrom-Json
$name = $requestBody.name
# GET method: each querystring parameter is its own variable
if ($req_query_name)
{
$name = $req_query_name
}
Out-File -Encoding Ascii -FilePath $res -inputObject "Hello $name"
Write-Output "Message entity: '$requestBody'"
$entity = [PSObject]#{
PartitionKey = $requestBody.role
RowKey = $req_query_rowkey
AccountId = $requestBody.id
}
$entity | ConvertTo-Json | Out-File -Encoding UTF8 $outputTable
Test in Postman:
Log view:
2017-07-04T17:21:17.095 Function started (Id=775a36ce-9d71-454c-887c-05f08cfdb877)
2017-07-04T17:21:17.314 Message entity: '#{name=Azure; role=admin; id=78910}'
2017-07-04T17:21:17.314 Function completed (Success, Id=775a36ce-9d71-454c-887c-05f08cfdb877, Duration=222ms)
Table entry view in Azure Storage Explorer:
I'm wondering if there's a simpler way to accomplish this. I have two (JSON) objects, where they have properties that are lists of IPs (the properties are individual IPs). I'm comparing the two object properties to find any matching items and want to remove any matches from the first object ($JSONConverted). I believe I can use the remove feature (which I haven't gotten working yet). I'm really wondering if there's a simpler way to accomplish this.
$JSONConverted = Get-Content -Raw -Path Output.json | ConvertFrom-Json
$FWJSONConverted = Get-Content -Raw -Path FWOutput.json | ConvertFrom-Json
$MatchingIPs = Compare-Object -IncludeEqual -ExcludeDifferent -ReferenceObject $FWJSONConverted.data.value -DifferenceObject $JSONConverted.data.value
$ListOfMatchingIPs = $MatchingIPs.InputObject
$JSONConverted.data.value | ForEach-Object {
foreach ($IP in $ListOfMatchingIPs) {
if ($_ -eq $IP) {
$JSONConverted.remove.($_)
}
}
}
Here's an example of the $JSONConverted data:
{
"number_of_elements": 1134,
"timeout_type": "LAST",
"name": "IP List",
"data": [
{
"last_seen": 1486571563476,
"source": "WORD: WORDS",
"value": "10.10.10.10",
"first_seen": 1486397213696
},
{
"last_seen": 1486736205285,
"source": "WORD: WORDS",
"value": "10.17.24.22",
"first_seen": 1486397813280
},
{
"last_seen": 1486637743793,
"source": "WORD: WORDS",
"value": "10.11.10.10",
"first_seen": 1486398713056
}
],
"creation_time": 1486394698941,
"time_to_live": "1 years 0 mons 3 days 0 hours 0 mins 0.00 secs",
"element_type":"IP"
}
Something like this should suffice (assuming you want to remove the entire child object from the data array):
$JSONConverted.data = $JSONConverted.data | Where-Object {
#($FWJSONConverted.data.value) -notcontains $_.value
}