Loop through individual id's in Powershell - powershell

I am running the following command in AWS:
$ aws ec2 describe-images \
--owners self \
--query 'reverse(sort_by(Images,&CreationDate))[:5].{id:ImageId,date:CreationDate}'
[
{
"id": "ami-0a1b2c3d4e5f60001",
"date": "2018-11-28T17:16:38.000Z"
},
{
"id": "ami-0a1b2c3d4e5f60002",
"date": "2018-09-15T13:51:22.000Z"
},
{
"id": "ami-0a1b2c3d4e5f60003",
"date": "2018-08-19T10:22:45.000Z"
},
{
"id": "ami-0a1b2c3d4e5f60004",
"date": "2018-05-03T12:04:02.000Z"
},
{
"id": "ami-0a1b2c3d4e5f60005",
"date": "2017-12-13T17:16:38.000Z"
}
]
I would like to use a foreach loop for each individual id. I have attempted this with text output, however, the foreach loop will only grab the first id. I am new to programming and not sure how I to make this work. I am limited to Powershell.

If you capture this output, which is in JSON format in a variable say $json, you can do the following to iterate over the individual id properties:
# convert the JSON text to an array of PSObjects and loop through
($json | ConvertFrom-Json) | ForEach-Object {
# do something using the items id property
"Found ID: $($_.id)"
}
Or, if you like do it a bit differently you can use
$allItems = $json | ConvertFrom-Json
foreach ($item in $allItems) {
# do something using the items id property
"Found ID: $($item.id)"
}
The $() construct is needed to make sure PowerShell expands the value of $item.id into the string. You can get the same output string by doing "Found ID: {0}" -f $item.id

Related

Remove PowerShell object properties that match pattern

Given an example object (coverted from JSON):
{
"Id": 1,
"Name": "Pablo",
"UnwantedProperty1XOXO": true,
"UnwantedProperty2XOXO": false,
"Things": [
{
"Name": "Something",
"UnwantedProperty3XOXO": true
}
]
...
}
How can I remove all properties that match a pattern? In the example I want to remove the three properties that end in XOXO.
My current approach is to use -ExcludeProperty like this:
$myObject | Select-Object -Property * -ExcludeProperty *XOXO
That only removes the first two properties. I need to reach deeper into the collection of Things as well. The object will change as well so I can't hardcode a check for Things and there could be many collections.
Indeed, Select-Object -ExcludeProperty does not act recursively - it only acts on the immediate properties - so a custom solution is needed.
Defining function Remove-Property, printed below, should provide the desired recursive logic:
$sampleJson = #'
{
"Id": 1,
"Name": "Pablo",
"UnwantedProperty1XOXO": true,
"UnwantedProperty2XOXO": false,
"Things": [
{
"Name": "Something",
"UnwantedProperty3XOXO": true
}
]
}
'#
$sampleJson | ConvertFrom-Json |
Remove-Property -NamePattern *XOXO |
ConvertTo-Json
An important aside: ConvertFrom-Json limits parsing to depth of just 2 levels by default, so you may have to specify a greater depth with -Depth <n>.
This problematic default behavior is discussed in GitHub issue #8393.
The result is as follows - note how all properties ending in XOXO, across all levels of the hierarchy, were removed:
{
"Id": 1,
"Name": "Pablo",
"Things": [
{
"Name": "Something"
}
]
}
Remove-Property source code
Important: Remove-Property:
assumes that the input objects are custom objects ([pscustomobject]), such as created by ConvertFrom-Json.
it modifies these objects in place, in addition to outputting the modified object; this differs from Select-Object, which creates new objects from the input.
function Remove-Property {
param(
[Parameter(Mandatory, ValueFromPipeline, Position = 0)]
[object] $InputObject,
[Parameter(Mandatory, Position = 1)]
[string] $NamePattern
)
process {
foreach ($el in $InputObject) {
foreach ($propName in $el.psobject.Properties.Name) {
if ($propName -like $NamePattern) {
$el.psobject.Properties.Remove($propName)
}
else {
$null = Remove-Property -InputObject $el.$propName -NamePattern $NamePattern
}
}
}
$InputObject
}
}
I don't prefer this solution, but it does seem easier than recursively traversing an object's nested properties of unknown depths.
$json = #'
{
"Id": 1,
"Name": "Pablo",
"UnwantedProperty1XOXO": true,
"UnwantedProperty2XOXO": false,
"Things": [
{
"Name": "Something",
"UnwantedProperty3XOXO": true
}
]
}
'#
$filter = "XOXO"
$json -replace ".*$filter.*\r?\n" -replace ",(?=\r?\n\W+})" | ConvertFrom-Json
Maybe this will work.
filter Remove-Property ($Name) {
$queue = [Collections.Generic.Queue[object]]::new(#(Get-Variable _))
while ($queue.Count) {
foreach ($elem in $queue.Dequeue().Value) {
$props = $elem.psobject.Properties
foreach ($p in $props) {
if ($p.Name -like $Name) { $props.Remove($p.Name) } else { $queue.Enqueue($p) }
}
}
}
}
The usage is as follows.
$myObject | Remove-Property -Name "*XOXO"
The ConvertFrom-Json Cmdlet by default has a depth of 2. This is most likely causing your issue.
To fix, use this ConvertFrom-Json command:
ConvertFrom-Json $input -Depth 10
Reference: ConvertFrom-Json

Why does ConvertTo-Json drop values

I am using powershell to explore a REST API. I discovered a strange anomaly. When I serialize/descrialize the following objct, the responses object are gone! I have verified that it is all still in tact when converted to .net object, so the problem happens when converting from .net object to JSON.
$json = #'
{
"stubs": [
{
"responses": [
{
"is": {
"body": "54"
}
},
{
"is": {
"body": "21"
}
},
{
"is": {
"body": "0"
}
}
]
}
]
}
'#
$json | ConvertFrom-Json | ConvertTo-Json
The result form the above conversion is this:
{
"stubs": [
{
"responses": " "
}
]
}
If I run this, I receive 54, as expected:
$json | ConvertFrom-Json | %{ $_.stubs.responses[0].is.body }
I am running on Ubuntu, but don't believe that should make a difference. This should be easy to verify for someone on Windows.
ConvertTo-Json takes an optional Depth parameter that (for whatever reason) defaults to 2. The parameter itself is described in the linked docs as follows:
Specifies how many levels of contained objects are included in the JSON representation. The default value is 2.
If you provide a higher value for this parameter, it’ll work - The maximum value is 100:
$json | ConvertFrom-Json | ConvertTo-Json -Depth 100

Replace a value in a json using powershell

I have a json file with the following content -
{
"IsEnabled": true,
"EngineConfiguration": {
"PollInterval": "00:00:15",
"Components": [{
"Id": "Logs",
"FullName": "AWS.EC2.Windows.CloudWatch.CustomLog.CustomLogInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"LogDirectoryPath": "C:\\log\\2018-05-25",
"TimestampFormat": "yyyy-MM-dd HH:mm:ss",
"Encoding": "UTF-8",
"Filter": "",
"CultureName": "en-US",
"TimeZoneKind": "UTC",
"LineCount": "1"
}
}]
}
}
I want to replace this date(mentioned in LogDirectoryPath) everyday using a managed task using powershell.
How can this be done using powershell?
This script will help you to update log directory.
Steps:
Get content of json file.
Update attribute value if exists. $JsonData.update | % {if(...)
Save content in same file.
Script:
$JsonData = Get-Content $JsonFilePath -raw | ConvertFrom-Json
$JsonData.update | % { if($JsonData.engineconfiguration.Components.Parameters.LogDirectoryPath)
{
$JsonData.engineconfiguration.Components.Parameters.LogDirectoryPath = "C:\log\$(Get-Date -Format 'yyyy-MM-dd')"
}
}
$JsonData | ConvertTo-Json -Depth 4 | set-content $JsonFilePath
The first step is converting the content to json:
$json = Convertfrom-json (get-content "\\path\To\Json\File.json")
Then editing the value as desired:
$json.engineconfiguration.Components.Parameters.LogDirectoryPath = "C:\log\$(Get-Date -Format 'yyyy-MM-dd')"
Then writing it back to file:
ConvertTo-Json $json -Depth 4 | Out-File C:\ProgramData\Temp\test.txt -Force
I have also faced the same kind of issue. I was looking to change the records of the below JSON file
{
"SQS_QUEUE_URL": "https://que-url.com/server1",
"SQS_EVENTS_QUEUE_URL": "https://events-server.com/server1/development_events",
"REGION": "region1",
"BUCKET": "test-bucket",
"AE_WORK_PATH": "C:\\workpath\\path1",
"ENV": "env"
}
Finally, I managed to find the easiest way to generate a JSON file from Powershell.
$json = Get-Content "c:\users\bharat.gadade\desktop\test.json" | ConvertFrom-Json
$json.SQS_QUEUE_URL = "https://que-url.com/server2"
$json.SQS_EVENTS_QUEUE_URL = "https://events-server.com/Server2/development_events"
$json.REGION = "region1 "
$json.BUCKET = "test-bucket"
$json.AE_WORK_PATH = "C:\workpath\path1"
$json.ENV = "env"
$json | ConvertTo-Json | Out-File "c:\users\bharat.gadade\desktop\test.json"

Compare-Object Find Matches and Remove Found from First Object

I'm wondering if there's a simpler way to accomplish this. I have two (JSON) objects, where they have properties that are lists of IPs (the properties are individual IPs). I'm comparing the two object properties to find any matching items and want to remove any matches from the first object ($JSONConverted). I believe I can use the remove feature (which I haven't gotten working yet). I'm really wondering if there's a simpler way to accomplish this.
$JSONConverted = Get-Content -Raw -Path Output.json | ConvertFrom-Json
$FWJSONConverted = Get-Content -Raw -Path FWOutput.json | ConvertFrom-Json
$MatchingIPs = Compare-Object -IncludeEqual -ExcludeDifferent -ReferenceObject $FWJSONConverted.data.value -DifferenceObject $JSONConverted.data.value
$ListOfMatchingIPs = $MatchingIPs.InputObject
$JSONConverted.data.value | ForEach-Object {
foreach ($IP in $ListOfMatchingIPs) {
if ($_ -eq $IP) {
$JSONConverted.remove.($_)
}
}
}
Here's an example of the $JSONConverted data:
{
"number_of_elements": 1134,
"timeout_type": "LAST",
"name": "IP List",
"data": [
{
"last_seen": 1486571563476,
"source": "WORD: WORDS",
"value": "10.10.10.10",
"first_seen": 1486397213696
},
{
"last_seen": 1486736205285,
"source": "WORD: WORDS",
"value": "10.17.24.22",
"first_seen": 1486397813280
},
{
"last_seen": 1486637743793,
"source": "WORD: WORDS",
"value": "10.11.10.10",
"first_seen": 1486398713056
}
],
"creation_time": 1486394698941,
"time_to_live": "1 years 0 mons 3 days 0 hours 0 mins 0.00 secs",
"element_type":"IP"
}
Something like this should suffice (assuming you want to remove the entire child object from the data array):
$JSONConverted.data = $JSONConverted.data | Where-Object {
#($FWJSONConverted.data.value) -notcontains $_.value
}

Create a dynamic ReST request body in PowerShell

I am currently working on a use case for invoking ReST request in PowerShell. The body of POST request is created dynamically, reading data from a CSV file.
Here is how my final request body should be
{
"#type": "mtTaskParameter",
"name": "$src_sfdc$",
"type": "EXTENDED_SOURCE",
"sourceConnectionId":"00002E0B00000000000C"
},
{
"#type": "mtTaskParameter",
"name": "$tgt_db_del$",
"type": "TARGET",
"targetConnectionId":"00002E0B00000000000D"
},
{
"#type": "mtTaskParameter",
"name": "$tgt_db_ups$",
"type": "TARGET",
"targetConnectionId":"00002E0B00000000000D"
},
{
"#type": "mtTaskParameter",
"name": "$tgt_status$",
"type": "TARGET",
"targetConnectionId":"00002E0B00000000000D"
}
}
Currently I have implemented like below
if($connectionParameterized -eq "true"){
$str = #"
"#type": "mtTaskParameter",
"name": "$name",
"type": "$type"
"#
if($type -eq "SOURCE"){
$sourceConnectionId = <get source id>
$str = $str+
#"
,"sourceConnectionId":"$sourceConnectionId"
"#
}
if($type -eq "TARGET"){
$targetConnectionId = <get target id>
$str = $str+
#"
,"targetConnectionId":"$targetConnectionId"
"#
}
$finalstr = $finalstr+#"
{
$str
},
"#
}
This works fine, but the code becomes really messy and so difficult to scale. Also while printing, the format is not proper.
Is there a better way to handle this?
Note: As evident from the example, the request body contains several special characters like #,$ etc.
This would be easier if you included your CSV, but basically, you can import the CSV as an array of objects, then convert that to JSON.
You can customize the objects that are created from importing the CSV by adding custom members so that the translation to JSON gives you the output you want.
You can also group or filter the array of objects to make different ones depending on certain conditions.
Here's a some sample code that probably won't work directly but should somewhat demonstrate the concept:
$json = Import-Csv -Path C:\my\data.csv |
ForEach-Object -Process {
$row = $_
$propName = $row.Type.ToLower() + 'ConnectionId'
$row | Add-Member -NotePropertyName $propName -NotePropertyValue $out["$mapping_name"].$name -Force -PassThru
} |
ConvertTo-Json