Remove PowerShell object properties that match pattern - powershell

Given an example object (coverted from JSON):
{
"Id": 1,
"Name": "Pablo",
"UnwantedProperty1XOXO": true,
"UnwantedProperty2XOXO": false,
"Things": [
{
"Name": "Something",
"UnwantedProperty3XOXO": true
}
]
...
}
How can I remove all properties that match a pattern? In the example I want to remove the three properties that end in XOXO.
My current approach is to use -ExcludeProperty like this:
$myObject | Select-Object -Property * -ExcludeProperty *XOXO
That only removes the first two properties. I need to reach deeper into the collection of Things as well. The object will change as well so I can't hardcode a check for Things and there could be many collections.

Indeed, Select-Object -ExcludeProperty does not act recursively - it only acts on the immediate properties - so a custom solution is needed.
Defining function Remove-Property, printed below, should provide the desired recursive logic:
$sampleJson = #'
{
"Id": 1,
"Name": "Pablo",
"UnwantedProperty1XOXO": true,
"UnwantedProperty2XOXO": false,
"Things": [
{
"Name": "Something",
"UnwantedProperty3XOXO": true
}
]
}
'#
$sampleJson | ConvertFrom-Json |
Remove-Property -NamePattern *XOXO |
ConvertTo-Json
An important aside: ConvertFrom-Json limits parsing to depth of just 2 levels by default, so you may have to specify a greater depth with -Depth <n>.
This problematic default behavior is discussed in GitHub issue #8393.
The result is as follows - note how all properties ending in XOXO, across all levels of the hierarchy, were removed:
{
"Id": 1,
"Name": "Pablo",
"Things": [
{
"Name": "Something"
}
]
}
Remove-Property source code
Important: Remove-Property:
assumes that the input objects are custom objects ([pscustomobject]), such as created by ConvertFrom-Json.
it modifies these objects in place, in addition to outputting the modified object; this differs from Select-Object, which creates new objects from the input.
function Remove-Property {
param(
[Parameter(Mandatory, ValueFromPipeline, Position = 0)]
[object] $InputObject,
[Parameter(Mandatory, Position = 1)]
[string] $NamePattern
)
process {
foreach ($el in $InputObject) {
foreach ($propName in $el.psobject.Properties.Name) {
if ($propName -like $NamePattern) {
$el.psobject.Properties.Remove($propName)
}
else {
$null = Remove-Property -InputObject $el.$propName -NamePattern $NamePattern
}
}
}
$InputObject
}
}

I don't prefer this solution, but it does seem easier than recursively traversing an object's nested properties of unknown depths.
$json = #'
{
"Id": 1,
"Name": "Pablo",
"UnwantedProperty1XOXO": true,
"UnwantedProperty2XOXO": false,
"Things": [
{
"Name": "Something",
"UnwantedProperty3XOXO": true
}
]
}
'#
$filter = "XOXO"
$json -replace ".*$filter.*\r?\n" -replace ",(?=\r?\n\W+})" | ConvertFrom-Json

Maybe this will work.
filter Remove-Property ($Name) {
$queue = [Collections.Generic.Queue[object]]::new(#(Get-Variable _))
while ($queue.Count) {
foreach ($elem in $queue.Dequeue().Value) {
$props = $elem.psobject.Properties
foreach ($p in $props) {
if ($p.Name -like $Name) { $props.Remove($p.Name) } else { $queue.Enqueue($p) }
}
}
}
}
The usage is as follows.
$myObject | Remove-Property -Name "*XOXO"

The ConvertFrom-Json Cmdlet by default has a depth of 2. This is most likely causing your issue.
To fix, use this ConvertFrom-Json command:
ConvertFrom-Json $input -Depth 10
Reference: ConvertFrom-Json

Related

Create the "skeleton" of a custom PSObject from scratch

I'm trying to create a PSCustomObject that will later on be converted to JSON as the body of an API request.
This body basically has a structure to add / remove different information to a security management tool policy (filehash / filenames / certificates / IPs / file extensions / folders exceptions, on which technology etc.)
Here is an example of a final JSON with all the possibilities.
As a proof of concept, I'm focusing on the following part which is adding a filename & a filehash to the exception policy
{
"add": {
"applications": [
{
"processfile": {
"sha2": "6ddc5c11925ab348eb0d390ec5179c1d655eb4bf70779f7a4e28b7db485d20ea",
"name": "myfilename"
}
}
]
...
My goal is not to import this specific JSON as a custom Object, (I know this can be done through convertfrom-Json). It's to create some empty object, but with the correct structure matching the JSON format. This way, I would just have to populate and access information in my object like this :
PS C:\PSSymantecCloud> $obj.add.applications
processfile
-----------
#{sha2=6ddc5c11925ab348eb0d390ec5179c1d655eb4bf70779f7a4e28b7db485d20ea; name=myfilename}
PS C:\PSSymantecCloud> $obj.add.applications.processfile
sha2 name
---- ----
6ddc5c11925ab348eb0d390ec5179c1d655eb4bf70779f7a4e28b7db485d20ea myfilename
I've been trying to play around with Format-Custom CmdLet but without success.
Any suggestions on how to create such a custom PSObject ?
I'm fond of PowerShell classes so here is one way to do it with them. You can add new processfile with the AddProcessFile method that takes 2 arguments as in your Json:
class addjson {
[object] $add
addjson() {
$this.add = [pscustomobject]#{
applications = [System.Collections.Generic.List[object]]::new()
}
}
[void] AddProcessFile([string] $sha2, [string] $name) {
$this.add.applications.Add([pscustomobject]#{
processfile = [pscustomobject]#{
sha2 = $sha2
name = $name
}
})
}
}
$add = [addjson]::new()
$add.AddProcessFile('6ddc5c11925ab348eb0d390ec5179c1d655eb4bf70779f7a4e28b7db485d20ea', 'myFile')
$add.AddProcessFile('6ddc5c11925ab348eb0d390ec5179c1d655eb4bf70779f7a4e28b7db485d20ea', 'myFile2')
$add | ConvertTo-Json -Depth 4
Resulting Json would be:
{
"add": {
"applications": [
{
"processfile": {
"sha2": "6ddc5c11925ab348eb0d390ec5179c1d655eb4bf70779f7a4e28b7db485d20ea",
"name": "myFile"
}
},
{
"processfile": {
"sha2": "6ddc5c11925ab348eb0d390ec5179c1d655eb4bf70779f7a4e28b7db485d20ea",
"name": "myFile2"
}
}
]
}
}
Create an psObject shell, assign empty strings ““, or $false to the properties
$Object = New-Object PSObject -Property #{
Name = ““
SecondProperty= ““
yetAnotherProoerty= ““
}
Later u can assign your results like this:
$Object.name= $myResult
Or even add new member aka properties to the object like this:
Object | Add-Member -NotePropertyName Status -NotePropertyValue Done
You can also just take your sample text and convert it with convertto-json. e.g.
$HereString = #"
{
"add": {
"applications": [
{
"processfile": {
"sha2": "6ddc5c11925ab348eb0d390ec5179c1d655eb4bf70779f7a4e28b7db485d20ea",
"name": "myfilename"
}
}
]
}
}
"#
$PSObject = $HereString | ConvertFrom-Json
$PSObject | get-member

Loop through individual id's in Powershell

I am running the following command in AWS:
$ aws ec2 describe-images \
--owners self \
--query 'reverse(sort_by(Images,&CreationDate))[:5].{id:ImageId,date:CreationDate}'
[
{
"id": "ami-0a1b2c3d4e5f60001",
"date": "2018-11-28T17:16:38.000Z"
},
{
"id": "ami-0a1b2c3d4e5f60002",
"date": "2018-09-15T13:51:22.000Z"
},
{
"id": "ami-0a1b2c3d4e5f60003",
"date": "2018-08-19T10:22:45.000Z"
},
{
"id": "ami-0a1b2c3d4e5f60004",
"date": "2018-05-03T12:04:02.000Z"
},
{
"id": "ami-0a1b2c3d4e5f60005",
"date": "2017-12-13T17:16:38.000Z"
}
]
I would like to use a foreach loop for each individual id. I have attempted this with text output, however, the foreach loop will only grab the first id. I am new to programming and not sure how I to make this work. I am limited to Powershell.
If you capture this output, which is in JSON format in a variable say $json, you can do the following to iterate over the individual id properties:
# convert the JSON text to an array of PSObjects and loop through
($json | ConvertFrom-Json) | ForEach-Object {
# do something using the items id property
"Found ID: $($_.id)"
}
Or, if you like do it a bit differently you can use
$allItems = $json | ConvertFrom-Json
foreach ($item in $allItems) {
# do something using the items id property
"Found ID: $($item.id)"
}
The $() construct is needed to make sure PowerShell expands the value of $item.id into the string. You can get the same output string by doing "Found ID: {0}" -f $item.id

Why does ConvertTo-Json drop values

I am using powershell to explore a REST API. I discovered a strange anomaly. When I serialize/descrialize the following objct, the responses object are gone! I have verified that it is all still in tact when converted to .net object, so the problem happens when converting from .net object to JSON.
$json = #'
{
"stubs": [
{
"responses": [
{
"is": {
"body": "54"
}
},
{
"is": {
"body": "21"
}
},
{
"is": {
"body": "0"
}
}
]
}
]
}
'#
$json | ConvertFrom-Json | ConvertTo-Json
The result form the above conversion is this:
{
"stubs": [
{
"responses": " "
}
]
}
If I run this, I receive 54, as expected:
$json | ConvertFrom-Json | %{ $_.stubs.responses[0].is.body }
I am running on Ubuntu, but don't believe that should make a difference. This should be easy to verify for someone on Windows.
ConvertTo-Json takes an optional Depth parameter that (for whatever reason) defaults to 2. The parameter itself is described in the linked docs as follows:
Specifies how many levels of contained objects are included in the JSON representation. The default value is 2.
If you provide a higher value for this parameter, it’ll work - The maximum value is 100:
$json | ConvertFrom-Json | ConvertTo-Json -Depth 100

Compare-Object Find Matches and Remove Found from First Object

I'm wondering if there's a simpler way to accomplish this. I have two (JSON) objects, where they have properties that are lists of IPs (the properties are individual IPs). I'm comparing the two object properties to find any matching items and want to remove any matches from the first object ($JSONConverted). I believe I can use the remove feature (which I haven't gotten working yet). I'm really wondering if there's a simpler way to accomplish this.
$JSONConverted = Get-Content -Raw -Path Output.json | ConvertFrom-Json
$FWJSONConverted = Get-Content -Raw -Path FWOutput.json | ConvertFrom-Json
$MatchingIPs = Compare-Object -IncludeEqual -ExcludeDifferent -ReferenceObject $FWJSONConverted.data.value -DifferenceObject $JSONConverted.data.value
$ListOfMatchingIPs = $MatchingIPs.InputObject
$JSONConverted.data.value | ForEach-Object {
foreach ($IP in $ListOfMatchingIPs) {
if ($_ -eq $IP) {
$JSONConverted.remove.($_)
}
}
}
Here's an example of the $JSONConverted data:
{
"number_of_elements": 1134,
"timeout_type": "LAST",
"name": "IP List",
"data": [
{
"last_seen": 1486571563476,
"source": "WORD: WORDS",
"value": "10.10.10.10",
"first_seen": 1486397213696
},
{
"last_seen": 1486736205285,
"source": "WORD: WORDS",
"value": "10.17.24.22",
"first_seen": 1486397813280
},
{
"last_seen": 1486637743793,
"source": "WORD: WORDS",
"value": "10.11.10.10",
"first_seen": 1486398713056
}
],
"creation_time": 1486394698941,
"time_to_live": "1 years 0 mons 3 days 0 hours 0 mins 0.00 secs",
"element_type":"IP"
}
Something like this should suffice (assuming you want to remove the entire child object from the data array):
$JSONConverted.data = $JSONConverted.data | Where-Object {
#($FWJSONConverted.data.value) -notcontains $_.value
}

Create a dynamic ReST request body in PowerShell

I am currently working on a use case for invoking ReST request in PowerShell. The body of POST request is created dynamically, reading data from a CSV file.
Here is how my final request body should be
{
"#type": "mtTaskParameter",
"name": "$src_sfdc$",
"type": "EXTENDED_SOURCE",
"sourceConnectionId":"00002E0B00000000000C"
},
{
"#type": "mtTaskParameter",
"name": "$tgt_db_del$",
"type": "TARGET",
"targetConnectionId":"00002E0B00000000000D"
},
{
"#type": "mtTaskParameter",
"name": "$tgt_db_ups$",
"type": "TARGET",
"targetConnectionId":"00002E0B00000000000D"
},
{
"#type": "mtTaskParameter",
"name": "$tgt_status$",
"type": "TARGET",
"targetConnectionId":"00002E0B00000000000D"
}
}
Currently I have implemented like below
if($connectionParameterized -eq "true"){
$str = #"
"#type": "mtTaskParameter",
"name": "$name",
"type": "$type"
"#
if($type -eq "SOURCE"){
$sourceConnectionId = <get source id>
$str = $str+
#"
,"sourceConnectionId":"$sourceConnectionId"
"#
}
if($type -eq "TARGET"){
$targetConnectionId = <get target id>
$str = $str+
#"
,"targetConnectionId":"$targetConnectionId"
"#
}
$finalstr = $finalstr+#"
{
$str
},
"#
}
This works fine, but the code becomes really messy and so difficult to scale. Also while printing, the format is not proper.
Is there a better way to handle this?
Note: As evident from the example, the request body contains several special characters like #,$ etc.
This would be easier if you included your CSV, but basically, you can import the CSV as an array of objects, then convert that to JSON.
You can customize the objects that are created from importing the CSV by adding custom members so that the translation to JSON gives you the output you want.
You can also group or filter the array of objects to make different ones depending on certain conditions.
Here's a some sample code that probably won't work directly but should somewhat demonstrate the concept:
$json = Import-Csv -Path C:\my\data.csv |
ForEach-Object -Process {
$row = $_
$propName = $row.Type.ToLower() + 'ConnectionId'
$row | Add-Member -NotePropertyName $propName -NotePropertyValue $out["$mapping_name"].$name -Force -PassThru
} |
ConvertTo-Json