I'm wondering if there's a simpler way to accomplish this. I have two (JSON) objects, where they have properties that are lists of IPs (the properties are individual IPs). I'm comparing the two object properties to find any matching items and want to remove any matches from the first object ($JSONConverted). I believe I can use the remove feature (which I haven't gotten working yet). I'm really wondering if there's a simpler way to accomplish this.
$JSONConverted = Get-Content -Raw -Path Output.json | ConvertFrom-Json
$FWJSONConverted = Get-Content -Raw -Path FWOutput.json | ConvertFrom-Json
$MatchingIPs = Compare-Object -IncludeEqual -ExcludeDifferent -ReferenceObject $FWJSONConverted.data.value -DifferenceObject $JSONConverted.data.value
$ListOfMatchingIPs = $MatchingIPs.InputObject
$JSONConverted.data.value | ForEach-Object {
foreach ($IP in $ListOfMatchingIPs) {
if ($_ -eq $IP) {
$JSONConverted.remove.($_)
}
}
}
Here's an example of the $JSONConverted data:
{
"number_of_elements": 1134,
"timeout_type": "LAST",
"name": "IP List",
"data": [
{
"last_seen": 1486571563476,
"source": "WORD: WORDS",
"value": "10.10.10.10",
"first_seen": 1486397213696
},
{
"last_seen": 1486736205285,
"source": "WORD: WORDS",
"value": "10.17.24.22",
"first_seen": 1486397813280
},
{
"last_seen": 1486637743793,
"source": "WORD: WORDS",
"value": "10.11.10.10",
"first_seen": 1486398713056
}
],
"creation_time": 1486394698941,
"time_to_live": "1 years 0 mons 3 days 0 hours 0 mins 0.00 secs",
"element_type":"IP"
}
Something like this should suffice (assuming you want to remove the entire child object from the data array):
$JSONConverted.data = $JSONConverted.data | Where-Object {
#($FWJSONConverted.data.value) -notcontains $_.value
}
Related
I have two json files (about 20mb each).
Structure is something like that
{
"Property1": [
"https://testweb.site"
],
"Property2": "abc",
"Property3": [
"7676"
],
"Property4": "some string",
"Property5": "some string",
"Property6": "test",
"Property7": "C:\\Folder\\Test\\file.txt"
}
What I do:
$oldContent = Get-Content file1.json | ConvertFrom-Json
$newContent = Get-Content file2.json | ConvertFrom-Json
$diff = Compare-Object $oldContent $newContent -Property Property1, Property2, Property3 ...
I should compare the selected properties and find what is new in json2.
The problem:
This has been working for the last few months, but now is like stuck in forever on the compare step.
Doesn't consume Memory or CPU. I'm not sure if the compare is too complicated or something else.
Questions:
Is there any other better way to make this comparison?
Any suggestion what is the reason?
I am running the following command in AWS:
$ aws ec2 describe-images \
--owners self \
--query 'reverse(sort_by(Images,&CreationDate))[:5].{id:ImageId,date:CreationDate}'
[
{
"id": "ami-0a1b2c3d4e5f60001",
"date": "2018-11-28T17:16:38.000Z"
},
{
"id": "ami-0a1b2c3d4e5f60002",
"date": "2018-09-15T13:51:22.000Z"
},
{
"id": "ami-0a1b2c3d4e5f60003",
"date": "2018-08-19T10:22:45.000Z"
},
{
"id": "ami-0a1b2c3d4e5f60004",
"date": "2018-05-03T12:04:02.000Z"
},
{
"id": "ami-0a1b2c3d4e5f60005",
"date": "2017-12-13T17:16:38.000Z"
}
]
I would like to use a foreach loop for each individual id. I have attempted this with text output, however, the foreach loop will only grab the first id. I am new to programming and not sure how I to make this work. I am limited to Powershell.
If you capture this output, which is in JSON format in a variable say $json, you can do the following to iterate over the individual id properties:
# convert the JSON text to an array of PSObjects and loop through
($json | ConvertFrom-Json) | ForEach-Object {
# do something using the items id property
"Found ID: $($_.id)"
}
Or, if you like do it a bit differently you can use
$allItems = $json | ConvertFrom-Json
foreach ($item in $allItems) {
# do something using the items id property
"Found ID: $($item.id)"
}
The $() construct is needed to make sure PowerShell expands the value of $item.id into the string. You can get the same output string by doing "Found ID: {0}" -f $item.id
Given an example object (coverted from JSON):
{
"Id": 1,
"Name": "Pablo",
"UnwantedProperty1XOXO": true,
"UnwantedProperty2XOXO": false,
"Things": [
{
"Name": "Something",
"UnwantedProperty3XOXO": true
}
]
...
}
How can I remove all properties that match a pattern? In the example I want to remove the three properties that end in XOXO.
My current approach is to use -ExcludeProperty like this:
$myObject | Select-Object -Property * -ExcludeProperty *XOXO
That only removes the first two properties. I need to reach deeper into the collection of Things as well. The object will change as well so I can't hardcode a check for Things and there could be many collections.
Indeed, Select-Object -ExcludeProperty does not act recursively - it only acts on the immediate properties - so a custom solution is needed.
Defining function Remove-Property, printed below, should provide the desired recursive logic:
$sampleJson = #'
{
"Id": 1,
"Name": "Pablo",
"UnwantedProperty1XOXO": true,
"UnwantedProperty2XOXO": false,
"Things": [
{
"Name": "Something",
"UnwantedProperty3XOXO": true
}
]
}
'#
$sampleJson | ConvertFrom-Json |
Remove-Property -NamePattern *XOXO |
ConvertTo-Json
An important aside: ConvertFrom-Json limits parsing to depth of just 2 levels by default, so you may have to specify a greater depth with -Depth <n>.
This problematic default behavior is discussed in GitHub issue #8393.
The result is as follows - note how all properties ending in XOXO, across all levels of the hierarchy, were removed:
{
"Id": 1,
"Name": "Pablo",
"Things": [
{
"Name": "Something"
}
]
}
Remove-Property source code
Important: Remove-Property:
assumes that the input objects are custom objects ([pscustomobject]), such as created by ConvertFrom-Json.
it modifies these objects in place, in addition to outputting the modified object; this differs from Select-Object, which creates new objects from the input.
function Remove-Property {
param(
[Parameter(Mandatory, ValueFromPipeline, Position = 0)]
[object] $InputObject,
[Parameter(Mandatory, Position = 1)]
[string] $NamePattern
)
process {
foreach ($el in $InputObject) {
foreach ($propName in $el.psobject.Properties.Name) {
if ($propName -like $NamePattern) {
$el.psobject.Properties.Remove($propName)
}
else {
$null = Remove-Property -InputObject $el.$propName -NamePattern $NamePattern
}
}
}
$InputObject
}
}
I don't prefer this solution, but it does seem easier than recursively traversing an object's nested properties of unknown depths.
$json = #'
{
"Id": 1,
"Name": "Pablo",
"UnwantedProperty1XOXO": true,
"UnwantedProperty2XOXO": false,
"Things": [
{
"Name": "Something",
"UnwantedProperty3XOXO": true
}
]
}
'#
$filter = "XOXO"
$json -replace ".*$filter.*\r?\n" -replace ",(?=\r?\n\W+})" | ConvertFrom-Json
Maybe this will work.
filter Remove-Property ($Name) {
$queue = [Collections.Generic.Queue[object]]::new(#(Get-Variable _))
while ($queue.Count) {
foreach ($elem in $queue.Dequeue().Value) {
$props = $elem.psobject.Properties
foreach ($p in $props) {
if ($p.Name -like $Name) { $props.Remove($p.Name) } else { $queue.Enqueue($p) }
}
}
}
}
The usage is as follows.
$myObject | Remove-Property -Name "*XOXO"
The ConvertFrom-Json Cmdlet by default has a depth of 2. This is most likely causing your issue.
To fix, use this ConvertFrom-Json command:
ConvertFrom-Json $input -Depth 10
Reference: ConvertFrom-Json
I'm working on a function to export audit logs from Office 365. When dumping the logs, the AuditData field contains what I feel is the useful information. I'm looking to export only that field and convert it to a CSV. Below is my function.
Function ExportAuditLog($User) {
$logResults = #()
if ($user -eq $null) {
$user = SelectUser "Please select a user to audit"
}
$logData = Search-UnifiedAuditLog -StartDate ((Get-Date).AddDays(-30)) -EndDate (Get-Date) -UserIds $user -Operations MailboxLogin -Formatted | select AuditData
foreach ($entry in $logData) {
$logResults += $entry.AuditData
}
return $logResults
}
This is the output.
{
"CreationTime": "2018-12-01T14:08:55",
"Id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"Operation": "MailboxLogin",
"OrganizationId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"RecordType": "ExchangeItem",
"ResultStatus": "Succeeded",
"UserKey": "xxxxxxxxxxxxxxxx",
"UserType": "Regular",
"Version": 1,
"Workload": "Exchange",
"UserId": "user#domain.com",
"ClientIPAddress": "1.2.3.4",
"ClientInfoString":
"Client=Microsoft.Exchange.Autodiscover; Microsoft Office/16.0 (Windows NT 10.0; Microsoft Outlook 16.0.11001; Pro)",
"ExternalAccess": false,
"InternalLogonType": 0,
"LogonType": 0,
"LogonUserSid": "x-x-x-xx-xxxxxxxxxx-xxxxxxxxxx-xxxxxxxxx-xxxxxxxx",
"MailboxGuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"MailboxOwnerSid": "x-x-x-xx-xxxxxxxxxx-xxxxxxxxxx-xxxxxxxxx-xxxxxxxx",
"MailboxOwnerUPN": "user#domain.com",
"OrganizationName": "domain.com",
"OriginatingServer": "",
"SessionId": ""
}
My goal is to have the first part of each line serve as the column header and the second part is the data.
Use the ConvertFrom-Json cmdlet to convert the field into a Powershell object:
$logResults += ConvertFrom-Json $entry.AuditData
Then use the Export-Csv cmdlet to export it to a csv file:
$logResults | Export-Csv -Path AuditData.csv -NoTypeInformation
I am currently working on a use case for invoking ReST request in PowerShell. The body of POST request is created dynamically, reading data from a CSV file.
Here is how my final request body should be
{
"#type": "mtTaskParameter",
"name": "$src_sfdc$",
"type": "EXTENDED_SOURCE",
"sourceConnectionId":"00002E0B00000000000C"
},
{
"#type": "mtTaskParameter",
"name": "$tgt_db_del$",
"type": "TARGET",
"targetConnectionId":"00002E0B00000000000D"
},
{
"#type": "mtTaskParameter",
"name": "$tgt_db_ups$",
"type": "TARGET",
"targetConnectionId":"00002E0B00000000000D"
},
{
"#type": "mtTaskParameter",
"name": "$tgt_status$",
"type": "TARGET",
"targetConnectionId":"00002E0B00000000000D"
}
}
Currently I have implemented like below
if($connectionParameterized -eq "true"){
$str = #"
"#type": "mtTaskParameter",
"name": "$name",
"type": "$type"
"#
if($type -eq "SOURCE"){
$sourceConnectionId = <get source id>
$str = $str+
#"
,"sourceConnectionId":"$sourceConnectionId"
"#
}
if($type -eq "TARGET"){
$targetConnectionId = <get target id>
$str = $str+
#"
,"targetConnectionId":"$targetConnectionId"
"#
}
$finalstr = $finalstr+#"
{
$str
},
"#
}
This works fine, but the code becomes really messy and so difficult to scale. Also while printing, the format is not proper.
Is there a better way to handle this?
Note: As evident from the example, the request body contains several special characters like #,$ etc.
This would be easier if you included your CSV, but basically, you can import the CSV as an array of objects, then convert that to JSON.
You can customize the objects that are created from importing the CSV by adding custom members so that the translation to JSON gives you the output you want.
You can also group or filter the array of objects to make different ones depending on certain conditions.
Here's a some sample code that probably won't work directly but should somewhat demonstrate the concept:
$json = Import-Csv -Path C:\my\data.csv |
ForEach-Object -Process {
$row = $_
$propName = $row.Type.ToLower() + 'ConnectionId'
$row | Add-Member -NotePropertyName $propName -NotePropertyValue $out["$mapping_name"].$name -Force -PassThru
} |
ConvertTo-Json