Json to CSV select second level columns - powershell

I have a json in the below format, where some columns are at 2nd level. I want to the column at the second level to be at the first level in CSV
{
"ApiKey": "123",
"BasicDetails": {
"ClientID": "1234",
"CompanyName": "C1",
"ContactName": "",
"EmailAddress": "",
"Country": "United Kingdom",
"TimeZone": "(GMT+00:00)"
}}
Required Output is:
APIKey, ClientID, CompanyName, ContactName, EmailAddress,Country,Timezone
123,1234,C1,,,United Kingdom,(GMT+00:00)
I have tried:
(GET-Content F:/my.json -RAW | ConvertFrom-Json)|Select Columns|Export-CSV F:/my.csv
EDIT
Actual Json:
{
"ApiKey": "123",
"BasicDetails": {
"ClientID": "1234",
"CompanyName": "C1",
"ContactName": "",
"EmailAddress": "",
"Country": "United Kingdom",
"TimeZone": "(GMT+00:00)"
}
"BillingDetails": {
"CurrentTier": "0 - 500",
"CurrentMonthlyRate": 9.0000,
"MarkupPercentage": 0,
"MonthlyScheme": "Basic",
"Currency": "USD",
"ClientPays": false
}}
I need Belling Detail columns, Basic Details column and API key at one level

This should work for a all JSON files with a depth of two (which you requested):
$tempObject = #{}
(GET-Content F:/my.json -RAW | convertfrom-json).PsObject.Properties | ForEach-Object {
if ($_.TypeNameOfValue -eq 'System.Management.Automation.PSCustomObject')
{
$_.Value.PsObject.Properties | ForEach-Object {
$tempObject += #{$_.Name = $_.Value}
}
}
else
{
$tempObject += #{$_.Name = $_.Value}
}
}
[PsCustomObject]$tempObject | Export-CSV F:/my.csv

Tricky question. Depends very much if you are looking for a solution for your specific case or a more general approach.
For the first, go with Martin Brandl's answer.
Else, there is also ConvertTo-FlatObject:
https://gallery.technet.microsoft.com/ConvertTo-FlatObject-396a6e0a

Related

PowerShell Replace value in JSON

In our Azure CICD Pipeline, we have an element where we are trying to deploy Policies. We have JSON file per policy in the repo and we bring all these json files together into one file as part of CI which later is deployed via the CD. The PowerShell wasn't written by me, but a Microsoft consultant who was on site a few years back.
The problem is that when all the JSON comes together, we get an illegal syntax e.g.
Altering the code to this works and deploys, but means we have to go through all our files manually replace [ with [[:
In summary the PowerShell bring all of this together, does some manipulation and outputs to a file in the artifacts folder.
This is just a small snippet of the json, but highlights the area and there are many areas like this in the total json that need replacing:
{
"functions": [
],
"variables": {
"location": "UK South"
},
"resources": [{
"properties": {
"displayName": "Allowed Locations for Resources",
"policyType": "Custom",
"mode": "Indexed",
"description": "description.",
"metadata": {
"version": "1.0.0",
"category": "General"
},
"parameters": {
"listOfAllowedLocations": {
"type": "Array",
"metadata": {
"description": "The list of locations that can be specified when deploying resources.",
"strongType": "location",
"displayName": "Allowed locations"
},
"allowedValues": [
"uksouth",
"ukwest"
],
"defaultValue": [
"uksouth",
"ukwest"
]
}
},
"policyRule": {
"if": {
"allOf": [{
"field": "location",
"notIn": "[parameters('listOfAllowedLocations')]"
},
{
"field": "location",
"notEquals": "global"
},
{
"field": "type",
"notEquals": "Microsoft.Resources/subscriptions/resourceGroups"
},
{
"field": "type",
"notEquals": "Microsoft.Resources/b2cDirectories"
}
]
},
"then": {
"effect": "audit"
}
}
},
"name": "Policy1",
"apiVersion": "2019-01-01",
"type": "Microsoft.Authorization/policyDefinitions",
"location": "[variables('location')]"
}]
}
My PowerShell is intro level at best, so I am struggling to get a replace working.
I can obtain the offending area and replace it in a Write-Host, but I don't know how to write the back to the originating object with without making a right mess of things:
if ($content.properties.policyRule.if.allOf -ne $null){
foreach ($param in $content.properties.policyRule.if.allOf){
Write-Host "were here..................."
#$param = ($param | ConvertTo-Json -Depth 100 | % { [System.Text.RegularExpressions.Regex]::Unescape($_) })
if ($param.notIn -ne $null){
$param.notIn.replace('[', '[[')
Write-Host $param.notIn
}
}
Any suggestions would be grateful.
The point is that the allOf node contains an array. Due to the member-access enumeration feature you will be able to conveniently read the notIn property but to write to it, you will need to be specific on the index ([0]) in the allOf node:
$Data = ConvertFrom-Json $Json # $Json contains your $Json snippet
$Data.resources.properties.policyRule.if.allOf[0].notIn = "[[parameters('listOfAllowedLocations')]"
$Data |ConvertTo-Json -Depth 9
In case you want to recursively find your items based on e.g. a specific name and value format from a specific property level, you might use this common reusable function to recursively find (and replace) a node in a complex PowerShell object:
function Get-Node {
[CmdletBinding()][OutputType([Object[]])] param(
[ScriptBlock]$Where,
[AllowNull()][Parameter(ValueFromPipeLine = $True, Mandatory = $True)]$InputObject,
[Int]$Depth = 10
)
process {
if ($_ -isnot [String] -and $Depth -gt 0) {
if ($_ -is [Collections.IDictionary]) {
if (& $Where) { $_ }
$_.get_Values() | Get-Node -Where $Where -Depth ($Depth - 1)
}
elseif ($_ -is [Collections.IEnumerable]) {
for ($i = 0; $i -lt $_.get_Count(); $i++) { $_[$i] | Get-Node -Where $Where -Depth ($Depth - 1) }
}
elseif ($Nodes = $_.PSObject.Properties.Where{ $_.MemberType -eq 'NoteProperty' }) {
$Nodes.ForEach{
if (& $Where) { $_ }
$_.Value | Get-Node -Where $Where -Depth ($Depth - 1)
}
}
}
}
}
Usage
Finding node(s) with a specific name and value (-format):
$Node = $Data.resources.properties.policyRule.if |Get-Node -Where {
$_.name -eq 'notIn' -and $_.value -Match "^\[\w+\('\w+'\)\]$"
}
$Node
Value : [parameters('listOfAllowedLocations')]
MemberType : NoteProperty
IsSettable : True
IsGettable : True
TypeNameOfValue : System.String
Name : notIn
IsInstance : True
Replacing the value of the found node(s):
$Node |ForEach-Object {
$_.Value = '[' + $_.Value
}
$Data |ConvertTo-Json -Depth 9
Results
{
"functions": [],
"variables": {
"location": "UK South"
},
"resources": [
{
"properties": {
"displayName": "Allowed Locations for Resources",
"policyType": "Custom",
"mode": "Indexed",
"description": "description.",
"metadata": {
"version": "1.0.0",
"category": "General"
},
"parameters": {
"listOfAllowedLocations": {
"type": "Array",
"metadata": {
"description": "The list of locations that can be specified when deploying resources.",
"strongType": "location",
"displayName": "Allowed locations"
},
"allowedValues": [
"uksouth",
"ukwest"
],
"defaultValue": [
"uksouth",
"ukwest"
]
}
},
"policyRule": {
"if": {
"allOf": [
{
"field": "location",
"notIn": "[[parameters('listOfAllowedLocations')]"
},
{
"field": "location",
"notEquals": "global"
},
{
"field": "type",
"notEquals": "Microsoft.Resources/subscriptions/resourceGroups"
},
{
"field": "type",
"notEquals": "Microsoft.Resources/b2cDirectories"
}
]
},
"then": {
"effect": "audit"
}
}
},
"name": "Policy1",
"apiVersion": "2019-01-01",
"type": "Microsoft.Authorization/policyDefinitions",
"location": "[variables('location')]"
}
]
}
Update 2022-11-21
Resolved an issue with $Null values in the Get-Node function, see also: PowerShell FilterScript error with some JSON Files (thanks
mklement0).

Convert JSON-String to proper Object in Powershell

I spend 3 hours debugging, but I don't get the rootcause.
I have text (in JSON Format) in a variable. I convert it with "Convert-From-JSON" to an actual object. I was hoping that I can then work with the properties of the object. But it does not work :( It seems that the conversion only gives me two tables (status and data). What am I doing wrong?
My code:
$results = {
"status": "FINISHED",
"data": {
"results": [
{
"id": "11565C230500",
"custom": {
"image": "XXXX",
"name": "XXXX",
"articleNumber": "4032423423505"
}
},
{
"id": "22739L5F16243",
"custom": {
"image": "XXXX",
"name": "XXXX",
"articleNumber": "4032423423505"
}
},
{
"id": "3304332724004",
"custom": {
"image": "XXXX",
"name": "XXXX",
"articleNumber": "4032423423505"
}
}
]
}
}
Now putting it as Object:
$resultObject = ConvertFrom-JSON -InputObject $result
Now trying to retrieve:
$resultObject.id | where {$_.id -eq '11565C230500'}
As it is now, you define $results as script block where it should be a string:
$results = #'
{
"status": "FINISHED",
"data": {
"results": [{
"id": "11565C230500",
"custom": {
"image": "XXXX",
"name": "XXXX",
"articleNumber": "4032423423505"
}
},
{
"id": "22739L5F16243",
"custom": {
"image": "XXXX",
"name": "XXXX",
"articleNumber": "4032423423505"
}
},
{
"id": "3304332724004",
"custom": {
"image": "XXXX",
"name": "XXXX",
"articleNumber": "4032423423505"
}
}
]
}
}
'#
Now you can convert it from the JSON string
$resultObject = $results | ConvertFrom-Json
Then, to parse out the nested property, you need to follow the data structure:
$resultObject.data.results | Where-Object {$_.id -eq '11565C230500'}
will give you this:
id custom
-- ------
11565C230500 #{image=XXXX; name=XXXX; articleNumber=4032423423505}
If you want to get the articleNumber from the custom object that contains, dig deeper still:
($resultObject.data.results | Where-Object {$_.id -eq '11565C230500'}).custom.articleNumber
which will return 4032423423505
Assuming the json is single quoted in that example, otherwise it's a scriptblock, and "$results" is used in place of "$result", the id property is underneath data.results. You couldn't examine the id property with where-object if you already expanded it. That's an alternate form of where-object. I'm expanding the "custom" property with the foreach-object alias "%". Note one id number is shorter. Beware convertto-json's default depth of 2.
$results = '{"status":"FINISHED","data":{"results":[
{"id":"11565C230500", "custom":{"image":"XXXX","name":"XXXX","articleNumber":"4032423423505"}},
{"id":"22739L5F16243","custom":{"image":"XXXX","name":"XXXX","articleNumber":"4032423423505"}},
{"id":"3304332724004","custom":{"image":"XXXX","name":"XXXX","articleNumber":"4032423423505"}}
]}}'
$resultObject = ConvertFrom-JSON -InputObject $results
$resultObject.data.results | where id -eq 11565C230500
id custom
-- ------
11565C230500 #{image=XXXX; name=XXXX; articleNumber=4032423423505}
$resultobject.data.results | where id -eq 11565C230500 | % custom
image name articleNumber
----- ---- -------------
XXXX XXXX 4032423423505

Powershell Iterate through multidimensional array of hashtables to find a match and combine values from both arrays

I need to combine values from 2 JSONs:
If there is a match in alerts IDs, I need to create structure, that will take data from both jsons
Result for a match should look like:
$array = #()
$hashtable = #{}
$hashtable.AlertID (does not matter what JSON is it from)
$hashtable.Tags (from JSON 1)
$hashtable.IncidentName (from JSON2)
$hashtable.IncidentID (from JSON2)
$array += $hashtable
I would prefer if this would be done with c style powershell loop.
c style for loop = for ($x = 0; $x -array.count; $x++)
JSON 1:
[
{
"Status": "Active",
"IncidentId": "3",
"tags": "SINC0008009",
"AlertId": [
"da637563185629568182_-638872186",
"da637563185631732095_1120592736",
"da637563185706412029_-614525914",
"da637563185760439486_-276692370",
"da637563185856325888_-1949235651",
"da637563186785996176_2128073884",
"da637563186789897000_1239551047",
"da637563186806513555_1512241399",
"da637563193194338043_-244132089"
],
"severity": "Medium"
},
{
"Status": "Active",
"IncidentId": "4",
"tags": "SINC0008008",
"AlertId": [
"da637643650725801726_1735022501",
"da637643650741237104_1473290917",
"da637643650748739479_-40211355",
"da637643652767933265_-1887823168",
"da637643670830160376_-443360743"
],
"severity": "Medium"
},
{
"Status": "Active",
"IncidentId": "2",
"tags": null,
"AlertId": [
"caD76232A5-F386-3C5D-94CD-7C82A7F778DC"
],
"severity": "Medium"
},
{
"Status": "Active",
"IncidentId": "1",
"tags": null,
"AlertId": [
"ca6534FF45-D62A-3FB7-BD6B-FF5029C553DB"
],
"severity": "Medium"
}
]
JSON 2:
{
"value": [
{
"incidentId": 3,
"incidentName": "Multi-stage incident involving Initial access & Discovery on one endpoint",
"status": "Active",
"severity": "Medium",
"tags": ["SINC0000001"],
"comments": [],
"alerts": [
{
"alertId": "da637563185629568182_-638872186",
"incidentId": 3,
"description": "A suspicious PowerShell activity was observed on the machine. ",
"status": "New",
"severity": "Medium",
"devices": [
{
"deviceDnsName": "xxxxx"
}
],
"entities": [
{
"entityType": "User",
"accountName": "xxxxxx",
"userPrincipalName": "xxx#xx.xx"
},
{
"entityType": "Process"
},
{
"entityType": "Process",
"verdict": "Suspicious"
},
{
"entityType": "File"
}
]
},
{
"alertId": "da637563185631732095_1120592736",
"incidentId": 3,
"devices": [
{
"osPlatform": "Windows10",
"version": "1909"
}
],
"entities": [
{
"entityType": "User",
"remediationStatus": "None"
}
]
}
]
},
{
"incidentId": 4,
"incidentName": "Multi-stage incident involving Initial access & Discovery on one endpoint",
"status": "Active",
"severity": "Medium",
"tags": ["SINC0000002"],
"comments": [],
"alerts": [
{
"alertId": "da637563185629568182_-638872186",
"incidentId": 3,
"description": "A suspicious PowerShell activity was observed on the machine. ",
"status": "New",
"severity": "Medium",
"devices": [
{
"deviceDnsName": "xxxxx"
}
],
"entities": [
{
"entityType": "User",
"accountName": "xxxxxx",
"userPrincipalName": "xxx#xx.xx"
},
{
"entityType": "Process"
},
{
"entityType": "Process",
"verdict": "Suspicious"
},
{
"entityType": "File"
}
]
},
{
"alertId": "da637563185631732095_1120592736",
"incidentId": 3,
"devices": [
{
"osPlatform": "Windows10",
"version": "1909"
}
],
"entities": [
{
"entityType": "User",
"remediationStatus": "None"
}
]
}
]
}
]
}
Till now, I was looking into using nested foreach loop to address it but it does not behave like I want. I am looking for for loop as I could use the indexes.
Instead of creating an array of Hashtables, I think it's better to create an array of PsCustomObjects, because outputting the result to console/file/json would be a lot easier then.
$json1 = Get-Content -Path 'X:\json1.json' -Raw | ConvertFrom-Json
$json2 = Get-Content -Path 'X:\json2.json' -Raw | ConvertFrom-Json
$result = foreach ($incident in $json1) {
foreach ($alertId in $incident.AlertId) {
$json2.value | Where-Object { $_.alerts.alertId -eq $alertId } | ForEach-Object {
# output an object with the wanted properties
[PsCustomObject]#{
AlertID = $alertId # from json1
Tags = $incident.Tags # from json1
IncidentName = $_.incidentName # from json2
IncidentID = $_.incidentId # from json2
}
}
}
}
# output on screen
$result | Format-Table -AutoSize # or use Out-GridView
# output to new JSON
$result | ConvertTo-Json
# output to CSV file
$result | Export-Csv -Path 'X:\incidents.csv' -NoTypeInformation
Using your examples, the output to console window is:
AlertID Tags IncidentName IncidentID
------- ---- ------------ ----------
da637563185629568182_-638872186 SINC0008009 Multi-stage incident involving Initial access & Discovery on one endpoint 3
da637563185629568182_-638872186 SINC0008009 Multi-stage incident involving Initial access & Discovery on one endpoint 4
da637563185631732095_1120592736 SINC0008009 Multi-stage incident involving Initial access & Discovery on one endpoint 3
da637563185631732095_1120592736 SINC0008009 Multi-stage incident involving Initial access & Discovery on one endpoint 4

Using Powershell scipt following json will show on CSV

JSON
{
"members": [
{
"id": "4b3556f1-df58-6c3c-848b-022fc6a8668d",
"user": "#{subjectKind=user; metaType=member; domain=2161a74d-1c3e-4d34-a8c8-131360d2e92c; principalName=abc#my.com; mailAddress=abc.xyz#my.com; origin=aad; originId=bc20990e-cf30-4c37-9e4b-b04e37ab2b04; displayName=ABC; }",
"accessLevel": "#{licensingSource=account; accountLicenseType=express; msdnLicenseType=none; licenseDisplayName=Basic; status=active; statusMessage=; assignmentSource=unknown}",
"lastAccessedDate": "2019-05-23T05:54:25.14Z",
"dateCreated": "2019-05-23T05:54:21.6Z",
"projectEntitlements": "",
"extensions": "",
"groupAssignments": ""
},
{
"id": "4c07118c-8dc0-4e85-97a5-5501003d620d",
"user": "#{subjectKind=user; domain=2161a74d-1c3e-4d34-a8c8-131360d2e92c; principalName=xyz#gmail.com; mailAddress=xyz#gmail.com; origin=aad; originId=; displayName=xyz}",
"accessLevel": "#{licensingSource=account; accountLicenseType=stakeholder; msdnLicenseType=none; licenseDisplayName=Stakeholder; status=pending; statusMessage=; assignmentSource=unknown}",
"lastAccessedDate": "0001-01-01T00:00:00Z",
"dateCreated": "2019-05-23T10:05:41.663Z",
"projectEntitlements": "",
"extensions": "",
"groupAssignments": ""
}
],
"continuationToken": "",
"totalCount": 2 }
In CSV, I want Following Output:
id principalName licenseDisplayName
4b3556f1-df58-6c3c-848b-022fc6a8668d abc#my.com Basic
This can be done using Select-Object, Select-String with a Regex pattern, and ConvertTo-Csv.
$j = #"
{
"members": [
{
"id": "4b3556f1-df58-6c3c-848b-022fc6a8668d",
"user": "#{subjectKind=user; metaType=member; domain=2161a74d-1c3e-4d34-a8c8-131360d2e92c; principalName=abc#my.com; mailAddress=abc.xyz#my.com; origin=aad; originId=bc20990e-cf30-4c37-9e4b-b04e37ab2b04; displayName=ABC; }",
"accessLevel": "#{licensingSource=account; accountLicenseType=express; msdnLicenseType=none; licenseDisplayName=Basic; status=active; statusMessage=; assignmentSource=unknown}",
"lastAccessedDate": "2019-05-23T05:54:25.14Z",
"dateCreated": "2019-05-23T05:54:21.6Z",
"projectEntitlements": "",
"extensions": "",
"groupAssignments": ""
},
{
"id": "4c07118c-8dc0-4e85-97a5-5501003d620d",
"user": "#{subjectKind=user; domain=2161a74d-1c3e-4d34-a8c8-131360d2e92c; principalName=xyz#gmail.com; mailAddress=xyz#gmail.com; origin=aad; originId=; displayName=xyz}",
"accessLevel": "#{licensingSource=account; accountLicenseType=stakeholder; msdnLicenseType=none; licenseDisplayName=Stakeholder; status=pending; statusMessage=; assignmentSource=unknown}",
"lastAccessedDate": "0001-01-01T00:00:00Z",
"dateCreated": "2019-05-23T10:05:41.663Z",
"projectEntitlements": "",
"extensions": "",
"groupAssignments": ""
}
],
"continuationToken": "",
"totalCount": 2 }
"# | ConvertFrom-Json
$j.members | Select-Object id,
#{n='principalName';e={($_.user |
Select-String "(?<=principalname=).*?(?=;)").matches.value}},
#{n='licenseDisplayName';e={($_.accessLevel |
Select-String "(?<=licenseDisplayName=).*?(?=;)").matches.value}} |
ConvertTo-Csv -NoTypeInformation
You can just remove the | ConvertTo-Csv if you want a tabular display in the console.
A somewhat alternative approach would be to remove the surrounding #{} from the property you want to access and then use ConvertFrom-StringData to return a hash table with sub-property values you can easily access.
$j.members | Select-Object id,
#{n='principalName';e={($_.user -replace "^#{|}$" -split ';' |
ConvertFrom-StringData).principalName}},
#{n='licenseDisplayName';e={($_.accesslevel -replace "^#{|}$" -split ';' |
ConvertFrom-StringData).licenseDisplayName}} |
ConvertTo-Csv -NoTypeInformation

How to get child object from PSObject using dot notation string

providing I have the following JSON
{
"firstName": "Frank",
"lastName": "Smith",
"age": "25",
"address": {
"streetAddress": "21 3rd st",
"city": "New York",
"state": "NY",
"postalCode": "10021"
},
"phoneNumber": [
{
"type": "home",
"number": "212 555-1234"
},
{
"type": "fax",
"number": "646 555-4567"
}
]
}
I need to be able to update a value using dotted notation.
$path = "C:\somePath\test.json"
$node = "address.streetAddress" # should also work with "phoneNumber[0].number"
$value = "21 Jump St."
$config = Get-Content -Path $path -Raw | ConvertFrom-Json
$config.$node = $value
Write-Host $config.$node
#Set-Content $path $($config | ConvertTo-Json)
The problem I'm getting is that the property cannot be found.
Exception setting "address.streetAddress": "The property 'address.streetAddress' cannot be found on this object. Verify that the property exists and can be set."
What do I need to do to be able to pass in dotted notation, and update the appropriate value?
While you can put a single property name in a variable and use that to access the property, you can't do that for multiple, dotted properties. You can work around this by using Invoke-Expression:
Invoke-Expression "`$config.$node = `$value"
Shortest way around it is:
$config.$($node) = $value
Level of nesting does not matter, You can do this:
$config.$($node).$($subnode).$($subSubNode) = $value
You can also refer to properties in objects like this:
$config.$($node.nodename)=$value