Extract value from json using powershell - powershell

I am having the below json
{
"clusterName": "IBDCluster",
"defaultReplicaSet": {
"name": "default",
"primary": "X92SL224XXX2XX:3306",
"ssl": "REQUIRED",
"status": "OK_NO_TOLERANCE",
"statusText": "Cluster is NOT tolerant to any failures. 1 member is not active.",
"topology": {
"X92SL224XXX1XX:3306": {
"address": "X92SL224XXXXXXX:3306",
"memberRole": "SECONDARY",
"mode": "R/O",
"readReplicas": {},
"role": "HA",
"status": "ONLINE",
"version": "5.7.36"
},
"X92SL224XXX2XX:3306": {
"address": "X92SL224XXX2XX:3306",
"memberRole": "PRIMARY",
"mode": "R/W",
"readReplicas": {},
"role": "HA",
"status": "ONLINE",
"version": "5.7.36"
},
"X92SL224XXXX3XX:3306": {
"address": "X92SL224XXX3XX:3306",
"instanceErrors": [
"ERROR: group_replication has stopped with an error."
],
"memberRole": "SECONDARY",
"memberState": "ERROR",
"mode": "R/O",
"readReplicas": {},
"role": "HA",
"status": "(MISSING)",
"version": "5.7.36"
}
},
"topologyMode": "Single-Primary"
},
"groupInformationSourceMember": "X92SL224XXXXXXX:3306"
}
I need to extract value like memberRole, status from the topology section.
when I go to the topology part
$ClusterDetails = $ClusterStatus.defaultReplicaSet.topology
the $ClusterDetails have value like (data visible only for 2 servers but all 3 servers are present)
PS C:\Windows\system32> $ClusterDetails
X92SL224XXXX1XX:3306 X92SL224XXXX2XX:3306
-------------------- --------------------
#{address=X92SL224XXXX1XX:3306; memberRole=SECONDARY; mode=R/O; readReplicas=; role=HA; status=ONLINE; version=5.7.36} #{address=X92SL224XXXX2XX:3306; memberRole=PRIM...
from shell I am able to see the individual output if i select like
PS C:\Windows\system32> $ClusterDetails.'X92SL224XXXX1XX:3306'
address : X92SL224XXXX1XX:3306
memberRole : PRIMARY
mode : R/W
readReplicas :
role : HA
status : ONLINE
version : 5.7.36
I need help to fetch the data from $ClusterDetails for individual servers like above but not getting how to get that dot part via script. please let me know how to do that.

Quite a long statement but this should work:
$json.defaultReplicaSet.topology.PSObject.Properties.Value | Select-Object memberRole, status
# Results in:
memberRole status
---------- ------
SECONDARY ONLINE
PRIMARY ONLINE
SECONDARY (MISSING)
You can access the Values of each Property of the Object in $json.defaultReplicaSet.topology accessing the PSObject Properties.
It's worth noting that .PSObject.Properties.Value works to enumerate all Property Values at once due to Member-Access Enumeration.
The same can be accomplished using a loop, for example:
foreach($property in $json.defaultReplicaSet.topology.PSObject.Properties) {
[pscustomobject]#{
ThisProperty = $property.Name
memberRole = $property.Value.memberRole
status = $property.Value.status
}
}

Related

Extract data from json using powershell

I executing below command using strem from powershell and I am getting response in form of json using the below code
$session = New-SSHSession -ComputerName $S_1 -Credential $cred
$Strem = New-SSHShellStream -SSHSession $Session
$cmd_4 = $Strem.WriteLine("shell.connect('USER#X92SL224XXX2XX:3306')")
sleep -Seconds 5
$Strem.read()
$pass = $Strem.WriteLine("password")
sleep -Seconds 5
$Strem.read()
$cmd_5 = $Strem.WriteLine("var cluster = dba.getCluster('IBDCluster')")
sleep -Seconds 5
$Strem.read()
$cmd_6 = $Strem.WriteLine("cluster.status()")
sleep -Seconds 5
$ClusterStatus = $Strem.read()
[DBG]: PS C:\Windows\system32>>
cluster.status()
{
"clusterName": "IBDCluster",
"defaultReplicaSet": {
"name": "default",
"primary": "X92SL224XXX2XX:3306",
"ssl": "REQUIRED",
"status": "OK_NO_TOLERANCE",
"statusText": "Cluster is NOT tolerant to any failures. 1 member is not active.",
"topology": {
"X92SL224XXX1XX:3306": {
"address": "X92SL224XXXXXXX:3306",
"memberRole": "SECONDARY",
"mode": "R/O",
"readReplicas": {},
"role": "HA",
"status": "ONLINE",
"version": "5.7.36"
},
"X92SL224XXX2XX:3306": {
"address": "X92SL224XXX2XX:3306",
"memberRole": "PRIMARY",
"mode": "R/W",
"readReplicas": {},
"role": "HA",
"status": "ONLINE",
"version": "5.7.36"
},
"X92SL224XXXX3XX:3306": {
"address": "X92SL224XXX3XX:3306",
"instanceErrors": [
"ERROR: group_replication has stopped with an error."
],
"memberRole": "SECONDARY",
"memberState": "ERROR",
"mode": "R/O",
"readReplicas": {},
"role": "HA",
"status": "(MISSING)",
"version": "5.7.36"
}
},
"topologyMode": "Single-Primary"
},
"groupInformationSourceMember": "X92SL224XXXXXXX:3306"
}
[48;5;254m[38;5;23m My[0m[48;5;254m[38;5;166mSQL [0m[48;5;237m[38;5;15m X92SL224QDBA2DB:3306 ssl [0m[48;5;221m[38;5;0m JS [0m[48;5;0m> [0m
I need to fetch all the address, memberRole, memberState from the above data.
I was trying to convert from Json but getting error like
ConvertFrom-Json : Invalid JSON primitive: cluster.status.
Please let me know how to get the data from above
It looks like you simply need to remove the cluster.status() line that precedes the JSON text in the multi-line string that $Strem.Read() returns, as well as the extra line that follows it:
$ClusterStatus = $Strem.Read() -replace '^.+|.+$' | ConvertFrom-Json
Note: -replace '^.+|.+$' removes the first and last line from the input string, without also trying to remove a newline that follows / precedes them; however, these newlines are incidental whitespace that doesn't affect the operation of ConvertFrom-Json.

Powershell Iterate through multidimensional array of hashtables to find a match and combine values from both arrays

I need to combine values from 2 JSONs:
If there is a match in alerts IDs, I need to create structure, that will take data from both jsons
Result for a match should look like:
$array = #()
$hashtable = #{}
$hashtable.AlertID (does not matter what JSON is it from)
$hashtable.Tags (from JSON 1)
$hashtable.IncidentName (from JSON2)
$hashtable.IncidentID (from JSON2)
$array += $hashtable
I would prefer if this would be done with c style powershell loop.
c style for loop = for ($x = 0; $x -array.count; $x++)
JSON 1:
[
{
"Status": "Active",
"IncidentId": "3",
"tags": "SINC0008009",
"AlertId": [
"da637563185629568182_-638872186",
"da637563185631732095_1120592736",
"da637563185706412029_-614525914",
"da637563185760439486_-276692370",
"da637563185856325888_-1949235651",
"da637563186785996176_2128073884",
"da637563186789897000_1239551047",
"da637563186806513555_1512241399",
"da637563193194338043_-244132089"
],
"severity": "Medium"
},
{
"Status": "Active",
"IncidentId": "4",
"tags": "SINC0008008",
"AlertId": [
"da637643650725801726_1735022501",
"da637643650741237104_1473290917",
"da637643650748739479_-40211355",
"da637643652767933265_-1887823168",
"da637643670830160376_-443360743"
],
"severity": "Medium"
},
{
"Status": "Active",
"IncidentId": "2",
"tags": null,
"AlertId": [
"caD76232A5-F386-3C5D-94CD-7C82A7F778DC"
],
"severity": "Medium"
},
{
"Status": "Active",
"IncidentId": "1",
"tags": null,
"AlertId": [
"ca6534FF45-D62A-3FB7-BD6B-FF5029C553DB"
],
"severity": "Medium"
}
]
JSON 2:
{
"value": [
{
"incidentId": 3,
"incidentName": "Multi-stage incident involving Initial access & Discovery on one endpoint",
"status": "Active",
"severity": "Medium",
"tags": ["SINC0000001"],
"comments": [],
"alerts": [
{
"alertId": "da637563185629568182_-638872186",
"incidentId": 3,
"description": "A suspicious PowerShell activity was observed on the machine. ",
"status": "New",
"severity": "Medium",
"devices": [
{
"deviceDnsName": "xxxxx"
}
],
"entities": [
{
"entityType": "User",
"accountName": "xxxxxx",
"userPrincipalName": "xxx#xx.xx"
},
{
"entityType": "Process"
},
{
"entityType": "Process",
"verdict": "Suspicious"
},
{
"entityType": "File"
}
]
},
{
"alertId": "da637563185631732095_1120592736",
"incidentId": 3,
"devices": [
{
"osPlatform": "Windows10",
"version": "1909"
}
],
"entities": [
{
"entityType": "User",
"remediationStatus": "None"
}
]
}
]
},
{
"incidentId": 4,
"incidentName": "Multi-stage incident involving Initial access & Discovery on one endpoint",
"status": "Active",
"severity": "Medium",
"tags": ["SINC0000002"],
"comments": [],
"alerts": [
{
"alertId": "da637563185629568182_-638872186",
"incidentId": 3,
"description": "A suspicious PowerShell activity was observed on the machine. ",
"status": "New",
"severity": "Medium",
"devices": [
{
"deviceDnsName": "xxxxx"
}
],
"entities": [
{
"entityType": "User",
"accountName": "xxxxxx",
"userPrincipalName": "xxx#xx.xx"
},
{
"entityType": "Process"
},
{
"entityType": "Process",
"verdict": "Suspicious"
},
{
"entityType": "File"
}
]
},
{
"alertId": "da637563185631732095_1120592736",
"incidentId": 3,
"devices": [
{
"osPlatform": "Windows10",
"version": "1909"
}
],
"entities": [
{
"entityType": "User",
"remediationStatus": "None"
}
]
}
]
}
]
}
Till now, I was looking into using nested foreach loop to address it but it does not behave like I want. I am looking for for loop as I could use the indexes.
Instead of creating an array of Hashtables, I think it's better to create an array of PsCustomObjects, because outputting the result to console/file/json would be a lot easier then.
$json1 = Get-Content -Path 'X:\json1.json' -Raw | ConvertFrom-Json
$json2 = Get-Content -Path 'X:\json2.json' -Raw | ConvertFrom-Json
$result = foreach ($incident in $json1) {
foreach ($alertId in $incident.AlertId) {
$json2.value | Where-Object { $_.alerts.alertId -eq $alertId } | ForEach-Object {
# output an object with the wanted properties
[PsCustomObject]#{
AlertID = $alertId # from json1
Tags = $incident.Tags # from json1
IncidentName = $_.incidentName # from json2
IncidentID = $_.incidentId # from json2
}
}
}
}
# output on screen
$result | Format-Table -AutoSize # or use Out-GridView
# output to new JSON
$result | ConvertTo-Json
# output to CSV file
$result | Export-Csv -Path 'X:\incidents.csv' -NoTypeInformation
Using your examples, the output to console window is:
AlertID Tags IncidentName IncidentID
------- ---- ------------ ----------
da637563185629568182_-638872186 SINC0008009 Multi-stage incident involving Initial access & Discovery on one endpoint 3
da637563185629568182_-638872186 SINC0008009 Multi-stage incident involving Initial access & Discovery on one endpoint 4
da637563185631732095_1120592736 SINC0008009 Multi-stage incident involving Initial access & Discovery on one endpoint 3
da637563185631732095_1120592736 SINC0008009 Multi-stage incident involving Initial access & Discovery on one endpoint 4

What is the difference between RestartRequired and RestartNeeded in Get-WindowsOptionalFeature output?

I run the following:
Get-WindowsOptionalFeature -Online -FeatureName TelnetClient | ConvertTo-Json -Depth 100
It produces the following output:
{
"DisplayName": "Telnet Client",
"Description": "Allows you to connect to other computers remotely.",
"RestartRequired": 1,
"CustomProperties": [
{
"Name": "Description",
"Value": "Telnet Client uses the Telnet protocol to connect to a remote Telnet server and run applications on that server.",
"Path": "ServerComponent"
},
{
"Name": "DisplayName",
"Value": "Telnet Client",
"Path": "ServerComponent"
},
{
"Name": "Id",
"Value": "44",
"Path": "ServerComponent"
},
{
"Name": "Type",
"Value": "Feature",
"Path": "ServerComponent"
},
{
"Name": "UniqueName",
"Value": "Telnet-Client",
"Path": "ServerComponent"
},
{
"Name": "Major",
"Value": "10",
"Path": "ServerComponent\\Version"
},
{
"Name": "Minor",
"Value": "0",
"Path": "ServerComponent\\Version"
},
{
"Name": "Name",
"Value": "TelnetClient",
"Path": "ServerComponent\\Deploys\\Update"
}
],
"FeatureName": "TelnetClient",
"State": 0,
"Path": null,
"Online": true,
"WinPath": null,
"SysDrivePath": null,
"RestartNeeded": false,
"LogPath": "C:\\Windows\\Logs\\DISM\\dism.log",
"ScratchDirectory": null,
"LogLevel": 2
}
My question is in particular about what is the difference between "RestartNeeded" and "RestartRequired"? They both appear to be about the same topic. What exactly is the difference between them?
"RestartRequired" is actually an enumeration (Microsoft.Dism.Commands.RestartType) with values:
Name
Value
No
0
Possible
1
Required
2
(It is a pity that I am still using PowerShell 5.1 which lacks the -EnumsAsStrings option to ConvertTo-Json which was added in PowerShell 6 and 7.)
Short Answer: On features like TelnetClient which return a [Microsoft.Dism.Commands.AdvancedFeatureObject]:
RestartRequired denotes whether a restart could be required after installing or enabling the feature
RestartNeeded is used to show whether a restart is currently pending for the feature.
Long Version: The AdvancedFeatureObject type never actually gets this value set though. Instead, the property is only used by Microsoft.Dism.Commands.ImageObject which is returned from Enable/Disable-WindowsOptionalFeature. For example:
# I already have it installed:
Get-WindowsOptionalFeature -Online -FeatureName 'TelnetClient' | fl FeatureName,State,Restart*
FeatureName : TelnetClient
State : Enabled
RestartNeeded : False
RestartRequired : Possible
# I remove it, and save the output:
$result = Get-WindowsOptionalFeature -Online -FeatureName 'TelnetClient'|
Disable-WindowsOptionalFeature
# Returns an ImageObject
($result|gm).TypeName
Microsoft.Dism.Commands.ImageObject
# RestartNeeded is set here
$result
Path :
Online : True
RestartNeeded : True
# It does not get set on the main object though (only State=Disabled):
FeatureName : TelnetClient
State : Disabled
RestartRequired : Possible
RestartNeeded : False
I think the RestartNeeded property is only present on the AdvancedFeatureObject due to some type inheritance, and is not actually a useful property.

can we used world update for France only server ? seems to make wrong query result

On my local overpass api server with only french data on which is applied hourly planet diff, some of the query responses are wrong.
It's not doing it for each query : but something like once every 200 requests , sometime more ...
for example :
[timeout:360][out:json];way(48.359900103518235,5.708088852670471,48.360439696481784,5.708900947329539)[highway];out ;
return 3 ways :
{
"version": 0.6,
"generator": "Overpass API 0.7.54.13 ff15392f",
"osm3s": {
"timestamp_osm_base": "2019-09-23T15:00:00Z",
},
"elements": [
{
"type": "way",
"id": 53290349,
"nodes": [...],
"tags": {
"highway": "secondary",
"maxspeed": "100",
"ref": "L 385"
}
},
{
"type": "way",
"id": 238493649,
"nodes": [...],
"tags": {
"highway": "residential",
"name": "Rue du Stand",
"ref": "C 3",
"source": "..."
}
},
{
"type": "way",
"id": 597978369,
"nodes": [...],
"tags": {
"highway": "service"
}
}
]
}
First one is in Germany, far East ...
My question :
On an overpass api server, is there a way to apply diff only for defined area ? it is not documented ( neither here : https://wiki.openstreetmap.org/wiki/Overpass_API/Installation
or here : https://wiki.openstreetmap.org/wiki/User:Breki/Overpass_API_Installation#Configuring_Diffs )
if not, how to get rid of those wrong results ?
Thanks,
Two questions, so two answer :
i found that there is French diff file existing : http://download.openstreetmap.fr/replication/europe/france/minute/ so i will restart my server with those diffs.
The best way to get rid of those wrong result is to have a consistant server : no world diff for just France Data.

How to use script arguments in AWS DataPipeline SQLActivity?

I am trying to execute an unload command on Redshift via Data Pipeline. The script looks something like:
unload ($$ SELECT *, count(*) FROM (SELECT APP_ID, CAST(record_date AS DATE) WHERE len(APP_ID)>0 AND CAST(record_date as DATE)=$1) GROUP BY APP_ID $$) to 's3://test/unload/' iam_role 'arn:aws:iam::xxxxxxxxxxx:role/Test' delimiter ',' addquotes;
The pipeline looks something like this:
{
"objects": [
{
"role": "DataPipelineDefaultRole",
"subject": "SuccessNotification",
"name": "SNS",
"id": "ActionId_xxxxx”,
"message": "SUCCESS: #{format(minusDays(node.#scheduledStartTime,1),'MM-dd-YYYY')}",
"type": "SnsAlarm",
"topicArn": "arn:aws:sns:us-west-2:xxxxxxxxxx:notification"
},
{
"connectionString": “connection-url”,
"password": “password”,
"name": “Test”,
"id": "DatabaseId_xxxxx”,
"type": "RedshiftDatabase",
"username": “username”
},
{
"subnetId": "subnet-xxxxxx”,
"resourceRole": "DataPipelineDefaultResourceRole",
"role": "DataPipelineDefaultRole",
"name": "EC2",
"id": "ResourceId_xxxxx”,
"type": "Ec2Resource"
},
{
"failureAndRerunMode": "CASCADE",
"resourceRole": "DataPipelineDefaultResourceRole",
"role": "DataPipelineDefaultRole",
"pipelineLogUri": "s3://test/logs/",
"scheduleType": "ONDEMAND",
"name": "Default",
"id": "Default"
},
{
"database": {
"ref": "DatabaseId_xxxxxx”
},
"scriptUri": "s3://test/script.sql",
"name": "SqlActivity",
"scriptArgument": "#{format(minusDays(node.#scheduledStartTime,1),"MM-dd-YYYY”)}”,
"id": "SqlActivityId_xxxxx”,
"runsOn": {
"ref": "ResourceId_xxxx”
},
"type": "SqlActivity",
"onSuccess": {
"ref": "ActionId_xxxxx”
}
}
],
"parameters": []
}
However, I keep getting the error: The column index is out of range: 1, number of columns: 0.
I just can't get it to work. I have tried using ?, $1 and I even tried putting the expression #{format(minusDays(node.#scheduledStartTime,1),"MM-dd-YYYY”)} directly in the script. None of them works.
I have looked at the answers to Amazon Data Pipline: How to use a script argument in a SqlActivity? but none of them are helpful.
Does anyone has idea how to use script argument in SQL Script in AWS Data Pipeline?