Getting extra variable in XML to JSON conversion using perl - perl

My Program is giving extra variable $t in output. Can anyone help me on this?
use XML::XML2JSON;
xml content
my $XML = '<file><sno>1</sno><process>VALID</process><validation_type>C</validation_type><file_type>HTML</file_type><line>2</line><column>78</column><status>0</status><type>Warning</type><code>001</code><rule>aligning content.</rule><desc>Check that non-breaking space.</desc></file>';
my $XML2JSON = XML::XML2JSON->new();
my $JSON = $XML2JSON->convert($XML);
print $JSON;
Output - Extra variable is coming $t
{
"#encoding": "UTF-8",
"#version": "1.0",
"file": {
"status": {
"$t": "0"
},
"rule": {
"$t": "aligning content."
},
"validation_type": {
"$t": "C"
},
"process": {
"$t": "VALID"
},
"sno": {
"$t": "1"
},
"file_type": {
"$t": "HTML"
},
"desc": {
"$t": "Check that non-breaking space."
},
"type": {
"$t": "Warning"
},
"code": {
"$t": "001"
},
"line": {
"$t": "2"
},
"column": {
"$t": "78"
}
}
}
Expected output is:
{
"sno": "1",
"process": "VALID",
"validation_type": "C",
"file_type": "HTML",
"line": "2",
"column": "78",
"status": "0",
"type": "Warning",
"code": "001",
"rule": "aligning content.",
"desc": "Check that non-breaking space."
}

The $t is content key as mentioned in the XML::XML2JSON documentation.
If your intention is to convert from XML to JSON, I would recommend to use XML::Simple and later you can encode using JSON.pm.
Code below:
#!/usr/bin/perl
use strict;
use warnings;
use JSON;
use XML::Simple;
#Create an object
my $xmlSimple = new XML::Simple;
my $XML = '<file><sno>1</sno><process>VALID</process><validation_type>C</validation_type><file_type>HTML</file_type><line>2</line><column>78</column><status>0</status><type>Warning</type><code>001</code><rule>aligning content.</rule><desc>Check that non-breaking space.</desc></file>';
#either we can pass a variable or a xml file which contains xml data
my $dataXML = $xmlSimple->XMLin($XML);
my $jsonString = encode_json($dataXML);
print $jsonString;
Output:
{
"process":"VALID",
"line":"2",
"column":"78",
"type":"Warning",
"file_type":"HTML",
"sno":"1",
"status":"0",
"rule":"aligning content.",
"code":"001",
"desc":"Check that non-breaking space.",
"validation_type":"C"
}

Related

Can we append a text file with punctuations in ahk?

I am trying to append a txt file with punctuations to text to a file in AHK but it seems to break the line even if i use round brackets. How can I append to a file without breaking it?
FileAppend, ({
"accounts": [
{
"active": true,
"type": "dummy",
"ygg": {
"extra": {
"clientToken": "123456789",
"userName": "BX0W"
},
"iat": 1655273051,
"token": "BX0W"
}
}
],
"formatVersion": 3
}), accounts.json
Try this instead. Even inside a function my experience with a continuation section is that the parentheses must be the beginning of each line (with no preceding whitespaces) for it to work. I am guessing that it has something to do with the compiler.
FileAppend,
(
{
"accounts": [
{
"active": true,
"type": "dummy",
"ygg": {
"extra": {
"clientToken": "123456789",
"userName": "BX0W"
},
"iat": 1655273051,
"token": "BX0W"
}
}
],
"formatVersion": 3
}
), accounts.json

Powershell Iterate through multidimensional array of hashtables to find a match and combine values from both arrays

I need to combine values from 2 JSONs:
If there is a match in alerts IDs, I need to create structure, that will take data from both jsons
Result for a match should look like:
$array = #()
$hashtable = #{}
$hashtable.AlertID (does not matter what JSON is it from)
$hashtable.Tags (from JSON 1)
$hashtable.IncidentName (from JSON2)
$hashtable.IncidentID (from JSON2)
$array += $hashtable
I would prefer if this would be done with c style powershell loop.
c style for loop = for ($x = 0; $x -array.count; $x++)
JSON 1:
[
{
"Status": "Active",
"IncidentId": "3",
"tags": "SINC0008009",
"AlertId": [
"da637563185629568182_-638872186",
"da637563185631732095_1120592736",
"da637563185706412029_-614525914",
"da637563185760439486_-276692370",
"da637563185856325888_-1949235651",
"da637563186785996176_2128073884",
"da637563186789897000_1239551047",
"da637563186806513555_1512241399",
"da637563193194338043_-244132089"
],
"severity": "Medium"
},
{
"Status": "Active",
"IncidentId": "4",
"tags": "SINC0008008",
"AlertId": [
"da637643650725801726_1735022501",
"da637643650741237104_1473290917",
"da637643650748739479_-40211355",
"da637643652767933265_-1887823168",
"da637643670830160376_-443360743"
],
"severity": "Medium"
},
{
"Status": "Active",
"IncidentId": "2",
"tags": null,
"AlertId": [
"caD76232A5-F386-3C5D-94CD-7C82A7F778DC"
],
"severity": "Medium"
},
{
"Status": "Active",
"IncidentId": "1",
"tags": null,
"AlertId": [
"ca6534FF45-D62A-3FB7-BD6B-FF5029C553DB"
],
"severity": "Medium"
}
]
JSON 2:
{
"value": [
{
"incidentId": 3,
"incidentName": "Multi-stage incident involving Initial access & Discovery on one endpoint",
"status": "Active",
"severity": "Medium",
"tags": ["SINC0000001"],
"comments": [],
"alerts": [
{
"alertId": "da637563185629568182_-638872186",
"incidentId": 3,
"description": "A suspicious PowerShell activity was observed on the machine. ",
"status": "New",
"severity": "Medium",
"devices": [
{
"deviceDnsName": "xxxxx"
}
],
"entities": [
{
"entityType": "User",
"accountName": "xxxxxx",
"userPrincipalName": "xxx#xx.xx"
},
{
"entityType": "Process"
},
{
"entityType": "Process",
"verdict": "Suspicious"
},
{
"entityType": "File"
}
]
},
{
"alertId": "da637563185631732095_1120592736",
"incidentId": 3,
"devices": [
{
"osPlatform": "Windows10",
"version": "1909"
}
],
"entities": [
{
"entityType": "User",
"remediationStatus": "None"
}
]
}
]
},
{
"incidentId": 4,
"incidentName": "Multi-stage incident involving Initial access & Discovery on one endpoint",
"status": "Active",
"severity": "Medium",
"tags": ["SINC0000002"],
"comments": [],
"alerts": [
{
"alertId": "da637563185629568182_-638872186",
"incidentId": 3,
"description": "A suspicious PowerShell activity was observed on the machine. ",
"status": "New",
"severity": "Medium",
"devices": [
{
"deviceDnsName": "xxxxx"
}
],
"entities": [
{
"entityType": "User",
"accountName": "xxxxxx",
"userPrincipalName": "xxx#xx.xx"
},
{
"entityType": "Process"
},
{
"entityType": "Process",
"verdict": "Suspicious"
},
{
"entityType": "File"
}
]
},
{
"alertId": "da637563185631732095_1120592736",
"incidentId": 3,
"devices": [
{
"osPlatform": "Windows10",
"version": "1909"
}
],
"entities": [
{
"entityType": "User",
"remediationStatus": "None"
}
]
}
]
}
]
}
Till now, I was looking into using nested foreach loop to address it but it does not behave like I want. I am looking for for loop as I could use the indexes.
Instead of creating an array of Hashtables, I think it's better to create an array of PsCustomObjects, because outputting the result to console/file/json would be a lot easier then.
$json1 = Get-Content -Path 'X:\json1.json' -Raw | ConvertFrom-Json
$json2 = Get-Content -Path 'X:\json2.json' -Raw | ConvertFrom-Json
$result = foreach ($incident in $json1) {
foreach ($alertId in $incident.AlertId) {
$json2.value | Where-Object { $_.alerts.alertId -eq $alertId } | ForEach-Object {
# output an object with the wanted properties
[PsCustomObject]#{
AlertID = $alertId # from json1
Tags = $incident.Tags # from json1
IncidentName = $_.incidentName # from json2
IncidentID = $_.incidentId # from json2
}
}
}
}
# output on screen
$result | Format-Table -AutoSize # or use Out-GridView
# output to new JSON
$result | ConvertTo-Json
# output to CSV file
$result | Export-Csv -Path 'X:\incidents.csv' -NoTypeInformation
Using your examples, the output to console window is:
AlertID Tags IncidentName IncidentID
------- ---- ------------ ----------
da637563185629568182_-638872186 SINC0008009 Multi-stage incident involving Initial access & Discovery on one endpoint 3
da637563185629568182_-638872186 SINC0008009 Multi-stage incident involving Initial access & Discovery on one endpoint 4
da637563185631732095_1120592736 SINC0008009 Multi-stage incident involving Initial access & Discovery on one endpoint 3
da637563185631732095_1120592736 SINC0008009 Multi-stage incident involving Initial access & Discovery on one endpoint 4

ElasticSearch Reindex API and painless script to access date field

I try to familiarize myself with the Reindexing API of ElasticSearch and the use of Painless scripts.
I have the following model:
"mappings": {
"customer": {
"properties": {
"firstName": {
"type": "text",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"lastName": {
"type": "text",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"dateOfBirth": {
"type": "date"
}
}
}
}
I would like to reindex all documents from test-v1 to test-v2 and apply a few transformations on them (for example extract the year part of dateOfBirth, convert a date value to a timestamp, etc) and save the result as a new field. But I got an issue when I tried to access it.
When I made the following call, I got an error:
POST /_reindex?pretty=true&human=true&wait_for_completion=true HTTP/1.1
Host: localhost:9200
Content-Type: application/json
{
"source": {
"index": "test-v1"
},
"dest": {
"index": "test-v2"
},
"script": {
"lang": "painless",
"inline": "ctx._source.yearOfBirth = ctx._source.dateOfBirth.getYear();"
}
}
And the response:
{
"error": {
"root_cause": [
{
"type": "script_exception",
"reason": "runtime error",
"script_stack": [
"ctx._source.yearOfBirth = ctx._source.dateOfBirth.getYear();",
" ^---- HERE"
],
"script": "ctx._source.yearOfBirth = ctx._source.dateOfBirth.getYear();",
"lang": "painless"
}
],
"type": "script_exception",
"reason": "runtime error",
"script_stack": [
"ctx._source.yearOfBirth = ctx._source.dateOfBirth.getYear();",
" ^---- HERE"
],
"script": "ctx._source.yearOfBirth = ctx._source.dateOfBirth.getYear();",
"lang": "painless",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "Unable to find dynamic method [getYear] with [0] arguments for class [java.lang.String]."
}
},
"status": 500
}
According to this tutorial Date fields are exposed as ReadableDateTime so they support methods like getYear, and getDayOfWeek. and indeed, the Reference mentions those as supported methods.
Still, the response mentions [java.lang.String] as the type of the dateOfBirth property. I could just parse it to e.g. an OffsetDateTime, but I wonder why it is a string.
Anyone has a suggestion what I'm doing wrong?

How to remove the curly braces and string from JSON file using Perl

{
"ticket": {
{
"id":"01",
"language": "Java",
"edition": "third",
"author": "Herbert Schildt"
},
{
"id":"07",
"language": "C++",
"edition": "second"
"author": "E.Balagurusamy"
}
}
,
"edition": "third",
"author": "Herbert Schildt"
}
That's my JSON file. I want to remove from file starting part {"ticket":
and ending part ,"edition": "third", "author": "Herbert Schildt" }.
Finally output I want like this:
{
"id":"01",
"language": "Java",
"edition": "third",
"author": "Herbert Schildt"
},
{
"id":"07",
"language": "C++",
"edition": "second"
"author": "E.Balagurusamy"
}
}
Note
Don't use regular expression, using only JSON Perl.
update
#!/usr/bin/perl
use JSON;
use Data::Dumper;
use Getopt::Long;
my($infile);
GetOptions('inFile=s' => \$infile);
my $json;
{
local $/;
open my $fh, "<", "$infile";
$json = <$fh>;
close $fh;
}
my $text = decode_json($json);
I have done this up to decode, but how to remove the starting and ending parts?
Assuming the syntax errors in your JSON file are by accident and fixing it with a comma and an array structure, this works:
#!/usr/bin/env perl
use strict;
use warnings;
use JSON;
my $json = do {
local $/;
<DATA>;
};
my $data = decode_json($json);
my $data2 = $data->{ticket};
my $json2 = encode_json($data2);
print $json2, "\n";
__DATA__
{
"ticket": [
{
"id":"01",
"language": "Java",
"edition": "third",
"author": "Herbert Schildt"
},
{
"id":"07",
"language": "C++",
"edition": "second",
"author": "E.Balagurusamy"
}
],
"edition": "third",
"author": "Herbert Schildt"
}
Prints out:
[{"language":"Java","edition":"third","author":"Herbert Schildt","id":"01"},{"language":"C++","edition":"second","author":"E.Balagurusamy","id":"07"}]

what is regular expression for matching single line with multi match?

$str = "Data = [ {"name": "test","Address": "UK" "currency": "£" },{"name": "test2","Address": "US" "currency": "$" },{"name": "test","Address": "eur" "currency": "E" }
I want to display all address
its not multi line string . It all a single string
Please help on this
Thanks ,
TREE J
Your string is JSON! Treat it as such!
edit: I'm an idiot and can't tell when a question is tagged as perl instead of PHP :-) Link ammended.
This should work:
while ($str =~ /\"Address\":\S+\"(.*?)\"/g) {
print "Address = $1\n";
}
You do it by using the right tool for the job. In this case you fix the corrupt JSON with a regex and then use JSON to get the data:
#!/usr/bin/perl
use strict;
use warnings;
use JSON;
my $input = <DATA>;
my ($json) = $input =~ /DATA = (.*)/;
my $data = decode_json $json;
for my $record (#$data) {
print "$record->{name} has address $record->{Address}\n";
}
__DATA__
DATA = [ {"name": "test", "Address": "UK", "currency": "£" }, {"name": "test2", "Address": "US", "currency": "$" }, {"name": "test", "Address": "eur", "currency": "E" } ]
something like:
my $str = q(Data = [ {"name": "test","Address": "UK" "currency": "£" },{"name": "test2","Address": "US" "currency": "$" },{"name": "test","Address": "eur" "currency": "E" });
my #addresses = $str =~ /"Address":\s*"([^"]*)"/g;
print "#addresses\n";
HTH,
Paul
(PS: post real code, not pseudo code...)