Replace string in text file with PowerShell - powershell

I am attempting to replace a string in a text file on a specific line number using PowerShell but the command is removing a majority of the content. I would like to leverage the line number because the string to be replaced occurs on several lines but only want the string on line 51 to be updated. I am currently attempting to use the following to perform the replacement:
$content = Get-Content "file.txt"
$contentUpdate = $content[51] -replace '"Endpoint": ""','"Endpoint": "bucket.s3.us-west-1.vpce.amazonaws.com",'
Set-Content file.txt $contentUpdate
When the above commands are run against the file referenced at the bottom of this question, only the following remains in the file:
"Region": "",
File content:
{
"Profile":{
"ShareCreds" : true,
"ShareProfile" : "",
"ForceUpdateCreds" : false,
"KeyAutoRotateDays": 0
},
"Mds": {
"CommandWorkersLimit" : 5,
"StopTimeoutMillis" : 20000,
"Endpoint": "",
"CommandRetryLimit": 15
},
"Ssm": {
"Endpoint": "",
"HealthFrequencyMinutes": 5,
"CustomInventoryDefaultLocation" : "",
"AssociationLogsRetentionDurationHours" : 24,
"RunCommandLogsRetentionDurationHours" : 336,
"SessionLogsRetentionDurationHours" : 336,
"PluginLocalOutputCleanup": "",
"OrchestrationDirectoryCleanup": ""
},
"Mgs": {
"Region": "us-west-1",
"Endpoint": "",
"StopTimeoutMillis" : 20000,
"SessionWorkersLimit" : 1000,
"DeniedPortForwardingRemoteIPs" : [
"169.254.169.254",
"fd00:ec2::254",
"169.254.169.253",
"fd00:ec2::253"
]
},
"Agent": {
"Region": "",
"OrchestrationRootDir": "",
"SelfUpdate": false,
"TelemetryMetricsToCloudWatch": false,
"TelemetryMetricsToSSM": true,
"AuditExpirationDay" : 7,
"LongRunningWorkerMonitorIntervalSeconds": 60
},
"Os": {
"Lang": "en-US",
"Name": "",
"Version": "1"
},
"S3": {
"Endpoint": "",
"Region": "",
"LogBucket":"",
"LogKey":""
},
"Kms": {
"Endpoint": ""
}
}

Well, seeing that this is a JSON you're working with, I would treat it as such:
$content = Get-Content "file.txt" -Raw | ConvertFrom-Json
$content.S3.Endpoint = "bucket.s3.us-west-1.vpce.amazonaws.com"
$content | ConvertTo-Json | Set-Content "file.txt"
This way you avoid trying to index through an array and work with the objects themselves.

Related

Invoke-WebRequest content ConvertFrom-Json: Conversion from JSON failed with error: Unexpected character encountered while parsing value (CR;LF)

I successfully sent an webrequest to a Website and got a responde:
$Uri = "https://URLXYZ"
$Method = "POST"
$Header = #{
"Accept" = "*/*";
"Connection" = "keep-alive";
"Accept-Encoding" = "gzip, deflate, br";
"Content-Type" = "text/json; charset=""UTF-8"""
}
$Body = #"
{
"Items": [
{
"Type": "XX",
"Value": "YY",
}
],
"TypeId": XY,
"LiveConditions": []
}
"#
$webrequest = Invoke-WebRequest -Uri $Uri -Body $Body -Headers $Header -Method $Method
Now, I am trying to convert it from JSON without success:
$webrequest.Content | ConvertFrom-Json
ConvertFrom-Json: Conversion from JSON failed with error: Unexpected character encountered while parsing value: . Path '', line 0, position 0.
When I copy the output ($webrequest.Content) to Notepad++ I can see a carriage return (CR) and line feed (LF):
$webrequest.Content
[{"MailG":[{"DisplayName":{"7":"Worker","9":"Person"},"Mails":"max.mustermann#gmai.com;"}],"ResultCount":1,"Rows":[{"ElementGroups":[{"ObjectContainer":{"Id":55
6677889900,"UID":"1122334455","Info":[],"PreCalculatedSelectedProperties":{"11":"Mustermann","22":"","33":"StreetName","44":"","55":"max.mustermann#gmai.com","66":"","77":"Berlin","88":"","99":"Max Mustermann","00":"+49 00 000 000","111":"userid","222":"xyz","333":"company","444":"1122334455","555":"roomnumber","666":"Germany","777":"Team","888":"+49 000 0 00 0","999":"Max","000":""},"TID":5,"RuleConditionPartResults":{"1448925":false,"1448926":false,"1448927":false,"1448928":false,"1448929":false,"1448930":false,"1448931":false,"1448932":false,"1448933":false,"1448934":false,"1448935":false,"1448936":false,"1448937":false,"1448938":false,"1448939":false,"1448940":false,"1448941":false,"1448942":false},"Img":{"3714":["picture"]},"Parents":[],"Childs":[],"UpObjects":0,"Down":0,"LinkCount":0,"FootObject":{},"BoxIds":[],"DisplayValue":"Max Mustermann","Key":"1122334455"},"Columns":[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17]}],"Blank":{},"Score":0.0,"SInd":0}],"Page":0,"TID":5}]
This is unexpected cause the CR;LF is e.g. in the middle of an email address.
If the response is longer, there are multiple CR;LF within the content of the webrequest.
The $webrequest.content is TypeName: System.String
If I manually remove the CR;CF in Notepad++ and PrettyPrint it is working:
[
{
"MailG": [
{
"DisplayName": {
"7": "Worker",
"9": "Person"
},
"Mails": "max.mustermann#gmai.com;"
}
],
"ResultCount": 1,
"Rows": [
{
"ElementGroups": [
{
"ObjectContainer": {
"Id": 556677889900,
"UID": "1122334455",
"Info": [],
"PreCalculatedSelectedProperties": {
"11": "Mustermann",
"22": "",
"33": "StreetName",
"44": "",
"55": "max.mustermann#gmai.com",
"66": "",
"77": "Berlin",
"88": "",
"99": "Max Mustermann",
"00": "+49 00 000 000",
"111": "userid",
"222": "xyz",
"333": "company",
"444": "1122334455",
"555": "roomnumber",
"666": "Germany",
"777": "Team",
"888": "+49 000 0 00 0",
"999": "Max",
"000": ""
},
"TID": 5,
"RuleConditionPartResults": {
"1448925": false,
"1448926": false,
"1448927": false,
"1448928": false,
"1448929": false,
"1448930": false,
"1448931": false,
"1448932": false,
"1448933": false,
"1448934": false,
"1448935": false,
"1448936": false,
"1448937": false,
"1448938": false,
"1448939": false,
"1448940": false,
"1448941": false,
"1448942": false
},
"Img": {
"3714": [
"picture"
]
},
"Parents": [],
"Childs": [],
"UpObjects": 0,
"Down": 0,
"LinkCount": 0,
"FootObject": {},
"BoxIds": [],
"DisplayValue": "Max Mustermann",
"Key": "1122334455"
},
"Columns": [
0,
1,
2,
3,
4,
5,
6,
7,
8,
9,
10,
11,
12,
13,
14,
15,
16,
17
]
}
],
"Blank": {},
"Score": 0.0,
"SInd": 0
}
],
"Page": 0,
"TID": 5
}
]
I already tried to convert it or even replace the CR;LF it in PowerShell without success.
The only workaround I found is to write the content to a .txt-file and read it again.
$webrequest.Content | Out-File "C:\Temp\WebRequestTemp.txt"
$json = Get-Content "C:\Temp\WebRequestTemp.txt" | ConvertFrom-Json
Afterwards I was able to convert it from JSON and work with the data.
Why can I not directly convert it from JSON as usual?
To help you with a more detailed answer. It is useful to share the contents of the web response. But maybe the line below will solve your problem.
$jsonCorrected = [Text.Encoding]::UTF8.GetString([Text.Encoding]::GetEncoding(28591).GetBytes(($webrequest.Content)))
I had a similar issue, and the issue was there was an unknown invisible character at the beginning of the content block. This character was "U+FEFF".
I managed to get it to be replaced by doing $webrequest.Content -replace '\uFEFF' | ConvertFrom-Json
I hope this works for anyone else too.

Powershell Iterate through multidimensional array of hashtables to find a match and combine values from both arrays

I need to combine values from 2 JSONs:
If there is a match in alerts IDs, I need to create structure, that will take data from both jsons
Result for a match should look like:
$array = #()
$hashtable = #{}
$hashtable.AlertID (does not matter what JSON is it from)
$hashtable.Tags (from JSON 1)
$hashtable.IncidentName (from JSON2)
$hashtable.IncidentID (from JSON2)
$array += $hashtable
I would prefer if this would be done with c style powershell loop.
c style for loop = for ($x = 0; $x -array.count; $x++)
JSON 1:
[
{
"Status": "Active",
"IncidentId": "3",
"tags": "SINC0008009",
"AlertId": [
"da637563185629568182_-638872186",
"da637563185631732095_1120592736",
"da637563185706412029_-614525914",
"da637563185760439486_-276692370",
"da637563185856325888_-1949235651",
"da637563186785996176_2128073884",
"da637563186789897000_1239551047",
"da637563186806513555_1512241399",
"da637563193194338043_-244132089"
],
"severity": "Medium"
},
{
"Status": "Active",
"IncidentId": "4",
"tags": "SINC0008008",
"AlertId": [
"da637643650725801726_1735022501",
"da637643650741237104_1473290917",
"da637643650748739479_-40211355",
"da637643652767933265_-1887823168",
"da637643670830160376_-443360743"
],
"severity": "Medium"
},
{
"Status": "Active",
"IncidentId": "2",
"tags": null,
"AlertId": [
"caD76232A5-F386-3C5D-94CD-7C82A7F778DC"
],
"severity": "Medium"
},
{
"Status": "Active",
"IncidentId": "1",
"tags": null,
"AlertId": [
"ca6534FF45-D62A-3FB7-BD6B-FF5029C553DB"
],
"severity": "Medium"
}
]
JSON 2:
{
"value": [
{
"incidentId": 3,
"incidentName": "Multi-stage incident involving Initial access & Discovery on one endpoint",
"status": "Active",
"severity": "Medium",
"tags": ["SINC0000001"],
"comments": [],
"alerts": [
{
"alertId": "da637563185629568182_-638872186",
"incidentId": 3,
"description": "A suspicious PowerShell activity was observed on the machine. ",
"status": "New",
"severity": "Medium",
"devices": [
{
"deviceDnsName": "xxxxx"
}
],
"entities": [
{
"entityType": "User",
"accountName": "xxxxxx",
"userPrincipalName": "xxx#xx.xx"
},
{
"entityType": "Process"
},
{
"entityType": "Process",
"verdict": "Suspicious"
},
{
"entityType": "File"
}
]
},
{
"alertId": "da637563185631732095_1120592736",
"incidentId": 3,
"devices": [
{
"osPlatform": "Windows10",
"version": "1909"
}
],
"entities": [
{
"entityType": "User",
"remediationStatus": "None"
}
]
}
]
},
{
"incidentId": 4,
"incidentName": "Multi-stage incident involving Initial access & Discovery on one endpoint",
"status": "Active",
"severity": "Medium",
"tags": ["SINC0000002"],
"comments": [],
"alerts": [
{
"alertId": "da637563185629568182_-638872186",
"incidentId": 3,
"description": "A suspicious PowerShell activity was observed on the machine. ",
"status": "New",
"severity": "Medium",
"devices": [
{
"deviceDnsName": "xxxxx"
}
],
"entities": [
{
"entityType": "User",
"accountName": "xxxxxx",
"userPrincipalName": "xxx#xx.xx"
},
{
"entityType": "Process"
},
{
"entityType": "Process",
"verdict": "Suspicious"
},
{
"entityType": "File"
}
]
},
{
"alertId": "da637563185631732095_1120592736",
"incidentId": 3,
"devices": [
{
"osPlatform": "Windows10",
"version": "1909"
}
],
"entities": [
{
"entityType": "User",
"remediationStatus": "None"
}
]
}
]
}
]
}
Till now, I was looking into using nested foreach loop to address it but it does not behave like I want. I am looking for for loop as I could use the indexes.
Instead of creating an array of Hashtables, I think it's better to create an array of PsCustomObjects, because outputting the result to console/file/json would be a lot easier then.
$json1 = Get-Content -Path 'X:\json1.json' -Raw | ConvertFrom-Json
$json2 = Get-Content -Path 'X:\json2.json' -Raw | ConvertFrom-Json
$result = foreach ($incident in $json1) {
foreach ($alertId in $incident.AlertId) {
$json2.value | Where-Object { $_.alerts.alertId -eq $alertId } | ForEach-Object {
# output an object with the wanted properties
[PsCustomObject]#{
AlertID = $alertId # from json1
Tags = $incident.Tags # from json1
IncidentName = $_.incidentName # from json2
IncidentID = $_.incidentId # from json2
}
}
}
}
# output on screen
$result | Format-Table -AutoSize # or use Out-GridView
# output to new JSON
$result | ConvertTo-Json
# output to CSV file
$result | Export-Csv -Path 'X:\incidents.csv' -NoTypeInformation
Using your examples, the output to console window is:
AlertID Tags IncidentName IncidentID
------- ---- ------------ ----------
da637563185629568182_-638872186 SINC0008009 Multi-stage incident involving Initial access & Discovery on one endpoint 3
da637563185629568182_-638872186 SINC0008009 Multi-stage incident involving Initial access & Discovery on one endpoint 4
da637563185631732095_1120592736 SINC0008009 Multi-stage incident involving Initial access & Discovery on one endpoint 3
da637563185631732095_1120592736 SINC0008009 Multi-stage incident involving Initial access & Discovery on one endpoint 4

Getting extra variable in XML to JSON conversion using perl

My Program is giving extra variable $t in output. Can anyone help me on this?
use XML::XML2JSON;
xml content
my $XML = '<file><sno>1</sno><process>VALID</process><validation_type>C</validation_type><file_type>HTML</file_type><line>2</line><column>78</column><status>0</status><type>Warning</type><code>001</code><rule>aligning content.</rule><desc>Check that non-breaking space.</desc></file>';
my $XML2JSON = XML::XML2JSON->new();
my $JSON = $XML2JSON->convert($XML);
print $JSON;
Output - Extra variable is coming $t
{
"#encoding": "UTF-8",
"#version": "1.0",
"file": {
"status": {
"$t": "0"
},
"rule": {
"$t": "aligning content."
},
"validation_type": {
"$t": "C"
},
"process": {
"$t": "VALID"
},
"sno": {
"$t": "1"
},
"file_type": {
"$t": "HTML"
},
"desc": {
"$t": "Check that non-breaking space."
},
"type": {
"$t": "Warning"
},
"code": {
"$t": "001"
},
"line": {
"$t": "2"
},
"column": {
"$t": "78"
}
}
}
Expected output is:
{
"sno": "1",
"process": "VALID",
"validation_type": "C",
"file_type": "HTML",
"line": "2",
"column": "78",
"status": "0",
"type": "Warning",
"code": "001",
"rule": "aligning content.",
"desc": "Check that non-breaking space."
}
The $t is content key as mentioned in the XML::XML2JSON documentation.
If your intention is to convert from XML to JSON, I would recommend to use XML::Simple and later you can encode using JSON.pm.
Code below:
#!/usr/bin/perl
use strict;
use warnings;
use JSON;
use XML::Simple;
#Create an object
my $xmlSimple = new XML::Simple;
my $XML = '<file><sno>1</sno><process>VALID</process><validation_type>C</validation_type><file_type>HTML</file_type><line>2</line><column>78</column><status>0</status><type>Warning</type><code>001</code><rule>aligning content.</rule><desc>Check that non-breaking space.</desc></file>';
#either we can pass a variable or a xml file which contains xml data
my $dataXML = $xmlSimple->XMLin($XML);
my $jsonString = encode_json($dataXML);
print $jsonString;
Output:
{
"process":"VALID",
"line":"2",
"column":"78",
"type":"Warning",
"file_type":"HTML",
"sno":"1",
"status":"0",
"rule":"aligning content.",
"code":"001",
"desc":"Check that non-breaking space.",
"validation_type":"C"
}

Using Powershell scipt following json will show on CSV

JSON
{
"members": [
{
"id": "4b3556f1-df58-6c3c-848b-022fc6a8668d",
"user": "#{subjectKind=user; metaType=member; domain=2161a74d-1c3e-4d34-a8c8-131360d2e92c; principalName=abc#my.com; mailAddress=abc.xyz#my.com; origin=aad; originId=bc20990e-cf30-4c37-9e4b-b04e37ab2b04; displayName=ABC; }",
"accessLevel": "#{licensingSource=account; accountLicenseType=express; msdnLicenseType=none; licenseDisplayName=Basic; status=active; statusMessage=; assignmentSource=unknown}",
"lastAccessedDate": "2019-05-23T05:54:25.14Z",
"dateCreated": "2019-05-23T05:54:21.6Z",
"projectEntitlements": "",
"extensions": "",
"groupAssignments": ""
},
{
"id": "4c07118c-8dc0-4e85-97a5-5501003d620d",
"user": "#{subjectKind=user; domain=2161a74d-1c3e-4d34-a8c8-131360d2e92c; principalName=xyz#gmail.com; mailAddress=xyz#gmail.com; origin=aad; originId=; displayName=xyz}",
"accessLevel": "#{licensingSource=account; accountLicenseType=stakeholder; msdnLicenseType=none; licenseDisplayName=Stakeholder; status=pending; statusMessage=; assignmentSource=unknown}",
"lastAccessedDate": "0001-01-01T00:00:00Z",
"dateCreated": "2019-05-23T10:05:41.663Z",
"projectEntitlements": "",
"extensions": "",
"groupAssignments": ""
}
],
"continuationToken": "",
"totalCount": 2 }
In CSV, I want Following Output:
id principalName licenseDisplayName
4b3556f1-df58-6c3c-848b-022fc6a8668d abc#my.com Basic
This can be done using Select-Object, Select-String with a Regex pattern, and ConvertTo-Csv.
$j = #"
{
"members": [
{
"id": "4b3556f1-df58-6c3c-848b-022fc6a8668d",
"user": "#{subjectKind=user; metaType=member; domain=2161a74d-1c3e-4d34-a8c8-131360d2e92c; principalName=abc#my.com; mailAddress=abc.xyz#my.com; origin=aad; originId=bc20990e-cf30-4c37-9e4b-b04e37ab2b04; displayName=ABC; }",
"accessLevel": "#{licensingSource=account; accountLicenseType=express; msdnLicenseType=none; licenseDisplayName=Basic; status=active; statusMessage=; assignmentSource=unknown}",
"lastAccessedDate": "2019-05-23T05:54:25.14Z",
"dateCreated": "2019-05-23T05:54:21.6Z",
"projectEntitlements": "",
"extensions": "",
"groupAssignments": ""
},
{
"id": "4c07118c-8dc0-4e85-97a5-5501003d620d",
"user": "#{subjectKind=user; domain=2161a74d-1c3e-4d34-a8c8-131360d2e92c; principalName=xyz#gmail.com; mailAddress=xyz#gmail.com; origin=aad; originId=; displayName=xyz}",
"accessLevel": "#{licensingSource=account; accountLicenseType=stakeholder; msdnLicenseType=none; licenseDisplayName=Stakeholder; status=pending; statusMessage=; assignmentSource=unknown}",
"lastAccessedDate": "0001-01-01T00:00:00Z",
"dateCreated": "2019-05-23T10:05:41.663Z",
"projectEntitlements": "",
"extensions": "",
"groupAssignments": ""
}
],
"continuationToken": "",
"totalCount": 2 }
"# | ConvertFrom-Json
$j.members | Select-Object id,
#{n='principalName';e={($_.user |
Select-String "(?<=principalname=).*?(?=;)").matches.value}},
#{n='licenseDisplayName';e={($_.accessLevel |
Select-String "(?<=licenseDisplayName=).*?(?=;)").matches.value}} |
ConvertTo-Csv -NoTypeInformation
You can just remove the | ConvertTo-Csv if you want a tabular display in the console.
A somewhat alternative approach would be to remove the surrounding #{} from the property you want to access and then use ConvertFrom-StringData to return a hash table with sub-property values you can easily access.
$j.members | Select-Object id,
#{n='principalName';e={($_.user -replace "^#{|}$" -split ';' |
ConvertFrom-StringData).principalName}},
#{n='licenseDisplayName';e={($_.accesslevel -replace "^#{|}$" -split ';' |
ConvertFrom-StringData).licenseDisplayName}} |
ConvertTo-Csv -NoTypeInformation

Paste multiline text as variable in PowerShell not working

I am trying to paste some code from MS Docs into PowerShell, the code is for a variable, but when I paste it and hit enter, nothing happens.
Here's the code for the variable:
$site_script = #'
{
"$schema": "schema.json",
"actions": [
{
"verb": "createSPList",
"listName": "Customer Tracking",
"templateType": 100,
"subactions": [
{
"verb": "SetDescription",
"description": "List of Customers and Orders"
},
{
"verb": "addSPField",
"fieldType": "Text",
"displayName": "Customer Name",
"isRequired": false,
"addToDefaultView": true
},
{
"verb": "addSPField",
"fieldType": "Number",
"displayName": "Requisition Total",
"addToDefaultView": true,
"isRequired": true
},
{
"verb": "addSPField",
"fieldType": "User",
"displayName": "Contact",
"addToDefaultView": true,
"isRequired": true
},
{
"verb": "addSPField",
"fieldType": "Note",
"displayName": "Meeting Notes",
"isRequired": false
}
]
}
],
"bindata": { },
"version": 1
}
'#
However, when I paste this and hit enter, nothing happens, and the pointer just goes to a new line that looks like this >>, below is a screenshot:
Is there something I need to do? (Still new to PowerShell)
The error in your Code post is that the ending of your final line includes a space.
$Test = #'
Stuff...
'# #This is wrong as there is a space before '#
$Test = #'
Stuff...
'# #This is correct