I am writing an IIS log parser and having trouble wrapping a variable value in quotes while doing some string processing.
Here is a truncated log file, as an example:
#Fields: date time s-ip cs-method ...
2021-08-09 19:00:16.367 0.0.0.0 GET ...
2021-08-09 19:01:42.184 0.0.0.0 POST ...
Here is how I am executing the code below:
.\Analyse.ps1 cs-method -eq `'POST`'
If the line marked with #PROBLEM is executed as is, the output looks like this:
> .\Analyse.ps1 cs-method -eq `'POST`'
"""""G""E""T""""" ""-""e""q"" ""'""P""O""S""T""'""
"""""P""O""S""T""""" ""-""e""q"" ""'""P""O""S""T""'""
But if I replace $quoted with $value, so that the code reads like this:
$thisInstruction = $thisInstruction -replace $key , $value #PROBLEM
The output looks like this:
> .\Analyse.ps1 cs-method -eq `'POST`'
GET -eq 'POST'
POST -eq 'POST'
The problem is that I want the first value on each line of the output (the GET and the POST before the -eq) to be wrapped in quotes.
How can I achieve this?
Here is my code:
# compile cli args into single line instruction
$instruction = $args -join " "
# define key array
$keys = #('date','time','s-ip','cs-method','cs(Host)','cs-uri-stem','cs-uri-query','s-computername','s-port','cs-username','c-ip','s-sitename','cs(User-Agent)','cs(Referer)','sc-status','sc-substatus','sc-win32-status','TimeTakenMS','x-forwarded-for')
# <#
# get current execution folder
$currentFolder = Get-Location
# define string splitter regex https://www.reddit.com/r/PowerShell/comments/2h5elx/split_string_by_spaces_unless_in_quotes/ckpkydh?utm_source=share&utm_medium=web2x&context=3
$splitter = ' +(?=(?:[^\"]*\"[^\"]*\")*[^\"]*$)'
# process *.log files in folder
Get-Childitem -Path $currentFolder *.log1 | ForEach-Object {
# process each line in the file
Get-Content $_.Name | ForEach-Object {
# duplicate instruction
$thisInstruction = $instruction
# exclude comment lines
if (!$_.StartsWith('#')) {
# split line into array
$logEntryArr = $_ -Split $splitter
# populate dictionary with contents of array
For ($i=0; $i -le $keys.length; $i++) {
# get key
$key = $keys[$i]
# get value
$value = $logEntryArr[$i]
$quoted = "`""+$value+"`""
# replace mention of key in instruction with dictionary reference
$thisInstruction = $thisInstruction -replace $key , $quoted #PROBLEM
}
# process rule from command line against dictionary
echo $thisInstruction
}
}
}
#>
I do know why, thanks to #mathias-r-jessen commenting
I don't know why, but altering the For loop to iterate one fewer fixed the problem and does not appear to leave out any keys. The only significant change is this:
For ($i=0; $i -le $keys.length-1; $i++) {
This PowerShell script can be used to query a folder of log files echo out matching rows, eg:
.\Analyse.ps1 cs-method -eq 'GET'
The above would print out all log entries with a cs-method value of GET.
Here's the code:
# compile cli args into single line instruction
$instruction = $args -join " "
# define key array
$keys = #('date','time','s-ip','cs-method','cs(Host)','cs-uri-stem','cs-uri-query','s-computername','s-port','cs-username','c-ip','s-sitename','cs(User-Agent)','cs(Referer)','sc-status','sc-substatus','sc-win32-status','TimeTakenMS','x-forwarded-for')
# get current execution folder
$currentFolder = Get-Location
# define string splitter regex https://www.reddit.com/r/PowerShell/comments/2h5elx/split_string_by_spaces_unless_in_quotes/ckpkydh?utm_source=share&utm_medium=web2x&context=3
$splitter = ' +(?=(?:[^\"]*\"[^\"]*\")*[^\"]*$)'
# process *.log files in folder
Get-Childitem -Path $currentFolder *.log | ForEach-Object {
# process each line in the file
Get-Content $_.Name | ForEach-Object {
# duplicate instruction
$thisInstruction = $instruction
# exclude comment lines
if (!$_.StartsWith('#')) {
# split line into array
$logEntryArr = $_ -Split $splitter
# populate dictionary with contents of array
For ($i=0; $i -lt $keys.length; $i++) {
# get key
$key = $keys[$i]
# get value
$quoted = "`'"+$logEntryArr[$i]+"`'"
# replace mention of key in instruction with dictionary reference
$thisInstruction = $thisInstruction -replace $key , $quoted
}
# process rule from command line against dictionary
$answer = Invoke-Expression $thisInstruction
if ($answer) {
echo $_
}
}
}
}
Related
My English may not be perfect but I do my best.
I'm trying to write a Powershell script where the filename has a number at the end and it should print exactly that often.
Is this somehow possible ?
With the script it prints it only 1 time.
For whatever reason..
param (
[string]$file = "C:\Scans\temp\*.pdf",
[int]$number_of_copies = 1
)
foreach ($onefile in (Get-ChildItem $file -File)) {
$onefile -match '\d$' | Out-Null
for ($i = 1; $i -le [int]$number_of_copies; $i++) {
cmd /C "lpr -S 10.39.33.204 -P optimidoc ""$($onefile.FullName)"""
}
}
There is no need for parameter $number_of_copies when the number of times it should be printed is taken from the file's BaseName anyway.
I would change your code to:
param (
[string]$path = 'C:\Scans\temp'
)
Get-ChildItem -Path $path -Filter '*.pdf' -File |
# filter only files that end with a number and capture that number in $matches[1]
Where-Object { $_.BaseName -match '(\d+)$' } |
# loop through the files and print
ForEach-Object {
for ($i = 1; $i -le [int]$matches[1]; $i++) {
cmd /C "lpr -S 10.39.33.204 -P optimidoc ""$($_.FullName)"""
}
}
Inside the ForEach-Object, on each iteration, the $_ automatic variable represents the current FileInfo object.
P.S. Your script prints each file only once because you set parameter $number_of_copies to 1 as default value, but the code never changes that to the number found in the file name.
BTW. Nothing wrong with your English
I am attempting to turn the file below into one that contains no comments '#', no blank lines, no unneeded spaces, and only one entry per line. I'm unsure how to run the following code without the need to output the file and then reimport it. There should be code that doesn't require that step but I can't find it. The way I wrote my script also doesn't look right to me even though it works. As if there was a more elegant way of doing what I'm attempting but I just don't see it.
Before File Change: TNSNames.ora
#Created 9_27_16
#Updated 8_30_19
AAAA.world=(DESCRIPTION =(ADDRESS_LIST =
(ADDRESS =
(COMMUNITY = tcp.world)
(PROTOCOL = TCP)
(Host = www.url1111.com)
(Port = 1111)
)
)
(CONNECT_DATA = (SID = SID1111)
)
)
#Created 9_27_16
BBBB.world=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(COMMUNITY=tcp.world)(PROTOCOL=TCP)(Host=url2222.COM)(Port=2222))(ADDRESS=(COMMUNITY=tcp.world)(PROTOCOL=TCP)(Host=url22222.COM)(Port=22222)))(CONNECT_DATA=(SID=SID2222)))
CCCC.world=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(Host=url3333.COM)(Port=3333))(CONNECT_DATA=(SID=SID3333)))
DDDD.url =(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(COMMUNITY=tcp.world)(PROTOCOL=TCP)(Host=URL4444 )(Port=4444))(ADDRESS=(COMMUNITY=TCP.world)(PROTOCOL=TCP)(Host=URL44444 )(Port=44444)))(CONNECT_DATA=(SID=SID4444 )(GLOBAL_NAME=ASDF.URL)))
#Created 9_27_16
#Updated 8_30_19
After File Change:
AAAA.world=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(COMMUNITY=tcp.world)(PROTOCOL=TCP)(Host=www.url1111.com)(Port=1111)))(CONNECT_DATA=(SID=SID1111)))
BBBB.world=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(COMMUNITY=tcp.world)(PROTOCOL=TCP)(Host=url2222.COM)(Port=2222))(ADDRESS=(COMMUNITY=tcp.world)(PROTOCOL=TCP)(Host=url22222.COM)(Port=22222)))(CONNECT_DATA=(SID=SID2222)))
CCCC.world=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(Host=url3333.COM)(Port=3333))(CONNECT_DATA=(SID=SID3333)))
DDDD.url=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(COMMUNITY=tcp.world)(PROTOCOL=TCP)(Host=URL4444)(Port=4444))(ADDRESS=(COMMUNITY=TCP.world)(PROTOCOL=TCP)(Host=URL44444)(Port=44444)))(CONNECT_DATA=(SID=SID4444)(GLOBAL_NAME=ASDF.URL)))
Code:
# Get the file
[System.IO.FileInfo] $File = 'C:\temp\TNSNames.ora'
[string] $data = (Get-Content $File.FullName | Where-Object { !$_.StartsWith('#') }).ToUpper()
# Convert the data. This part is where any (CONNECT_DATA entry ends up on it's own line.
$Results = $data.Replace(" ", "").Replace("`t", "").Replace(")))", ")))`n")
# Convert $Results from BaseType of System.Object to System.Array
$Path = '.\.vscode\StringResults.txt'
$Results | Out-File -FilePath $Path
$Results = Get-Content $Path
# Find all lines that start with '(CONNECT_DATA'
for ($i = 0; $i -lt $Results.Length - 1; $i++) {
if ($Results[$i + 1].StartsWith("(CONNECT_DATA")) {
# Add the '(CONNECT_DATA' line to the previous line
$Results[$i] = $Results[$i] + $Results[$i + 1]
# Blank out the '(CONNECT_DATA' line
$Results[$i + 1] = ''
}
}
# Remove all blank lines
$FinalForm = $null
foreach ($Line in $Results) {
if ($Line -ne "") {
$FinalForm += "$Line`n"
}
}
$FinalForm
So the crux of your problem is that you have declared $data as a [string] which is fine because probably some of your replace operations work better as a single string. Its just that $Results also then ends up being a string so when you try to index into $Results near the bottom these operations fail. You can however easily turn your $Results variable into a string array using the -split operator this would eliminate the need to save the string to disk and import back in just to accomplish the same. See comments below.
# Get the file
[System.IO.FileInfo] $File = 'C:\temp\TNSNames.ora'
[string] $data = (Get-Content $File.FullName | Where-Object { !$_.StartsWith('#') }).ToUpper()
# Convert the data. This part is where any (CONNECT_DATA entry ends up on it's own line.
$Results = $data.Replace(' ', '').Replace("`t", '').Replace(')))', ")))`n")
# You do not need to do this next section. Essentially this is just saving your multiline string
# to a file and then using Get-Content to read it back in as a string array
# Convert $Results from BaseType of System.Object to System.Array
# $Path = 'c:\temp\StringResults.txt'
# $Results | Out-File -FilePath $Path
# $Results = Get-Content $Path
# Instead split your $Results string into multiple lines using -split
# this will do the same thing as above without writing to file
$Results = $Results -split "\r?\n"
# Find all lines that start with '(CONNECT_DATA'
for ($i = 0; $i -lt $Results.Length - 1; $i++) {
if ($Results[$i + 1].StartsWith('(CONNECT_DATA')) {
# Add the '(CONNECT_DATA' line to the previous line
$Results[$i] = $Results[$i] + $Results[$i + 1]
# Blank out the '(CONNECT_DATA' line
$Results[$i + 1] = ''
}
}
# Remove all blank lines
$FinalForm = $null
foreach ($Line in $Results) {
if ($Line -ne '') {
$FinalForm += "$Line`n"
}
}
$FinalForm
Also, for fun, try this out
((Get-Content 'C:\temp\tnsnames.ora' |
Where-Object {!$_.StartsWith('#') -and ![string]::IsNullOrWhiteSpace($_)}) -join '' -replace '\s' -replace '\)\s?\)\s?\)', ")))`n" -replace '\r?\n\(Connect_data','(connect_data').ToUpper()
Hello my input file will be like below,my requiremnet is to add 4 line if the macthing pattern folled by the next pattern is unmacthed along with count number.
i will check look for the socket and if matches will incrremnt the line count to +1 toi get the next line and look for the word "address",if the address is not present i need to insert a set of line "communication.manageraddress_9,communication.manageraddress_10,communication.manageraddress_11" netx to the line.
communication.manageraddress_7=xxx.com
communication.managerid_7=xxx
communication.managerport_7=xxx
communication.socket_7=xx
communication.manageraddress_8=xxx.com
communication.managerid_8=xxx
communication.managerport_8=xxx
communication.socket_8=plain
Added by Manager
communication.managerhealthmon_4=true
communication.protocolrev_4=3
communication.managerhealthmon_1=true
communication.protocolrev_1=2
output will be like this
communication.manageraddress_7=xxx.com
communication.managerid_7=xxx
communication.managerport_7=xxx
communication.socket_7=xx
communication.manageraddress_8=xxx.com
communication.managerid_8=xxx
communication.managerport_8=xxx
communication.socket_8=plain
communication.manageraddress_9=xxx.com
communication.managerid_9=xxx
communication.managerport_9=xxx
communication.socket_9=plain
communication.manageraddress_10=xxx.com
communication.managerid_10=xxx
communication.managerport_1o=xxx
communication.socket_1o=plain
Added by Manager
communication.managerhealthmon_4=true
communication.protocolrev_4=3
communication.managerhealthmon_1=true
communication.protocolrev_1=2
this my script and i am struck with insert into text file along with increment number,can some one help in power shell.
$files = $File = 'C:\Users\rseerala\Desktop\ARUN\in.txt'
#$NewContent = Get-Content -Path $File
foreach($file in $files){
$content = Get-Content $file
for($i = 0; $i -lt $content.Count; $i++){
$line = $content[$i]
if ($line.Contains("socket"))
{
$line = $content[$i+2]
if ($line.Contains("address"))
{
Write-Host "This line starts with 6"
}}}}
Ok, so if I understand correctly, this is what you want:
#read the file as a single multiline string
$txt = Get-Content -Path 'C:\Users\rseerala\Desktop\ARUN\in.txt' -Raw
# if it contains the magic word '.socket_' followed by a number
if ($txt -match '\.socket_\d+') {
# first split off the 'Added by Manager' stuff
$content, $managerAdded = ($txt -split 'Added by Manager').Trim()
# split the content part into separate blocks of 4 lines
$blocks = $content -split '(\r?\n){2}' | Where-Object { $_ -match '\S' }
# get the index value from the last block
$index = [int]([regex] '(?i)\.socket_(\d+)').Match($blocks[-1]).Groups[1].Value
# now repeat the blocks you already have and output copies with incremented indices
$newBlocks = ($blocks | ForEach-Object {
$_ -replace '_\d+=', ('_{0}=' -f ++$index)
}) -join "`r`n`r`n"
# finally, combine the content part with the new blocks
# and the 'Added by Manager' lines with double newlines
$result = $content, $newBlocks, 'Added by Manager', $managerAdded -join "`r`n`r`n"
# output on screen
$result
# write to a new file
$result | Set-Content -Path 'C:\Users\rseerala\Desktop\ARUN\out.txt'
}
else {
Write-Warning "The file does not contain the word '.socket_' followed by a number.."
}
Output:
communication.manageraddress_7=xxx.com
communication.managerid_7=xxx
communication.managerport_7=xxx
communication.socket_7=xx
communication.manageraddress_8=xxx.com
communication.managerid_8=xxx
communication.managerport_8=xxx
communication.socket_8=plain
communication.manageraddress_9=xxx.com
communication.managerid_9=xxx
communication.managerport_9=xxx
communication.socket_9=plain
communication.manageraddress_10=xxx.com
communication.managerid_10=xxx
communication.managerport_10=xxx
communication.socket_10=plain
Added by Manager
communication.managerhealthmon_4=true
communication.protocolrev_4=3
communication.managerhealthmon_1=true
communication.protocolrev_1=2
I am creating the below script to search through and replace data in a set of files. The problem I'm running into is I need to ONLY match if it's the beginning of the line, and I'm not sure how/where would I use regex in the below example (e.g. ^A, ^B) when doing the comparison? I tried putting the caret in front of the name values in the table, but that didn't work...
$lookupTable = #{
'A'='1';
'B'='2'
#etc
}
Get-ChildItem 'c:\windows\system32\dns' -Filter *.dns |
Foreach-Object {
$file = $_
Write-Host "$file"
(Get-Content -Path $file -Raw) | ForEach-Object {
$line = $_
$lookupTable.GetEnumerator() | ForEach-Object {
$line = $line -replace $_.Name, $_.Value
}
$line
} | Set-Content -Path $file
}
The -replace operator accepts Regex. Just $line = $line -replace "^$($_.Name)", "$_.Value".
the way that regex works makes getting a proper "start of line" marker into the regex pattern along with the $VarName a tad iffy. so i broke it out into it's own line and used the -f string format operator to build the regex pattern.
then i used the way that -replace works on an array of strings that one usually gets from Get-Content to work on the whole array at each pass.
note that the strings have lower case items where they otta be replaced, and uppercase items where the item should NOT be replaced. [grin]
$LookUpTable = #{
A = 'Wizbang Shadooby'
Z = '666 is the number of the beast'
}
$LineList = #(
'a sdfq A er Z xcv'
'qwertyuiop A'
'z xcvbnm'
'z A xcvbnm'
'qwertyuiop Z'
)
$LookUpTable.GetEnumerator() |
ForEach-Object {
$Target = '^{0}' -f $_.Name
$LineList = $LineList -replace $Target, $_.Value
}
$LineList
output ...
Wizbang Shadooby sdfq A er Z xcv
qwertyuiop A
666 is the number of the beast xcvbnm
666 is the number of the beast A xcvbnm
qwertyuiop Z
# Here is a complete, working script that beginners can read.
# This thread
# Using regex in a key/value lookup table in powershell?
# https://stackoverflow.com/questions/57277282/using-regex-in-a-key-value-lookup-table-in-powershell
# User-modifiable variables.
# substitutions
# We need to specify what we're looking for (keys).
# We need to specify our substitutions (values).
# Example: Looking for A and substituting 1 in its place.
# Add as many pairs as you like.
# Here I use an array of objects instead of a Hashtable so that I can specify upper- and lowercase matches.
# Use the regular expression caret (^) to match the beginning of a line.
$substitutions = #(
[PSCustomObject]#{ Key = '^A'; Value = '1' },
[PSCustomObject]#{ Key = '^B'; Value = '2' },
[PSCustomObject]#{ Key = '^Sit'; Value = '[Replaced Text]' }, # Example for my Latin placeholder text.
[PSCustomObject]#{ Key = 'nihil'; Value = '[replaced text 2]' }, # Lowercase example.
[PSCustomObject]#{ Key = 'Nihil'; Value = '[Replaced Text 3]' } # Omit comma for the last array item.
)
# Folder where we are looking for files.
$inputFolder = 'C:\Users\Michael\PowerShell\Using regex in a key value lookup table in powershell\input'
# Here I've created some sample files using Latin placeholder text from
# https://lipsum.com/
# Folder where we are saving the modified files.
# This can be the same as the input folder.
# I'm creating this so we can test without corrupting the original files.
$outputFolder = 'C:\Users\Michael\PowerShell\Using regex in a key value lookup table in powershell\output'
#$outputFolder = $inputFolder
# We are only interested in files ending with .dns
$filterString = '*.dns'
# Here is an example for text files.
#$filterString = '*.txt'
# For all files.
#$filterString = '*.*'
# More info.
# https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.management/get-childitem?view=powershell-6#parameters
# Search on the page for -Filter
# You won't need to update any variables after this line.
# ===================================================================
# Generate a list of files to look at.
$fileList = Get-ChildItem $inputFolder -Filter $filterString
# Simple example.
# get-content .\apple.dns | % { $_ -replace "sit", "michael" } | set-content "C:\output\apple.dns"
# input file substitutions output
# Set up loops.
# For each file.
#{
# For each key-value pair.
#}
# "For each key-value pair."
# Create a function.
# Pipe in a string.
# Specify a list of substitutions.
# Make the substitutions.
# Output a modified string.
filter find_and_replace ([object[]] $substitutions)
{
# The automatic variable $_ will be a line from the file.
# This comes from the pipeline.
# Copy the input string.
# This avoids modifying a pipeline object.
$myString = $_
# Look at each key-value pair passed to the function.
# In practice, these are the ones we defined at the top of the script.
foreach ($pair in $substitutions)
{
# Modify the strings.
# Update the string after each search.
# case-sensitive -creplace instead of -replace
$myString = $myString -creplace $pair.Key, $pair.Value
}
# Output the final, modified string.
$myString
}
# "For each file."
# main
# Do something with each file.
foreach ($file in $fileList)
{
# Where are we saving the output?
$outputFile = Join-Path -Path $outputFolder -ChildPath $file.Name
# Create a pipeline.
# Pipe strings to our function.
# Let the function modify the strings.
# Save the output to the output folder.
# This mirrors our simple example but with dynamic files and substitutions.
# find_and_replace receives strings from the pipeline and we pass $substitutions into it.
Get-Content $file | find_and_replace $substitutions | Set-Content $outputFile
# The problem with piping files into a pipeline is that
# by the time the pipeline gets to Set-Content,
# we only have modified strings
# and we have no information to create the path for an output file.
# ex [System.IO.FileInfo[]] | [String[]] | [String] | Set-Content ?
#
# Instead, we're in a loop that preserves context.
# And we have the opportunity to create and use the variable $outputFile
# ex foreach ($file in [System.IO.FileInfo[]])
# ex $outputFile = ... $file ...
# ex [String[]] | [String] | Set-Content $outputFile
# Quote
# (Get-Content -Path $file -Raw)
# By omitting -Raw, we get: one string for each line.
# This is instead of getting: one string for the whole file.
# This keeps us from having to use
# the .NET regular expression multiline option (and the subexpression \r?$)
# while matching.
#
# What it is.
# Multiline Mode
# https://learn.microsoft.com/en-us/dotnet/standard/base-types/regular-expression-options#Multiline
#
# How you would get started.
# Miscellaneous Constructs in Regular Expressions
# https://learn.microsoft.com/en-us/dotnet/standard/base-types/miscellaneous-constructs-in-regular-expressions
}
I'm looping through some files and pulling values into a dictionary around a ":" delimiter.
The data in the txt files looks like this:
AD ID: 9999
Ad Placement: Computers
Landing Page: www.something.com
Interests: this and that and this
Interests: also this thing and one final thing
My script for creating a dictionary looks like the following:
$files = ls "*.txt"
$dictionary = #{}
[System.Collections.Generic.List[String]]$list = #()
foreach ($f in $files) {
$in = Get-Content -Raw $f
$in.Split([Environment]::NewLine) | ForEach-Object {
$key, $value = $_.Split(':')
$dictionary[$key] = $value
}
[void]$list.Add($dictionary['Ad ID'] + ',' + $dictionary['Ad Text'] + ',' +
$dictionary['Ad Landing Page'] + ',' + $dictionary['Interests'])
}
That's the basic idea at least. I've gotten unpredictable results when I come across a file that has a key twice, as is the case in the entry in the sample data above called "Interests."
What occurs when adding dictionary items to a list from a file?
In the above example, what is the value of $dictionary['interests'] as it goes through the script?
Since the data can contain duplicate keys, you cannot use the ConvertFrom-StringData cmdlet.
To get the data in a dictionary (hashtable) manually is not that hard to do and you can decide for yourself what to do with duplicate keys: either overwrite the values so the last entry found 'wins' or not:
# this decides which duplicate value you want to store in the hashtable
$allowOverwrite = $false
$hash = #{}
# get the content of the file as string array and loop through
Get-Content -Path 'THE FULL PATH AND FILENAME OF YOUR TEXTFILE' | ForEach-Object {
if ( -not [string]::IsNullOrWhiteSpace($_)) {
# split string to get the key and the value
$key, $value = $_ -split ':', 2 | ForEach-Object { $_.Trim() }
# if a key is found that already exists in the hashtable
if ($hash.ContainsKey($key)) {
# either overwrite the value 'Last-One-Wins'
# or do nothing 'First-One-Wins'
if ($allowOverwrite) { $hash[$key] = $value }
}
else {
$hash[$key] = $value
}
}
}
$hash["interests"]
shows "this and that and this" in case of $allowOverwrite = $false
shows "also this thing and one final thing" in case of $allowOverwrite = $true