Powershell text search - multiple matches - powershell

I have a group of .txt files that contain one or two of the following strings.
"red", "blue", "green", "orange", "purple", .... many more (50+) possibilities in the list.
If it helps, I can tell if the .txt file contains one or two items, but don't know which one/ones they are. The string patterns are always on their own line.
I'd like the script to tell me specifically which one or two string matches (from the master list) it found, and the order in which it found them. (Which one was first)
Since I have a lot of text files to search, I'd like to write the output results to a CSV file as I search.
FILENAME1,first_match,second_match
file1.txt,blue,red
file2.txt,red, blue
file3.txt,orange,
file4.txt,purple,red
file5.txt,purple,
...
I've tried using many individual Select-Strings returning Boolean results to set variables with any matches found, but with the number of possible strings it gets ugly real fast. My search results for this issue has provided me with no new ideas to try. (I'm sure I'm not asking in the correct way)
Do I need to loop through each line of text in each file?
Am I stuck with the process of elimination method by checking for the existence of each search string?
I'm looking for a more elegant approach to this problem. (if one exists)

Not very intuïtive but elegant...
Following switch statement
$regex = "(purple|blue|red)"
Get-ChildItem $env:TEMP\test\*.txt | Foreach-Object{
$result = $_.FullName
switch -Regex -File $_
{
$regex {$result = "$($result),$($matches[1])"}
}
$result
}
returns
C:\Users\Lieven Keersmaekers\AppData\Local\Temp\test\file1.txt,blue,red
C:\Users\Lieven Keersmaekers\AppData\Local\Temp\test\file2.txt,red,blue
where
file1 contains first blue, then red
file2 contains first red, then blue

You can use regex to search to get index (startpos. in line) combine with Select-String which returns linenumber and you're good to go.
Select-String supports an array as value for -Pattern, but unfortunately it stops on a line after first match even when you use -AllMatches (bug?). Because of this we have to search one time per word/pattern. Try:
#List of words. Had to escape them because Select-String doesn't return Matches-objects (with Index/location) for SimpleMatch
$words = "purple","blue","red" | ForEach-Object { [regex]::Escape($_) }
#Can also use a list with word/sentence per line using $words = Get-Content patterns.txt | % { [regex]::Escape($_.Trim()) }
#Get all files to search
Get-ChildItem -Filter "test.txt" -Recurse | Foreach-Object {
#Has to loop words because Select-String -Pattern "blue","red" won't return match for both pattern. It stops on a line after first match
foreach ($word in $words) {
$_ | Select-String -Pattern $word |
#Select the properties we care about
Select-Object Path, Line, Pattern, LineNumber, #{n="Index";e={$_.Matches[0].Index}}
}
} |
#Sort by File (to keep file-matches together), then LineNumber and Index to get the order of matches
Sort-Object Path, LineNumber, Index |
Export-Csv -NoTypeInformation -Path Results.csv -Encoding UTF8
Results.csv
"Path","Line","Pattern","LineNumber","Index"
"C:\Users\frode\Downloads\test.txt","file1.txt,blue,red","blue","3","10"
"C:\Users\frode\Downloads\test.txt","file1.txt,blue,red","red","3","15"
"C:\Users\frode\Downloads\test.txt","file2.txt,red, blue","red","4","10"
"C:\Users\frode\Downloads\test.txt","file2.txt,red, blue","blue","4","15"
"C:\Users\frode\Downloads\test.txt","file4.txt,purple,red","purple","6","10"
"C:\Users\frode\Downloads\test.txt","file4.txt,purple,red","red","6","17"
"C:\Users\frode\Downloads\test.txt","file5.txt,purple,","purple","7","10"

Related

Is there a way to merge similar lines using Powershell?

Suppose I have two csv files. One is
id_number,location_code,category,animal,quantity
12212,3,4,cat,2
29889,7,6,dog,2
98900,
33221,1,8,squirrel,1
the second one is:
98900,2,1,gerbil,1
The second file may have a newline or something at the end (maybe or maybe not, I haven't checked), but only the one line of content. There may be three or four or more different varieties of the "second" file, but each one will have a first element (98900 in this example) that corresponds to an incomplete line in the first file similar to what is in this example.
Is there a way using powershell to automatically merge the line in the second (plus any additional similar) csv file into the matching line(s) of the first file, so that the resulting file is:
12212,3,4,cat,2
29889,7,6,dog,2
98900,2,1,gerbil,1
33221,1,8,squirrel,1
main.csv
id_number,location_code,category,animal,quantity
12212,3,4,cat,2
29889,7,6,dog,2
98900,
33221,1,8,squirrel,1
correction_001.csv
98900,2,1,gerbil,1
merge code used at the commandline, or in the .ps1 file of your choice
$myHeader = #('id_number','location_code','category','animal','quantity')
#Stage all the correction files: last correction in the most recent file wins
$ToFix = #{}
filter Plumbing_Import-Csv($Header){import-csv -LiteralPath $_ -Header $Header}
ls correction*.csv | sort -Property LastWriteTime | Plumbing_Import-Csv $myHeader | %{$ToFix[$_.id_number]=$_}
function myObjPipe($Header){
begin{
function TextTo-CsvField([String]$text){
#text fields which contain comma, double quotes, or new-line are a special case for CSV fields and need to be accounted for
if($text -match '"|,|\n'){return '"'+($text -replace '"','""')+'"'}
return $text
}
function myObjTo-CsvRecord($obj){
return ''+
$obj.id_number +','+
$obj.location_code +','+
$obj.category +','+
(TextTo-CsvField $obj.animal)+','+
$obj.quantity
}
$Header -join ','
}
process{
if($ToFix.Contains($_.id_number)){
$out = $ToFix[$_.id_number]
$ToFix.Remove($_.id_number)
}else{$out = $_}
myObjTo-CsvRecord $out
}
end{
#I assume you'd append any leftover fixes that weren't used
foreach($out in $ToFix.Values){
myObjTo-CsvRecord $out
}
}
}
import-csv main.csv | myObjPipe $myHeader | sc combined.csv -encoding ascii
You could also use ConvertTo-Csv, but my preference is to not have all the extra " cruft.
Edit 1: reduced code redundancy, accounted for \n, fixed appends, and used #OwlsSleeping suggestion about the -Header commandlet parameter
also works with these files:
correction_002.csv
98900,2,1,I Win,1
correction_new.csv
98901,2,1,godzilla,1
correction_too.csv
98902,2,1,gamera,1
98903,2,1,mothra,1
Edit 2: convert gc | ConvertTo-Csv over to Import-Csv to fix the front-end \n issues. Now also works with:
correction_003.csv
29889,7,6,"""bad""
monkey",2
This is a simple solution assuming there's always exactly one match, and you don't care about output order. Change the output path to csv1 to overwrite.
I added headers manually in both input files, but you can specify them in Import-Csv instead if you'd rather avoid changing your files.
[array]$MissingLine = Import-Csv -Path "C:\Users\me\Documents\csv2.csv"
[string]$MissingId = $MissingLine[0].id_number
[array]$BigCsv = Import-Csv -Path "C:\Users\me\Documents\csv1.csv" |
Where-Object {$_.id_number -ne $MissingId}
($BigCsv + $MissingLine) |
Export-Csv -Path "C:\Users\me\Documents\Combined.csv"

Powershell replace text once per line

I have a Powershell script that I am trying to work out part of it, so the text input to this is listing the user group they are part of. This PS script is supposed to replace the group with the groups that I am assigning them in active directory(I am limited to only changing groups in active directory). My issue is that when it reaches HR and replaces it, it will then proceed to contine and replace all the new but it all so replaces the HR in CHRL, so my groups look nuts right now. But I am looking it over and it doesn't do it with every line. But for gilchrist it will put something in there for the HR in the name. Is there anything can I do to keep it for changing or am I going to have to change my HR to Human Resources? Thanks for the help.
$lookupTable = #{
'Admin' = 'W_CHRL_ADMIN_GS,M_CHRL_ADMIN_UD,M_CHRL_SITE_GS'
'Security' = 'W_CHRL_SECURITY_GS,M_CHRL_SITE_GS'
'HR' = 'M_CHRL_HR_UD,W_CHRL_HR_GS,M_CHRL_SITE_GS'
$original_file = 'c:\tmp\test.txt'
$destination_file = 'c:\tmp\test2.txt'
Get-Content -Path $original_file | ForEach-Object {
$line = $_
$lookupTable.GetEnumerator() | ForEach-Object {
if ($line -match $_.Key)
{
$line = $line -replace $_.Key, $_.Value
}
}
$line
} | Set-Content -Path $destination_file
Get-Content $destination_file
test.txt:
user,group
john.smith,Admin
joanha.smith,HR
john.gilchrist,security
aaron.r.smith,admin
abby.doe,secuity
abigail.doe,admin
Your input appears to be in CSV format (though note that your sample rows have trailing spaces, which you'd have to deal with, if they're part of your actual data).
Therefore, use Import-Csv and Export-Csv to read / rewrite your data, which allows a more concise and convenient solution:
Import-Csv test.txt |
Select-Object user, #{ Name='group'; Expression = { $lookupTable[$_.group] } } |
Export-Csv -NoTypeInformation -Encoding Utf8 test2.txt
Import-Csv reads the CSV file as a collection of custom objects whose properties correspond to the CSV column values; that is, each object has a .user and .name property in your case.
$_.group therefore robustly reports the abstract group name only, which you can directly pass to your lookup hashtable; Select-Object is used to pass the original .user value through, and to replace the original .group value with the lookup result, using a calculated property.
Export-Csv re-converts the custom objects to a CSV file:
-NoTypeInformation suppresses the (usually useless) data-type-information line at the top of the output file
-Encoding Utf8 was added to prevent potential data loss, because it is ASCII encoding that is used by default.
Note that Export-Csv blindly double-quotes all field values, whether they need it or not; that said, CSV readers should be able to deal with that (and Import-Csv certainly does).
As for what you tried:
The -replace operator replaces all occurrences of a given regex (regular expression) in the input.
Your regexes amounts to looking for (case-insensitive) substrings, which explains why HR matches both the HR group name and substring hr in username gilchrist.
A simple workaround would be to add assertions to your regex so that the substrings only match where you want them; e.g.: ,HR$ would only match after a , at the end of a line ($).
However, your approach of enumerating the hashtable keys for each input CSV row is inefficient, and you're better off splitting off the group name and doing a straight lookup based on it:
# Split the row into fields.
$fields = $line -split ','
# Update the group value (last field)
$fields[-1] = $lookupTable[$fields[-1]]
# Rebuild the line
$line = $fields -join ','
Note that you'd have to make an exception for the header row (e.g., test if the lookup result is empty and refrain from updating, if so).
Why don't you load your text file as a CSV file, using Import-CSV and use "," as a delimiter?
This will allow you to have a Powershell Object you can work on. and then export it as text o CSV. if I use your file & lookup table this code may help you :
$file = Import-Csv -Delimiter "," -Path "c:\ps\test.txt"
$lookupTable = #{
'Admin' = 'W_CHRL_ADMIN_GS,M_CHRL_ADMIN_UD,M_CHRL_SITE_GS'
'Security' = 'W_CHRL_SECURITY_GS,M_CHRL_SITE_GS'
'HR' = 'M_CHRL_HR_UD,W_CHRL_HR_GS,M_CHRL_SITE_GS'}
foreach ($i in $file) {
#Compare and replace
...
}
Export-CSV $file -Delimiter ","
You can then iterate over $file and compare and replace. you can also Export-CSV after you're done.

Read numbers from multiple files and sum

I have a logfile C:\temp\data.log
It contains the following data:
totalSize = 222,6GB
totalSize = 4,2GB
totalSize = 56,2GB
My goal is to extract the numbers from the file and sum them up including the number after the comma. So far it works if I don't regex the number included with value after comma, and only use the number in front of the comma. The other problem I have is if the file only contains one row like below example, if it only contains one line it splits up the number 222 into three file containing the number 2 in three files. If the above logfile contains 2 lines or more it works and sums up as it should, as long I don't use value with comma.
totalSize = 222,6GB
Here is a bit of the code for the regex to add to end of existing variable $regex included with comma is:
[,](\d{1,})
I haven't included the above regex, as it does not sum up properly then.
The whole script is below:
#Create path variable to store contents grabbed from $log_file
$extracted_strings = "C:\temp\amount.txt"
#Create path variable to read from original file
$log_file = "C:\temp\data.log"
#Read data from file $log_file
Get-Content -Path $log_file | Select-String "(totalSize = )" | out-file $extracted_strings
#Create path variable to write only numbers to file $output_numbers
$output_numbers = "C:\temp\amountresult.log"
#Create path variable to write to file jobblog1
$joblog1_file = "C:\temp\joblog1.txt"
#Create path variable to write to file jobblog2
$joblog2_file = "C:\temp\joblog2.txt"
#Create path variable to write to file jobblog3
$joblog3_file = "C:\temp\joblog3.txt"
#Create path variable to write to file jobblog4
$joblog4_file = "C:\temp\joblog4.txt"
#Create path variable to write to file jobblog5
$joblog5_file = "C:\temp\joblog5.txt"
#Create pattern variable to read with select string
$regex = "[= ](\d{1,})"
select-string -Path $extracted_strings -Pattern $regex -AllMatches | % { $_.Matches } | % { $_.Value } > $output_numbers
(Get-Content -Path $output_numbers)[0..0] -replace '\s' > $joblog1_file
(Get-Content -Path $output_numbers)[1..1] -replace '\s' > $joblog2_file
(Get-Content -Path $output_numbers)[2..2] -replace '\s' > $joblog3_file
(Get-Content -Path $output_numbers)[3..3] -replace '\s' > $joblog4_file
(Get-Content -Path $output_numbers)[4..4] -replace '\s' > $joblog5_file
$jobdata0 = (Get-Content -Path $joblog1_file)
$jobdata1 = (Get-Content -Path $joblog2_file)
$jobdata2 = (Get-Content -Path $joblog3_file)
$jobdata3 = (Get-Content -Path $joblog4_file)
$jobdata4 = (Get-Content -Path $joblog5_file)
$result = $jobdata0 + $jobdata1 + $jobdata2 + $jobdata3 + $jobdata4
$result
So my questions is:
How can I get this to work if the file C:\temp\data.log only contains one string without dividing that single number into multiple files. It should also work if it contains multiple strings, as it is now it works with multiple strings.
And how can I include the comma values in the calculation?
The result I get if I run this script should be 282, maybe its even possible to shorten the script?
Where $log_file has contents like the example above.
Get-Content $log_file | Where-Object{$_ -match "\d+(,\d+)?"} |
ForEach-Object{[double]($matches[0] -replace ",",".")} |
Measure-Object -Sum |
Select-Object -ExpandProperty sum
Match all of the lines that have numerical values with optional commas. I am assuming they could be optional as I do not know how whole numbers appear. Replace the comma with a period and cast as a double. Using measure object we sum up all the values and expand the result.
Not the only way to do it but it is simple enough to understand what is going on.
You can always wrap the above up in a loop so that you can use it for multiple files. Get-ChildItem "C:temp\" -Filter "job*" | ForEach-Object... etc.
Matt's helpful answer shows a concise and effective solution.
As for what you tried:
As for why a line with a single token such as 222,6 can result in multiple outputs in this command:
select-string -Path $extracted_strings -Pattern $regex -AllMatches |
% { $_.Matches } | % { $_.Value } > $output_numbers
Your regex, [= ](\d{1,}), does not explain the symptom, but just \d{1,} would, because that would capture 222 and 6 separately, due to -AllMatches.
[= ](\d{1,}) probably doesn't do what you want, because [= ] matches a single character that can be either a = or a space; with your sample input, this would only ever match the space before the numbers.
To match characters in sequence, simply place them next to each other: = (\d{1,})
Also note that even though you're enclosing \d{1,} in (...) to create a capture group, your later code doesn't actually use what that capture group matched; use (...) only if you need it for precedence (in which case you can even opt out of subexpression capturing with (?:...)) or if you do have a need to access what the subexpression matched.
That said, you could actually utilize a capture group here (an alternative would be to use a look-behind assertion), which allows you to both match the leading =<space> for robustness and extract only the numeric token of interest (saving you the need to trim whitespace later).
If we simplify \d{1,} to \d+ and append ,\d+ to also match the number after the comma, we get:
= (\d+,\d+)
The [System.Text.RegularExpressions.Match] instances returned by Select-String then allow us to access what the capture group captured, via the .Groups property (the following simplified example also works with multiple input lines):
> 'totalSize = 222,6GB' | Select-String '= (\d+,\d+)' | % { $_.Matches.Groups[1].Value }
222,6
On a side note: your code contains a lot of repetition that could be eliminated with arrays and pipelines; for instance:
$joblog1_file = "C:\temp\joblog1.txt"
$joblog2_file = "C:\temp\joblog2.txt"
$joblog3_file = "C:\temp\joblog3.txt"
$joblog4_file = "C:\temp\joblog4.txt"
$joblog5_file = "C:\temp\joblog5.txt"
could be replaced with (create an array of filenames, using a pipeline):
$joblog_files = 1..5 | % { "C:\temp\joblog$_.txt" }
and
$jobdata0 = (Get-Content -Path $joblog1_file)
$jobdata1 = (Get-Content -Path $joblog2_file)
$jobdata2 = (Get-Content -Path $joblog3_file)
$jobdata3 = (Get-Content -Path $joblog4_file)
$jobdata4 = (Get-Content -Path $joblog5_file)
$result = $jobdata0 + $jobdata1 + $jobdata2 + $jobdata3 + $jobdata4
could then be replaced with (pass the array of filenames to Get-Content):
$result = Get-Content $joblog_files

powershell Parsing for multiple keywords and sending output to a text file

I'm trying to write a powershell cmdlet to find multiple words in lines in file. Example. I need to parse "word1", "word2", "word3" are in the same line of a file. I'm doing something wrong because I tried this with no success:
(gci -File -Filter FileName | Select-String -SimpleMatch word1, word2,word3) > outputFileName.txt
where FileName = name of file, outputFileName = generated file from my search of the three words. Thank you.
Select-String doesn't have any combination operator that I can think of. If your words were always in that order, then you could do -Pattern 'word1.*word2.*word3' as your match, but if they could be in any order that would get complex very quickly. Instead, I'd look at
.. | Select-String 'word1' | Select-String 'word2' | Select-String 'Word3'
so, all the lines which match word1. Out of those, the ones which match word2 somewhere. Out of that even smaller result, the ones which also match word3.
try this:
$wordlist=#("word1", "word2", "word3")
Get-ChildItem "c:\temp\" -file | %{$currentfile=$_.FullName; Get-Content $_.FullName |
%{
$founded=$true
foreach ($item in $wordlist)
{
if (!$_.Contains($item))
{
$founded=$false
break
}
}
if ($founded)
{
$currentfile
}
}
}

Prepend / append data to one column in csv in powershell

I'm really liking what I have seen of Powershell. But I'm really confused by some things, as I have so much to learn. I've been reading everything on the site here, but I've not been able to figure this out. Hopefully this is simple. I have a csv like this:
Title,Name,Office,Phone
Boss,Bob,101,323.555-1212
Office-Manager-Level-2,Helen,202,5-1213
Time-Waster-Level-5,Nemo,105,5-1214
Widget-Maker,Zack,10,5-1215
Temp,Larry,102,5-1000
I have been trying to figure out an easy way to prepend & append data to the first column, "Title", that will take eventually become a static webpage with the user's information. I'm trying this so far:
$file = ("\\web\users.csv")
$urlbase="<a href`=`"file:///web/users/info/"
$urlend="_info.html`">"
$data = import-csv ($file) -header ("Title","Name","Office","Phone")
$data | select -Skip 1 | % { $_.Title -replace '$_.Title', "'$urlbase'$_.Title'$urlend'`">'$_.Title'</a>"} | Export-CSV -Path "links_output.csv" -NoTypeInformation
However - all that I'm matching or replacing it appears is the length of the string (??) of the first column of data. My output file is this:
"Length"
"4"
"23"
"19"
"12"
"4"
What I would desire as my output would be:
<a href="file:///web/users/info/Boss_info.html"Boss</a>"
Office-Manager-Level-2"
Time-Waster-Level-5"
Widget-Maker"
Temp"
Also, besides my basic issue, if I could use set-content I'd be happy because I'd really like this to be like a sed -i type of action/function, on the original file, but a new file with the same contents as the old with the updated first column will satisfy if I cannot set-content on the original.
This section of my script will become an html file later and because of issues with regex find and replacing with tags, I'm trying to add the html tags before I use ConvertTo-Html, because that is all working already. Thanks in advance!!
Here's one solution:
$file = ("\\web\users.csv")
$urlbase='<a href="file:///web/users/info/'
$urlend='_info.html">'
get-content $file |
select -Skip 1 |
foreach {
"$Urlbase{0}$urlend" -f $_.split(',')[0]
}
The split(',') is an object method of [string] that will split the string at the commas, producing an array. The trailing [0] takes the first element of that array, whic will be the Name. That gets inserted at {0} in the format string between the other two variables by the format (-f) operator.
You can use the -replace operator, but you can't use PS variables in the replacement string. You can include the literal text:
(get-content $file | select -Skip 1) -replace '^([^,]+)(.+)','<a href="file:///web/users/info/$1_info.html">$2'