Powershell: Search data in *.txt files to export into *.csv - powershell

First of all, this is my first question here. I often come here to browse existing topics, but now I'm hung on my own problem. And I didn't found a helpful resource right now. My biggest concern would be, that it won't work in Powershell... At the moment I try to get a small Powershell tool to save me a lot of time. For those who don't know cw-sysinfo, it is a tool that collects information of any host system (e.g. Hardware-ID, Product Key and stuff like that) and generates *.txt files.
My point is, if you have 20, 30 or 80 server in a project, it is a huge amount of time to browse all files and just look for those lines you need and put them together in a *.csv file.
What I have working is more like the basic of the tool, it browses all *.txt in a specific path and checks for my keywords. And here is the problem that I just can use the words prior to those I really need, seen as follow:
Operating System: Windows XP
Product Type: Professional
Service Pack: Service Pack 3
...
I don't know how I can tell Powershell to search for "Product Type:"-line and pick the following "Professional" instead. Later on with keys or serial numbers it will be the same problem, that is why I just can't browse for "Standard" or "Professional".
I placed my keywords($controls) in an extra file that I can attach the project folders and don't need to edit in Powershell each time. Code looks like this:
Function getStringMatch
{
# Loop through the project directory
Foreach ($file In $files)
{
# Check all keywords
ForEach ($control In $controls)
{
$result = Get-Content $file.FullName | Select-String $control -quiet -casesensitive
If ($result -eq $True)
{
$match = $file.FullName
# Write the filename according to the entry
"Found : $control in: $match" | Out-File $output -Append
}
}
}
}
getStringMatch

I think this is the kind of thing you need, I've changed Select-String to not use the -quiet option, this will return a matches object, one of the properties of this is the line I then split the line on the ':' and trim any spaces. These results are then placed into a new PSObject which in turn is added to an array. The array is then put back on the pipeline at the end.
I also moved the call to get-content to avoid reading each file more than once.
# Create an array for results
$results = #()
# Loop through the project directory
Foreach ($file In $files)
{
# load the content once
$content = Get-Content $file.FullName
# Check all keywords
ForEach ($control In $controls)
{
# find the line containing the control string
$result = $content | Select-String $control -casesensitive
If ($result)
{
# tidy up the results and add to the array
$line = $result.Line -split ":"
$results += New-Object PSObject -Property #{
FileName = $file.FullName
Control = $line[0].Trim()
Value = $line[1].Trim()
}
}
}
}
# return the results
$results
Adding the results to a csv is just a case of piping the results to Export-Csv
$results | Export-Csv -Path "results.csv" -NoTypeInformation

If I understand your question correctly, you want some way to parse each line from your report files and extract values for some "keys". Here are a few lines to give you an idea of how you could proceede. The example is for one file, but can be generalized very easily.
$config = Get-Content ".\config.txt"
# The stuff you are searching for
$keys = #(
"Operating System",
"Product Type",
"Service Pack"
)
foreach ($line in $config)
{
$keys | %{
$regex = "\s*?$($_)\:\s*(?<value>.*?)\s*$"
if ($line -match $regex)
{
$value = $matches.value
Write-Host "Key: $_`t`tValue: $value"
}
}
}

Related

Scanning log file using ForEach-Object and replacing text is taking a very long time

I have a Powershell script that scans log files and replaces text when a match is found. The list is currently 500 lines, and I plan to double/triple this. the log files can range from 400KB to 800MB in size. 
Currently, when using the below, a 42MB file takes 29mins, and I'm looking for help if anyone can see any way to make this faster?
I tried changing ForEach-Object with ForEach-ObjectFast but it's causing the script to take sufficiently longer. also tried changing the first ForEach-Object to a forloop but still took ~29 mins. 
$lookupTable= #{
'aaa:bbb:123'='WORDA:WORDB:NUMBER1'
'bbb:ccc:456'='WORDB:WORDBC:NUMBER456'
}
Get-Content -Path $inputfile | ForEach-Object {
$line=$_
$lookupTable.GetEnumerator() | ForEach-Object {
if ($line-match$_.Key)
{
$line=$line-replace$_.Key,$_.Value
}
}
$line
}|Set-Content -Path $outputfile
Since you say your input file could be 800MB in size, reading and updating the entire content in memory could potentially not fit.
The way to go then is to use a fast line-by-line method and the fastest I know of is switch
# hardcoded here for demo purposes.
# In real life you get/construct these from the Get-ChildItem
# cmdlet you use to iterate the log files in the root folder..
$inputfile = 'D:\Test\test.txt'
$outputfile = 'D:\Test\test_new.txt' # absolute full file path because we use .Net here
# because we are going to Append to the output file, make sure it doesn't exist yet
if (Test-Path -Path $outputfile -PathType Leaf) { Remove-Item -Path $outputfile -Force }
$lookupTable= #{
'aaa:bbb:123'='WORDA:WORDB:NUMBER1'
}
# create a regex string from the Keys of your lookup table,
# merging the strings with a pipe symbol (the regex 'OR').
# your Keys could contain characters that have special meaning in regex, so we need to escape those
$regexLookup = '({0})' -f (($lookupTable.Keys | ForEach-Object { [regex]::Escape($_) }) -join '|')
# create a StreamWriter object to write the lines to the new output file
# Note: use an ABSOLUTE full file path for this
$streamWriter = [System.IO.StreamWriter]::new($outputfile, $true) # $true for Append
switch -Regex -File $inputfile {
$regexLookup {
# do the replacement using the value in the lookup table.
# because in one line there may be multiple matches to replace
# get a System.Text.RegularExpressions.Match object to loop through all matches
$line = $_
$match = [regex]::Match($line, $regexLookup)
while ($match.Success) {
# because we escaped the keys, to find the correct entry we now need to unescape
$line = $line -replace $match.Value, $lookupTable[[regex]::Unescape($match.Value)]
$match = $match.NextMatch()
}
$streamWriter.WriteLine($line)
}
default { $streamWriter.WriteLine($_) } # write unchanged
}
# dispose of the StreamWriter object
$streamWriter.Dispose()

How to pull a sentence from a log file on multiple remote computers

I am needing to pull a specific sentence from a log file on multiple remote computers. I have all of the computer names already but I do not know how to go about pulling contents of a file from them and copying all of it to a file so that I can reference the sentence from each computer with its machine name. Basically each machine has a specific number unique to itself that we need.
Before explaining, I assume powershell is the tool to use for this.
There are about 1800 machines and I have a variable for all of those. Then I assume I have to make a loop of some kind that runs on each of those machines.
the loop would pull the text from that file that I need and save it all to a file. I am basically pretty new in my Net Admin position with not a lot of PowerShell experience and I wondered if anyone could help.
$computers = ***list of computers***
$computers | ForEachObject{
Add-Content -Path C:\Users\Public\Activant\Eagle\3log.log -Value "Terminal information for ***need the info that is here***"
}
Get-Content -Path .\TERMINAL NUMBERS.txt
this seems to do what you want. [grin] it builds a scriptblock that does the work, hands that off to Invoke-Command with a list of systems to run it on, gathers the results, creates a list of $Non-Responders, removes unwanted properties added by the I-C cmdlet, and finally shows the two collections.
#requires -RunAsAdministrator
# fake reading in a text file
# in real life, use Get-Content
$ComputerNameList = #'
LocalHost
10.0.0.1
127.0.0.1
BetterNotBeThere
'# -split [System.Environment]::NewLine
$IC_ScriptBlock = {
$TargetFileName = 'C:\Temp\Grouping-Strings-List_2019-07-31.log'
# the " \b\w+\b \b\w+\b " is two words delimited by spaces
# so this will find any line that has two words between the listed phrases
$LinePattern = '^Acid Drum \b\w+\b \b\w+\b Psychedelic$'
# the next line is a no-match patern for testing
#$LinePattern = '^Acid Drum \b\w+\b$'
$Line = (Get-Content -LiteralPath $TargetFileName |
Select-String -Pattern $LinePattern).Line
if ([string]::IsNullOrEmpty($Line))
{
$Line = '__Not Found__'
}
[PSCustomObject]#{
ComputerName = $env:COMPUTERNAME
LineText = $Line
}
}
$IC_Params = #{
ComputerName = $ComputerNameList
ScriptBlock = $IC_ScriptBlock
# comment out the next line to see any errors in the I-C call
ErrorAction = 'SilentlyContinue'
}
$Responders = Invoke-Command #IC_Params
$Non_Responders = $ComputerNameList.Where({$_ -notin $Responders.PSComputerName})
# the next line removes unwated properties added by "Invoke-Command"
$Responders = $Responders |
Select-Object -Property * -ExcludeProperty PSComputerName, PSShowComputerName, RunspaceId
$Responders
'=' * 40
$Non_Responders
output ...
ComputerName LineText
------------ --------
[MySysName] Acid Drum Instrumental Live Psychedelic
[MySysName] Acid Drum Instrumental Live Psychedelic
========================================
10.0.0.1
BetterNotBeThere
if needed, you can create a single collection from the two above fairly directly. [grin]
I think what you are trying to do is to READ the line from a file all computers in your list should have, located at C:\Users\Public\Activant\Eagle\3log.log
In that case, something like below should work:
# use UNC naming for the remote file path
$inputFile = 'C$\Users\Public\Activant\Eagle\3log.log' # I'm guessing this is the file you want to read
$outputFile = 'C:\TERMINAL NUMBERS.txt'
$computers = ***list of computers*** # the array of computer names
$result = $computers | ForEach-Object {
# test if the computer is online
if (Test-Connection -ComputerName $_ -Count 1 -Quiet) {
# create the full UNC path by prepending the common file path with the computer name
$file = '\\{0}\{1}' -f $_, $inputFile
# test if the file can be found or not
if (Test-Path -LiteralPath $file -PathType Leaf) {
# using non regex string search
$line = (Select-String -LiteralPath $file -Pattern "whatever you're looking for" -SimpleMatch).Line
if (!$line) {
# the file is there, but the pattern was not found
$line = "Pattern not found"
}
}
else {
$line = "File '$inputFile' not found."
}
}
else {
$line = 'Computer is Off-Line'
}
# Finally, add this info to your text file
Add-Content -Path $outputFile -Value "$_ --> $line"
# And emit an object for the $result collection. This will display nicely on screen,
# but also allow for the creation of a CSV file which might be better as output.
[PsCustomObject]#{
'Computer' = $_
'LogInfo' = $line
}
}
Afterwards you can read the output text file or (better I think) use the $result collection for output:
On screen:
$result | Format-Table -AutoSize
To CSV file
$result | Export-Csv -Path 'C:\TERMINAL NUMBERS.csv' -NoTypeInformation -Force

Delete lines from multiple textfiles in PowerShell

I am trying to delete lines with a defined content from multiple textfiles.
It works in the core, but it will rewrite every file even if no changes are made, which is not cool if you are just modifying 50 out of about 3000 logonscripts.
I even made a if statement but it seems like it doesn't work.
Alright this is what I already have:
#Here $varFind will be escaped from potential RegEx triggers.
$varFindEscaped = [regex]::Escape($varFind)
#Here the deletion happens.
foreach ($file in Get-ChildItem $varPath*$varEnding) {
$contentBefore = Get-Content $file
$contentAfter = Get-Content $file | Where-Object {$_ -notmatch $varFindEscaped}
if ($contentBefore -ne $contentAfter) {Set-Content $file $contentAfter}
}
What the variables mean:
$varPath is the path in which the logonscripts are.
$varEnding is the file ending of the files to modify.
$varFind is the string that triggers the deletion of the line.
Any help would be highly appreciated.
Greetings
Löwä Cent
You have to read the file regardless but some improvement on your change condition could help.
#Here the deletion happens.
foreach ($file in Get-ChildItem $varPath*$varEnding) {
$data = (Get-Content $file)
If($data -match $varFindEscaped){
$data | Where-Object {$_ -notmatch $varFindEscaped} | Set-Content $file
}
}
Read the file into $data. Check to see if the pattern $varFindEscaped is present in the file. If it is than filter out those matching the same pattern. Else we move onto the next file.

Powershell 3: Remove last line of text file

I am using the following script to iterate through a list of files in a folder, then it will regex search for a string containing the 'T|0-9' which is the trailer record and will be present at the end of each text file.
$path = "D:\Test\"
$filter = "*.txt"
$files = Get-ChildItem -path $path -filter $filter
foreach ($item in $files)
{
$search = Get-content $path$item
($search)| ForEach-Object { $_ -replace 'T\|[0-9]*', '' } | Set-Content $path$item
}
This script works fine, however, it may take a long time to go through large file, I therefore used the '-tail 5' parameter so that it will start searching from the last 5 lines, the problem is that it is deleting everything and only leaving the last lines in the feed.
Is there any other way to acomplish this?
I tried another sample code I found but it doesnt really work, can someone guide me please
$stream = [IO.File]::OpenWrite($path$item)
$stream.SetLength($stream.Length - 2)
$stream.Close()
$stream.Dispose()
Since Get-Content returns an array, you can access the last item (last line) using [-1]:
foreach ($item in $files)
{
$search = Get-content $item.FullName
$search[-1] = $search[-1] -replace 'T\|[0-9]*', ''
$search | Set-Content $item.FullName
}

Powershell: Move Items Based on Destination from Hashtable

I'm attempting to write a PowerShell script to move files from one directory to another based on a few conditions. For example:
An example of a file name: testingcenter123456-testtype-222-412014.pdf.
The script should look for "testingcenter123456" before the first dash ("-") and then refer to a hash table for a matching key. All the files follow the format shown above.
Once its finds that key, it should use that key's corresponding value (example: "c:\temp\destination\customer7890") as the destination file path and copy the file there.
I looked around StackOverflow and found a few Q&As that seemed to answer parts of similar questions but the fact that I'm very new to this has led to the script I pieced together not working at all.
Here's what I have so far:
$hashTable = ConvertFrom-StringData ([IO.File]::ReadAllText("c:\temp\filepaths.txt"))
$directory = "c:\temp\source"
Get-ChildItem $directory |
where {!($_.PsIsContainer)} |
Foreach-Object {
Foreach ($key in $hashTable.GetEnumerator()){
if ($_.Name.Substring(0,$_.Name.IndexOf("-")) -eq $key.Name){
Copy-Item -Path $_.fullname -Destination $key.Value
}
}
}
If anyone can help me get un-stuck and hopefully learn a little something about PowerShell in the process, I'd appreciate it.
Honestly, I'm not seeing why this shouldn't work. It would be helpful if you told us which line was generating an error.
Foreach ($key in $hashTable.GetEnumerator()) {
if ($_.Name.Substring(0,$_.Name.IndexOf("-")) -eq $key.Name) {
Copy-Item -Path $_.fullname -Destination $key.Value
}
}
That said, you're missing the point of using hashtable by looping through its entries, manually matching on key. With a hashtable, you don't need to loop e.g.
$hashTable = ConvertFrom-StringData ([IO.File]::ReadAllText("c:\temp\filepaths.txt"))
Get-ChildItem c:\temp\source |
Where {!($_.PsIsContainer)} |
Foreach-Object {
$key = $_.Name.Substring(0,$_.Name.IndexOf("-"))
$val = $hashtable.$key
if ($val) {
$_ | Copy-Item -Dest $val -WhatIf
}
else {
Write-Warning "No entry for $key"
}
}