Powershell issues out-file - powershell

I am trying to search a text file for a certain text then output that text to a different text file only if it matches. I'm having troubles creating this. I attached the code I have, but the issue is no matter what it is creating a results.txt file and the file is blank. I only want the results.txt file to be created if the multiplatform_201604110718.txt has CCTK STATUS CODE : SUCCESS inside it.
$Path = "C:\multiplatform_201604110718.txt"
$Text = "CCTK STATUS CODE : SUCCESS"
$PathArray = #()
$Results = "C:\results.txt"
# This code snippet gets all the files in $Path that end in “.txt”.
Get-ChildItem $Path -Filter “*.txt” |
Where-Object { $_.Attributes -ne “Directory”} |
ForEach-Object {
If (Get-Content $_.FullName | Select-String -Pattern $Text) {
$PathArray += $_.FullName
$PathArray += $_.FullName
}
}
Write-Host “Contents of ArrayPath:”
$PathArray | ForEach-Object {$_}
$PathArray | % {$_} | Out-File "C:\Resuts.txt"

If you really only need to check one file you can do it with much less code:
$Text = "CCTK STATUS CODE : SUCCESS"
$p = "C:\Users\kzbx\Desktop\test.txt"
if($ln = Select-String -pattern $text -path $p){
$ln.Line | Out-File C:\result.txt
}
This writes the line in which the text occurs to your result file (if i understood you correctly that is what you need, if not please clarify ;))

Related

Memory exception while filtering large CSV files

getting memory exception while running this code. Is there a way to filter one file at a time and write output and append after processing each file. Seems the below code loads everything to memory.
$inputFolder = "C:\Change\2019\October"
$outputFile = "C:\Change\2019\output.csv"
Get-ChildItem $inputFolder -File -Filter '*.csv' |
ForEach-Object { Import-Csv $_.FullName } |
Where-Object { $_.machine_type -eq 'workstations' } |
Export-Csv $outputFile -NoType
May be can you export and filter your files one by one and append result into your output file like this :
$inputFolder = "C:\Change\2019\October"
$outputFile = "C:\Change\2019\output.csv"
Remove-Item $outputFile -Force -ErrorAction SilentlyContinue
Get-ChildItem $inputFolder -Filter "*.csv" -file | %{import-csv $_.FullName | where machine_type -eq 'workstations' | export-csv $outputFile -Append -notype }
Note: The reason for not using Get-ChildItem ... | Import-Csv ... - i.e., for not directly piping Get-ChildItem to Import-Csv and instead having to call Import-Csv from the script block ({ ... } of an auxiliary ForEach-Object call, is a bug in Windows PowerShell that has since been fixed in PowerShell Core - see the bottom section for a more concise workaround.
However, even output from ForEach-Object script blocks should stream to the remaining pipeline commands, so you shouldn't run out of memory - after all, a salient feature of the PowerShell pipeline is object-by-object processing, which keeps memory use constant, irrespective of the size of the (streaming) input collection.
You've since confirmed that avoiding the aux. ForEach-Object call does not solve the problem, so we still don't know what causes your out-of-memory exception.
Update:
This GitHub issue contains clues as to the reason for excessive memory use, especially with many properties that contain small amounts of data.
This GitHub feature request proposes using strongly typed output objects to help the issue.
The following workaround, which uses the switch statement to process the files as text files, may help:
$header = ''
Get-ChildItem $inputFolder -Filter *.csv | ForEach-Object {
$i = 0
switch -Wildcard -File $_.FullName {
'*workstations*' {
# NOTE: If no other columns contain the word `workstations`, you can
# simplify and speed up the command by omitting the `ConvertFrom-Csv` call
# (you can make the wildcard matching more robust with something
# like '*,workstations,*')
if ((ConvertFrom-Csv "$header`n$_").machine_type -ne 'workstations') { continue }
$_ # row whose 'machine_type' column value equals 'workstations'
}
default {
if ($i++ -eq 0) {
if ($header) { continue } # header already written
else { $header = $_; $_ } # header row of 1st file
}
}
}
} | Set-Content $outputFile
Here's a workaround for the bug of not being able to pipe Get-ChildItem output directly to Import-Csv, by passing it as an argument instead:
Import-Csv -LiteralPath (Get-ChildItem $inputFolder -File -Filter *.csv) |
Where-Object { $_.machine_type -eq 'workstations' } |
Export-Csv $outputFile -NoType
Note that in PowerShell Core you could more naturally write:
Get-ChildItem $inputFolder -File -Filter *.csv | Import-Csv |
Where-Object { $_.machine_type -eq 'workstations' } |
Export-Csv $outputFile -NoType
Solution 2 :
$inputFolder = "C:\Change\2019\October"
$outputFile = "C:\Change\2019\output.csv"
$encoding = [System.Text.Encoding]::UTF8 # modify encoding if necessary
$Delimiter=','
#find header for your files => i take first row of first file with data
$Header = Get-ChildItem -Path $inputFolder -Filter *.csv | Where length -gt 0 | select -First 1 | Get-Content -TotalCount 1
#if not header founded then not file with sise >0 => we quit
if(! $Header) {return}
#create array for header
$HeaderArray=$Header -split $Delimiter -replace '"', ''
#open output file
$w = New-Object System.IO.StreamWriter($outputfile, $true, $encoding)
#write header founded
$w.WriteLine($Header)
#loop on file csv
Get-ChildItem $inputFolder -File -Filter "*.csv" | %{
#open file for read
$r = New-Object System.IO.StreamReader($_.fullname, $encoding)
$skiprow = $true
while ($line = $r.ReadLine())
{
#exclude header
if ($skiprow)
{
$skiprow = $false
continue
}
#Get objet for current row with header founded
$Object=$line | ConvertFrom-Csv -Header $HeaderArray -Delimiter $Delimiter
#write in output file for your condition asked
if ($Object.machine_type -eq 'workstations') { $w.WriteLine($line) }
}
$r.Close()
$r.Dispose()
}
$w.close()
$w.Dispose()
You have to read and write to the .csv files one row at a time, using StreamReader and StreamWriter:
$filepath = "C:\Change\2019\October"
$outputfile = "C:\Change\2019\output.csv"
$encoding = [System.Text.Encoding]::UTF8
$files = Get-ChildItem -Path $filePath -Filter *.csv |
Where-Object { $_.machine_type -eq 'workstations' }
$w = New-Object System.IO.StreamWriter($outputfile, $true, $encoding)
$skiprow = $false
foreach ($file in $files)
{
$r = New-Object System.IO.StreamReader($file.fullname, $encoding)
while (($line = $r.ReadLine()) -ne $null)
{
if (!$skiprow)
{
$w.WriteLine($line)
}
$skiprow = $false
}
$r.Close()
$r.Dispose()
$skiprow = $true
}
$w.close()
$w.Dispose()
get-content *.csv | add-content combined.csv
Make sure combined.csv doesn't exist when you run this, or it's going to go full Ouroboros.

Splitting file into smaller files, working script, but need some tweaks

I have a script here that looks for a delimiter in a text file with several reports in it.  The script saves each individual report as it's own text document. The tweaks I'm trying to achieve are:
In the middle of the data of each page there is - SPEC #: RX:<string>.  I want that string to be saved as the filename.
it currently saves from the delimiter down to the next one. This ignores the first report and grabs every one after. I want it to go from the delimiter UP to the next one, but I haven't figured out how to achieve that.
$InPC = "C:\Users\path"
Get-ChildItem -Path $InPC -Filter *.txt | ForEach-Object -Process {
$basename= $_.BaseName
$m = ( ( Get-Content $_.FullName | Where { $_ | Select-String "END OF
REPORT" -Quiet } | Measure-Object | ForEach-Object { $_.Count } ) -ge 2)
$a = 1
if ($m) {
Get-Content $_.FullName | % {
If ($_ -match "END OF REPORT") {
$OutputFile = "$InPC\$basename _$a.txt"
$a++
}
Add-Content $OutputFile $_
}
Remove-Item $_.FullName
}
}
This works, as stated it outputs the file with END OF REPORT on top, the first report in the file gets omitted as it does not have END OF REPORT above it.
Edited code:
$InPC = 'C:\Path' #
ForEach($File in Get-ChildItem -Path $InPC -Filter *.txt){
$RepNum=0
ForEach($Report in (([IO.File]::ReadAllText('C:\Path'$File) -split 'END OF REPORT\r?\n?' -ne '')){
if ($Report -match 'SPEC #: RX:(?<ReportFile>.*?)\.'){
$ReportFile=$Matches.ReportFile
}
$OutputFile = "{0}\{1}_{2}_{3}.txt" -f $InPC,$File.BaseName,$ReportFile,++$RepNum
$Report | Add-Content $OutputFile
}
# Remove-Item $File.FullName
}
I suggest to use Regular Expressions to
read in the file with -raw parameter and
split the file at the marker END OF REPORT into sections
use the 'SPEC #: RX:(?<ReportFile>.*?)\.' with a named capture group to extract the string
Edit adapted to PowerShell v2
## Q:\Test\2019\09\12\SO_57911471.ps1
$InPC = 'C:\Users\path' # 'Q:\Test\2019\09\12\' #
ForEach($File in Get-ChildItem -Path $InPC -Filter *.txt){
$RepNum=0
ForEach($Report in (((Get-Content $File.FullName) -join "`n") -split 'END OF REPORT\r?\n?' -ne '')){
if ($Report -match 'SPEC #: RX:(?<ReportFile>.*?)\.'){
$ReportFile=$Matches.ReportFile
}
$OutputFile = "{0}\{1}_{2}_{3}.txt" -f $InPC,$File.BaseName,$ReportFile,++$RepNum
$Report | Add-Content $OutputFile
}
# Remove-Item $File.FullName
}
This construed sample text:
## Q:\Test\2019\09\12\SO_57911471.txt
I have a script here that looks for a delimiter in a text file with several reports in it.
In the middle of the data of each page there is -
SPEC #: RX:string1.
I want that string to be saved as the filename.
END OF REPORT
I have a script here that looks for a delimiter in a text file with several reports in it.
In the middle of the data of each page there is -
SPEC #: RX:string2.
I want that string to be saved as the filename.
END OF REPORT
yields:
> Get-ChildItem *string* -name
SO_57911471_string1_1.txt
SO_57911471_string2_2.txt
The added ReportNum is just a precaution in case the string could not be grepped.

Search txt file and return PC name if string exist

I probably should not have added the script I was trying to tweak as it might not have been the best choice to start with. I just knew it had to ability to get a string from a text file and return some result. Here is basically what I need. Wanting to add a variable such as this :$env:COMPUTERNAME
Each PC has a file in C:\someSoftware\version\ and it has a txt file.
Inside this text file, there is a version number formatted like: VER = 1.425.1
I wanting the script to tell me which computers are not version 1.425.1
Any help on this matter is greatly appreciated.
$Path = "C:\Admin\"
$Text = "SomeText"
$PathArray = #()
$Results = "C:\temp\test.txt"
# This code snippet gets all the files in $Path that end in ".txt".
Get-ChildItem $Path -Filter "*.txt" |
Where-Object { $_.Attributes -ne "Directory"} |
ForEach-Object {
If (Get-Content $_.FullName | Select-String -Pattern $Text) {
$PathArray += $_.FullName
$PathArray += $_.FullName
}
}
Write-Host "Contents of ArrayPath:"
$PathArray | % {$_} | Out-File "C:\SearchString\Output.txt"

How can I look for file paths within .conf files?

There are a lot of directories and .conf/.xml files I have to open and search through. I have this right now:
$Path = "D:\Logs"
$Text = "*\log"
$PathArray = #()
$Results = "D:\Logs\Search.txt"
Get-Childitem $Path -Filter "*.conf" -Recurse |
Where-Object {$_.Attributes -ne "Directory"} |
ForEach-Object
{
If (Get-Content $_.FullName | Select-String -Pattern $Text -AllMatches)
{
$PathArray += $_.FullName
$PathArray += $_.FullName
}
}
Write-Host "Contents of ArrayPath:"
$PathArray | ForEach-Object {$_}
$PathArray | % {$_} | Out-File “D:\Logs\Search.txt”
I am trying to create this script, so I can have the out-file report all of the .conf files in a txt file with the correct path the .conf files are located in. I will also do the same with the .xml files by simply replacing the .conf with .xml. As of now, I am getting the .txt file up, but no paths. I know I am missing one or two things but I can not figure out what it is. I will have to manually change the old paths with the news ones I have already created. I would like to run this script to search for all the .conf/.xml files with a *\log or *\logs in them.
There's an issue with your regex not escaping backslash and it apparently not matching what you've shown as the typically content of a .conf file. Plus it can be simplified. Try this - adjusting the $Text regex to actually match the desired text in the .conf files:
$Path = "D:\Logs"
$Text = "\\log"
$Results = "D:\Logs\Search.txt"
$PathArray = #(Get-Childitem $Path -Filter *.conf -Recurse -File |
Where {(Get-Content $_.FullName -Raw) -match $Text})
Write-Host "Contents of ArrayPath:"
$PathArray
$PathArray | Out-File $Results -Encoding ascii
There were a couple of issues. Keith got the biggest one being that Select-String uses regex by default. Again, you had some redundancies like adding to the $pathArray twice and the use of ForEach-Object {$_}.
I wanted to show a solution that still uses select-string but uses some of the switch therein to get the use you want out of it. Main one being -simplematch which treats the pattern literally and not as regex. I saw an underscore leading the log in your sample text so I use that here. If you don't want to or it does not match your data simply remove it.
$Path = "D:\Logs"
$Text = "_log"
$Results = "D:\Logs\Search.txt"
$PathArray = Get-Childitem $Path -Filter "*.conf" -Recurse |
Where-Object {$_.Attributes -ne "Directory"} |
Where-Object{Select-string $_ -Pattern $Text -SimpleMatch -Quiet} |
Select-Object -ExpandProperty FullName
# Show results on screen
$PathArray
# Export results to file
$PathArray | Set-Content $Results

Check log file for access from Tor exit nodes

I am trying to write a script that will download the current Tor exit node list and check a local log file for access from Tor.
The code below outputs matches from Tor IPs in the log file to the console, but I would like to output the Select-String output in the foreach loop to a file (in addition to writing it to the console).
Once I have the results output to a file, I'll need to check the results to see if there was a match.
Any tips on how to output the foreach Select-String to a file and how to check for matches would be appreciated.
$source = "https://check.torproject.org/cgi-bin/TorBulkExitList.py?ip=[serverip]&port=[port]"
$dest = "C:\tor.txt"
$wc = New-Object System.Net.WebClient
$wc.DownloadFile($source, $dest)
Get-Content "C:\tor.txt" | where { $_ -notmatch "^#" } | Set-Content "C:\filteredtor.txt"
Remove-Item "C:\tor.txt"
$list = get-content "C:\filteredtor.txt"
ForEach ($_ in $list) {Select-String "C:\logfile.log" -pattern $_}
The Tee-Object cmdlet will save a copy of the input stream to a file while passing it to the output stream. Also, by using DownloadString() instead of DownloadFile() you could streamline your code to a single pipeline and remove the need for temporary files:
$source = 'https://...'
$log = 'C:\logfile.log'
$results = 'C:\results.txt'
$wc = New-Object System.Net.WebClient
($wc.DownloadString($source)) -split "`n" | ? { $_ -notmatch '^(#|\s*$)' } | % {
Select-String $log -pattern $_ | Tee-Object $results -Append
}
if ((Test-Path -LiteralPath $results) -and (Get-Content $results)) {
# do something
}