Powershell script to automate gci - powershell

I'm tying to automate gci in order to work on each row in a config file, where for each row I have as first column the path, and following it a list of files. Something like this:
C:\Users\*\AppData\Roaming\* *.dll
C:\Test file.txt,file2.txt
This means that gci will search for:
*.dll in C:\Users*\AppData\Roaming*
file.txt in C:\Test
file2.txt in C:\Test
In order to do this I'm creating dynamically the where condition in the script below. Here the ps script I'm using
foreach($line in Get-Content .\List.txt) {
try {
$path,$files = $line.split(' ')
$files = $files.split(',')
}
catch {
$path = $line
$files = "*.*"
}
if([string]::IsNullOrEmpty($files)){
$files = "*.*"
}
$filter = $files -join(" -or `$_.Name` -like ")
$filter = "`$_.Name` -like " + $filter
echo "Searching Path: $path, Pattern: $filter" | out-file -append -encoding ASCII -filepath .\result.txt
if ($path.Contains("*"))
{
gci -Path $path -Recurse | Where {$filter} | Select -ExpandProperty FullName | Out-String -Width 2048 | out-file -append -encoding UTF8 -filepath .\result.txt
}
else
{
gci -Path $path | Where {$filter} | Select -ExpandProperty FullName | Out-String -Width 2048 | out-file -append -encoding UTF8 -filepath .\result.txt
}
}
The problem is that the where filter is not considered. All files are returned
First attempt, suggested by
foreach($line in Get-Content .\List.txt) {
try {
$path,$files = $line.split(' ')
$files = $files.split(',')
}
catch {
$path = $line
$files = "*.*"
}
if([string]::IsNullOrEmpty($files)){
$files = "*.*"
}
$filter = $files -join(" -or `$_.Name -like ")
$filter = "`$_.Name -like " + $filter
$gciParams = #{
Path = $Path
Recurse = $Path.Contains('*')
}
"Searching Path: $path, Pattern(s): [$($files -join ',')]" | Add-Content -Path .\result.txt -Encoding ASCII
Get-ChildItem #gciParams | Where $filter | Select -ExpandProperty FullName | Add-Content -Path .\result.txt -Encoding UTF8
}

If you want to create a piece of code and defer execution of it until later, you need a Script Block.
A Script Block literal in PowerShell is just {}, so for constructing script block to filter based on a single comparison, you'd want to define $filter like this:
$filter = {$_.Name -like $filter}
At which point you can pass it directly as an argument to Where-Object:
Get-ChildItem $path |Where-Object $filter
... but since you want to test against multiple wildcard patterns, we'll need to write a slightly different filtering routine:
$filter = {
# Store file name of file we're filtering
$FileName = $_.Name
# Test ALL the patterns in $files and see if at least 1 matches
$files.Where({$FileName -like $_}, 'First').Count -eq 1
}
Since the $filter block now references $files to get the patterns, we can simplify your loop as:
foreach($line in Get-Content .\List.txt) {
try {
$path,$files = $line.split(' ')
$files = $files.split(',')
}
catch {
$path = $line
$files = "*.*"
}
if([string]::IsNullOrEmpty($files)){
$files = "*.*"
}
$gciParams = #{
Path = $Path
Recurse = $Path.Contains('*')
}
"Searching Path: $path, Pattern(s): [$($files -join ',')]" | Add-Content -Path .\result.txt -Encoding ASCII
Get-ChildItem #gciParams | Where $filter | Select -ExpandProperty FullName | Add-Content -Path .\result.txt -Encoding UTF8
}
Note that we no longer need to re-define $filter everytime the loop runs - the condition is based on the value of $files at runtime, so you can define $filter once before entering the loop and then reuse $filter every time.
The "trick" with using #gciParams (which allows us to remove the big if/else block) is known as splatting, but you could achieve the same result with Get-ChildItem -Path:$Path -Recurse:$Path.Contains('*') :)

Related

Write a script to sort words in alphabet order from specific file and put them into 26 text files named A.txt, B.txt, and so on up to Z.txt

I need to sort words alphabetically from a specific file and put them into 26 text files named A.txt, B.txt and so on up to Z.txt.
$Content = Get-Content ".\\1.txt"
$Content = ($Content.Split(" .,:;?!/()\[\]{}-\`\`\`"")|sort)
$linecount = 0
$filenumber = 0
$destPath = "C:\\test"
$destFileSize = 26
$Content |Group {$_.Substring(0,1).ToUpper()} |ForEach-Object {
$path = Join-Path $destPath $_.Name
$\_.Group |Set-Content $path
}
$Content | % {
Add-Content $destPath$filenumber.txt "$\_"
$linecount++
If ($linecount -eq $destFileSize) {
$filenumber++
$linecount = 0
}
}
You could do something like this, but this also could mean some files may not be written if there are no words beginning with a certain letter found in the file:
$destPath = "D:\test"
(Get-Content -Path 'D:\Test\Lorem.txt' -Raw) -split '\W' -ne '' |
Group-Object {$_.Substring(0,1).ToUpperInvariant()} |
Where-Object {$_.Name -cmatch '[A-Z]'} | ForEach-Object {
$_.Group | Sort-Object | Set-Content -Path (Join-Path -Path $destPath -ChildPath ('{0}.txt' -f $_.Name))
}
If you always want exactly 26 files even if some may contain nothing, use this instead
$destPath = "D:\test"
$wordGroups = (Get-Content -Path 'D:\Test\Lorem.txt' -Raw) -split '\W' -ne '' |
Group-Object {$_.Substring(0,1).ToUpperInvariant()}
foreach ($char in ('ABCDEFGHIJKLMNOPQRSTUVWXYZ' -split '(.)' -ne '')) {
$outFile = Join-Path -Path $destPath -ChildPath ('{0}.txt' -f $char)
$group = $wordGroups | Where-Object { $_.Name -eq $char }
if ($group) { $group.Group | Sort-Object | Set-Content -Path $outFile } # output the found words
else { $null | Set-Content -Path $outFile } # or create an empty file
}
The Where-Object {$_.Name -cmatch '[A-Z]'} clause makes it ignore words starting with some other character than A to Z

Memory exception while filtering large CSV files

getting memory exception while running this code. Is there a way to filter one file at a time and write output and append after processing each file. Seems the below code loads everything to memory.
$inputFolder = "C:\Change\2019\October"
$outputFile = "C:\Change\2019\output.csv"
Get-ChildItem $inputFolder -File -Filter '*.csv' |
ForEach-Object { Import-Csv $_.FullName } |
Where-Object { $_.machine_type -eq 'workstations' } |
Export-Csv $outputFile -NoType
May be can you export and filter your files one by one and append result into your output file like this :
$inputFolder = "C:\Change\2019\October"
$outputFile = "C:\Change\2019\output.csv"
Remove-Item $outputFile -Force -ErrorAction SilentlyContinue
Get-ChildItem $inputFolder -Filter "*.csv" -file | %{import-csv $_.FullName | where machine_type -eq 'workstations' | export-csv $outputFile -Append -notype }
Note: The reason for not using Get-ChildItem ... | Import-Csv ... - i.e., for not directly piping Get-ChildItem to Import-Csv and instead having to call Import-Csv from the script block ({ ... } of an auxiliary ForEach-Object call, is a bug in Windows PowerShell that has since been fixed in PowerShell Core - see the bottom section for a more concise workaround.
However, even output from ForEach-Object script blocks should stream to the remaining pipeline commands, so you shouldn't run out of memory - after all, a salient feature of the PowerShell pipeline is object-by-object processing, which keeps memory use constant, irrespective of the size of the (streaming) input collection.
You've since confirmed that avoiding the aux. ForEach-Object call does not solve the problem, so we still don't know what causes your out-of-memory exception.
Update:
This GitHub issue contains clues as to the reason for excessive memory use, especially with many properties that contain small amounts of data.
This GitHub feature request proposes using strongly typed output objects to help the issue.
The following workaround, which uses the switch statement to process the files as text files, may help:
$header = ''
Get-ChildItem $inputFolder -Filter *.csv | ForEach-Object {
$i = 0
switch -Wildcard -File $_.FullName {
'*workstations*' {
# NOTE: If no other columns contain the word `workstations`, you can
# simplify and speed up the command by omitting the `ConvertFrom-Csv` call
# (you can make the wildcard matching more robust with something
# like '*,workstations,*')
if ((ConvertFrom-Csv "$header`n$_").machine_type -ne 'workstations') { continue }
$_ # row whose 'machine_type' column value equals 'workstations'
}
default {
if ($i++ -eq 0) {
if ($header) { continue } # header already written
else { $header = $_; $_ } # header row of 1st file
}
}
}
} | Set-Content $outputFile
Here's a workaround for the bug of not being able to pipe Get-ChildItem output directly to Import-Csv, by passing it as an argument instead:
Import-Csv -LiteralPath (Get-ChildItem $inputFolder -File -Filter *.csv) |
Where-Object { $_.machine_type -eq 'workstations' } |
Export-Csv $outputFile -NoType
Note that in PowerShell Core you could more naturally write:
Get-ChildItem $inputFolder -File -Filter *.csv | Import-Csv |
Where-Object { $_.machine_type -eq 'workstations' } |
Export-Csv $outputFile -NoType
Solution 2 :
$inputFolder = "C:\Change\2019\October"
$outputFile = "C:\Change\2019\output.csv"
$encoding = [System.Text.Encoding]::UTF8 # modify encoding if necessary
$Delimiter=','
#find header for your files => i take first row of first file with data
$Header = Get-ChildItem -Path $inputFolder -Filter *.csv | Where length -gt 0 | select -First 1 | Get-Content -TotalCount 1
#if not header founded then not file with sise >0 => we quit
if(! $Header) {return}
#create array for header
$HeaderArray=$Header -split $Delimiter -replace '"', ''
#open output file
$w = New-Object System.IO.StreamWriter($outputfile, $true, $encoding)
#write header founded
$w.WriteLine($Header)
#loop on file csv
Get-ChildItem $inputFolder -File -Filter "*.csv" | %{
#open file for read
$r = New-Object System.IO.StreamReader($_.fullname, $encoding)
$skiprow = $true
while ($line = $r.ReadLine())
{
#exclude header
if ($skiprow)
{
$skiprow = $false
continue
}
#Get objet for current row with header founded
$Object=$line | ConvertFrom-Csv -Header $HeaderArray -Delimiter $Delimiter
#write in output file for your condition asked
if ($Object.machine_type -eq 'workstations') { $w.WriteLine($line) }
}
$r.Close()
$r.Dispose()
}
$w.close()
$w.Dispose()
You have to read and write to the .csv files one row at a time, using StreamReader and StreamWriter:
$filepath = "C:\Change\2019\October"
$outputfile = "C:\Change\2019\output.csv"
$encoding = [System.Text.Encoding]::UTF8
$files = Get-ChildItem -Path $filePath -Filter *.csv |
Where-Object { $_.machine_type -eq 'workstations' }
$w = New-Object System.IO.StreamWriter($outputfile, $true, $encoding)
$skiprow = $false
foreach ($file in $files)
{
$r = New-Object System.IO.StreamReader($file.fullname, $encoding)
while (($line = $r.ReadLine()) -ne $null)
{
if (!$skiprow)
{
$w.WriteLine($line)
}
$skiprow = $false
}
$r.Close()
$r.Dispose()
$skiprow = $true
}
$w.close()
$w.Dispose()
get-content *.csv | add-content combined.csv
Make sure combined.csv doesn't exist when you run this, or it's going to go full Ouroboros.

Filter lines according to word by powershell

I want to filter lines according to specific word from file in powershell.
For example: the files animal1.txt and animal2.txt. Every file contain lines
dog
cat
dog
dog
bird
Then I want to create two derived files:
animal1_bak.txt that stores lines which contains the word 'dog' from animal1.txt
animal2_bak.txt that stores lines which contains the word 'dog' from animal2.txt
What I found on web is:
Select-String -Path "*.*" -Pattern "dog"
But the instruction to create the derived word is missing.
What can I do?
You can first get-content and use set-content like below
Get-Content -Path E:\KTDocs\Scripts\animal1.txt | where {
$_ -like '*dog*'} |Set-Content e:\animalbak.txt
try Something like this
select-string -Path "c:\temp\animal*.txt" -Pattern "dog" | Group Path | %{
$FileName="{0}_bak.txt" -f $_.Name
$_.Group.Line | select -unique | Out-File $FileName -Append
}
$folderpath = "D:\AnimalFolder" # your folder path here
$Allfiles = Get-ChildItem -Path $folderpath -Recurse -File -Force -ErrorAction SilentlyContinue |where{$_.Name -match ".txt"} |Select-Object -expandproperty FullName
foreach($filepath in $allfiles)
{
$Data = get-content $filepath
foreach($line in $data)
{
if($line -match "dog")
{
$newpath = $filepath.split('.')[0]
$getfullpath = $newpath + "_bak.txt"
$line | out-file $getfullpath -append
}
}
}

Read txt file in folder and fint some text by per sample

I try to read big data log file, in folder C: \ log \ 1 \ i put 2 txt files, i need open-> read all file .txt and find with filter some text like whis: [text]
# Filename: script.ps1
$Files = Get-ChildItem "C:\log\1\" -Filter "*.txt"
foreach ($File in $Files)
{
$StringMatch = $null
$StringMatch = select-string $File -pattern "[Error]"
if ($StringMatch) {out-file -filepath C:\log\outputlog.txt -inputobject $StringMatch}
}
# end of script
not work
Would doing something like a select-string work?
Select-String C:\Scripts\*.txt -pattern "SEARCH STRING HERE" | Format-List
Or if there are multiple files you are wanting to parse maybe use the same select-string but within a loop and output the results.
$Files = Get-ChildItem "C:\log\1\" -Filter "*.txt"
foreach ($File in $Files)
{
$StringMatch = $null
$StringMatch = select-string $File -pattern "SEARCH STRING HERE"
if ($StringMatch) {out-file -filepath c:\outputlog.txt -inputobject $StringMatch}
}
This will print out the file name along with the line number in the file. I hope this is what you are looking for.
Remove-Item -Path C:\log\outlog.txt
$Files = Get-ChildItem "C:\log\1\" -Filter "*.txt"
foreach ($File in $Files)
{
$lineNumber = 0
$content = Get-Content -Path "C:\log\1\$File"
foreach($line in $content)
{
if($line.Contains('[Error]'))
{
Add-Content -Path C:\log\outlog.txt -Value "$File -> $lineNumber"
}
$lineNumber++
}
}
Code below works
It selects strings in txt files in your folder based on -SimpleMatch and then appends it to new.txt file.
Though i do not know how to put two simple matches in one line. Maybe someone does and can post it here
Select-String -Path C:\log\1\*.txt -SimpleMatch "[Error]" -ca | select -exp line | out-file C:\log\1\new.txt -Append
Select-String -Path C:\log\1\*.txt -SimpleMatch "[File]" -ca | select -exp line | out-file C:\log\1\new.txt -Append
Regards
-----edit-----
If you want to you may not append it anywhere just display - simply dont pipe it to out-file
use index then check it :
New-Item C:\log\outputlog.txt
$Files = Get-ChildItem "C:\log\1\" -Include "*.txt"
foreach ($File in $Files)
{
$StringMatch = $null
$StringMatch = Get-Content $File
if($StringMatch.IndexOf("[Error]") -ne -1)
{
Add-Content -Path C:\log\outputlog.txt -Value ($StringMatch+"
-------------------------------------------------------------
")
}
}
# end of script

Comparing csv files with -like in Powershell

I have two csv files, each that contain a PATH column. For example:
CSV1.csv
PATH,Data,NF
\\server1\folderA,1,1
\\server1\folderB,1,1
\\server2\folderA,1,1
\\server2\folderB,1,1
CSV2.csv
PATH,User,Access,Size
\\server1\folderA\file1,don,1
\\server1\folderA\file2,don,1
\\server1\folderA\file3,sue,1
\\server2\folderB\file1,don,1
What I'm attempting to do is create a script that will result in separate csv exports based on the paths in CSV1 such that the new files contain file values from CSV2 that match. For example, from the above, I'd end up with 2 results:
result1.csv
\\server1\folderA\file1,don,1
\\server1\folderA\file2,don,1
\\server1\folderA\file3,sue,1
result2.csv
\\server2\folderB\file1,don,1
Previously I've used a script lime this when the two values are exact:
$reportfile = import-csv $apireportoutputfile -delimiter ';' -encoding unicode
$masterlist = import-csv $pathlistfile
foreach ($record in $masterlist)
{
$path=$record.Path
$filename = $path -replace '\\','_'
$filename = '.\Working\sharefiles\' + $filename + '.csv'
$reportfile | where-object {$_.path -eq $path} | select FilePath,UserName,LastAccessDate,LogicalSize | export-csv -path $filename
write-host " Creating files list for $path" -foregroundcolor red -backgroundcolor white
}
however since the two path values are not the same, it returns nothing. I found a -like operator but am not sure how to use it in this code to get the results I want. where-object is a filter while -like ends up returning a true/false. Am I on the right track? Any ideas for a solution?
Something like this, maybe?
$ht = #{}
Import-Csv csv1.csv |
foreach { $ht[$_.path] = New-Object collections.arraylist }
Import-Csv csv2.csv |
foreach {
$path = $_.path | Split-Path -Parent
$ht[$path].Add($_) > $null
}
$i=1
$ht.Values |
foreach { if ($_.count)
{
$_ | Export-Csv "result$i.csv" -NoTypeInformation
$i++
}
}
My suggestion:
$1=ipcsv .\csv1.CSV
$2=ipcsv .\csv2.CSV
$equal = diff ($2|select #{n='PATH';e={Split-Path $_.PATH}}) $1 -Property PATH -IncludeEqual -ExcludeDifferent -PassThru
0..(-1 + $equal.Count) | %{%{$i = $_}{
$2 | ?{ (Split-Path $_.PATH) -eq $equal[$i].PATH } | epcsv ".\Result$i.CSV"
}}