pipe foreach loop CSV PowerShell - powershell

I've written a script but cannot get it to export to CSV or output in any way.
PowerShell does not like foreach loops and exporting.
For each "folderpath/filename" in the .txt file it checks to see if the file is still there and outputs true or false + folderpath/file name.
Script works fine, just cannot get the thing to export to CSV.
Any ideas what I'm doing wrong?
foreach ($WantFile in Get-Content "C:\scripts\folderpaths.txt") {
$FileExists = Test-Path $WantFile
if ($FileExists -eq $True) {
Write-Output $wantfile "True"
} else {
Write-Output $wantfile "False"
}
} | Export-Csv C:\scripts\output.csv -noType

Change your code to this:
Get-Content 'C:\scripts\folderpaths.txt' | % {
if (Test-Path -LiteralPath $_) {
Write-Output $_ "True"
} else {
Write-Output $_ "False"
}
} | Export-Csv 'C:\scripts\output.csv' -NoType
I doubt that the resulting file will contain what you expect, though. Export-Csv exports the properties of objects. The output you generate are string objects (2 with each Write-Output statement, actually), and their only property is Length, so your result will be one column with the lengths of the strings you echo.
To create a CSV with 2 columns, one for path and the other for existence of the path you need to create objects with the desired properties, e.g. like this:
Get-Content 'C:\scripts\folderpaths.txt' `
| select #{n='Path';e={$_}}, #{n='Exists';e={Test-Path -LiteralPath $_}} `
| Export-Csv 'C:\scripts\output.csv' -NoType

With regard to the original question (exporting the output of a foreach loop to CSV), you can make that output to the pipeline by wrapping it in a subexpression, but that's not going to solve the other problems in your script with regard to what it is you're trying to export:
$(ForEach ($WantFile in Get-Content "C:\scripts\folderpaths.txt"){
$FileExists = Test-Path $WantFile
If ($FileExists -eq $True) {Write-Output $wantfile "True"}
Else {Write-Output $wantfile "False"}
})| export-csv C:\scripts\output.csv -noType

I got the same problem and i got it worked by doing as follow.
$forloop = foreach ( $i in $computers)
{
$i
$out = .\psexec.exe \\$i C:\hp\hpsmh\bin\smhlogreader.exe --version
$out
}
$forloop | Out-file C:\scripts\output.txt -Append

Related

Memory exception while filtering large CSV files

getting memory exception while running this code. Is there a way to filter one file at a time and write output and append after processing each file. Seems the below code loads everything to memory.
$inputFolder = "C:\Change\2019\October"
$outputFile = "C:\Change\2019\output.csv"
Get-ChildItem $inputFolder -File -Filter '*.csv' |
ForEach-Object { Import-Csv $_.FullName } |
Where-Object { $_.machine_type -eq 'workstations' } |
Export-Csv $outputFile -NoType
May be can you export and filter your files one by one and append result into your output file like this :
$inputFolder = "C:\Change\2019\October"
$outputFile = "C:\Change\2019\output.csv"
Remove-Item $outputFile -Force -ErrorAction SilentlyContinue
Get-ChildItem $inputFolder -Filter "*.csv" -file | %{import-csv $_.FullName | where machine_type -eq 'workstations' | export-csv $outputFile -Append -notype }
Note: The reason for not using Get-ChildItem ... | Import-Csv ... - i.e., for not directly piping Get-ChildItem to Import-Csv and instead having to call Import-Csv from the script block ({ ... } of an auxiliary ForEach-Object call, is a bug in Windows PowerShell that has since been fixed in PowerShell Core - see the bottom section for a more concise workaround.
However, even output from ForEach-Object script blocks should stream to the remaining pipeline commands, so you shouldn't run out of memory - after all, a salient feature of the PowerShell pipeline is object-by-object processing, which keeps memory use constant, irrespective of the size of the (streaming) input collection.
You've since confirmed that avoiding the aux. ForEach-Object call does not solve the problem, so we still don't know what causes your out-of-memory exception.
Update:
This GitHub issue contains clues as to the reason for excessive memory use, especially with many properties that contain small amounts of data.
This GitHub feature request proposes using strongly typed output objects to help the issue.
The following workaround, which uses the switch statement to process the files as text files, may help:
$header = ''
Get-ChildItem $inputFolder -Filter *.csv | ForEach-Object {
$i = 0
switch -Wildcard -File $_.FullName {
'*workstations*' {
# NOTE: If no other columns contain the word `workstations`, you can
# simplify and speed up the command by omitting the `ConvertFrom-Csv` call
# (you can make the wildcard matching more robust with something
# like '*,workstations,*')
if ((ConvertFrom-Csv "$header`n$_").machine_type -ne 'workstations') { continue }
$_ # row whose 'machine_type' column value equals 'workstations'
}
default {
if ($i++ -eq 0) {
if ($header) { continue } # header already written
else { $header = $_; $_ } # header row of 1st file
}
}
}
} | Set-Content $outputFile
Here's a workaround for the bug of not being able to pipe Get-ChildItem output directly to Import-Csv, by passing it as an argument instead:
Import-Csv -LiteralPath (Get-ChildItem $inputFolder -File -Filter *.csv) |
Where-Object { $_.machine_type -eq 'workstations' } |
Export-Csv $outputFile -NoType
Note that in PowerShell Core you could more naturally write:
Get-ChildItem $inputFolder -File -Filter *.csv | Import-Csv |
Where-Object { $_.machine_type -eq 'workstations' } |
Export-Csv $outputFile -NoType
Solution 2 :
$inputFolder = "C:\Change\2019\October"
$outputFile = "C:\Change\2019\output.csv"
$encoding = [System.Text.Encoding]::UTF8 # modify encoding if necessary
$Delimiter=','
#find header for your files => i take first row of first file with data
$Header = Get-ChildItem -Path $inputFolder -Filter *.csv | Where length -gt 0 | select -First 1 | Get-Content -TotalCount 1
#if not header founded then not file with sise >0 => we quit
if(! $Header) {return}
#create array for header
$HeaderArray=$Header -split $Delimiter -replace '"', ''
#open output file
$w = New-Object System.IO.StreamWriter($outputfile, $true, $encoding)
#write header founded
$w.WriteLine($Header)
#loop on file csv
Get-ChildItem $inputFolder -File -Filter "*.csv" | %{
#open file for read
$r = New-Object System.IO.StreamReader($_.fullname, $encoding)
$skiprow = $true
while ($line = $r.ReadLine())
{
#exclude header
if ($skiprow)
{
$skiprow = $false
continue
}
#Get objet for current row with header founded
$Object=$line | ConvertFrom-Csv -Header $HeaderArray -Delimiter $Delimiter
#write in output file for your condition asked
if ($Object.machine_type -eq 'workstations') { $w.WriteLine($line) }
}
$r.Close()
$r.Dispose()
}
$w.close()
$w.Dispose()
You have to read and write to the .csv files one row at a time, using StreamReader and StreamWriter:
$filepath = "C:\Change\2019\October"
$outputfile = "C:\Change\2019\output.csv"
$encoding = [System.Text.Encoding]::UTF8
$files = Get-ChildItem -Path $filePath -Filter *.csv |
Where-Object { $_.machine_type -eq 'workstations' }
$w = New-Object System.IO.StreamWriter($outputfile, $true, $encoding)
$skiprow = $false
foreach ($file in $files)
{
$r = New-Object System.IO.StreamReader($file.fullname, $encoding)
while (($line = $r.ReadLine()) -ne $null)
{
if (!$skiprow)
{
$w.WriteLine($line)
}
$skiprow = $false
}
$r.Close()
$r.Dispose()
$skiprow = $true
}
$w.close()
$w.Dispose()
get-content *.csv | add-content combined.csv
Make sure combined.csv doesn't exist when you run this, or it's going to go full Ouroboros.

How to check duplicate multiple file using powershell?

I want to check duplicate file.If the condition of the file like this, it means duplicate. The same name but different extension.
AAA18WWQ6BT602.PRO
AAA18WWQ6BT602.XML
I can figure out this case with my script. But I have problem if I have this more than 1 .XML file like this
AAA18WWQ6BT602.PRO
AAA18WWQ6BT602.XML
AAA18WWQ6BT601.XML
AAA18WWQ6BT604.XML
This case, it will not detect that file AAA18WWQ6BT602.PRO and AAA18WWQ6BT602.XML duplicated.
Anyone can help me please.
Thanks
$duplicate = #()
#(Get-ChildItem "$Flag_Path\*.xml") | ForEach-Object { $duplicate += $_.basename }
if(Test-Path -Path "$Flag_Path\*$duplicate*" -Exclude *.xml)
{
Get-ChildItem -Path "$Flag_Path\*$duplicate*" -Include *.xml | Out-File $Flag_Path\Flag_Duplicate
Write-Host "Flag duplicated, continue for Error_Monitoring"
pause
Error_Monitoring
}
else{
Write-Host "Flag does not duplicate, continue the process"
}
The -Include parameter only works if the path on Get-ChildItem ends in \* OR if the -Recurse switch is used.
The following should do what you want:
$flagFolder = 'D:\*'
$dupeReport = 'D:\Flag_Duplicate.txt'
$duplicates = Get-ChildItem -Path $flagFolder -File -Include '*.xml', '*.pro' |
Group-Object -Property BaseName | Where-Object { $_.Count -gt 1 }
if ($duplicates) {
# output the duplicate XML to Flag_Duplicate.txt
$duplicates.Group | Where-Object {$_.Extension -eq '.xml' } | ForEach-Object {
$_.FullName | Out-File -FilePath $dupeReport -Append
}
# do the rest of your code
Write-Host "Flag duplicated, continue for Error_Monitoring"
Error_Monitoring
}
else {
Write-Host "Flag does not duplicate, continue the process"
}
Your script does not iterate correctly. You need to have an iteration to check. The Test-Path logic looks mixed up to me. I tried to keep as much of your code as possible.
This script checks for a any xml basename filename against any suffix duplicate (not only pro):
$Flag_Path = "C:\dir_to_be_checked"
$xmlFilesArray = #()
$allFilesExceptXml = #() # all files excluding xml files
# Get all the xml files
Get-ChildItem -Path $Flag_Path -Include "*.xml" | ForEach-Object { $xmlFilesArray += $_.basename }
# Get all files from the directory the xml files
Get-ChildItem -Path $Flag_Path -Exclude "*.xml" | ForEach-Object { $allFilesExceptXml += $_.basename }
# Iterate over list of files names without suffix
ForEach ($xmlFile in $xmlFilesArray) {
ForEach ($fileToCheck in $allFilesExceptXml) {
If ($xmlFile -eq $fileToCheck) {
# logging the duplicate file (specifying utf8 or the output would be UTF-16)
Write-Output "$Flag_Path\$xmlFile.xml" | Out-File -Append -Encoding utf8 $Flag_Path\Flag_Duplicate
Write-Host "Flag duplicated, continue with duplicate search"
# pause
Write-Host "Press any key to continue ..."
$x = $host.UI.RawUI.ReadKey("NoEcho,IncludeKeyDown")
Error_Monitoring
} Else {
Write-Host "Flag is not duplicated. Continue with the search."
}
}
}

Powershell check whether a file exists

On Windows 7 x64, Powershell 3.0:
I have a directory where I download files to, this is done with a .csv file.
A sample of the files is:
008KS04.zip
008KS05.zip
008KS06.zip
008KS07.zip
008KS08.zip
008KS09.zip
I have a Csv, op wich this is a sample:
Documentnummer,Versie,Status,Geocode,IDS datum,Documentgroep (Omsch),Documenttype (Wrd),Tekeningnummer
550164,I,AB,008,26.03.2016,Themalagen OI,KS,008KS09.zip
550163,D,AB,008,19.07.2013,Themalagen OI,KS,008KS08.zip
550162,D,AB,008,19.07.2013,Themalagen OI,KS,008KS07.zip
550161,C,AB,008,19.07.2013,Themalagen OI,KS,008KS06.zip
550160,D,AB,008,19.07.2013,Themalagen OI,KS,008KS05.zip
550159,E,AB,008,15.08.2014,Themalagen OI,KS,008KS04.zip
So, lets say the files are in c:\test\
and the csv in C:\drawings.csv
I want the files that are missing in a csv, lets say C:\missing.csv
At first I tried with one drawing in a variable:
$drawnr = '008KS09.zip'
$destcsv = "C:\missing.csv"
Set-Location C:\test
If(Test-Path -Path ${drawnr}) {
Write-Output "${drawnr}'.zip' exists"
} Else {
Write-Output "${drawnr}'.zip' doesnt exist"
${drawnr} | Export-Csv $destcsv -Append -NoTypeInformation
}
The output when having an existing and a non existsing drawing in the variable
This works perfect, see image but using the CSV as input, I can't get it to work.
This is what I have so far:
$sourcecsv = 'C:\drawings.csv'
$destcsv = 'C:\missing.csv'
Set-Location 'C:\test\'
Import-Csv $sourcecsv | Select-object -Property Tekeningnummer | ForEach {
Write-Output "${_} is the current object" #For Testing purpuse
If (Test-Path -Path ${_}) {
Write-Output "${_} does exist"
} Else {
Write-Output "${_} doesnt exist"
${_} | Export-Csv $destcsv -Append -NoTypeInformation
}
}
This tells me that none of the files exist, while all do exist.
See the picture to see the output of PowerShell and the content written to the missing.csv
The picture of the output where all files are reported as non existend
I'm not sure I get your intent, but it looks like this might work.
$sourcecsv = 'C:\drawings.csv'
$destcsv = 'C:\missing.csv'
Set-Location 'C:\test\'
Import-Csv $sourcecsv | ForEach {
Write-Output "${_} is the current object" #For Testing purpuse
$TK = $_.Tekeningnummer
If (Test-Path -Path $TK) {
Write-Output "$TK does exist"
} Else {
Write-Output "$TK doesnt exist"
$_ | Export-Csv $destcsv -Append -NoTypeInformation
}
}
not sure if ${_} is a new syntax I'm not aware of but I can't get it to work on my PS4 host so I'm changing that to the normal $_ in my answer. beyond that you just need to do an ExpandProperty as #DaveSexton suggests.
Import-Csv $sourcecsv | Select-object -ExpandProperty Tekeningnummer | ForEach {
Write-Output "$($_) is the current object" #For Testing purpuse
If (Test-Path -Path $($_) {
Write-Output "$($_) does exist"
} Else {
Write-Output "$($_) doesnt exist"
$_ | Export-Csv $destcsv -Append -NoTypeInformation
}
}
just do it ;)
$sourcecsv = 'C:\drawings.csv'
$destcsv = 'C:\missing.csv'
$dirtest='C:\test\'
Import-Csv $sourcecsv | where {-not (Test-Path "$dirtest$($_.Tekeningnummer)") } | export-csv $destcsv -NoTypeInformation

Comparing csv files with -like in Powershell

I have two csv files, each that contain a PATH column. For example:
CSV1.csv
PATH,Data,NF
\\server1\folderA,1,1
\\server1\folderB,1,1
\\server2\folderA,1,1
\\server2\folderB,1,1
CSV2.csv
PATH,User,Access,Size
\\server1\folderA\file1,don,1
\\server1\folderA\file2,don,1
\\server1\folderA\file3,sue,1
\\server2\folderB\file1,don,1
What I'm attempting to do is create a script that will result in separate csv exports based on the paths in CSV1 such that the new files contain file values from CSV2 that match. For example, from the above, I'd end up with 2 results:
result1.csv
\\server1\folderA\file1,don,1
\\server1\folderA\file2,don,1
\\server1\folderA\file3,sue,1
result2.csv
\\server2\folderB\file1,don,1
Previously I've used a script lime this when the two values are exact:
$reportfile = import-csv $apireportoutputfile -delimiter ';' -encoding unicode
$masterlist = import-csv $pathlistfile
foreach ($record in $masterlist)
{
$path=$record.Path
$filename = $path -replace '\\','_'
$filename = '.\Working\sharefiles\' + $filename + '.csv'
$reportfile | where-object {$_.path -eq $path} | select FilePath,UserName,LastAccessDate,LogicalSize | export-csv -path $filename
write-host " Creating files list for $path" -foregroundcolor red -backgroundcolor white
}
however since the two path values are not the same, it returns nothing. I found a -like operator but am not sure how to use it in this code to get the results I want. where-object is a filter while -like ends up returning a true/false. Am I on the right track? Any ideas for a solution?
Something like this, maybe?
$ht = #{}
Import-Csv csv1.csv |
foreach { $ht[$_.path] = New-Object collections.arraylist }
Import-Csv csv2.csv |
foreach {
$path = $_.path | Split-Path -Parent
$ht[$path].Add($_) > $null
}
$i=1
$ht.Values |
foreach { if ($_.count)
{
$_ | Export-Csv "result$i.csv" -NoTypeInformation
$i++
}
}
My suggestion:
$1=ipcsv .\csv1.CSV
$2=ipcsv .\csv2.CSV
$equal = diff ($2|select #{n='PATH';e={Split-Path $_.PATH}}) $1 -Property PATH -IncludeEqual -ExcludeDifferent -PassThru
0..(-1 + $equal.Count) | %{%{$i = $_}{
$2 | ?{ (Split-Path $_.PATH) -eq $equal[$i].PATH } | epcsv ".\Result$i.CSV"
}}

Delete Files Older Than X Days: Presentation of Data/Logging Files

I'm new to Powershell, I'm creating a code to delete a file/s if more than "x" days.
I'm almost done. Need your help in representing my date (table) and should not produce a log file if no files will be delete.
Here's my code:
$max_days = "-30"
$curr_date = Get-Date
$del_date = $curr_date.AddDays($max_days)
$Path = "C:\Desktop\Code"
$DateTime = Get-Date -Format "D=yyyy-MM-dd_T=HH-mm-ss"
$itemsearch = Get-ChildItem C:\Test -Recurse | Where-Object { $_.LastWriteTime -lt $del_date}
Foreach ($item in $itemsearch)
{
Write "File:", $item.Name "Modified:", $item.LastWriteTime "Path:", $item.FullName "Date Deleted:" $del_date | Out-File "C:\Desktop\Code\Deleted\SFTP_DeleteFiles_WORKSPACE_$DateTime.txt" -append
$item | Remove-Item
}
Can anyone please help me? It's already working by the way.
Just need to present the data in table form and don't create a log file if there's nothing to delete.
Update:
Already solved the condition statement by doing:
if($itemsearch)
{
Foreach ($item in $itemsearch)
{
Write "File:", $item.Name "Modified:", $item.LastWriteTime "Path:", $item.FullName "Date Deleted:" $del_date | Out-File "C:\Desktop\Code\Deleted\SFTP_DeleteFiles_WORKSPACE_$DateTime.txt" -append
$item | Remove-Item
}
}
else
{
Write "No files will be deleted."
}
Thanks!
What I want to display it in Excel/Text file is like this one:
http://i59.tinypic.com/30wv33d.jpg
Anyone?
It returns me with this one:
IsReadOnly;"IsFixedSize";"IsSynchronized";"Keys";"Values";"SyncRoot";"Count"
False;"False";"False";"System.Collections.Hashtable+KeyCollection";"System.Collections.Hashtable+ValueCollection";"System.Object";"4"
False;"False";"False";"System.Collections.Hashtable+KeyCollection";"System.Collections.Hashtable+ValueCollection";"System.Object";"4"
False;"False";"False";"System.Collections.Hashtable+KeyCollection";"System.Collections.Hashtable+ValueCollection";"System.Object";"4"
False;"False";"False";"System.Collections.Hashtable+KeyCollection";"System.Collections.Hashtable+ValueCollection";"System.Object";"4"
In Excel. Do you have any idea? I have to search it though.
To introduce tabular logging I would use a CSV file as output by replacing your foreach block by this code:
$results = #()
foreach ($item in $itemsearch)
{
$success = $true
try
{
$item | Remove-Item
}
catch
{
$success = $false
}
if( $success -eq $true )
{
Write-Host $item.FullName 'successfully deleted.'
$results += [PSCustomObject]#{'File'=$item.Name;'Modified'=$item.LastWriteTime;'Path'=$item.FullName;'Date Deleted'=$del_date;'State'='SUCCESS'}
}
else
{
Write-Host 'Error deleting' $item.FullName
$results += [PSCustomObject]#{'File'=$item.Name;'Modified'=$item.LastWriteTime;'Path'=$item.FullName;'Date Deleted'=$del_date;'State'='ERROR'}
}
}
$results | Export-Csv -Path "C:\Desktop\Code\Deleted\SFTP_DeleteFiles_WORKSPACE_$DateTime.csv" -Encoding UTF8 -Delimiter ';' -NoTypeInformation
First an empty array is created ($results).
The try/catch block is here to detect if the deletion succeeded or not, then the appropriate line is added to $results.
At the end the $results array is exported to CSV with ';' separator so you can open it right away with Excel.