Use Import-Csv cmdlet with all files in directory - powershell

Using PowerShell, I have created a script that adds headers to a CSV, select unique value within specified columns, and exports it as a new CSV.
That script is:
import-csv -Header (1..39) -Path 'J:\consult evv\splitfile_385.csv' | select '30', '31', '39' -Unique | Export-Csv -Path "C:\Users\CFOSTER\Desktop\stuff\modsplitfile_385.csv" -NoTypeInformation
This script only works for one item: splitfile_385.csv. I'd like to use this for all files in a directory.
I can loop through items in the specified path but am having trouble combining the loop with the script above.
Currently, I have:
$files = Get-ChildItem "J:\consult evv"
for ($i=0; $i -lt $files.Count; $i++) {
$outfile = "C:\Users\CFOSTER\Desktop\Stuff\new" + $files[$i]
Get-Content $files[$i].FullName | Where-Object { !($_ -match 'step4' -or $_ -match 'step9') } | import-csv -Header (1..39) -Path $files[$i].FullName | select '30', '31', '39' -Unique | Set-Content $outfile
}
Using this gives me the error: import-csv : You must specify either the -Path or -LiteralPath parameters, but not both.
How can I combine my script with a loop for all files in a directory?

You can use ForEach-Object and import one at a time.
Get-ChildItem "C:\CSV Files\*.csv" | ForEach-Object {
Import-Csv $_ | Where-Object { ... }
}
If you want to output to a new CSV file, make sure the output filename doesn't match the pattern used with Get-ChildItem or you'll run into trouble.

try somethng like this:
Get-ChildItem "J:\consult evv" -file -Filter "*.csv" | %{
$outfile = "C:\Users\CFOSTER\Desktop\Stuff\new" + $_.Name
Get-Content $_.FullName | select -skip 1 | Where { !($_ -match 'step4' -or $_ -match 'step9') } |
ConvertFrom-Csv -Header (1..39) | select '30', '31', '39' -Unique | Out-File $outfile -Append
}

Related

Powershell Piping Variable to Export-CSV not giving the data I'm looking for

In this script I'm getting a collection of CSV files, performing a replace, storing in an empty array and attempting to export it to CSV.
$CSVFiles = Get-ChildItem "C:\GALIC\Test\Test2\WindowsLists\*.csv" -Exclude M*
$AllJobsList = $CSVFiles | ForEach { (Import-CSV $_ -Delimiter ',' | Select 'Agent', 'Name', 'Folder' | Where-Object {$_.Agent -like "*AGENTGROUP*"})}
$UpdatedGroupsList = #()
$AllJobsList | Export-Csv -Path "C:\GALIC\Test\Test2\WindowsLists\FullJobs-Test.csv" -NoTypeInformation -Force
**$CSVContent = Get-Content "C:\GALIC\Test\Test2\WindowsLists\FullJobs-Test.csv"
foreach($line in $CSVContent)
{
if($line.Contains('|') -and $line.Contains('HOSTG'))
{
#Write-Host $line
$null = $line.Replace('|', '').Replace('HOSTG', '')
#Write-Host $LineReplace
$UpdatedGroupsList += $line
}
}
$UpdatedGroupsList | Export-CSV -Path "C:\GALIC\Test\Test2\WindowsLists\UpdatedFullJobs.csv" -NoTypeInformation -Force**
($CSVContent on down is what's giving me issues.)
After opening the CSV file, the content looks nothing like what I'm expecting. Any ideas/suggestions?
enter image description here

Powershell - Combine CSV files and append a column

I'm trying (badly) to work through combining CSV files into one file and prepending a column that contains the file name. I'm new to PowerShell, so hopefully someone can help here.
I tried initially to do the well documented approach of using Import-Csv / Export-Csv, but I don't see any options to add columns.
Get-ChildItem -Filter *.csv | Select-Object -ExpandProperty FullName | Import-Csv | Export-Csv CombinedFile.txt -UseQuotes Never -NoTypeInformation -Append
Next I'm trying to loop through the files and append the name, which kind of works, but for some reason this stops after the first row is generated. Since it's not a CSV process, I have to use the switch to skip the first title row of each file.
$getFirstLine = $true
Get-ChildItem -Filter *.csv | Where-Object {$_.Name -NotMatch "Combined.csv"} | foreach {
$filePath = $_
$collection = Get-Content $filePath
foreach($lines in $collection) {
$lines = ($_.Basename + ";" + $lines)
}
$linesToWrite = switch($getFirstLine) {
$true {$lines}
$false {$lines | Select -Skip 1}
}
$getFirstLine = $false
Add-Content "Combined.csv" $linesToWrite
}
This is where the -PipelineVariable parameter comes in real handy. You can set a variable to represent the current iteration in the pipeline, so you can do things like this:
Get-ChildItem -Filter *.csv -PipelineVariable File | Where-Object {$_.Name -NotMatch "Combined.csv"} | ForEach-Object { Import-Csv $File.FullName } | Select *,#{l='OriginalFile';e={$File.Name}} | Export-Csv Combined.csv -Notypeinfo
Merging your CSVs into one and adding a column for the file's name can be done as follows, using a calculated property on Select-Object:
Get-ChildItem -Filter *.csv | ForEach-Object {
$fileName = $_.Name
Import-Csv $_.FullName | Select-Object #{
Name = 'FileName'
Expression = { $fileName }
}, *
} | Export-Csv path/to/merged.csv -NoTypeInformation

Run Powershell script on multiple files

I have the following code to add new columns to a csv file. I would like to amend the code to run on multiple csv files within a folder and output it to a different folder.
$source = "C:\input_folder\input.csv"
$destination = "C:\output_folder\output.csv"
(Import-CSV $source |
Select-Object *,#{Name='column1';Expression={'data1'}} |
Select-Object *,#{Name='column2';Expression={'data2'}} |
ConvertTo-csv -NoTypeInformation |
Select-Object -Skip 0) -replace '"' | Set-Content $destination
You could do something like this. You might also want to add a check that the files in the folder are .csv type and not something else.
$source = "C:\input_folder"
$destinationFolder = "C:\output_folder"
$folderContents = Get-ChildItem $source
foreach ($item in $folderContents) {
if (Test-Path -Path $item -PathType Leaf == True) {
(Import-CSV $item |
Select-Object *,#{Name='column1';Expression={'data1'}} |
Select-Object *,#{Name='column2';Expression={'data2'}} |
ConvertTo-csv -NoTypeInformation |
Select-Object -Skip 0) -replace '"' | Set-Content ("$destinationFolder\$item" + "_formatted.csv")
}
}

How to output multiple file extensions stored in a .txt file on Powershell

my purpose is to output the files with the extensions that match the ones I store on a .txt file but I don't know how to make it work. The way I am trying to do now does not generate any output or output the files of the extensions that are not on the text file (Extension.txt) I indicate. How am I supposed to fix this?
The content in my .txt file is:
*.xlsx,*.xlsm,*.xlsb,*.xltx,*.xltm,*.xls,*.xml
My current code is as followed:
$fileHeaders = #('country','cDrive','dDrive')
$extensions = ${C:temp:Extension.txt}
$LocContent = Import-Csv "C:\temp\Location.txt" -Header $fileHeaders
$NumberOfDays = Read-Host 'Within how many days the files created would you like to output?'
$SizeOfFile = Read-Host 'Above what size of the files would you like to output (in kb or mb)?'
$Output = ForEach($Row in $LocContent){
if (($Row.country -ne $null) -and ($Row.cDrive -ne $null) -and ($Row.dDrive -ne $null)){
Get-ChildItem $Row.cDrive,$Row.dDrive -Force -Include -Recurse |
$extensions Where-Object LastWriteTime -gt (Get-Date).AddDays(-$NumberOfDays) |
Where-Object {$_.length/$SizeOfFile -gt 1} |
Select-Object -Property #{N='File Basename';E={$_.BaseName}},
#{N='File Extension';E={$_.Extension}},
#{N='size in MB';E={$_.Length/1024kb}},
Directory,
CreationTime,
LastWriteTime,
#{N="Location";E={$Row.country}}
}
$Output | Format-Table -Auto
$Output | Out-Gridview
$Output | Export-Csv '\NewData.csv' -NoTypeInformation
This both generated the files, outputs the files and makes sure they are in your .txt file.
I am not surprised your code wont work. You were doing it fairly poorly. Look forward to seeing you improve.
$extensions = ((Get-Content C:\temp\Extensions.txt) -join ',') -split ',' -replace '\*',''
Foreach($ext in $extensions){
Get-ChildItem "C:\temp" -Recurse | select Name, FullName, CreationTime, Extension | Where-Object {$_.Extension -like $ext} | export-csv C:\Files.csv -NoTypeInformation -append
}
I just cleaned up your code.
Without knowing the content of Location.txt,
if there are repetitions in the drives, wouldn't Output contain dublettes of the files with different country?
Without your environment untested.
## Q:\Test\2018\07\05\SO_51183354.ps1
$fileHeaders = #('country','cDrive','dDrive')
$Extensions = (Get-Content 'C:\temp\Extension.txt') -replace '\*' -split ','
$LocContent = Import-Csv "C:\temp\Location.txt" -Header $fileHeaders |
Where-Object {($_.country -ne $null) -and
($_.cDrive -ne $null) -and
($_.dDrive -ne $null) }
$NumberOfDays = Read-Host 'Max file age in days?'
$SizeOfFile = Read-Host 'Min file size (in kb or mb)?'
$FileAge = (Get-Date).AddDays(-$NumberOfDays)
$Output = ForEach($Row in $LocContent){
Get-ChildItem $Row.cDrive,$Row.dDrive -Force -Recurse |
Where-Object {($_.Extension -in $Extensions) -and
($_.LastWriteTime -gt $FileAge) -and
($_.Length -gt $SizeOfFile) } |
Select-Object -Property `
#{N='File Basename' ;E={$_.BaseName}},
#{N='File Extension';E={$_.Extension}},
#{N='size in MB' ;E={$_.Length/1MB}},
Directory,
CreationTime,
LastWriteTime,
#{N="Location" ;E={$Row.country}}
}
$Output | Format-Table -Auto
$Output | Out-Gridview
$Output | Export-Csv '\NewData.csv' -NoTypeInformation

PowerShell - searching for existing files generates empty output

I wish to search for specific files listed in searchFiles and pipe their locations to TestFileLocation.CSV. However, my current script only generates an empty CSV. What am I missing?
My TestFindFile.csv is of the form:
Name
123.pdf
321.pdf
aaa.pdf
SNIPPET
$searchFiles = Import-CSV 'C:\Data\SCRIPTS\PS1\TestFindFile.csv' -Header ("Name")
$source = 'C:\Data'
ForEach($File in $searchFiles)
{
Get-ChildItem $source -Filter $File -rec | where {!$_.PSIsContainer} | select-object FullName | export-csv -notypeinformation -delimiter '|' -path c:\data\scripts\ps1\TestFileLocation.csv
}
You were overwriting the CSV for each iteration of the loop.
$searchFiles = Import-CSV 'C:\Data\SCRIPTS\PS1\TestFindFile.csv' -Header ("Name")
$source = 'C:\Data'
$outputPath = 'c:\data\scripts\ps1\TestFileLocation.csv'
$searchFiles | ForEach-Object {
# Silently continue to try to ignore error like
# not being able to read path's which are too long
Get-ChildItem $source -Filter $_ -rec -ErrorAction SilentlyContinue | where {!$_.PSIsContainer} | select-object FullName
} | export-csv -notypeinformation -delimiter '|' -path $outputPath
Example using AlphaFS
A comment asked for an example using AlphaFS because it claims to overcome the long path issue. I'm not going into all the details, but here is how I got it to work.
# download and unzip to c:\alpahfs
# dir C:\AlphaFS\* -Recurse -File | Unblock-File
[System.Reflection.Assembly]::LoadFrom('C:\AlphaFS\lib\net451\AlphaFS.dll')
$searchFiles = Import-CSV 'C:\Data\SCRIPTS\PS1\TestFindFile.csv' -Header ("Name")
$source = 'C:\Data'
$outputPath = 'c:\data\scripts\ps1\TestFileLocation.csv'
$searchFiles | ForEach-Object {
$files = [Alphaleonis.Win32.Filesystem.Directory]::EnumerateFiles($source,'*',[System.IO.SearchOption]::AllDirectories)
$files | ForEach-Object { [PSCustomObject] #{FileName = $_} }
} | export-csv -notypeinformation -delimiter '|' -path $outputPath
# type $outputPath
If your .csv file contains the header "Name", there is no need to again declare it when running Import-Csv.
The reason the output is empty is that you are searching for an Object which contains the property Name (imported from the TestFindFile.csv). Search for $File.Name. Also pull commands outside the loop that don't need to be there:
$searchFiles | Select -ExpandProperty Name | % {
Get-ChildItem $source -Filter $_ -Recurse | where {!$_.PSIsContainer}
} | select-object FullName | export-csv -notypeinformation -delimiter '|' -path c:\data\scripts\ps1\TestFileLocation.csv