I have a directory structure as below.
Root_dir
Sub_dir1
Sub_dir2
....
Here I have multiple sub folders in the Root directory. Now each sub folder contains a message.csv file. I want to append them and create a new csv file.
Assuming that all CSVs have the same columns something like this should work:
$root = 'C:\path\to\Root_dir'
$csv = 'C:\path\to\output.csv'
Get-ChildItem $root -Filter 'message.csv' -Recurse | % {
Import-Csv $_.FullName
} | Export-Csv $csv -NoTypeInformation
To remove duplicates from the output try this instead:
$root = 'C:\path\to\Root_dir'
$csv = 'C:\path\to\output.csv'
Get-ChildItem $root -Filter 'message.csv' -Recurse | % {
Import-Csv $_.FullName
} | ConvertTo-Csv -NoTypeInformation | select -Unique | Out-File $csv
Related
I have the following code to add new columns to a csv file. I would like to amend the code to run on multiple csv files within a folder and output it to a different folder.
$source = "C:\input_folder\input.csv"
$destination = "C:\output_folder\output.csv"
(Import-CSV $source |
Select-Object *,#{Name='column1';Expression={'data1'}} |
Select-Object *,#{Name='column2';Expression={'data2'}} |
ConvertTo-csv -NoTypeInformation |
Select-Object -Skip 0) -replace '"' | Set-Content $destination
You could do something like this. You might also want to add a check that the files in the folder are .csv type and not something else.
$source = "C:\input_folder"
$destinationFolder = "C:\output_folder"
$folderContents = Get-ChildItem $source
foreach ($item in $folderContents) {
if (Test-Path -Path $item -PathType Leaf == True) {
(Import-CSV $item |
Select-Object *,#{Name='column1';Expression={'data1'}} |
Select-Object *,#{Name='column2';Expression={'data2'}} |
ConvertTo-csv -NoTypeInformation |
Select-Object -Skip 0) -replace '"' | Set-Content ("$destinationFolder\$item" + "_formatted.csv")
}
}
I have a task that requires to scan the property of all the files indicated by certain directories where the files are stored. I need my code to read the following line of information separated by the delimiter "," stored in a .txt file as follows (the directory is made up by myself on my own device and I went ahead making up some blank .xlsx files to test my code:
Jakarta,C:\\temp\Hfolder,C:\temp\Lfolder
I currently have code that looks like this:
$LocContent = Import-Csv "C:\temp\Location.txt" # -Header $fileHeaders
ForEach($line in $LocContent){C:\temp\test1.csv -NoTypeInformation
#split fields into values
$line = $LocContent -split (",")
$country = $line[0]
$hDrivePath = $line[1]
$lDrivePath = $line[2]
Get-ChildItem $hDrivePath -force -include *.xlsx, *.accdb, *.accde, *.accdt, *.accdr -Recurse
Get-ChildItem $lDrivePath -force -include *.xlsx, *.accdb, *.accde, *.accdt, *.accdr -Recurse
? {
$_.LastWriteTime -gt (Get-Date).AddDays(-5)
}
Select-Object -Property Name, Directory, #{Name="Owner";Expression={(Get-ACL $_.Fullname).Owner}}, CreationTime, LastAccessTime, #{N="Location";E={$country}}, #{N='size in MB';E={$_.Length/1024kb}} | Export-Csv
}
However there is no output on the .csv file I assigned to output the information. What is wrong in my code?
Thanks!
There are several flaws within your code:
The Select has neither an -InputObject nor is anything piped to it so there can't be an output
You should decide whether you treat C:\temp\Location.txt as
a text file with Get-Contentand a split
or as a csv with headers
or without headers and supply them to the import.
The Get-ChildItem output isn't piped anywhere nor stored in a variable so it goes to the screen.
Export-Csv needs a file name to export to.
Try this untested script:
## Q:\Test\2018\06\26\SO_51038180.ps1
$fileHeaders = #('country','hDrivePath','lDrivePath')
$extensions = #('*.xlsx','*.accdb','*.accde','*.accdt','*.accdr')
$LocContent = Import-Csv "C:\temp\Location.txt" -Header $fileHeaders
$NewData = ForEach($Row in $LocContent){
Get-ChildItem $Row.hDrivePath,$Row.lDrivePath -Force -Include $extensions -Recurse |
Where-Object LastWriteTime -gt (Get-Date).AddDays(-5) |
Select-Object -Property Name,
Directory,
#{Name="Owner";Expression={(Get-ACL $_.Fullname).Owner}},
CreationTime,
LastAccessTime,
#{N="Location";E={$Row.country}},
#{N='size in MB';E={$_.Length/1024kb}}
}
# you choose what to do with the result uncomment the desired
$NewData | Format-Table -Auto
# $NewData | Out-Gridview
# $NewData | Export-Csv '.\NewData.csv' -NoTypeInformation
I'm trying to load several .csv files (with the same columns) into the same array from different directories.
$csv1 = Import-Csv "PATH1"
$csv1 = Import-Csv "PATH2"
$csv1 | Export-Csv C:\test.csv
This just outputs the last .csv loaded, what would be the best way to do this?
When in doubt, read the documentation. The Import-Csv cmdlet accepts an array of path strings as input, so all you need to do (assuming that all your CSVs have the same fields) is something like this:
$src = 'C:\path\to\input1.csv', 'C:\path\to\input2.csv', ...
$dst = 'C:\path\to\output.csv'
Import-Csv $src | Export-Csv $dst -NoType
If you want an additional column with the path of the source file you need some additional steps, though:
$src | ForEach-Object {
$path = $_
Import-Csv $path | Select-Object *,#{n='Path';e={$path}}
} | Export-Csv $dst -NoType
After beginning this task at the command line I realised I need to get down and dirty with Powershell. I have about 100 folders and each folder has a few thousand CSV files that I would like to merge together inside each folder. Ideally the merged CSV file(s) in each folder would use the parent folders name. For example, here is a top level folder conatining the 100 folders
E:\CSVFolders
The subfolders are named in a semi-random fashion like this:
E:\CSVFolders\Folder1
E:\CSVFolders\Folder18
So far I am at this point:
# Merge csv files and use the parent folder name
Import-Csv (Get-ChildItem File*.csv) |
Export-Csv $folderName.csv -NoTypeInformation -Encoding UTF8
I am struggling to make the script enumerate the subfolders and then use their name as the basis for the merged CSV file so if anyone is able to shed light on this I would appreciate it!
Use two loops:
Get-ChildItem 'E:\CSVFolders' | Where-Object {
$_.PSIsContainer
} | ForEach-Object {
$csv = Join-Path $_.FullName ($_.Name + '.csv')
Get-ChildItem $_.FullName -Filter File*.csv | ForEach-Object {
Import-Csv $_.FullName
} | Export-Csv $csv -NoType -Encoding UTF8
}
you can group by directory like this:
Get-ChildItem "c:\temp" -file -Filter "*.csv" -Recurse |
group DirectoryName |
%{$dir=$_.Name; $_.Group.FullName | %{import-csv -path $_} | export-csv "$dir\global.csv" -NoTypeInformation}
short version (for no purist) :
gci "c:\temp" -file -Filter "*.csv" -Rec |
group DirectoryName |
%{$dir=$_.Name; $_.Group.FullName | %{ipcsv -path $_} | epcsv "$dir\global.csv" -NoType}
I wish to search for specific files listed in searchFiles and pipe their locations to TestFileLocation.CSV. However, my current script only generates an empty CSV. What am I missing?
My TestFindFile.csv is of the form:
Name
123.pdf
321.pdf
aaa.pdf
SNIPPET
$searchFiles = Import-CSV 'C:\Data\SCRIPTS\PS1\TestFindFile.csv' -Header ("Name")
$source = 'C:\Data'
ForEach($File in $searchFiles)
{
Get-ChildItem $source -Filter $File -rec | where {!$_.PSIsContainer} | select-object FullName | export-csv -notypeinformation -delimiter '|' -path c:\data\scripts\ps1\TestFileLocation.csv
}
You were overwriting the CSV for each iteration of the loop.
$searchFiles = Import-CSV 'C:\Data\SCRIPTS\PS1\TestFindFile.csv' -Header ("Name")
$source = 'C:\Data'
$outputPath = 'c:\data\scripts\ps1\TestFileLocation.csv'
$searchFiles | ForEach-Object {
# Silently continue to try to ignore error like
# not being able to read path's which are too long
Get-ChildItem $source -Filter $_ -rec -ErrorAction SilentlyContinue | where {!$_.PSIsContainer} | select-object FullName
} | export-csv -notypeinformation -delimiter '|' -path $outputPath
Example using AlphaFS
A comment asked for an example using AlphaFS because it claims to overcome the long path issue. I'm not going into all the details, but here is how I got it to work.
# download and unzip to c:\alpahfs
# dir C:\AlphaFS\* -Recurse -File | Unblock-File
[System.Reflection.Assembly]::LoadFrom('C:\AlphaFS\lib\net451\AlphaFS.dll')
$searchFiles = Import-CSV 'C:\Data\SCRIPTS\PS1\TestFindFile.csv' -Header ("Name")
$source = 'C:\Data'
$outputPath = 'c:\data\scripts\ps1\TestFileLocation.csv'
$searchFiles | ForEach-Object {
$files = [Alphaleonis.Win32.Filesystem.Directory]::EnumerateFiles($source,'*',[System.IO.SearchOption]::AllDirectories)
$files | ForEach-Object { [PSCustomObject] #{FileName = $_} }
} | export-csv -notypeinformation -delimiter '|' -path $outputPath
# type $outputPath
If your .csv file contains the header "Name", there is no need to again declare it when running Import-Csv.
The reason the output is empty is that you are searching for an Object which contains the property Name (imported from the TestFindFile.csv). Search for $File.Name. Also pull commands outside the loop that don't need to be there:
$searchFiles | Select -ExpandProperty Name | % {
Get-ChildItem $source -Filter $_ -Recurse | where {!$_.PSIsContainer}
} | select-object FullName | export-csv -notypeinformation -delimiter '|' -path c:\data\scripts\ps1\TestFileLocation.csv