I'm looking for a way to add a header line into multiple CSV files.
Problem with this code below is that it will add an extra empty row at the end of each file. I don't understand why there is extra empty line but I need to delete those lines.
$header="Column1,Column2,Column3,Column4,Column5,Column6"
Get-ChildItem .\ -Recurse -Filter *.csv| Foreach-Object {
$header+"`r`n"+ (Get-Content $_.FullName | Out-String) |
Set-Content -Path $_.FullName
}
The canonical way would be to import the files specifying the headers and then re-export them:
$header = 'Column1', 'Column2', 'Column3', 'Column4', 'Column5', 'Column6'
Get-ChildItem .\ -Recurse -Filter '*.csv' | ForEach-Object {
$file = $_.FullName
(Import-Csv -Path $file -Header $header) | Export-Csv -Path $file -NoType
}
Export-Csv does add double quotes around all fields of the CSV, though. Also, parsing the data into objects does have a performance impact. If you don't want the double quotes added or are pressed for performance the solution suggested by #PetSerAl in the comments to your question might be a better approach for you:
$header = 'Column1,Column2,Column3,Column4,Column5,Column6'
Get-ChildItem .\ -Recurse -Filter '*.csv' | ForEach-Object {
$file = $_.FullName
#($header; Get-Content -Path $file) | Set-Content -Path $file
}
Related
cd 'A:\P\E\D'
$files = Get-ChildItem . *.CSV -rec
ForEach ($file in $files) {
(Get-Content $file -Raw) | ForEach-Object {
*some simple code*
} | Set-Content $file
}
How to modify this powershell script to locate only files starting with letters A/a to O/o and ending with .csv in specified directory cd?
I thought the solution below would work, but the test file M_K_O_X.CSV stored in the cd directory was not found and modified. The solution above will find and modify the file. It's possible that I have the regex expression wrong or the problem is somewhere else? I tried also this regex -- "[A-O]..CSV"
cd 'A:\P\E\D'
$files = Get-ChildItem . -rec | Where-Object { $_.Name -like "[a-oA-O]*.*.CSV" }
ForEach ($file in $files) {
(Get-Content $file -Raw) | ForEach-Object {
*some simple code*
} | Set-Content $file
}
Looking at your wildcard pattern, seems like you have an extra *. that shouldn't be there:
'M_K_O_X.CSV' -like '[a-oA-O]*.*.CSV' # False
'M_K_O_X.CSV' -like '[a-oA-O]*.CSV' # True
In this case you could simply use the -Include Parameter which supports character ranges. Also PowerShell is case insensitive by default, [a-oA-O]*.CSV can be reduced to [a-o]*.CSV:
Get-ChildItem 'A:\P\E\D' -Recurse -Include '[a-o]*.csv' | ForEach-Object {
($_ | Get-Content -Raw) | ForEach-Object {
# *some simple code*
} | Set-Content -LiteralPath $_.FullName
}
As commented, I would use the standard wildcard -Filter to filter for all files with a .csv extension.
Then pipe to a Where-Object clause in which you can use regex -match
$files = Get-ChildItem -Path 'A:\P\E\D' -Filter '*.csv' -File -Recurse |
Where-Object { $_.Name -match '^[a-o]' }
foreach ($file in $files) {
# switch `-Raw` makes Get-Content return a single multiline string, so no need for a loop
$content = Get-Content -Path $file.FullName -Raw
# *some simple code manipulating $content*
$content | Set-Content -Path $file.FullName
}
However, if these are valid csv files, I would not recommend using a pure textual manipulation on them, instead use Import-Csv -Path $file.FullName and work on the properties on each of the objects returned.
I'm looking to replace the character 'a' with 'o' in all txt files in a directory: "D:\Aither\Pca\Stat\"
For now I use this, but lose time because there are a lot of files.
(Get-Content D:\Aither\Pca\Stat\StrEtb.dat).replace('a', 'o') | Set-Content D:\Aither\Pca\Stat\StrEtb.dat
(Get-Content D:\Aither\Pca\Stat\StrEtb2.dat).replace('a', 'o') | Set-Content D:\Aither\Pca\Stat\StrEtb2.dat
....
I want something like this:
(Get-Content D:\Aither\Pca\Stat\*).replace('a', 'o') | Set-Content D:\Aither\Pca\Stat\*
Is this possible in PowerShell?
Using the -Filter and -File parameters, you will have better performance in collecting the files. Also, you can use a ForEach-Object to simply pipe these files through to the next cmdlet.
Because you are overwriting the file(s), you need to use brackets around Get-Content, so you are not trying to read and write to the same file at the same time.
Get-ChildItem -Path 'D:\Aither\Pca\Stat' -Filter '*.dat' -File -Recurse | ForEach-Object {
($_ | Get-Content).Replace('a', 'o') | Set-Content $_.FullName
}
Try this:
$collection = Get-ChildItem -Path 'D:\Aither\Pca\Stat' -Recurse -Filter '*.dat'
foreach( $file in $collection ) {
(Get-Content $file.FullName).replace('a', 'o') | Set-Content ($file.FullName) | Out-Null
}
I'm trying to merge CSV files in Powershell. I've read numerous answers here but I'm stuck on this problem.
I have a list of csv files, 2 difficulties :
[A] each file has a metadataline, the headers are in the second line.
[B] each file has the same structure, but sometimes quotes surround the column to escape the content.
Thanks to this question : Merging multiple CSV files into one using PowerShell,
I'm able to solve these two problems individually.
However, I'm stuck at combining the solutions.
Partial solution A
Skips every metadata line as well as header for subsequent files
Adapting the answer from kemiller2002:
$sourcefilefolderPath = "C:\CSV_folder"
$destinationfilePath = "C:\appended_files.csv"
$getHeader = $true
Get-ChildItem -Path $sourcefilefolderPath -Filter *.csv -Recurse| foreach {
$filePath = $_.FullName
$lines = $lines = Get-Content $filePath
$linesToWrite = switch($getHeader) {
$true {$lines | Select -Skip 1} # skips only the metadata line
$false {$lines | Select -Skip 2} # skips both the metadata line as well as headers
}
$getHeader = False
Add-Content $destination_file $linesToWrite
}
The problem : Import-Csv $destination_file give inconsistent results, as the quoting can be different for each source file.
Partial solution B
handles successfully random quoted columns
Solution provided by stinkyfriend.
Import-Csv seems to import the data gracefully when the column quoting, however different from one column to the other, is consistent for each line of the source file.
I could not combine this solution with the one above.
Get-ChildItem -Path $sourcefilefolderPath -File -Filter *.csv -Recurse |
Select-Object -ExpandProperty FullName |
Import-Csv |
Export-Csv $destination_file -NoTypeInformation -Append
Thanks a lot for your help !
Solution C
produces blank file on my PC
using suggestion from Mathias R. Jessen
Get-ChildItem -Path $sourcefilefolderPath -File -Filter *.csv -Recurse | foreach {
Write-Host $_.FullName |
Get-Content $_.FullName | Select-Object -Skip 1 | ConvertFrom-Csv |
Export-Csv $destinationfilePath -NoTypeInformation -Append
--- EDIT ---
RESULT
I could solve the problem by creating appended_files.csv using the first matching source file and then append to it.
$pattern_sourceFile = "*.csv*"
$list_files = Get-ChildItem -Path $sourcefilefolderPath -File -Recurse | Where {
$_FullName -match $pattern_sourcefile }
Get-Content $list_files[0].FullName |
Select-Object -Skip 1 | # skips metadataline
ConvertFrom-Csv | Export-Csv $destinationfilePath -NoTypeInformation
$list_files |
Select-Object -Skip 1 | # skips $array_files[0]
foreach { Get-Content $_.FullName |
Select-Object -Skip 1 | # skips metadata line
ConvertFrom-Csv |
Export-Csv $destinationfilePath -NoTypeInformation -Append }
Use ConvertFrom-Csv instead of Import-Csv, this way you can still control how many lines to skip:
Get-Content $file |Select -Skip 1 |ConvertFrom-Csv
So you'll end up with something like:
$sourcefilefolderPath = "C:\CSV_folder"
$destinationfilePath = "C:\appended_files.csv"
Get-ChildItem -Path $sourcefilefolderPath -Filter *.csv -Recurse | foreach {
Get-Content $_.FullName |Select-Object -Skip 1 |ConvertFrom-Csv |Export-Csv -Path $destinationfilePath -NoTypeInformation -Append
}
Newbie stuck doing the basics.
I'm trying to copy files from A to B & then replace text within each file copied. Pretty basic - but it fails What am I doing wrong? Any assistance would be welcome - please. nb I believe I need to use copy-item, alternatives okay as long as same result.
$files = Copy-Item -Path "C:\from" -Filter *.* -Recurse -Destination "C:\to" -Force -PassThru
foreach ($file in $files) {
Get-Item $file |
Where-Object {-not $_.PsIsContainer} |
(Get-Content .) |
Foreach-Object {
$_ -replace ( "maskkey", $maskvalue} |
Set-Content $file
}
}
You need to move the get-content part within the foreach-object script block, and use $_ instead of .
So, something like this:
foreach($file in $files){
Get-Item $file | Where-Object {-not ($_.PsIsContainer)} | Foreach-Object{
$content = Get-Content $_
$newContent = $content.replace("maskkey", $maskvalue)
Set-Content -Value $newContent -Path $file
}
}
Note. I haven't actually tested this code, but you should at least have the basic structure right.
I'm trying to do a replace in content of all files in a certain directory structure.
get-childItem temp\*.* -recurse |
get-content |
foreach-object {$_.replace($stringToFind1, $stringToPlace1)} |
set-content [original filename]
Can I get the filename from the original get-childItem to use it in the set-content?
Add processing for each file:
get-childItem *.* -recurse | % `
{
$filepath = $_.FullName;
(get-content $filepath) |
% { $_ -replace $stringToFind1, $stringToPlace1 } |
set-content $filepath -Force
}
Key points:
$filepath = $_.FullName; — get path to file
(get-content $filepath) — get content and close file
set-content $filepath -Force — save modified content
You can simply use $_, but you need a foreach-object around each file, too. While #akim's answer will work, the use of $filepath is unnecessary:
gci temp\*.* -recurse | foreach-object { (Get-Content $_) | ForEach-Object { $_ -replace $stringToFind1, $stringToPlace1 } | Set-Content $_ }