I would like to iterate over all js files in a folder, and add the exact line to all of them. The text should be appended at the end of the file in a new line. I'd like it to be written in a windows supported scripting language like batch or powershell.
My pseudo code would look like:
foreach file in folder
append some text to file in a new line
This can be easily done with a batch-file, using the >> redirection operator, given that every file ends with a line-break:
for %%F in ("%TargetFolder%\*.js") do (
>> "%%~F" echo/This is the new line of text.
)
Or if you want to recurse into sub-directories also:
for /R "%TargetFolder%" %%F in ("*.js") do (
>> "%%~F" echo/This is the new line of text.
)
You can of course first explicitly append a line-break:
for %%F in ("%TargetFolder%\*.js") do (
>> "%%~F" (
echo/
echo/This is the new line of text.
)
)
Or equivalently for the recursive approach, of course.
If you want to append a line-break only in case the file does not already end with such, you could do the following:
for %%F in ("%TargetFolder%\*.js") do (
rem /* Count the number of lines that contain zero or more characters at the end;
rem this is true for every line except for the last when it is not terminated
rem by a line-break, because the `$` is anchored to such: */
for /F %%D in ('findstr ".*$" "%%~F" ^| find /C /V ""') do (
rem // Count the total number of lines in the file:
for /F %%C in ('^< "%%~F" find /C /V ""') do (
>> "%%~F" (
rem // Compare the line counts and conditionally append a line-break:
if %%D lss %%C echo/
echo/This is the new line of text.
)
)
)
)
And again equivalently for the recursive approach.
This is easily done using PowerShell.
Add-Content -Path 'C:\src\t\copyt\*.txt' -Encoding ascii -Value "`nEND"
Or, in a .bat file script.
powershell -NoLogo -NoProfile -Command ^
Add-Content -Path 'C:\src\t\copyt\*.txt' -Encoding ascii -Value "`nEND"
If the files are not large, then to add the line before the last line:
$TempFile = New-TemporaryFile
Get-ChildItem -File -Path 'C:\src\t\copyt' -Filter '*.txt' |
ForEach-Object {
$f = Get-Content -Path $_.FullName
$f[0..($f.Length -2)] + "BOTTOM LINE" + $f[-1] |
Out-File -FilePath $TempFile -Encoding default
Remove-Item -Path $_.FullName
Move-Item -Path $TempFile -Destination $_.FullName
}
Remove-Item -Path $TempFile
I have a lot of files named like
Files_01(some text).png
Files_02(some different text).png
Files_03(some other text).png
Files_04(totally different text).png
What I'm looking for is a way to remove everything in the brackets so that I'm left with:
Files_01.png
Files_02.png
etc.
I tried the following, but it didn't remove the text in parentheses:
Get-ChildItem .png | foreach {
Rename-Item $_ $_.Name.Replace("()", "")
}
This is how I completed the task. I read in the file objects from the directory, then check if the filename contains parentheses. If the name does, I replace the parentheses and any text in between with an empty string. Then call Rename-Item and pass it the new file name without the parentheses.
Get-ChildItem -Path C:\PowershellTest\ | % {if($_.Name -match '.*\(.*\).*') {Rename-Item -Path $_.FullName -NewName $($_.Name -replace '\(.*\)', '')}}
Try like this :
#echo off
set "$ext=.png"
for /f "delims=" %%b in ('dir /b/a-d *%$ext%') do for /f "tokens=1 delims=()" %%c in ('echo %%b') do ren "%%~nb%$ext%" "%%~nc%$ext%"
In powershell I'd probably create something like this:
$myFolder = "C:\Users\Aarron\Pictures\"
Foreach ($file in Get-Childitem $myFolder\*.png)
{
$ReName = $file.basename.split("(")[0] + $file.extension
Rename-Item $file.FullName $ReName
}
…and a possible batch solution:
#Echo Off
Set "myFolder=C:\Users\Aarron\Pictures\"
For %%A In ("%myFolder%\*.png") Do Call :Sub "%%~dpA" "%%~nA" "%%~xA"
Exit/B
:Sub
Set "ReName=%~2"
Set ReName=%ReName:(=&:%
Ren "%~1%~2%~3" "%ReName%%~3"
Hello I'm looking for powershell script which would merge all csv files in a directory into one text file (.txt) . All csv files have same header which is always stored in a first row of every file. So I need to take header from the first file, but in rest of the files the first row should be skipped.
I was able to find batch file which is doing exactly what I need, but I have more than 4000 csv files in a single directory and it takes more than 45 minutes to do the job.
#echo off
ECHO Set working directory
cd /d %~dp0
Deleting existing combined file
del summary.txt
setlocal ENABLEDELAYEDEXPANSION
set cnt=1
for %%i in (*.csv) do (
if !cnt!==1 (
for /f "delims=" %%j in ('type "%%i"') do echo %%j >> summary.txt
) else (
for /f "skip=1 delims=" %%j in ('type "%%i"') do echo %%j >> summary.txt
)
set /a cnt+=1
)
Any suggestion how to create powershell script which would be more efficient than this batch code?
Thank you.
John
If you're after a one-liner you can pipe each csv to an Import-Csv and then immediately pipe that to Export-Csv. This will retain the initial header row and exclude the remaining files header rows. It will also process each csv one at a time rather than loading all into memory and then dumping them into your merged csv.
Get-ChildItem -Filter *.csv | Select-Object -ExpandProperty FullName | Import-Csv | Export-Csv .\merged\merged.csv -NoTypeInformation -Append
This will append all the files together reading them one at a time:
get-childItem "YOUR_DIRECTORY\*.txt"
| foreach {[System.IO.File]::AppendAllText
("YOUR_DESTINATION_FILE", [System.IO.File]::ReadAllText($_.FullName))}
# Placed on seperate lines for readability
This one will place a new line at the end of each file entry if you need it:
get-childItem "YOUR_DIRECTORY\*.txt" | foreach
{[System.IO.File]::AppendAllText("YOUR_DESTINATION_FILE",
[System.IO.File]::ReadAllText($_.FullName) + [System.Environment]::NewLine)}
Skipping the first line:
$getFirstLine = $true
get-childItem "YOUR_DIRECTORY\*.txt" | foreach {
$filePath = $_
$lines = $lines = Get-Content $filePath
$linesToWrite = switch($getFirstLine) {
$true {$lines}
$false {$lines | Select -Skip 1}
}
$getFirstLine = $false
Add-Content "YOUR_DESTINATION_FILE" $linesToWrite
}
Try this, it worked for me
Get-Content *.csv| Add-Content output.csv
This is pretty trivial in PowerShell.
$CSVFolder = 'C:\Path\to\your\files';
$OutputFile = 'C:\Path\to\output\file.txt';
$CSV = Get-ChildItem -Path $CSVFolder -Filter *.csv | ForEach-Object {
Import-Csv -Path $_
}
$CSV | Export-Csv -Path $OutputFile -NoTypeInformation -Force;
Only drawback to this approach is that it does parse every file. It also loads all files into memory, so if we're talking about 4000 files that are 100 MB each you'll obviously run into problems.
You might get better performance with System.IO.File and System.IO.StreamWriter.
Your Batch file is pretty inefficient! Try this one (you'll be surprised :)
#echo off
ECHO Set working directory
cd /d %~dp0
ECHO Deleting existing combined file
del summary.txt
setlocal
for %%i in (*.csv) do set /P "header=" < "%%i" & goto continue
:continue
(
echo %header%
for %%i in (*.csv) do (
for /f "usebackq skip=1 delims=" %%j in ("%%i") do echo %%j
)
) > summary.txt
How this is an improvement
for /f ... in ('type "%%i"') requires to load and execute cmd.exe in order to execute the type command, capture its output in a temporary file and then read data from it, and this is done with each input file. for /f ... in ("%%i") directly reads data from the file.
The >> redirection opens the file, appends data at end and closes the file, and this is done with each output *line*. The > redirection keeps the file open all the time.
If you need to scan folder recursively then you can use the approach below
Get-ChildItem -Recurse -Path .\data\*.csv | Get-Content | Add-Content output.csv
what this basically does is:
Get-ChildItem -Recurse -Path .\data\*.csv Find the requested files recursively
Get-Content Get content for each
Add-Content output.csv append it to output.csv
Here is a version also using System.IO.File,
$result = "c:\temp\result.txt"
$csvs = get-childItem "c:\temp\*.csv"
#read and write CSV header
[System.IO.File]::WriteAllLines($result,[System.IO.File]::ReadAllLines($csvs[0])[0])
#read and append file contents minus header
foreach ($csv in $csvs) {
$lines = [System.IO.File]::ReadAllLines($csv)
[System.IO.File]::AppendAllText($result, ($lines[1..$lines.Length] | Out-String))
}
Get-ChildItem *.csv|select -First 1|Get-Content|select -First 1|Out-File -FilePath .\input.csv -Force #Get the header from one of the CSV Files, write it to input.csv
Get-ChildItem *.csv|foreach {Get-Content $_|select -Skip 1|Out-File -FilePath .\Input.csv -Append} #Get the content of each file, excluding the first line and append it to input.csv
stinkyfriend's helpful answer shows an elegant, PowerShell-idiomatic solution based on Import-Csv and Export-Csv.
Unfortunately,
it is quite slow because it involves ultimately unnecessary round-trip conversion to and from objects.
also, even though it shouldn't matter to a CSV parser, the specific format of the files can get altered in the process, because Export-Csv double-quotes all column values, invariably so in Windows PowerShell, by default in PowerShell (Core) 7+, which now offers opt-in control via -UseQuotes and -QuoteFields).
When performance matters, a plain-text solution is required, which also avoids any inadvertent format alteration (just like the linked answer it assumes that all input CSV files have the same column structure).
The following PSv5+ solution:
reads each input file's content into memory in full, as a single multi-line string, using Get-Content -Raw (which is much faster than the default line-by-line reading),
skips the header line for all but the first file with -replace '^.+\r?\n', using the regex-based -replace operator,
and saves the results to the target file with Set-Content -NoNewLine.
Character-encoding caveat:
PowerShell never preserves the input character encoding of files, so you may have to use the -Encoding parameter to override Set-Content's default encoding (the same applies to Export-Csv and any other file-writing cmdlets; in PowerShell (Core) 7+ all cmdlets now consistently default to BOM-less UTF-8; but not only do Windows PowerShell cmdlets not default to UTF-8, they use varying encodings - see the bottom section of this answer).
# Determine the output file and remove a preexisting one, if any.
$outFile = 'summary.csv'
if (Test-Path $outFile) { Remove-Item -ErrorAction Stop $outFile }
# Process all *.csv files in the current folder and merge their contents,
# skipping the header line for all but the first file.
$first = $true
Get-ChildItem -Filter *.csv |
Get-Content -Raw |
ForEach-Object {
$content =
if ($first) { # first file: output content as-is
$_; $first = $false
} else { # subsequent file: skip the header line.
$_ -replace '^.+\r?\n'
}
# Make sure that each file content ends in a newline
if (-not $content.EndsWith("`n")) { $content += [Environment]::NewLine }
$content # Output
} |
Set-Content -NoNewLine $outFile # add -Encoding as needed.
The modern Powershell 7 answer:
(Assuming all csv files are on the same directory and have the same amount of fields.)
#(Get-ChildItem -Filter *.csv).fullname | Import-Csv |Export-Csv ./merged.csv -NoTypeInformation
First part of the pipeline gets all the .csv files and parses the fullname (Path + filename + extension), then import CSV takes each and creates an object and then each object gets merged into a single CSV file with only one header.
I found the previous solutions quite inefficient for large csv-files in terms of performance, so here is a performant alternative.
Here is an alternative which simply appends the files:
cmd /c copy ((gci "YOUR_DIRECTORY\*.csv" -Name) -join '+') "YOUR_OUTPUT_FILE.csv"
Thereafter, you probably want to get rid of the multiple csv-headers.
The following batch script is very fast. It should work well as long as none of your CSV files contain tab characters, and all source CSV files have fewer than 64k lines.
#echo off
set "skip="
>summary.txt (
for %%F in (*.csv) do if defined skip (
more +1 "%%F"
) else (
more "%%F"
set skip=1
)
)
The reason for the restrictions is that MORE converts tabs into a series of spaces, and redirected MORE hangs at 64k lines.
#Input path
$InputFolder = "W:\My Documents\... input folder"
$FileType = "*.csv"
#Output path
$OutputFile = "W:\My Documents\... some folder\merged.csv"
#Read list of files
$AllFilesFullName = #(Get-ChildItem -LiteralPath $InputFolder -Filter $FileType | Select-Object -ExpandProperty FullName)
#Loop and write
Write-Host "Merging" $AllFilesFullName.Count $FileType "files."
foreach ($FileFullName in $AllFilesFullName) {
Import-Csv $FileFullName | Export-Csv $OutputFile -NoTypeInformation -Append
Write-Host "." -NoNewline
}
Write-Host
Write-Host "Merge Complete"
$pathin = 'c:\Folder\With\CSVs'
$pathout = 'c:\exported.txt'
$list = Get-ChildItem -Path $pathin | select FullName
foreach($file in $list){
Import-Csv -Path $file.FullName | Export-Csv -Path $pathout -Append -NoTypeInformation
}
type *.csv >> folder\combined.csv
first of all: I'm on a Windows 7 machine ;).
I got a folder with several dozens files. Each file contains about 240.000 rows. But only half of that rows are needed.
What I would like to do is: have a script that runs over these files, filters out every row that contains the string "abcd" and have it either saved in a new directory or just saved in the same file.
I would try using Powershell as below:
$currentPath = "the path these files currently in"
$newPath = "the path you want to put the new files"
$files = Get-ChildItem $currentPath
foreach ($item in $files) {
Get-Content $item | Where-Object {$_ -notmatch 'abcd'} |Set-Content $newPath\$item
}
#echo off
setlocal enableextensions
set "_where=c:\some\where\*.txt"
set "_filter=abcd"
rem find files which needs filter
for /f "tokens=*" %%f in ('findstr /m "%_filter%" "%_where%"') do (
rem generate a temporary file with the valid content
findstr /v /c:"%_filter%" "%%~ff" > "%%~ff.tmp"
rem rename original file to .old
ren "%%~ff" *.old > nul
rem rename temporary file as original file
ren "%%~ff.tmp" "%%~nxf" > nul
)
rem if needed, delete *.old files
you might use sed for Windows
sed -i.bak "/abcd/!d" *.txt
Find all abcd containing rows in .txt files, make a backup file .bak and store the the detected lines in the original file.
#echo on
For %%a in (*.txt) do (CALL:FILTER "%%a")
echo/Done.&pause>nul&exit/b
:FILTER
type "%~1"|find "abcd" 1>nul 2>nul
if %errorlevel% EQU 0 find /n "abcd" "%~1">"Found abcd in %~1.txt"
The command Find returns error level = 0 if him find something
If the files are that big, I'd so something like this:
$Folder = 'C:\MyOldFiles'
$NewFolder = 'C:\MyNewFiles'
New-Item -ItemType Directory -Path $NewFolder -Force
foreach ($file in Get-ChildItem $Folder)
{
Get-Content $file -ReadCount 1500 |
foreach { $_ -notmatch 'abcd' } |
Add-Content "$NewFolder\$($file.name)"
}
OK what I am trying to do is make a script that will read each line of a text file, "directory.txt" and export every line that is to a file and not to a directory. Example below.
I'm just trying to remove the paths to directories like "C:\users\"
and keep any path that is to a file like "C:\users\file.txt"
In the test file, Direcory.txt" there will be the following:
C:\path\path\folder\
C:\path\path\file.ext
C:\path\path\path\path\folder
The script will need to read the text file above and export the following line to a new text file.
C:\path\path\file.ext
The batch script equivalent would be the following:
#ECHO OFF
FOR /F %%A IN (directory.txt) DO CALL:NoDir "%%A"
pause
EXIT /B
:NoDir
IF "%~x1" NEQ "" ECHO %~1>>nodir.txt
EXIT /B
Batch script can't handle a file of 400mb so need to use powershell to do it o.0
FTR: The condition if "%~x1" neq "" does not do what you seem to expect. It will match not only folders, but also files without an extension.
Anyway, in PowerShell you'd probably do something like this to list only items that are not directories:
Get-Content \PATH\TO\directory.txt `
| Get-Item `
| Where-Object { -not $_.PSIsContainer } `
| Select-Object FullName
I'm not sure I understand the question (batch example shows all paths), maybe that's what you're looking for:
Get-Content directory.txt | Where-Object {$_ -eq 'C:\path\path\file.ext'}
You didn't say whether you're concerned if they are valid paths or not. The following outputs to the host the lines that are either valid files (using the '-PathType leaf' parameter of the Test-Path cmdlet), or if the last 4 characters of the last item in the path are a dot followed by 3 letters.
$Lines = Get-Content C:\Path\to\file.txt
foreach ( $line in $Lines )
{
if ( (Test-Path $line -PathType leaf) -OR ($line -match "\.\w{3}$") )
{
Write-Host $line
}
}
If you find your file extensions are longer than 3 letters, you can change the regex appropriately ("\.\w{3,4}$" for 4 character extensions, or "\.[\w\d]{3,4}$ to match extensions that are 3 or 4 characters long and might include numbers)
And the one-liner:
Get-Content C:\Path\to\file.txt | % { if ((Test-Path $_ -PathType Leaf) -OR ($_ -match "\.\w{3}$")) { $_ } }