My Old Bat file
Copy F:\File.hdr+F:*.csv F:\FinalOutput.csv
the HDR file is a single entry file that has only header data for the CSV files
Is there a way to perform this in PowerShell (to combine all the CSV files into a single file)?
Here is my powershell script that doesn't work
$CSVFolder = 'F:\Input\';
$OutputFile = 'F:\Output\NewOutput.csv';
$CSV= #();
Get-ChildItem -Path $CSVFolder -Filter *.inv | ForEach-Object {
$CSV += #(Import-Csv -Path $CSVFolder\$_)
}
$CSVHeader = Import-Csv 'F:\Input\Headings.hdr'
$CSV = $CSVHeader + $CSV
$CSV | Export-Csv -Path $OutputFile -NoTypeInformation -Force;
I get the list of FileNames that are exported and not the content of the Files.
The script is also modifying the date/time stamp on my INV files. It shouldn't be doing that.
You can skip the whole CSV bit if you just append the files as you would before.
Something like this should work:
# First we create the new file and add the header.
get-content $headerfile | set-content $outputfile
# Then we get the input files, read them out with get-content
# and append them to the output file (add-content).
get-childitem -path $csvfolder *.inv | get-content | add-content $outputfile
The CSV commandlets are handy if you want to be processing the CSV data in your script, but in your case simply appending the files will do the trick. Not bothering with the CSV conversion will be a lot faster as Powershell doesn't have to parse the CSV lines and create PS-objects. It's really fast with pure text though.
Another trick here is how the get-content and add-content are used in the pipeline. Since they are aware of the pipeline you can pass in file objects without having to use a foreach loop. This makes your statements a lot shorter.
How about:
get-childitem *.inv | foreach-object {
import-csv $_ -header (get-content Headings.hdr)
} | export-csv NewOutput.csv -notypeinformation
Related
I am just starting to learn PowerShell and have run into a hurdle where I'm trying to use gci and import-csv. My goal is to run a script in a folder directory that has numerous subfolders that contain a specific csv file that I would like to import and consolidate the data. These subfolders have additional subfolders that have other file types including csv that I don't have any use for. I am interested in the specific path below. The csv files have a specific header type called location that I care about and have to parse out into a string.
Folder directory example
This is my code so far:
$files = gci .\ -Recurse
foreach($file in $files) {
$foldername = $file.Name
$csv = Import-Csv -Path ".\$foldername\$foldername.csv"
foreach ($line in $csv) {
$outputlines = $line.location -split '/'
Export-csv -Path .\Test.csv -NoTypeInformation -Append
}
}
This is the message I get when I run it:
cmdlet Export-Csv at command pipeline position 1
Supply values for the following parameters:
InputObject:
Can someone please guide me in the right direction on what I'm doing wrong?
As others have commented and strictly speaking, Export-Csv needs something to export. However, that's not the only issue. Import-Csv will return objects based on the data in the csv file. So -split is likely to provide strange results as it's designed to run against strings and/or arrays of strings.
Withstanding the -split that is unlikely to work, you can address consolidation more simply by simply feeding the results from many Import-Csv commands to a single Export-csv command:
$OutputCsv = 'c:\temp\Output.csv'
Get-ChildItem .\ -Directory |
ForEach-Object{
$FolderName = $_.Name
".\$FolderName \$FolderName .csv"
} |
Import-Csv |
Export-Csv -Path $OutputCsv -NoTypeInformation -Append
The loop will output a series of strings that are piped to Import-Csv. So, all the files will get imported and the resulting objects will be streamed to Export-Csv consolidating everything into $Outputcsv which is c:\temp\Output.csv.
Note: The use of the -Directory parameter. Since you are only leveraging the folder names that should prevent a few errors, particularly for file names that may not exist.
If you want to clarify the question with an example of the CSV contents and the desired output we can take this further.
I created a short powershell script to convert csv files from Unicode to UTF-8 encoding. My script outputs new files with the the original file name preceded by UTF8. I'm running into two issues:
I'm trying to only run the powershell script on csv files. Currently the script runs on every file in the directory, including the powershell script (it outputs a new file called UTF8pshell_script if the powershell script was called pshell_script for example). The other methods where I've tried to only run the script on csv files just end up making the script not do anything.
I'm trying to run the script on sub-directories. The first issue is that output files created from csv files in subdirectories have no content inside them whatsoever. If the script is ran in the same directory as the csv file this problem does not arise. This is not crucial but I am also uncertain how to get output files created from those in subdirectories to be outputted in the same subdirectories (currently they are outputted in the main directory where the powershell script is).
as
Get-Content -Encoding Unicode $_ | Out-File -Encoding UTF8
Get-ChildItem -Recurse | ForEach-Object {Get-Content -Encoding Unicode $_ | Out-File -Encoding UTF8 "UTF8$_"}
The desired output is the powershell script running on only csv files, and outputting files to the same subdirectories where the files they were created form are.
Get-ChildItem takes a -Filter parameter, which for files is the simple wildcard pattern. This will allow you to restrict your cmdlet to CSV files only:
Get-ChildItem -Filter *.csv
To process subdirectories, you may also use the -Recurse switch
Get-ChildItem -Filter *.csv -Recurse
Now, I'm never quite sure how $_ changes as you pass different objects through the pipe, so I'm probably not doing the next steps the most efficient way - but it will be clear what I'm trying to do:
Each file object that we find needs to be processed as follows:
Dissect it into a path and a filename: $filepath = $_.PSParentPath; $filename = $_.PSChildName
Load up the CSV: Import-CSV -Path $_
Output the new CSV with the proper encoding: Export-CSV -Path ("{0}\UTF8{1}" -f $filepath,$filename) -Encoding UTF8
So, we put it all together:
Get-ChildItem -Filter *.csv -Recurse -exclude UTF8* | ForEach-Object {
$filepath = $_.PSParentPath
$filename = $_.PSChildName
Import-CSV -Path $_ |
Export-CSV -Encoding UTF8 -Path ("{0}\UTF8{1}" -f $filepath,$filename) -NoTypeInformation
}
The -Exclude UTF8* in the Get-ChildItem ensures that when you create a file, it doesn't get picked up later and re-processed. The -NoTypeInformation on the Export-CSV compensates for a stupidity built in to the cmdlet that causes an extra line with a meaningless object type name at the beginning of the file.
Depending on the original encoding (and presence of a BOM) you might have to specify an encoding also on the input side.
ForEach($Csv in (Get-ChildItem -Filter *.csv -Recurse -Exclude UTF8*)){
(Get-Content $Csv.FullName -raw) |
Set-Content -Path {Join-Path $Csv.Directory ("UTF8"+$Csv.Name)} -Encoding UTF8
}
LotPings beat me to this by 10 minutes with a virtually identical answer, but I'm leaving this for the 'passing an empty file to the pipeline' bit that I have. I also realize after the fact that you don't need a pipeline variable for that same reason, as you only need it if you pass things through the pipeline within the loop.
If all you want to do is change the encoding I would use a ForEach($x in $y){} loop, or a ForEach-Object{} loop with a PipelineVariable on the Get-ChildItem. I'll show that since I think pipeline variables are under used. I would also not read the file and pipe it to something, since if the file is empty you won't create a new file as nothing is passed down the pipeline.
Get-ChildItem *.csv -Recurse -PipelineVariable File | ForEach-Object{
Set-Content -Value (Get-Content $File.FullName -Encoding Unicode) -Path {Join-Path $File.Directory "UTF8$($File.Name)"} -Encoding UTF8
}
if you specify the file extension at the end of Get-ChildItem.
This will get only the files with the .csv extension.
By specifying the File path in Out-File it will send it to the specified directory.
Get-ChildItem -Path C:\folder\*.csv -Recurse | ForEach-Object {Get-Content -Encoding Unicode $_ | Out-File -FilePath C:\Folder -Encoding UTF8 "UTF8$_"}
I have a CSV file with logging information. Date/time info in the file is written in UNIX time-stamp format.
I want to remove all lines older than 5 minutes. I use the following code to do this:
$FiveMinutesAgo=(Get-Date ((get-date).toUniversalTime()) -UFormat +%s).SubString(0,10)-300
$Content = Import-Csv "logfile.csv" | where {$_.DateCompleted -gt $FiveMinutesAgo}
$Content | Export-Csv -delimiter ";" -NoTypeInformation -encoding UTF8 -Path 'logfile.csv'
The CSV file looks like this:
"DateInitiated";"DateStarted";"DateCompleted";"InitiatedBy";"Action";"Status"
"1496659208";"1496659264";"1496752840";"administrator";"Reboot server";"Completed"
No matter whether the five minutes have already passed or not, my CSV file ends up completely empty after executing the script lines above.
I couldn't replicate it in testing (but potentially need the actual source file). I think the issue is that you need to specify both the encoding and delimiter on the Import-CSV as well (as you already have on Export).
Try this:
$FiveMinutesAgo=(Get-Date ((get-date).toUniversalTime()) -UFormat +%s).SubString(0,10)-300
$Content = Import-Csv "test.csv" -Delimiter ';' -Encoding UTF8 | where {$_.DateCompleted -gt $FiveMinutesAgo}
$Content | Export-Csv -delimiter ";" -NoTypeInformation -encoding UTF8 -Path 'logfile.csv'
I have multiple .tsv file that I want to convert to a .csv file using Windows Powershell.
I used
Import-Csv C:\temp\file\users.txt -Delimiter "`t"
But I can't export to a csv file.
Can someone help please?
Use the Export-Csv cmdlet to export the data.
Now, Import-Csv only takes one file at a time, so wrap that in a loop or the ForEach-Object cmdlet:
Get-ChildItem C:\temp\file\ -Filter *.txt |ForEach-Object {
Import-Csv $_.FullName -Delimiter "`t"
} |Export-Csv C:\path\to\output.csv -Delimiter "`t" -NoTypeInformation
I have five .sql files and know the name of each file. For this example, call them one.sql, two.sql, three.sql, four.sql and five.sql. I want to append the text of all files and create one file called master.sql. How do I do this in PowerShell? Feel free to post multiple answers to this problem because I am sure there are several ways to do this.
My attempt does not work and creates a file with several hundred thousand lines.
PS C:\sql> get-content '.\one.sql' | get-content '.\two.sql' | get-content '.\three.sql' | get-content '.\four.sql' | get-content '.\five.sql' | out-file -encoding UNICODE master.sql
Get-Content one.sql,two.sql,three.sql,four.sql,five.sql > master.sql
Note that > is equivalent to Out-File -Encoding Unicode. I only tend to use Out-File when I need to specify a different encoding.
There are some good answers here but if you have a whole lot of files and maybe you don't know all of the names this is what I came up with:
$vara = get-childitem -name "path"
$varb = foreach ($a in $vara) {gc "path\$a"}
example
$vara = get-childitem -name "c:\users\test"
$varb = foreach ($a in $vara) {gc "c:\users\test\$a"}
You can obviously pipe this directly into | add-content or whatever but I like to capture in variables so I can manipulate later on.
See if this works better
get-childitem "one.sql","two.sql","three.sql","four.sql","five.sql" | get-content | out-file -encoding UNICODE master.sql
I needed something similar, Chris Berry's post helped, but I think this is more efficient:
gci -name "*PathToFiles*" | gc > master.sql
The first part gci -name "*PathToFiles*" gets you your file list. This can be done with wildcards to just get your .sql files i.e. gci -name "\\share\folder\*.sql"
Then pipes to Get-Content and redirects the output to your master.sql file. As noted by Kieth Hill, you can use Out-File in place of > to better control your output if needed.
I think logical way of solving this is to use Add-Content
$files = Get-ChildItem '.\one.sql', '.\two.sql', '.\three.sql', '.\four.sql', '.\five.sql'
$files | foreach { Get-Content $_ | Add-Content '.\master.sql' -encoding UNICODE }
hovewer Get-Content is usually very slow when reading multiple very large files. If its your case this article could help: http://keithhill.spaces.live.com/blog/cns!5A8D2641E0963A97!756.entry
What about:
Get-Content .\one.sql,.\two.sql,.\three.sql,.\four.sql,.\five.sql | Set-Content .\master.sql
Here is how I do concatenate sql files from the Sql folder:
# Set the current location of the script to use relative path
Set-Location $PSScriptRoot
# Concatenate all the sql files
$concatSql = Get-Content -Path .\Sql\*.sql
# Write/overwrite sql to single file
Add-Content -Path concatFile.sql -Value $concatSql