I am just starting to learn PowerShell and have run into a hurdle where I'm trying to use gci and import-csv. My goal is to run a script in a folder directory that has numerous subfolders that contain a specific csv file that I would like to import and consolidate the data. These subfolders have additional subfolders that have other file types including csv that I don't have any use for. I am interested in the specific path below. The csv files have a specific header type called location that I care about and have to parse out into a string.
Folder directory example
This is my code so far:
$files = gci .\ -Recurse
foreach($file in $files) {
$foldername = $file.Name
$csv = Import-Csv -Path ".\$foldername\$foldername.csv"
foreach ($line in $csv) {
$outputlines = $line.location -split '/'
Export-csv -Path .\Test.csv -NoTypeInformation -Append
}
}
This is the message I get when I run it:
cmdlet Export-Csv at command pipeline position 1
Supply values for the following parameters:
InputObject:
Can someone please guide me in the right direction on what I'm doing wrong?
As others have commented and strictly speaking, Export-Csv needs something to export. However, that's not the only issue. Import-Csv will return objects based on the data in the csv file. So -split is likely to provide strange results as it's designed to run against strings and/or arrays of strings.
Withstanding the -split that is unlikely to work, you can address consolidation more simply by simply feeding the results from many Import-Csv commands to a single Export-csv command:
$OutputCsv = 'c:\temp\Output.csv'
Get-ChildItem .\ -Directory |
ForEach-Object{
$FolderName = $_.Name
".\$FolderName \$FolderName .csv"
} |
Import-Csv |
Export-Csv -Path $OutputCsv -NoTypeInformation -Append
The loop will output a series of strings that are piped to Import-Csv. So, all the files will get imported and the resulting objects will be streamed to Export-Csv consolidating everything into $Outputcsv which is c:\temp\Output.csv.
Note: The use of the -Directory parameter. Since you are only leveraging the folder names that should prevent a few errors, particularly for file names that may not exist.
If you want to clarify the question with an example of the CSV contents and the desired output we can take this further.
Related
Beginner user...
I have 30+ .dat files in one folder and need to export row 10 from each file and compile into one csv file.
I know I am on the right lines but am not sure of the middle section - this is where I'm at...
Get-ChildItem -Path C:\Users\pitters\Folder\* -Include *.dat -Recurse
ForEach-Object {
Select-Object -Skip 9 -First 1 }
Export-CSV -Path Users\pitters\Folder\output.csv
Am I missing Get-Content and can anyone help with what needs correcting?
Thanks in advance.
Matt
As you mentioned yourself, you need to invoke Get-Content to actually read the file contents. In addition, you also need to construct an object with appropriate properties corresponding to the coumns you want in your CSV file - something we can do with Select-Object directly:
Get-ChildItem -Path C:\Users\pitters\Folder\* -Include *.dat -Recurse |ForEach-Object {
$file = $_
$file |Get-Content |Select-Object #{Name='Line10';Expression={$_}},#{Name='File';Expression={$file.FullName}} -Skip 9 -First 1
} |Export-CSV -Path Users\pitters\Folder\output.csv
i'm really new to Powershell (2 days) and i am not good yet. :(
My Question ist:
How to change a character in multiple .txt-files and save/ overwrite the existing file in Powershell
My goal is to Copy multiple RAW-Files in a new folder, change the file-name from .tsv to .txt and at least change one character in these files from % to percent.
What i've got so far:
The first two steps are working, but i'm losing my mind with the third step (the replacement).
Copy-Item -Path "C:\Users\user\Desktop\RAW\*.tsv" -Destination "C:\Users\user\Desktop\TXT" -Recurse
Set-Location "C:\Users\User\Desktop\TXT"
Get-ChildItem *.tsv | Rename-Item -NewName { $_.Name -replace '.tsv','.txt' }
This works fine for me and now i am not able to get further ...
I am able to replace the "%" in one specific file, and save it in a new file, but this doesn't work for a batch processing with changing file-names.
$file = "A.txt"
Get-Content $file | Foreach {$_ -replace "%", "percent"} | Set-Content A_1.txt
It would be perfect, if "$file = "A.txt"" would be "all the files in this path with .txt" and
"Set-Content A_1.txt" would be "overwrite the existing file".
I hope someone will help me, thank you! <3 <3 <3
You already have some of the solution in your first code snippet, you need to iterate over the files again to perform the replace and save.
$txtFiles = Get-ChildItem -Name *.txt
ForEach ($file in $txtFiles) {
(Get-Content $file) | ForEach-Object {
$_ -replace '%','percent'
} | Set-Content $file
}
The first line adds all the text files to an array, the foreach loop iterates over the files of the array and grabs the content of the file and unloads it - that's the reason for the parenthesis, the Foreach-Object then iterates over the content of the file and saves it to the same file name as before.
If you skip the parentheses around Get-Content $file the file would still be loaded into memory and you would get an error message about not being able to save the file.
i am quite new to powershell and i am trying to make a script that copy files to certain folders that are declared in a CSV file. But till now i am getting errors from everywhere and can't find nothing to resolve this issue.
I have this folders and .txt files created in the same folder as the script.
Till now i could only do this:
$files = Import-Csv .\files.csv
$files
foreach ($file in $files) {
$name = $file.name
$final = $file.destination
Copy-Item $name -Destination $final
}
This is my CSV
name;destination
file1.txt;folderX
file2.txt;folderY
file3.txt;folderZ
As the comments indicate, if you are not using default system delimiters, you should make sure to specify them.
I also recommend typically to use quotes for your csv to ensure no problems with accidentally including an entry that includes the delimiter in the name.
#"
"taco1.txt";"C:\temp\taco2;.txt"
"# | ConvertFrom-CSV -Delimiter ';' -Header #('file','destination')
will output
file destination
---- -----------
taco1.txt C:\temp\taco2;.txt
The quotes make sure the values are correctly interpreted. And yes... you can name a file foobar;test..txt. Never underestimate what users might do. 😁
If you take the command Get-ChildItem | Select-Object BaseName,Directory | ConvertTo-CSV -NoTypeInformation and review the output, you should see it quoted like this.
Sourcing Your File List
One last tip. Most of the time I've come across a CSV for file input lists a CSV hasn't been needed. Consider looking at grabbing the files you in your script itself.
For example, if you have a folder and need to filter the list down, you can do this on the fly very easily in PowerShell by using Get-ChildItem.
For example:
$Directory = 'C:\temp'
$Destination = $ENV:TEMP
Get-ChildItem -Path $Directory -Filter *.txt -Recurse | Copy-Item -Destination $Destination
If you need to have more granular matching control, consider using the Where-Object cmdlet and doing something like this:
Get-ChildItem -Path $Directory -Filter *.txt -Recurse | Where-Object Name -match '(taco)|(burrito)' | Copy-Item -Destination $Destination
Often you'll find that you can easily use this type of filtering to keep CSV and input files out of the solution.
example
Using techniques like this, you might be able to get files from 2 directories, filter the match, and copy all in a short statement like this:
Get-ChildItem -Path 'C:\temp' -Filter '*.xlsx' -Recurse | Where-Object Name -match 'taco' | Copy-Item -Destination $ENV:TEMP -Verbose
Hope that gives you some other ideas! Welcome to Stack Overflow. 👋
I created a short powershell script to convert csv files from Unicode to UTF-8 encoding. My script outputs new files with the the original file name preceded by UTF8. I'm running into two issues:
I'm trying to only run the powershell script on csv files. Currently the script runs on every file in the directory, including the powershell script (it outputs a new file called UTF8pshell_script if the powershell script was called pshell_script for example). The other methods where I've tried to only run the script on csv files just end up making the script not do anything.
I'm trying to run the script on sub-directories. The first issue is that output files created from csv files in subdirectories have no content inside them whatsoever. If the script is ran in the same directory as the csv file this problem does not arise. This is not crucial but I am also uncertain how to get output files created from those in subdirectories to be outputted in the same subdirectories (currently they are outputted in the main directory where the powershell script is).
as
Get-Content -Encoding Unicode $_ | Out-File -Encoding UTF8
Get-ChildItem -Recurse | ForEach-Object {Get-Content -Encoding Unicode $_ | Out-File -Encoding UTF8 "UTF8$_"}
The desired output is the powershell script running on only csv files, and outputting files to the same subdirectories where the files they were created form are.
Get-ChildItem takes a -Filter parameter, which for files is the simple wildcard pattern. This will allow you to restrict your cmdlet to CSV files only:
Get-ChildItem -Filter *.csv
To process subdirectories, you may also use the -Recurse switch
Get-ChildItem -Filter *.csv -Recurse
Now, I'm never quite sure how $_ changes as you pass different objects through the pipe, so I'm probably not doing the next steps the most efficient way - but it will be clear what I'm trying to do:
Each file object that we find needs to be processed as follows:
Dissect it into a path and a filename: $filepath = $_.PSParentPath; $filename = $_.PSChildName
Load up the CSV: Import-CSV -Path $_
Output the new CSV with the proper encoding: Export-CSV -Path ("{0}\UTF8{1}" -f $filepath,$filename) -Encoding UTF8
So, we put it all together:
Get-ChildItem -Filter *.csv -Recurse -exclude UTF8* | ForEach-Object {
$filepath = $_.PSParentPath
$filename = $_.PSChildName
Import-CSV -Path $_ |
Export-CSV -Encoding UTF8 -Path ("{0}\UTF8{1}" -f $filepath,$filename) -NoTypeInformation
}
The -Exclude UTF8* in the Get-ChildItem ensures that when you create a file, it doesn't get picked up later and re-processed. The -NoTypeInformation on the Export-CSV compensates for a stupidity built in to the cmdlet that causes an extra line with a meaningless object type name at the beginning of the file.
Depending on the original encoding (and presence of a BOM) you might have to specify an encoding also on the input side.
ForEach($Csv in (Get-ChildItem -Filter *.csv -Recurse -Exclude UTF8*)){
(Get-Content $Csv.FullName -raw) |
Set-Content -Path {Join-Path $Csv.Directory ("UTF8"+$Csv.Name)} -Encoding UTF8
}
LotPings beat me to this by 10 minutes with a virtually identical answer, but I'm leaving this for the 'passing an empty file to the pipeline' bit that I have. I also realize after the fact that you don't need a pipeline variable for that same reason, as you only need it if you pass things through the pipeline within the loop.
If all you want to do is change the encoding I would use a ForEach($x in $y){} loop, or a ForEach-Object{} loop with a PipelineVariable on the Get-ChildItem. I'll show that since I think pipeline variables are under used. I would also not read the file and pipe it to something, since if the file is empty you won't create a new file as nothing is passed down the pipeline.
Get-ChildItem *.csv -Recurse -PipelineVariable File | ForEach-Object{
Set-Content -Value (Get-Content $File.FullName -Encoding Unicode) -Path {Join-Path $File.Directory "UTF8$($File.Name)"} -Encoding UTF8
}
if you specify the file extension at the end of Get-ChildItem.
This will get only the files with the .csv extension.
By specifying the File path in Out-File it will send it to the specified directory.
Get-ChildItem -Path C:\folder\*.csv -Recurse | ForEach-Object {Get-Content -Encoding Unicode $_ | Out-File -FilePath C:\Folder -Encoding UTF8 "UTF8$_"}
My Old Bat file
Copy F:\File.hdr+F:*.csv F:\FinalOutput.csv
the HDR file is a single entry file that has only header data for the CSV files
Is there a way to perform this in PowerShell (to combine all the CSV files into a single file)?
Here is my powershell script that doesn't work
$CSVFolder = 'F:\Input\';
$OutputFile = 'F:\Output\NewOutput.csv';
$CSV= #();
Get-ChildItem -Path $CSVFolder -Filter *.inv | ForEach-Object {
$CSV += #(Import-Csv -Path $CSVFolder\$_)
}
$CSVHeader = Import-Csv 'F:\Input\Headings.hdr'
$CSV = $CSVHeader + $CSV
$CSV | Export-Csv -Path $OutputFile -NoTypeInformation -Force;
I get the list of FileNames that are exported and not the content of the Files.
The script is also modifying the date/time stamp on my INV files. It shouldn't be doing that.
You can skip the whole CSV bit if you just append the files as you would before.
Something like this should work:
# First we create the new file and add the header.
get-content $headerfile | set-content $outputfile
# Then we get the input files, read them out with get-content
# and append them to the output file (add-content).
get-childitem -path $csvfolder *.inv | get-content | add-content $outputfile
The CSV commandlets are handy if you want to be processing the CSV data in your script, but in your case simply appending the files will do the trick. Not bothering with the CSV conversion will be a lot faster as Powershell doesn't have to parse the CSV lines and create PS-objects. It's really fast with pure text though.
Another trick here is how the get-content and add-content are used in the pipeline. Since they are aware of the pipeline you can pass in file objects without having to use a foreach loop. This makes your statements a lot shorter.
How about:
get-childitem *.inv | foreach-object {
import-csv $_ -header (get-content Headings.hdr)
} | export-csv NewOutput.csv -notypeinformation