Below is the data I have in 2 csv
CSV_1
"Username","UserCreationStatus","GroupAdditionStatus"
"WA92J4063641OAD","Success","Success"
CSV_2
"GroupName","GroupCreationStatus"
"WA92GRP-ADAdminAccount-CAP-OAD","Already exist"
I need to merge them in to single csv file like below
"Username","UserCreationStatus","GroupAdditionStatus","GroupName","GroupCreationStatus"
"WA92J4063641OAD","Success","Success","WA92GRP-ADAdminAccount-CAP-OAD","Already exist"
I tried the below code
Get-ChildItem -Path $RootPath -Filter *.csv | Select-Object * | Import-Csv | Export-Csv $RootPath\merged.csv -NoTypeInformation -Append
But getting below error
Import-Csv : You must specify either the -Path or -LiteralPath parameters, but not both.
Please let me know what is wrong here
You could do simething like this.
It does not work but shows the logic.
Let me know, if you have any questions.
$CSV1 = ".\first.csv"
$CSV2 = ".\second.csv"
$NewCSV = ".\new.csv"
$Data1 = Get-Content -Path $CSV1
$Data2 = Get-Content -Path $CSV2
foreach ($Line in $CSV1)
{
Add-Content -Value "$($Line),$($CSV2[$index])" -Path $NewCSV
}
Related
I'm trying (badly) to work through combining CSV files into one file and prepending a column that contains the file name. I'm new to PowerShell, so hopefully someone can help here.
I tried initially to do the well documented approach of using Import-Csv / Export-Csv, but I don't see any options to add columns.
Get-ChildItem -Filter *.csv | Select-Object -ExpandProperty FullName | Import-Csv | Export-Csv CombinedFile.txt -UseQuotes Never -NoTypeInformation -Append
Next I'm trying to loop through the files and append the name, which kind of works, but for some reason this stops after the first row is generated. Since it's not a CSV process, I have to use the switch to skip the first title row of each file.
$getFirstLine = $true
Get-ChildItem -Filter *.csv | Where-Object {$_.Name -NotMatch "Combined.csv"} | foreach {
$filePath = $_
$collection = Get-Content $filePath
foreach($lines in $collection) {
$lines = ($_.Basename + ";" + $lines)
}
$linesToWrite = switch($getFirstLine) {
$true {$lines}
$false {$lines | Select -Skip 1}
}
$getFirstLine = $false
Add-Content "Combined.csv" $linesToWrite
}
This is where the -PipelineVariable parameter comes in real handy. You can set a variable to represent the current iteration in the pipeline, so you can do things like this:
Get-ChildItem -Filter *.csv -PipelineVariable File | Where-Object {$_.Name -NotMatch "Combined.csv"} | ForEach-Object { Import-Csv $File.FullName } | Select *,#{l='OriginalFile';e={$File.Name}} | Export-Csv Combined.csv -Notypeinfo
Merging your CSVs into one and adding a column for the file's name can be done as follows, using a calculated property on Select-Object:
Get-ChildItem -Filter *.csv | ForEach-Object {
$fileName = $_.Name
Import-Csv $_.FullName | Select-Object #{
Name = 'FileName'
Expression = { $fileName }
}, *
} | Export-Csv path/to/merged.csv -NoTypeInformation
I am trying to Remove unnecessary commas in a column in the CSV file. For now, I know a few issues and hard-coded it, But I wanted the code to be dynamic. Any suggestions are greatly appreciated.
$FilePath = "C:\Test\"
Get-ChildItem $FilePath -Filter .csv | ForEach-Object {
(Get-Content $_.FullName -Raw) | Foreach-Object {
$_ -replace ',"Frederick, Fred",' , ',"Frederick Fred",' `
-replace ',"Brian, Josiah",' , ',"Brian Josiah",' `
-replace ',"Lisinopril ,Tablet / 20MG",' , ',"Lisinopril Tablet / 20MG",'
} | Set-Content $_.FullName
}
Try this, also note that I worked with the csv sample that you gave here.It might not work with other csv files.
also make sure that you change the path of %YOURCSVFILE% to the real path of your file
#import the csv
$csv = Import-Csv -Path %YOURCSVFILE% -Delimiter ','
#going each row and replacing commas
foreach ($desc in $csv){
$desc.Desc = $desc.Desc -replace ',',''
}
#exporting the csv
$csv | Export-csv -NoTypeInformation "noCommas.csv"
Here's a few more alteratives for you:
Method 1. Loop through the rows with foreach(..) and capture the output:
$result = foreach ($row in (Import-Csv -Path 'D:\Test\FileWithCommasInDescription.csv')) {
$row.Desc = $row.Desc -replace ','
$row # output the updated item
}
$result | Export-Csv -Path 'D:\Test\FileWithoutCommasInDescription.csv' -NoTypeInformation
Method 2. Use ForEach-Object and the automatic variable $_. Pipe the results through:
Import-Csv -Path 'D:\Test\FileWithCommasInDescription.csv' | ForEach-Object {
$_.Desc = $_.Desc -replace ','
$_ # output the updated item
} | Export-Csv -Path 'D:\Test\FileWithoutCommasInDescription.csv' -NoTypeInformation
Method 3. Use a calculated property:
Import-Csv -Path 'D:\Test\FileWithCommasInDescription.csv' |
Select-Object ID, #{Name = 'Desc'; Expression = {$_.Desc -replace ','}}, Nbr -ExcludeProperty Desc |
Export-Csv -Path 'D:\Test\FileWithoutCommasInDescription.csv' -NoTypeInformation
All will result in a new CSV file
"ID","Desc","Nbr"
"12","Frederick Fred","11"
"21","Brian Josiah","31"
"13","Lisinopril Tablet / 20MG","17"
Below is one of the file data I have in text file
B97SW | CHANGED | rc=0 >>
Server Name";"SystemFolderPath";"IdenityReference";"FileSystemRights";"Vulnerable
B97SW;C:\Windows\system32;CREATOR OWNER;268435456;No
B97SW;C:\Windows\system32;NT AUTHORITY\SYSTEM;268435456;No
B97SW;C:\Windows\system32;NT AUTHORITY\SYSTEM;Modify, Synchronize;No
........
I am trying to replace ";" with "," and write to csv.
Below is the code I wrote but it is not writing the data in csv.
$FileList = Get-ChildItem -Path "C:\Files"
$props=[ordered]#{
ServerName=''
SystemFolderPath=''
IdenityReference=''
FileSystemRights=''
Vulnerable=''
}
New-Object PsObject -Property $props |
Export-Csv C:\2021.csv -NoTypeInformation
$FinalData = #()
foreach($n_file in $FileList)
{
$FileName = $n_file.FullName
$FileContent = Get-Content -Path $FileName | Select-Object -Skip 2
foreach($line in $FileContent)
{
$line = $line -replace(";",",")
$line | Export-Csv -Path C:\2021.csv -Append -NoTypeInformation -Force
}
}
output I am getting
"ServerName","SystemFolderPath","IdenityReference","FileSystemRights","Vulnerable"
"","","","",""
,,,,
,,,,
Please let me know what is wrong I am doing here.
$line | Export-Csv -Path C:\2021.csv -Append -NoTypeInformation -Force
This doesn't work because Export-Csv expects object(s) with properties, but $line is just a string. You need to parse it into an object first, using ConvertFrom-Csv.
Try this:
$FileList = Get-ChildItem -Path "C:\Files"
foreach($n_file in $FileList)
{
$FileName = $n_file.FullName
Get-Content -Path $FileName |
Select-Object -Skip 2 |
ConvertFrom-Csv -Delimiter ';' -Header ServerName, SystemFolderPath, IdenityReference, FileSystemRights, Vulnerable |
Export-Csv -Path C:\2021.csv -Append -NoTypeInformation -Force
}
As we have skipped the original headers, we have to supply these through the -Header parameter of ConvertFrom-Csv.
Your CSV file is goofed up in two ways. First, there is a line of garbage before the header line. Second, in the header line the semi-colons are surrounded by double quotes. The correct form would be to surround the header names with quotes instead.
Once these format errors are fixed, you can read the csv file with this:
Import-Csv myfile.csv -delimiter ";"
Or if you want to produce a comma delimited csv file, try this:
Import-Csv myfile.csv -delimiter ";" | Export-Csv newfile.csv
The result will be correct but it will have a lot of unnecessary double quotes.
I'm trying to merge CSV files in Powershell. I've read numerous answers here but I'm stuck on this problem.
I have a list of csv files, 2 difficulties :
[A] each file has a metadataline, the headers are in the second line.
[B] each file has the same structure, but sometimes quotes surround the column to escape the content.
Thanks to this question : Merging multiple CSV files into one using PowerShell,
I'm able to solve these two problems individually.
However, I'm stuck at combining the solutions.
Partial solution A
Skips every metadata line as well as header for subsequent files
Adapting the answer from kemiller2002:
$sourcefilefolderPath = "C:\CSV_folder"
$destinationfilePath = "C:\appended_files.csv"
$getHeader = $true
Get-ChildItem -Path $sourcefilefolderPath -Filter *.csv -Recurse| foreach {
$filePath = $_.FullName
$lines = $lines = Get-Content $filePath
$linesToWrite = switch($getHeader) {
$true {$lines | Select -Skip 1} # skips only the metadata line
$false {$lines | Select -Skip 2} # skips both the metadata line as well as headers
}
$getHeader = False
Add-Content $destination_file $linesToWrite
}
The problem : Import-Csv $destination_file give inconsistent results, as the quoting can be different for each source file.
Partial solution B
handles successfully random quoted columns
Solution provided by stinkyfriend.
Import-Csv seems to import the data gracefully when the column quoting, however different from one column to the other, is consistent for each line of the source file.
I could not combine this solution with the one above.
Get-ChildItem -Path $sourcefilefolderPath -File -Filter *.csv -Recurse |
Select-Object -ExpandProperty FullName |
Import-Csv |
Export-Csv $destination_file -NoTypeInformation -Append
Thanks a lot for your help !
Solution C
produces blank file on my PC
using suggestion from Mathias R. Jessen
Get-ChildItem -Path $sourcefilefolderPath -File -Filter *.csv -Recurse | foreach {
Write-Host $_.FullName |
Get-Content $_.FullName | Select-Object -Skip 1 | ConvertFrom-Csv |
Export-Csv $destinationfilePath -NoTypeInformation -Append
--- EDIT ---
RESULT
I could solve the problem by creating appended_files.csv using the first matching source file and then append to it.
$pattern_sourceFile = "*.csv*"
$list_files = Get-ChildItem -Path $sourcefilefolderPath -File -Recurse | Where {
$_FullName -match $pattern_sourcefile }
Get-Content $list_files[0].FullName |
Select-Object -Skip 1 | # skips metadataline
ConvertFrom-Csv | Export-Csv $destinationfilePath -NoTypeInformation
$list_files |
Select-Object -Skip 1 | # skips $array_files[0]
foreach { Get-Content $_.FullName |
Select-Object -Skip 1 | # skips metadata line
ConvertFrom-Csv |
Export-Csv $destinationfilePath -NoTypeInformation -Append }
Use ConvertFrom-Csv instead of Import-Csv, this way you can still control how many lines to skip:
Get-Content $file |Select -Skip 1 |ConvertFrom-Csv
So you'll end up with something like:
$sourcefilefolderPath = "C:\CSV_folder"
$destinationfilePath = "C:\appended_files.csv"
Get-ChildItem -Path $sourcefilefolderPath -Filter *.csv -Recurse | foreach {
Get-Content $_.FullName |Select-Object -Skip 1 |ConvertFrom-Csv |Export-Csv -Path $destinationfilePath -NoTypeInformation -Append
}
I need to take a slew of csv files from a directory and get them into an array in Powershell (to eventually manipulate and write back to a CSV).
The problem is there are 5 file types. I need around 8 columns from each. The columns are essentially the same, but have different headings.
Is there an easy way to do this? I started creating a custom object with my 8 fields, looping through the files importing each one, looking at the filename (which tells me the column names I need) and then a bunch of ifs to add it to my custom object array.
I was wondering if there is a simpler way...like with a template saying which columns from each file.
wound up doing this. It may have not been the most efficient, but works. I wound up writing out each file separately and combining at the end as PS really got bogged down (over a million rows combined).
$Newcsv = #()
$path = "c:\scrap\BWFILES\"
$files = gci -path $path -recurse -filter *.csv | Where-Object { ! ($_.psiscontainer) }
$counter=1
foreach($file in $files)
{
$csv = Import-Csv $file.FullName
if ($file.Name -like '*SAV*')
{
$Newcsv = $csv | Select-Object #{Name="PRODUCT";Expression={"SV"}},DMBRCH,DMACCT,DMSHRT
}
if ($file.Name -like '*TIME*')
{
$Newcsv = $csv | Select-Object #{Name="PRODUCT";Expression={"TM"}},TMBRCH,TMACCT,TMSHRT
}
if ($file.Name -like '*TRAN*')
{
$Newcsv = $csv | Select-Object #{Name="PRODUCT";Expression={"TR"}},DMBRCH,DMACCT,DMSHRT
}
if ($file.Name -like '*LN*')
{
$Newcsv = $csv | Select-Object #{Name="PRODUCT";Expression={"LN"}},LNBRCH,LNNOTE,LNSHRT
}
$Newcsv | Export-Csv "C:\scrap\$file.name$counter.csv" -force -notypeinformation
$counter++
}
get-childItem "c:\scrap\*.csv" | foreach {
$filePath = $_
$lines = $lines = Get-Content $filePath
$linesToWrite = switch($getFirstLine) {
$true {$lines}
$false {$lines | Select -Skip 1}
}
$getFirstLine = $false
Add-Content "c:\scrap\combined.csv" $linesToWrite
}
With a hashtable for reference, a little RegEx matching, and using the automatic variable $Matches in a ForEach-Object loop (alias % used) that could all be shortened to:
$path = "c:\scrap\BWFILES\"
$Reference = #{
'SAV' = 'SV'
'TIME' = 'TM'
'TRAN' = 'TR'
'LN'='LN'
}
Set-Content -Value "PRODUCT,BRCH,ACCT,SHRT" -Path 'c:\scrap\combined.csv'
gci -path $path -recurse -filter *.csv | Where-Object { !($_.psiscontainer) -and $_.Name -match ".*(SAV|TIME|TRAN|LN).*"}|%{
$Product = $Reference[($Matches[1])]
Import-CSV $_.FullName | Select-Object #{Name="PRODUCT";Expression={$Product}},*BRCH,#{l='Acct';e={$_.LNNOTE, $_.DMACCT, $_.TMACCT|?{$_}}},*SHRT | ConvertTo-Csv -NoTypeInformation | Select -Skip 1 | Add-Content 'c:\scrap\combined.csv'
}
That should produce the exact same file. Only kind of tricky part was the LNNOTE/TMACCT/DMACCT field since obviously you can't just do the same as like *SHRT.