Remove unnecessary commas in a column in csv file by using PowerShell - powershell

I am trying to Remove unnecessary commas in a column in the CSV file. For now, I know a few issues and hard-coded it, But I wanted the code to be dynamic. Any suggestions are greatly appreciated.
$FilePath = "C:\Test\"
Get-ChildItem $FilePath -Filter .csv | ForEach-Object {
(Get-Content $_.FullName -Raw) | Foreach-Object {
$_ -replace ',"Frederick, Fred",' , ',"Frederick Fred",' `
-replace ',"Brian, Josiah",' , ',"Brian Josiah",' `
-replace ',"Lisinopril ,Tablet / 20MG",' , ',"Lisinopril Tablet / 20MG",'
} | Set-Content $_.FullName
}

Try this, also note that I worked with the csv sample that you gave here.It might not work with other csv files.
also make sure that you change the path of %YOURCSVFILE% to the real path of your file
#import the csv
$csv = Import-Csv -Path %YOURCSVFILE% -Delimiter ','
#going each row and replacing commas
foreach ($desc in $csv){
$desc.Desc = $desc.Desc -replace ',',''
}
#exporting the csv
$csv | Export-csv -NoTypeInformation "noCommas.csv"

Here's a few more alteratives for you:
Method 1. Loop through the rows with foreach(..) and capture the output:
$result = foreach ($row in (Import-Csv -Path 'D:\Test\FileWithCommasInDescription.csv')) {
$row.Desc = $row.Desc -replace ','
$row # output the updated item
}
$result | Export-Csv -Path 'D:\Test\FileWithoutCommasInDescription.csv' -NoTypeInformation
Method 2. Use ForEach-Object and the automatic variable $_. Pipe the results through:
Import-Csv -Path 'D:\Test\FileWithCommasInDescription.csv' | ForEach-Object {
$_.Desc = $_.Desc -replace ','
$_ # output the updated item
} | Export-Csv -Path 'D:\Test\FileWithoutCommasInDescription.csv' -NoTypeInformation
Method 3. Use a calculated property:
Import-Csv -Path 'D:\Test\FileWithCommasInDescription.csv' |
Select-Object ID, #{Name = 'Desc'; Expression = {$_.Desc -replace ','}}, Nbr -ExcludeProperty Desc |
Export-Csv -Path 'D:\Test\FileWithoutCommasInDescription.csv' -NoTypeInformation
All will result in a new CSV file
"ID","Desc","Nbr"
"12","Frederick Fred","11"
"21","Brian Josiah","31"
"13","Lisinopril Tablet / 20MG","17"

Related

customize a CSV-file in PowerShell and export it

We are supposed to edit a CSV-file in PowerShell and export the file afterwards
The CSV-file contains:
"ID";"date";"number"
"YYY-12345";"24.01.2023";"123456910"
Now we should add "-001" after the ID-block..but for the whole column
thats the code for now:
$folder_csv = 'C:\Abschluss'
$folder_fileout = 'C:\Abschluss\Ausgabe'
$files = Get-ChildItem $folder_csv -File -Filter *.csv
foreach ($file in $files) {
$data = Import-csv $($file.Fullname) -Delimiter ";"
foreach($dataset in $data) {
$data.ID + "-001"
$data | export-csv "$folder_fileout\test.txt" -Delimiter ";" -NoType -Encoding UTF8 -append
}
}
it shows the results in the console of PowerShell but not in the created .txt-file.
You're pretty close with your code, the main issue is that you're currently doing $data.ID + "-001" and $data is actually the complete array of objects, you want to refer to $dataset instead (the object being enumerated). Aside from that, it seems you're looking to merge all Csvs into one, hence you could leverage the PowerShell pipeline using an outer ForEach-Object loop instead of foreach:
$folder_csv = 'C:\Abschluss'
$folder_fileout = 'C:\Abschluss\Ausgabe'
Get-ChildItem $folder_csv -File -Filter *.csv | ForEach-Object {
foreach($line in $_ | Import-csv -Delimiter ';') {
# update the Id property of this object
$line.Id = $line.Id + '-001'
# output the updated object
$line
}
} | Export-Csv "$folder_fileout\test.txt" -Delimiter ";" -NoTypeInformation -Encoding UTF8

Powershell Piping Variable to Export-CSV not giving the data I'm looking for

In this script I'm getting a collection of CSV files, performing a replace, storing in an empty array and attempting to export it to CSV.
$CSVFiles = Get-ChildItem "C:\GALIC\Test\Test2\WindowsLists\*.csv" -Exclude M*
$AllJobsList = $CSVFiles | ForEach { (Import-CSV $_ -Delimiter ',' | Select 'Agent', 'Name', 'Folder' | Where-Object {$_.Agent -like "*AGENTGROUP*"})}
$UpdatedGroupsList = #()
$AllJobsList | Export-Csv -Path "C:\GALIC\Test\Test2\WindowsLists\FullJobs-Test.csv" -NoTypeInformation -Force
**$CSVContent = Get-Content "C:\GALIC\Test\Test2\WindowsLists\FullJobs-Test.csv"
foreach($line in $CSVContent)
{
if($line.Contains('|') -and $line.Contains('HOSTG'))
{
#Write-Host $line
$null = $line.Replace('|', '').Replace('HOSTG', '')
#Write-Host $LineReplace
$UpdatedGroupsList += $line
}
}
$UpdatedGroupsList | Export-CSV -Path "C:\GALIC\Test\Test2\WindowsLists\UpdatedFullJobs.csv" -NoTypeInformation -Force**
($CSVContent on down is what's giving me issues.)
After opening the CSV file, the content looks nothing like what I'm expecting. Any ideas/suggestions?
enter image description here

Powershell - Combine CSV files and append a column

I'm trying (badly) to work through combining CSV files into one file and prepending a column that contains the file name. I'm new to PowerShell, so hopefully someone can help here.
I tried initially to do the well documented approach of using Import-Csv / Export-Csv, but I don't see any options to add columns.
Get-ChildItem -Filter *.csv | Select-Object -ExpandProperty FullName | Import-Csv | Export-Csv CombinedFile.txt -UseQuotes Never -NoTypeInformation -Append
Next I'm trying to loop through the files and append the name, which kind of works, but for some reason this stops after the first row is generated. Since it's not a CSV process, I have to use the switch to skip the first title row of each file.
$getFirstLine = $true
Get-ChildItem -Filter *.csv | Where-Object {$_.Name -NotMatch "Combined.csv"} | foreach {
$filePath = $_
$collection = Get-Content $filePath
foreach($lines in $collection) {
$lines = ($_.Basename + ";" + $lines)
}
$linesToWrite = switch($getFirstLine) {
$true {$lines}
$false {$lines | Select -Skip 1}
}
$getFirstLine = $false
Add-Content "Combined.csv" $linesToWrite
}
This is where the -PipelineVariable parameter comes in real handy. You can set a variable to represent the current iteration in the pipeline, so you can do things like this:
Get-ChildItem -Filter *.csv -PipelineVariable File | Where-Object {$_.Name -NotMatch "Combined.csv"} | ForEach-Object { Import-Csv $File.FullName } | Select *,#{l='OriginalFile';e={$File.Name}} | Export-Csv Combined.csv -Notypeinfo
Merging your CSVs into one and adding a column for the file's name can be done as follows, using a calculated property on Select-Object:
Get-ChildItem -Filter *.csv | ForEach-Object {
$fileName = $_.Name
Import-Csv $_.FullName | Select-Object #{
Name = 'FileName'
Expression = { $fileName }
}, *
} | Export-Csv path/to/merged.csv -NoTypeInformation

Convert text file data to csv powershell

Below is one of the file data I have in text file
B97SW | CHANGED | rc=0 >>
Server Name";"SystemFolderPath";"IdenityReference";"FileSystemRights";"Vulnerable
B97SW;C:\Windows\system32;CREATOR OWNER;268435456;No
B97SW;C:\Windows\system32;NT AUTHORITY\SYSTEM;268435456;No
B97SW;C:\Windows\system32;NT AUTHORITY\SYSTEM;Modify, Synchronize;No
........
I am trying to replace ";" with "," and write to csv.
Below is the code I wrote but it is not writing the data in csv.
$FileList = Get-ChildItem -Path "C:\Files"
$props=[ordered]#{
ServerName=''
SystemFolderPath=''
IdenityReference=''
FileSystemRights=''
Vulnerable=''
}
New-Object PsObject -Property $props |
Export-Csv C:\2021.csv -NoTypeInformation
$FinalData = #()
foreach($n_file in $FileList)
{
$FileName = $n_file.FullName
$FileContent = Get-Content -Path $FileName | Select-Object -Skip 2
foreach($line in $FileContent)
{
$line = $line -replace(";",",")
$line | Export-Csv -Path C:\2021.csv -Append -NoTypeInformation -Force
}
}
output I am getting
"ServerName","SystemFolderPath","IdenityReference","FileSystemRights","Vulnerable"
"","","","",""
,,,,
,,,,
Please let me know what is wrong I am doing here.
$line | Export-Csv -Path C:\2021.csv -Append -NoTypeInformation -Force
This doesn't work because Export-Csv expects object(s) with properties, but $line is just a string. You need to parse it into an object first, using ConvertFrom-Csv.
Try this:
$FileList = Get-ChildItem -Path "C:\Files"
foreach($n_file in $FileList)
{
$FileName = $n_file.FullName
Get-Content -Path $FileName |
Select-Object -Skip 2 |
ConvertFrom-Csv -Delimiter ';' -Header ServerName, SystemFolderPath, IdenityReference, FileSystemRights, Vulnerable |
Export-Csv -Path C:\2021.csv -Append -NoTypeInformation -Force
}
As we have skipped the original headers, we have to supply these through the -Header parameter of ConvertFrom-Csv.
Your CSV file is goofed up in two ways. First, there is a line of garbage before the header line. Second, in the header line the semi-colons are surrounded by double quotes. The correct form would be to surround the header names with quotes instead.
Once these format errors are fixed, you can read the csv file with this:
Import-Csv myfile.csv -delimiter ";"
Or if you want to produce a comma delimited csv file, try this:
Import-Csv myfile.csv -delimiter ";" | Export-Csv newfile.csv
The result will be correct but it will have a lot of unnecessary double quotes.

Powershell 5: ConvertTo-Csv a CSV with quotes in some but not all columns

I am building am updating a script which imports a large CSV file and then splits it into lots of separate CSV files based on the value in the first two columns
so POIMP_NL_20210306.csv which contains:
DOC_NUMBER|COMMENTS|ITEM|QTY|SUPPLIER
P-100-1234|JANE|5059585896978|2|"JOES SUPPLIES"
P-100-1234|JANE|5059585896985|2|"JOES SUPPLIES"
P-100-6666|TED|5059585896992|1|"ACTION TOYS"
must be split into POIMP_P-100-1234_JANE.csv containing
P-100-1234|JANE|5059585896978|2|"JOES SUPPLIES"
P-100-1234|JANE|5059585896985|2|"JOES SUPPLIES"
and POIMP_P-100-6666_TED.csv
P-100-6666|TED|5059585896992|1|"ACTION TOYS"
The problem I am trying to solve is preserving the quotes in just the SUPPLIER column
Since ConvertTo-Csv adds quotes to everything, I use a % { $_ -replace '"', ""} to remove these all before the out-file is created but of course it removes these from the SUPPLIER column 2
Here is my script which perfectly splits the big file into smaller files by DOC_NUMBER and COMMENTS but removes all quotes:
$basePath = "C:\"
$archivePath = "$basePath\archive\"
$todaysDate = $(get-date -Format yyyyMMdd)
$todaysFiles = #(
(Get-ChildItem -Path $basePath | Where-Object { $_.Name -match 'POIMP_' + $todaysDate })
)
cd $basePath
foreach ($file in $todaysFiles ) {
$fileName = $file.ToString()
Import-Csv $fileName -delimiter "|" | Group-Object -Property "DOC_NUMBER","COMMENTS" |
Foreach-Object {
$newName = $_.Name -replace ",","_" -replace " ",""; $path=$fileName.SubString(0,8) + $newName+".csv" ; $_.group |
ConvertTo-Csv -NoTypeInformation -delimiter "|" | % { $_ -replace '"', ""} | out-file $path -fo -en ascii
}
Rename-Item $fileName -NewName ([io.path]::GetFileNameWithoutExtension("$fileName") + "_Original.csv")
Move-Item (Get-ChildItem -Path $basePath | Where-Object { $_.Name -match '_Original' }) $archivePath -force
}
And here is another script which I found online and amended and which successfully leaves quotes in just the SUPPLIER column by first adding double back ticks and then replacing these with quotes after all others have been removed
$ImportedCSV = Import-CSV "C:\POIMP_NL_20210306.csv" -delimiter "|"
$NewCSV = Foreach ($Entry in $ImportedCsv) {
$Entry.SUPPLIER = '¬¬' + $Entry.SUPPLIER + '¬¬'
$Entry
}
$NewCSV |
ConvertTo-Csv -NoTypeInformation -delimiter "|" | % { $_ -replace '"', ""} | % { $_ -replace '¬¬', '"'} | out-file "C:\updatedPO.csv" -fo -en ascii
I just can't merge these scripts to achieve the desired result as I can't seem to reference the correct object. I'd really appreciate your help! Thanks
Any good CSV reader should be able to handle quotes around csv fields, even when not really needed.
Having said that, It is your explicit wish to only have quotes around the field in the SUPPLIER column. (Note, in your example there is a trailing space after that column name)
In this case, I think this would help.
Not only does it surround the SUPPLIER fields with quotes, but also saves the data as separate files using the values from column DOC_NUMBER and COMMENTS per group found in the csv
$path = 'D:\Test'
$fileIn = Join-Path -Path $path -ChildPath 'POIMP_NL_20210306.csv'
# import the csv file and group first two columns
Import-Csv -Path $fileIn -Delimiter '|' | Group-Object -Property "DOC_NUMBER","COMMENTS" | ForEach-Object {
$headerDone = $false
$data = foreach ($item in $_.Group) {
if (!$headerDone) {
$item.PsObject.Properties.Name -join '|'
$headerDone = $true
}
$item.SUPPLIER = '"{0}"' -f $item.SUPPLIER
$item.PsObject.Properties.Value -join '|'
}
# create a new filename like 'POIMP_P-100-1234_JANE.csv'
$fileOut = Join-Path -Path $path -ChildPath ('POIMP_{0}_{1}.csv' -f $_.Group[0].DOC_NUMBER, $_.Group[0].COMMENTS)
# save the data not using Export-Csv because that will add quotes around everything (in PowerShell 5)
$data | Set-Content -Path $fileOut -Force
}
Output
POIMP_P-100-1234_JANE.csv
DOC_NUMBER|COMMENTS|ITEM|QTY|SUPPLIER
P-100-1234|JANE|5059585896978|2|"JOES SUPPLIES"
P-100-1234|JANE|5059585896985|2|"JOES SUPPLIES"
POIMP_P-100-6666_TED.csv
DOC_NUMBER|COMMENTS|ITEM|QTY|SUPPLIER
P-100-6666|TED|5059585896992|1|"ACTION TOYS"
If you are Powershell 7 or later, you can use
$yourdata | ConvertTo-Csv -NoTypeInformation -QuoteFields "SUPPLIER" -Delimiter "|" |
Out-File ...
or you could use
$yourdata | Export-Csv -NoTypeInformation -QuoteFields "SUPPLIER" `
-Delimiter "|" -Path <path-to-output-file>.csv
You can also use -UseQuotes AsNeeded to let the converter add quoting where it thinks it makes sense, otherwise just specify the fields you want quoted.