Convert text file data to csv powershell - powershell

Below is one of the file data I have in text file
B97SW | CHANGED | rc=0 >>
Server Name";"SystemFolderPath";"IdenityReference";"FileSystemRights";"Vulnerable
B97SW;C:\Windows\system32;CREATOR OWNER;268435456;No
B97SW;C:\Windows\system32;NT AUTHORITY\SYSTEM;268435456;No
B97SW;C:\Windows\system32;NT AUTHORITY\SYSTEM;Modify, Synchronize;No
........
I am trying to replace ";" with "," and write to csv.
Below is the code I wrote but it is not writing the data in csv.
$FileList = Get-ChildItem -Path "C:\Files"
$props=[ordered]#{
ServerName=''
SystemFolderPath=''
IdenityReference=''
FileSystemRights=''
Vulnerable=''
}
New-Object PsObject -Property $props |
Export-Csv C:\2021.csv -NoTypeInformation
$FinalData = #()
foreach($n_file in $FileList)
{
$FileName = $n_file.FullName
$FileContent = Get-Content -Path $FileName | Select-Object -Skip 2
foreach($line in $FileContent)
{
$line = $line -replace(";",",")
$line | Export-Csv -Path C:\2021.csv -Append -NoTypeInformation -Force
}
}
output I am getting
"ServerName","SystemFolderPath","IdenityReference","FileSystemRights","Vulnerable"
"","","","",""
,,,,
,,,,
Please let me know what is wrong I am doing here.

$line | Export-Csv -Path C:\2021.csv -Append -NoTypeInformation -Force
This doesn't work because Export-Csv expects object(s) with properties, but $line is just a string. You need to parse it into an object first, using ConvertFrom-Csv.
Try this:
$FileList = Get-ChildItem -Path "C:\Files"
foreach($n_file in $FileList)
{
$FileName = $n_file.FullName
Get-Content -Path $FileName |
Select-Object -Skip 2 |
ConvertFrom-Csv -Delimiter ';' -Header ServerName, SystemFolderPath, IdenityReference, FileSystemRights, Vulnerable |
Export-Csv -Path C:\2021.csv -Append -NoTypeInformation -Force
}
As we have skipped the original headers, we have to supply these through the -Header parameter of ConvertFrom-Csv.

Your CSV file is goofed up in two ways. First, there is a line of garbage before the header line. Second, in the header line the semi-colons are surrounded by double quotes. The correct form would be to surround the header names with quotes instead.
Once these format errors are fixed, you can read the csv file with this:
Import-Csv myfile.csv -delimiter ";"
Or if you want to produce a comma delimited csv file, try this:
Import-Csv myfile.csv -delimiter ";" | Export-Csv newfile.csv
The result will be correct but it will have a lot of unnecessary double quotes.

Related

customize a CSV-file in PowerShell and export it

We are supposed to edit a CSV-file in PowerShell and export the file afterwards
The CSV-file contains:
"ID";"date";"number"
"YYY-12345";"24.01.2023";"123456910"
Now we should add "-001" after the ID-block..but for the whole column
thats the code for now:
$folder_csv = 'C:\Abschluss'
$folder_fileout = 'C:\Abschluss\Ausgabe'
$files = Get-ChildItem $folder_csv -File -Filter *.csv
foreach ($file in $files) {
$data = Import-csv $($file.Fullname) -Delimiter ";"
foreach($dataset in $data) {
$data.ID + "-001"
$data | export-csv "$folder_fileout\test.txt" -Delimiter ";" -NoType -Encoding UTF8 -append
}
}
it shows the results in the console of PowerShell but not in the created .txt-file.
You're pretty close with your code, the main issue is that you're currently doing $data.ID + "-001" and $data is actually the complete array of objects, you want to refer to $dataset instead (the object being enumerated). Aside from that, it seems you're looking to merge all Csvs into one, hence you could leverage the PowerShell pipeline using an outer ForEach-Object loop instead of foreach:
$folder_csv = 'C:\Abschluss'
$folder_fileout = 'C:\Abschluss\Ausgabe'
Get-ChildItem $folder_csv -File -Filter *.csv | ForEach-Object {
foreach($line in $_ | Import-csv -Delimiter ';') {
# update the Id property of this object
$line.Id = $line.Id + '-001'
# output the updated object
$line
}
} | Export-Csv "$folder_fileout\test.txt" -Delimiter ";" -NoTypeInformation -Encoding UTF8

Remove unnecessary commas in a column in csv file by using PowerShell

I am trying to Remove unnecessary commas in a column in the CSV file. For now, I know a few issues and hard-coded it, But I wanted the code to be dynamic. Any suggestions are greatly appreciated.
$FilePath = "C:\Test\"
Get-ChildItem $FilePath -Filter .csv | ForEach-Object {
(Get-Content $_.FullName -Raw) | Foreach-Object {
$_ -replace ',"Frederick, Fred",' , ',"Frederick Fred",' `
-replace ',"Brian, Josiah",' , ',"Brian Josiah",' `
-replace ',"Lisinopril ,Tablet / 20MG",' , ',"Lisinopril Tablet / 20MG",'
} | Set-Content $_.FullName
}
Try this, also note that I worked with the csv sample that you gave here.It might not work with other csv files.
also make sure that you change the path of %YOURCSVFILE% to the real path of your file
#import the csv
$csv = Import-Csv -Path %YOURCSVFILE% -Delimiter ','
#going each row and replacing commas
foreach ($desc in $csv){
$desc.Desc = $desc.Desc -replace ',',''
}
#exporting the csv
$csv | Export-csv -NoTypeInformation "noCommas.csv"
Here's a few more alteratives for you:
Method 1. Loop through the rows with foreach(..) and capture the output:
$result = foreach ($row in (Import-Csv -Path 'D:\Test\FileWithCommasInDescription.csv')) {
$row.Desc = $row.Desc -replace ','
$row # output the updated item
}
$result | Export-Csv -Path 'D:\Test\FileWithoutCommasInDescription.csv' -NoTypeInformation
Method 2. Use ForEach-Object and the automatic variable $_. Pipe the results through:
Import-Csv -Path 'D:\Test\FileWithCommasInDescription.csv' | ForEach-Object {
$_.Desc = $_.Desc -replace ','
$_ # output the updated item
} | Export-Csv -Path 'D:\Test\FileWithoutCommasInDescription.csv' -NoTypeInformation
Method 3. Use a calculated property:
Import-Csv -Path 'D:\Test\FileWithCommasInDescription.csv' |
Select-Object ID, #{Name = 'Desc'; Expression = {$_.Desc -replace ','}}, Nbr -ExcludeProperty Desc |
Export-Csv -Path 'D:\Test\FileWithoutCommasInDescription.csv' -NoTypeInformation
All will result in a new CSV file
"ID","Desc","Nbr"
"12","Frederick Fred","11"
"21","Brian Josiah","31"
"13","Lisinopril Tablet / 20MG","17"

Merge Multiple CSV files to one with different columns

Below is the data I have in 2 csv
CSV_1
"Username","UserCreationStatus","GroupAdditionStatus"
"WA92J4063641OAD","Success","Success"
CSV_2
"GroupName","GroupCreationStatus"
"WA92GRP-ADAdminAccount-CAP-OAD","Already exist"
I need to merge them in to single csv file like below
"Username","UserCreationStatus","GroupAdditionStatus","GroupName","GroupCreationStatus"
"WA92J4063641OAD","Success","Success","WA92GRP-ADAdminAccount-CAP-OAD","Already exist"
I tried the below code
Get-ChildItem -Path $RootPath -Filter *.csv | Select-Object * | Import-Csv | Export-Csv $RootPath\merged.csv -NoTypeInformation -Append
But getting below error
Import-Csv : You must specify either the -Path or -LiteralPath parameters, but not both.
Please let me know what is wrong here
You could do simething like this.
It does not work but shows the logic.
Let me know, if you have any questions.
$CSV1 = ".\first.csv"
$CSV2 = ".\second.csv"
$NewCSV = ".\new.csv"
$Data1 = Get-Content -Path $CSV1
$Data2 = Get-Content -Path $CSV2
foreach ($Line in $CSV1)
{
Add-Content -Value "$($Line),$($CSV2[$index])" -Path $NewCSV
}

Powershell 5: ConvertTo-Csv a CSV with quotes in some but not all columns

I am building am updating a script which imports a large CSV file and then splits it into lots of separate CSV files based on the value in the first two columns
so POIMP_NL_20210306.csv which contains:
DOC_NUMBER|COMMENTS|ITEM|QTY|SUPPLIER
P-100-1234|JANE|5059585896978|2|"JOES SUPPLIES"
P-100-1234|JANE|5059585896985|2|"JOES SUPPLIES"
P-100-6666|TED|5059585896992|1|"ACTION TOYS"
must be split into POIMP_P-100-1234_JANE.csv containing
P-100-1234|JANE|5059585896978|2|"JOES SUPPLIES"
P-100-1234|JANE|5059585896985|2|"JOES SUPPLIES"
and POIMP_P-100-6666_TED.csv
P-100-6666|TED|5059585896992|1|"ACTION TOYS"
The problem I am trying to solve is preserving the quotes in just the SUPPLIER column
Since ConvertTo-Csv adds quotes to everything, I use a % { $_ -replace '"', ""} to remove these all before the out-file is created but of course it removes these from the SUPPLIER column 2
Here is my script which perfectly splits the big file into smaller files by DOC_NUMBER and COMMENTS but removes all quotes:
$basePath = "C:\"
$archivePath = "$basePath\archive\"
$todaysDate = $(get-date -Format yyyyMMdd)
$todaysFiles = #(
(Get-ChildItem -Path $basePath | Where-Object { $_.Name -match 'POIMP_' + $todaysDate })
)
cd $basePath
foreach ($file in $todaysFiles ) {
$fileName = $file.ToString()
Import-Csv $fileName -delimiter "|" | Group-Object -Property "DOC_NUMBER","COMMENTS" |
Foreach-Object {
$newName = $_.Name -replace ",","_" -replace " ",""; $path=$fileName.SubString(0,8) + $newName+".csv" ; $_.group |
ConvertTo-Csv -NoTypeInformation -delimiter "|" | % { $_ -replace '"', ""} | out-file $path -fo -en ascii
}
Rename-Item $fileName -NewName ([io.path]::GetFileNameWithoutExtension("$fileName") + "_Original.csv")
Move-Item (Get-ChildItem -Path $basePath | Where-Object { $_.Name -match '_Original' }) $archivePath -force
}
And here is another script which I found online and amended and which successfully leaves quotes in just the SUPPLIER column by first adding double back ticks and then replacing these with quotes after all others have been removed
$ImportedCSV = Import-CSV "C:\POIMP_NL_20210306.csv" -delimiter "|"
$NewCSV = Foreach ($Entry in $ImportedCsv) {
$Entry.SUPPLIER = '¬¬' + $Entry.SUPPLIER + '¬¬'
$Entry
}
$NewCSV |
ConvertTo-Csv -NoTypeInformation -delimiter "|" | % { $_ -replace '"', ""} | % { $_ -replace '¬¬', '"'} | out-file "C:\updatedPO.csv" -fo -en ascii
I just can't merge these scripts to achieve the desired result as I can't seem to reference the correct object. I'd really appreciate your help! Thanks
Any good CSV reader should be able to handle quotes around csv fields, even when not really needed.
Having said that, It is your explicit wish to only have quotes around the field in the SUPPLIER column. (Note, in your example there is a trailing space after that column name)
In this case, I think this would help.
Not only does it surround the SUPPLIER fields with quotes, but also saves the data as separate files using the values from column DOC_NUMBER and COMMENTS per group found in the csv
$path = 'D:\Test'
$fileIn = Join-Path -Path $path -ChildPath 'POIMP_NL_20210306.csv'
# import the csv file and group first two columns
Import-Csv -Path $fileIn -Delimiter '|' | Group-Object -Property "DOC_NUMBER","COMMENTS" | ForEach-Object {
$headerDone = $false
$data = foreach ($item in $_.Group) {
if (!$headerDone) {
$item.PsObject.Properties.Name -join '|'
$headerDone = $true
}
$item.SUPPLIER = '"{0}"' -f $item.SUPPLIER
$item.PsObject.Properties.Value -join '|'
}
# create a new filename like 'POIMP_P-100-1234_JANE.csv'
$fileOut = Join-Path -Path $path -ChildPath ('POIMP_{0}_{1}.csv' -f $_.Group[0].DOC_NUMBER, $_.Group[0].COMMENTS)
# save the data not using Export-Csv because that will add quotes around everything (in PowerShell 5)
$data | Set-Content -Path $fileOut -Force
}
Output
POIMP_P-100-1234_JANE.csv
DOC_NUMBER|COMMENTS|ITEM|QTY|SUPPLIER
P-100-1234|JANE|5059585896978|2|"JOES SUPPLIES"
P-100-1234|JANE|5059585896985|2|"JOES SUPPLIES"
POIMP_P-100-6666_TED.csv
DOC_NUMBER|COMMENTS|ITEM|QTY|SUPPLIER
P-100-6666|TED|5059585896992|1|"ACTION TOYS"
If you are Powershell 7 or later, you can use
$yourdata | ConvertTo-Csv -NoTypeInformation -QuoteFields "SUPPLIER" -Delimiter "|" |
Out-File ...
or you could use
$yourdata | Export-Csv -NoTypeInformation -QuoteFields "SUPPLIER" `
-Delimiter "|" -Path <path-to-output-file>.csv
You can also use -UseQuotes AsNeeded to let the converter add quoting where it thinks it makes sense, otherwise just specify the fields you want quoted.

how to prepend filename to every record in a csv?

How do we prepend the filename to ALL the csv files in a specific directory?
I've got a bunch of csv files that each look like this:
ExampleFile.Csv
2323, alex, gordon
4382, liza, smith
The output I'd like is:
ExampleFile.Csv, 2323, alex, gordon
ExampleFile.Csv, 4382, liza, smith
How do we prepend the filename to ALL the csv files in a specific directory?
I've attempted the following solution:
Get-ChildItem *.csv | ForEach-Object {
$CSV = Import-CSV -Path $_.FullName -Delimiter ","
$FileName = $_.Name
$CSV | Select-Object *,#{E={$FileName}} | Export-CSV $_.FullName -NTI -Delimiter ","
}
However, this did not work because it was altering the first row. (My data does not have a header row). Also, this script will append to each record at the end rather than prepend at the beginning.
You're missing the column header name I think. Take a look at the duplicate (or original, rather) and see Shay's answer. Your Select-Object should look like:
$CSV | Select-Object #{Name='FileName';Expression={"$filename"}},* | Export-Csv -Path $FileName -NoTypeInformation -Delimiter ','
That worked fine for me with multiple CSVs in a directory when using the rest of your sample code verbatim.
If your files do not have headers and the column count is unknown or unpredictable, you can read each line with Get-Content, make the changes, and then use Set-Content to make the update.
Get-ChildItem *.csv | ForEach-Object {
$Filename = $_.Name
$Fullname = $_.FullName
$contents = Get-Content -Path $Fullname | Foreach-Object {
"{0}, {1}" -f $Filename,$_
}
$contents | Set-Content -Path $Fullname
}