How to select required columns in csv file using powershell - powershell

I have a csv file with data looking like this.
Div,Date,HomeTeam,AwayTeam,FTHG,FTAG,FTR,HTHG,HTAG,HTR
I1,20/08/16,Juventus,Fiorentina,2,1,H,1,0,H
I1,20/08/16,Roma,Udinese,4,0,H,0,0,D
I1,21/08/16,Atalanta,Lazio,3,4,A,0,3,A
I1,21/08/16,Bologna,Crotone,1,0,H,0,0,D
I am trying to export first six columns like below to a new csv file.
Div,Date,HomeTeam,AwayTeam,FTHG,FTAG
I1,20/08/16,Juventus,Fiorentina,2,1
I1,20/08/16,Roma,Udinese,4,0
I1,21/08/16,Atalanta,Lazio,3,4
I1,21/08/16,Bologna,Crotone,1,0
I am using the following powershell command.
Import-Csv $infile -DeLimiter ","|
Select'Div','Date','HomeTeam','AwayTeam','FTHG','FTAG'|
Export-Csv $trimfile -DeLimiter "," -NoTypeInformation
But my output csv file looks like this
"Div","Date","HomeTeam","AwayTeam","FTHG","FTAG"
"I1","20/08/16","Juventus","Fiorentina","2","1"
"I1","20/08/16","Roma","Udinese","4","0"
"I1","21/08/16","Atalanta","Lazio","3","4"
"I1","21/08/16","Bologna","Crotone","1","0"
The no.of columns are much higher in the actual file. Simplified it here. What am I doing wrong?


Try with this code :
Import-Csv $infile -DeLimiter "," |
Select-Object 'Div','Date','HomeTeam','AwayTeam','FTHG','FTAG' |
ConvertTo-Csv -NoTypeInformation |
ForEach-Object { $_ -replace '"', ""} |
Out-File $trimfile

Related

Powershell adds quotes to splitted CSV file

I'm trying to split a csv file by the first digits of the longitude column. Here is a sample:
X,Y,TYPE,SPEED,DirType,Direction
-44.058251,-19.945982,1,30,1,339
-54.629503,-20.497509,1,30,1,263
-54.646202,-20.496151,1,30,1,86
I have no powershell knowledge but I found some script online and it did what I wanted:
Import-Csv maparadar.csv
| Group-Object -Property {($_.x)[0..2] -join ""}
| Foreach-Object {$path=$_.name+".csv" ; $_.group
| Export-Csv -Path $path -NoTypeInformation}
With this I get output files like -44.csv, -54.csv
But it adds unwanted quotes to every field in the output file like:
"X","Y","TYPE","SPEED","DirType","Direction"
"-46.521991","-23.690235","1","30","1","169"
"-46.670774","-23.756021","1","30","1","281"
"-46.549897","-23.120720","1","30","1","99"
Is there any way I can export the csv without adding those quotes?

The following should provide the desired output:
Import-Csv maparadar.csv |
Group-Object -Property {($_.x)[0..2] -join ""} |
Foreach-Object { $path=$_.name+".csv" ; ($_.group |
ConvertTo-Csv -NoTypeInformation) -Replace '"' |
Set-Content -Path $path }
Explanation:
We replaced your Export-Csv with ConvertTo-Csv, which provides the CSV output to the console/pipeline rather than outputting to the file. Those CSV formatted outputs are sent through the -Replace operator to replace the literal " characters. Finally the formatted output is sent to the desired file using Set-Content -Path $path.

Merges csv files from directory into a single csv file PowerShell

How can I run one single PowerShell script that does the following in series?
Adds a the filename of all csv files in a directory as a column in the end of each file using this script:
Get-ChildItem *.csv | ForEach-Object {
$CSV = Import-CSV -Path $_.FullName -Delimiter ","
$FileName = $_.Name
$CSV | Select-Object *,#{N='Filename';E={$FileName}} | Export-CSV $_.FullName -NTI -Delimiter ","}
Merges all csv files in the directory into a single csv file
Keeping only a header (first row) only from first csv and excluding all other first rows from files.
Similiar to what kemiller2002 has done here, except one script with csv inputs and a csv output.

Bill's answer allows you to combine CSVs, but doesn't tack file names onto the end of each row. I think the best way to do that would be to use the PipelineVariable common parameter to add that within the ForEach loop.
Get-ChildItem \inputCSVFiles\*.csv -PipelineVariable File |
ForEach-Object { Import-Csv $_ | Select *,#{l='FileName';e={$File.Name}}} |
Export-Csv \outputCSVFiles\newOutputFile.csv -NoTypeInformation
That should accomplish what you're looking for.

This is the general pattern:
Get-ChildItem \inputCSVFiles\*.csv |
ForEach-Object { Import-Csv $_ } |
Export-Csv \outputCSVFiles\newOutputFile.csv -NoTypeInformation
Make sure the output CSV file has a different filename pattern, or use a different directory name (like in this example).

If your csv files dont have always same header you can do it :
$Dir="c:\temp\"
#get header first csv file founded
$header=Get-ChildItem $Dir -file -Filter "*.csv" | select -First 1 | Get-Content -head 1
#header + all rows without header into new file
$header, (Get-ChildItem $Dir -file -Filter "*.csv" | %{Get-Content $_.fullname | select -skip 1}) | Out-File "c:\temp\result.csv"

Extract differences of CSV files into a seperate file

I have a CSV file (with headers) filled with assortment data. The file will be updated once every day. I need to find the differences in those files (the old and the new one) and extract them into a separate file.
For instance: in the old file there could be a price of "18,50" and now it's an updated one of "17,90". The script should now extract this row into a new file.
So far, I was able to import both CSV files (via Import-Csv) but my current solution is to compare each row by findstr.
The problems are:
In 9 of 10 cases the strings are too long to compare.
What if a new row will be inserted - I guess the comparison wouldn't work any longer if the row isn't inserted at the end of the file.
My current code is:
foreach ($oldData in (Import-Csv $PSScriptRoot\old.csv -Delimiter ";" -Encoding "default")) {
foreach ($newData in (Import-Csv $PSScriptRoot\new.csv -Delimiter ";" -Encoding "default")) {
findstr.exe /v /c:$oldData $newData > $PSScriptRoot\diff.txt
}
}

Read both files into separate variables and use Compare-Object for the comparison:
$fields = 'idArtikel', 'Preis', ...
$csv1 = Import-Csv $PSScriptRoot\old.csv -Delimiter ';'
$csv2 = Import-Csv $PSScriptRoot\new.csv -Delimiter ';'
Compare-Object -ReferenceObject $csv1 -DifferenceObject $csv2 -Property $fields -PassThru | Where-Object {
$_.SideIndicator -eq '=>'
} | Select-Object $fields | Export-Csv 'C:\path\to\diff.csv' -Delimiter ';'

$csv1 | Join $csv2 idArtikel -Merge {$Right.$_} | Export-CSV 'C:\path\to\diff.csv' -Delimiter ';'
For details on Join (Join-Object), see: https://stackoverflow.com/a/45483110/1701026

Spliting a csv file based on the value of one column

I have a csv file that I have to split based on the value of a column.
I'm using the following script to do so:
Import-Csv test.csv | Group-Object -Property "Nr dep" |
Foreach-Object {$path=$_.name+".csv" ; $_.group |
Export-Csv -Path E:\PowerShell\script\$path -NoTypeInformation}
The file is split to files based on the Nr dep value column but with quotes and it works only with comma delimited csv files.
I tried to use -replace but still no result (maybe I'am writing it in a bad manner)
Import-Csv test.csv | Group-Object -Property "Nr dep" |
Foreach-Object {$path=$_.name+".csv" ; ($_.group |
ConvertTo-Csv -NoTypeInformation) -replace '"', "" | Out-File E:\PowerShell\script\$path -Force}
1) How can I make the delimitation a semi comma instead of a comma
2) how can I get rid of the quotes
3) is it possible to have an .xlsx outfile instead of a .csv file

1) You can specify the delimiter by using the -Delimiter parameter:
Import-Csv test.csv -Delimiter ';'
2) By using -replace '"' which you already do.
3) You will need a framework // application for that.

PowerShell write integer to file after x number of tabs

I'm sure this is ridiculously easy, but I'm a noob and trying to learn PowerShell.
I want to write an integer to each line of a tab delimited file, i.e. each line has 20 tabs; put a 1 after the nth tab.
No need to overwrite what's already there because in the current scenario there isn't anything.
Thanks!

If there is a header line then just import the file as a CSV, run it through a ForEach-Object loop and set that column to the integer that you want, then export the CSV again.
Import-CSV $File -Delimiter "`t" | ForEach{$_.ColumnName = $Integer} | Export-CSV $File -Delimiter "`t" -NoTypeInfo
If there is no header you could do the same thing and define your own headers. Except you would use ConvertTo-CSV instead of Export-CSV and then use Select to skip the header row, and use Set-Content to write the file. For my example I set the 7th column to $Integer.
$Headers = 1..20|ForEach{"Col$_"}
Import-CSV $File -Delimiter "`t" -Header $Headers | ForEach{$_.Col7 = $Integer} | ConvertTo-CSV -Delimiter "`t" -NoTypeInfo | Select -Skip 1 | Set-Content $File