Import-Csv Select -Skip - powershell

I have multiple CSV files that need to be merged to one. In every single CSV file there is a header and in the second row some text that I don't need.
I noticed the | Select -Skip 1 statement for the headers. Now I was wondering how I can skip the 3rd row?
I tried this, but this gives me an empty file
Get-ChildItem -Path $CSVFolder -Recurse -Filter "*.csv" | %{
Import-Csv $_.FullName -Header header1, header3, header4 |
Select -Skip 1 | Select -Skip 2
} | Export-Csv "C:\Export\result.csv" -NoTypeInformation

Select-Object doesn't allow you to skip arbitrary rows in between other rows. If you want to remove a particular row from a text input file, you can do so with a counter, e.g. like this:
$cnt = 0
Import-Csv 'C:\path\to\input.csv' |
Where-Object { ($cnt++) -ne 3 } |
Export-Csv 'C:\path\to\output.csv' -NoType
If the records in your input CSV don't have nested line breaks you could also use Get-Content/Set-Content, which is probably a little faster than Import-Csv/Export-Csv (due to less parsing overhead). Increase the line number you want to skip by one to account for the header line.
$cnt = 0
Get-Content 'C:\path\to\input.csv' |
Where-Object { ($cnt++) -ne 4 } |
Set-Content 'C:\path\to\output.csv'

try this
$i=0;
import-csv "C:\temp2\missing.csv" | %{$i++; if ($i -ne 3) {$_}} | export-csv "C:\temp2\result.csv" -NoTypeInformation

If all you are doing si skipping the first the rows in all user cases, just use -skip 3.
Get-Content -Path 'D:\Temp\UserRecord.csv'
# Results
<#
Name Codes
------- ---------
John AJFKC,EFUY
Ben EFOID, EIUF
Alex OIPORE, OUOIJE
#>
# Return all text after row the Header and row 3
(Get-Content -Path 'D:\Temp\UserRecord.csv') |
Select -Skip 3
# Results
<#
Ben EFOID, EIUF
Alex OIPORE, OUOIJE
#>
See also:
Parsing Text with PowerShell (1/3)

Related

PowerShell CSV, take a specific row from each line and combine it into one CSV

I have 300 CSV files all separated in a directory.
I want to get one specific criteria from each CSV and put it into another using PowerShell.
This is the line I have, but doesn't seem to work.
Get-ChildItem -Filter "*Results.csv" | Get-Content | Where-Object {$_.NAME -eq "Cage,Johnny"} | Add-Content "test.csv"
I filtered for the specific CSVs I wanted in my directory with gci, Got the content of each using Get-Content and Where the value is Johnny Cage in the NAME column, and Add-Content into a test.csv file but doesn't work.
Any help would be great!
You need to deserialize your CSV text into objects with properties that can be referenced. Then you can compare the Name property. You can do the following if all your csv files have the same headers.
Get-ChildItem -Filter "*Results.csv" | Foreach-Object {
Import-Csv $_.FullName |
Where-Object {$_.NAME -eq "Cage,Johnny"} } |
Export-Csv "test.csv"
If your CSV files contain different headers, then you have a couple of options. One, you could create your output CSV with all possible headers that exist across all files (or just the headers you want as long as they are the same across all files). Second, you could just output your data rows and have a broken CSV.
# Broken CSV Approach
Get-ChildItem -Filter "*Results.csv" | Foreach-Object {
Import-Csv $_.FullName |
Where-Object {$_.NAME -eq "Cage,Johnny"}} | Foreach-Object {
$_ | ConvertTo-Csv -Notype | Select-Object -Skip 1
} | Add-Content test.csv
I think I got it.
Get-ChildItem -Filter *Results.csv |
ForEach-Object{
Import-Csv $_.NAME | ? { $_.EMPLID -eq "Cage,Johnny"}
} | Export-Csv "test.csv"

export csv rows where duplicate values found in column

I have a csv file where I am trying to export rows into another csv file only where the values in the id column have duplicates.
I have the following csv file...
"id","blablah"
"valOne","valTwo"
"valOne","asdfdsa"
"valThree","valFour"
"valFive","valSix"
"valFive","qwreweq"
"valSeven","valEight"
I need the output csv file to look like the following...
"valOne","valTwo"
"valOne","asdfdsa"
"valFive","valSix"
"valFive","qwreweq"
Here is the code I have so far:
$inputCsv = Import-CSV './test.csv' -delimiter ","
#$output = #()
$inputCsv | Group-Object -prop id, blablah | Where-Object {$_.id -gt 1} |
Select-Object
##{n='id';e={$_.Group[0].id}},
##{n='blablah';e={$_.Group[0].blablah}}
#Export-Csv 'C:\scripts\powershell\output.csv' -NoTypeInformation
#Write-Host $output
#$output | Export-Csv 'C:\scripts\powershell\output.csv' -NoTypeInformation
I've searched multiple how-to's but can't seem to find the write syntax. Can anyone help with this?
Just group on the ID property and if there is more than 1 count in the group then expand those and export.
$inputCsv = Import-CSV './test.csv' -delimiter ","
$inputCsv |
Group-Object -Property ID |
Where-Object count -gt 1 |
Select-Object -ExpandProperty group |
Export-Csv output.csv -NoTypeInformation
output.csv will contain
"id","blablah"
"valOne","valTwo"
"valOne","asdfdsa"
"valFive","valSix"
"valFive","qwreweq"

Use the name of the original file for the output file

I have been able to get this script working but the only thing I cant figure out is to take the original file name and apply it to the output file name with _new added to it. Could you please push me in the write direction?
Import-Csv '\\DESKTOP-QC1GB24\Allpay DD\Processing\*.csv' -Header (1..5|%{"Column$_"}) |
Select-Object Column2,Column3,Column5 -SkipLast 1 |
ConvertTo-Csv -NoTypeInformation |
Select-Object -Skip 1 |
Set-Content -Path "\\DESKTOP-QC1GB24\Allpay DD\Completed\New.CSV"
You can get a list of .csv files before the import and loop through them:
Get-ChildItem -Filter '*.csv' -Path '\\DESKTOP-QC1GB24\Allpay DD\Processing\'|
% {Import-Csv $_.Fullname -Header (1..5|%{"Column$_"}) |
Select-Object Column2,Column3,Column5 -SkipLast 1 |
ConvertTo-Csv -NoTypeInformation |
Select-Object -Skip 1 |
Set-Content -Path "\\DESKTOP-QC1GB24\Allpay DD\Completed\$($_.BaseName)_new.CSV"}
This way $_.Fullname points to each Full Path of an file and $_.BaseName gives you the Name of each file without the extension (in this case .csv). This way you can add the "_new" string during the Set-Content.

Export-Csv adding unwanted header double quotes

I have got a source CSV file (without a header, all columns delimited by a comma) which I am trying split out into separate CSV files based upon the value in the first column and using that column value as the output file name.
Input file:
S00000009,2016,M04 01/07/2016,0.00,0.00,0.00,0.00,0.00,0.00,750.00,0.00,0.00
S00000009,2016,M05 01/08/2016,0.00,0.00,0.00,0.00,0.00,0.00,600.00,0.00,0.00
S00000009,2016,M06 01/09/2016,0.00,0.00,0.00,0.00,0.00,0.00,600.00,0.00,0.00
S00000010,2015,W28 05/10/2015,2275.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00
S00000010,2015,W41 04/01/2016,0.00,0.00,0.00,0.00,0.00,0.00,568.75,0.00,0.00
S00000010,2015,W42 11/01/2016,0.00,0.00,0.00,0.00,0.00,0.00,568.75,0.00,0.00
S00000012,2015,W10 01/06/2015,0.00,0.00,0.00,0.00,0.00,0.00,650.00,0.00,0.00
S00000012,2015,W11 08/06/2015,0.00,0.00,0.00,0.00,0.00,0.00,650.00,0.00,0.00
S00000012,2015,W12 15/06/2015,0.00,0.00,0.00,0.00,0.00,0.00,650.00,0.00,0.00
My PowerShell script looks like this:
Import-Csv INPUT_FILE.csv -Header service_id,year,period,cash_exp,cash_inc,cash_def,act_exp,act_inc,act_def,comm_exp,comm_inc,comm_def |
Group-Object -Property "service_id" |
Foreach-Object {
$path = $_.Name + ".csv";
$_.group | Export-Csv -Path $path -NoTypeInformation
}
Output files:
S00000009.csv:
"service_id","year","period","cash_exp","cash_inc","cash_def","act_exp","act_inc","act_def","comm_exp","comm_inc","comm_def"
"S00000009","2016","M04 01/07/2016","0.00","0.00","0.00","0.00","0.00","0.00","750.00","0.00","0.00"
"S00000009","2016","M05 01/08/2016","0.00","0.00","0.00","0.00","0.00","0.00","600.00","0.00","0.00"
"S00000009","2016","M06 01/09/2016","0.00","0.00","0.00","0.00","0.00","0.00","600.00","0.00","0.00"
S00000010.csv:
"service_id","year","period","cash_exp","cash_inc","cash_def","act_exp","act_inc","act_def","comm_exp","comm_inc","comm_def"
"S00000010","2015","W28 05/10/2015","2275.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00"
"S00000010","2015","W41 04/01/2016","0.00","0.00","0.00","0.00","0.00","0.00","568.75","0.00","0.00"
"S00000010","2015","W42 11/01/2016","0.00","0.00","0.00","0.00","0.00","0.00","568.75","0.00","0.00"
It is generating the new files using the header value in column 1 (service_id).
There are 2 problems.
The output CSV file contains a header row which I don't need.
The columns are enclosed with double quotes which I don't need.
First of all the .csv file needs headers and the quote marks as a csv file structure. But if you don't want them then you can go on with a text file or...
$temp = Import-Csv INPUT_FILE.csv -Header service_id,year,period,cash_exp,cash_inc,cash_def,act_exp,act_inc,act_def,comm_exp,comm_inc,comm_def | Group-Object -Property "service_id" |
Foreach-Object {
$path=$_.name+".csv"
$temp0 = $_.group | ConvertTo-Csv -NoTypeInformation | Select-Object -Skip 1
$temp1 = $temp0.replace("""","")
$temp1 > $path
}
But this output is not a "real" csv file.
Hope that helps.
For your particular scenario you could probably use a simpler approach. Read the input file as a plain text file, group the lines by splitting off the first field, then write the groups to output files named after the groups:
Get-Content 'INPUT_FILE.csv' |
Group-Object { $_.Split(',')[0] } |
ForEach-Object { $_.Group | Set-Content ($_.Name + '.csv') }
Another solution,
using no named headers but simply numbers (as they aren't wanted in output anyway)
avoiding unneccessary temporary files.
removing only field delimiting double quotes.
Import-Csv INPUT_FILE.csv -Header (1..12) |
Group-Object -Property "1" | Foreach-Object {
($_.Group | ConvertTo-Csv -NoType | Select-Object -Skip 1).Trim('"') -replace '","',',' |
Set-Content -Path ("{0}.csv" -f $_.Name)
}

Get-Content | Select-String return top only row from resultset

Fetching top row only from below command:
$G = Get-Content $SourceFile | select-string -pattern $SearchKeyword2`**
Above command return multiple line numbers it has found in document, we want to select the top 1 .. returned. How can we apply filter to this to return only one top row.. like we can do in SQL Server
Use Select-Object -First 1:
$G = Get-Content $SourceFile | Select-String -Pattern $SearchKeyword2 | Select-Object -First 1