Convert txt file to csv and ignore specific text Line - powershell

$userObj = [PSCustomObject]((Get-Content -Raw C:\Automation\sam.txt) -replace ':', '=' | ConvertFrom-StringData)
$name = Get-Item C:\Automation\sam.txt | Select-Object -ExpandProperty BaseName
$userObj | Export-Csv C:\Automation\$name.csv
I am using the above script to convert txt file to CSV but I don't know how to exclude word Line with underlined characters .
The script works if I delete "Line ---- " and try.


Skip the first 2 lines when importing the file:
$data = (Get-Content 'C:\path\to\input.txt') -replace ':', '=' |
Select-Object -Skip 2 |
Out-String |
ConvertFrom-StringData
[PSCustomObject]$data | Export-Csv 'C:\path\to\output.csv' -NoType
Better yet, change the process that creates your input file, so that it doesn't produce the offending header. The data seems to be generated by something like
... | Select-String ... | Select-Object Line | Set-Content ...
To remove the header from the output you just need to change Select-Object Line to Select-Object -Expand Line.

Related

Powershell adds quotes to splitted CSV file

I'm trying to split a csv file by the first digits of the longitude column. Here is a sample:
X,Y,TYPE,SPEED,DirType,Direction
-44.058251,-19.945982,1,30,1,339
-54.629503,-20.497509,1,30,1,263
-54.646202,-20.496151,1,30,1,86
I have no powershell knowledge but I found some script online and it did what I wanted:
Import-Csv maparadar.csv
| Group-Object -Property {($_.x)[0..2] -join ""}
| Foreach-Object {$path=$_.name+".csv" ; $_.group
| Export-Csv -Path $path -NoTypeInformation}
With this I get output files like -44.csv, -54.csv
But it adds unwanted quotes to every field in the output file like:
"X","Y","TYPE","SPEED","DirType","Direction"
"-46.521991","-23.690235","1","30","1","169"
"-46.670774","-23.756021","1","30","1","281"
"-46.549897","-23.120720","1","30","1","99"
Is there any way I can export the csv without adding those quotes?

The following should provide the desired output:
Import-Csv maparadar.csv |
Group-Object -Property {($_.x)[0..2] -join ""} |
Foreach-Object { $path=$_.name+".csv" ; ($_.group |
ConvertTo-Csv -NoTypeInformation) -Replace '"' |
Set-Content -Path $path }
Explanation:
We replaced your Export-Csv with ConvertTo-Csv, which provides the CSV output to the console/pipeline rather than outputting to the file. Those CSV formatted outputs are sent through the -Replace operator to replace the literal " characters. Finally the formatted output is sent to the desired file using Set-Content -Path $path.

Blank first line when using "select-string -pattern" to strip lines from file

I have a simple text file that looks like this...
A,400000051115,null,null,null,null,null,null,null,20190312,090300,Answer Machine,2019,3,14,10,0
A,400000051117,null,null,null,null,null,null,null,20190312,090300,Confirmed,2019,3,14,10,30
A,400000051116,null,null,null,null,null,null,null,20190312,090300,Answer Machine,2019,3,14,11,0
A,400000051114,null,null,null,null,null,null,null,20190312,090300,Wants to Cancel,2019,3,14,9,0
A,400000051117,null,null,null,null,null,null,null,20190312,091800,SMS Sent,2019,3,14,10,30
A,400000051116,null,null,null,null,null,null,null,20190312,091800,SMS Sent,2019,3,14,11,0
A,400000051115,null,null,null,null,null,null,null,20190312,091800,SMS Sent,2019,3,14,10,0
A,400000051116,null,null,null,null,null,null,null,20190312,093000,Appointment Cancelled/Rescheduled Via SMS,2019,3,14,11,0
I need to save all the lines except those that have "SMS Sent" in them to a new file. I am using the following...
get-content $SourceFile.FullName | select-string -pattern 'SMS Sent' -notmatch | Out-File $targetFile
Why in the resulting file do I get a blank first line?

If you change Out-File $targetFile to Out-Host or even just omit that last segment in the pipeline, you will see a blank line in the console output, too.
The output analog of Get-Content is Set-Content, so if you change Out-File $targetFile to Set-Content $targetFile the first line is no longer blank.
Also, since you're working with a CSV file you could use Import-CSV to read the data and Where-Object to filter on that specific column, although a little extra work is required to specify the headers and omit them from the output file...
$csvHeaders = 1..17 | ForEach-Object -Process { "Column $_" }
$csvHeaders[11] = 'Status'
Import-Csv -Path $SourceFile.FullName -Header $csvHeaders `
| Where-Object -Property 'Status' -NE -Value 'SMS Sent' `
| ConvertTo-Csv -NoTypeInformation `
| Select-Object -Skip 1 `
| Set-Content $targetFile
...which writes...
"A","400000051115","null","null","null","null","null","null","null","20190312","090300","Answer Machine","2019","3","14","10","0"
"A","400000051117","null","null","null","null","null","null","null","20190312","090300","Confirmed","2019","3","14","10","30"
"A","400000051116","null","null","null","null","null","null","null","20190312","090300","Answer Machine","2019","3","14","11","0"
"A","400000051114","null","null","null","null","null","null","null","20190312","090300","Wants to Cancel","2019","3","14","9","0"
"A","400000051116","null","null","null","null","null","null","null","20190312","093000","Appointment Cancelled/Rescheduled Via SMS","2019","3","14","11","0"
...to $targetFile. Note that all of the values are quoted now. If your input file does have headers then you could use simply...
Import-Csv -Path $SourceFile.FullName `
| Where-Object -Property 'Status' -NE -Value 'SMS Sent' `
| Export-Csv -NoTypeInformation -LiteralPath $targetFile
In either case the output file will not contain a leading blank line.

Export-Csv adding unwanted header double quotes

I have got a source CSV file (without a header, all columns delimited by a comma) which I am trying split out into separate CSV files based upon the value in the first column and using that column value as the output file name.
Input file:
S00000009,2016,M04 01/07/2016,0.00,0.00,0.00,0.00,0.00,0.00,750.00,0.00,0.00
S00000009,2016,M05 01/08/2016,0.00,0.00,0.00,0.00,0.00,0.00,600.00,0.00,0.00
S00000009,2016,M06 01/09/2016,0.00,0.00,0.00,0.00,0.00,0.00,600.00,0.00,0.00
S00000010,2015,W28 05/10/2015,2275.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00
S00000010,2015,W41 04/01/2016,0.00,0.00,0.00,0.00,0.00,0.00,568.75,0.00,0.00
S00000010,2015,W42 11/01/2016,0.00,0.00,0.00,0.00,0.00,0.00,568.75,0.00,0.00
S00000012,2015,W10 01/06/2015,0.00,0.00,0.00,0.00,0.00,0.00,650.00,0.00,0.00
S00000012,2015,W11 08/06/2015,0.00,0.00,0.00,0.00,0.00,0.00,650.00,0.00,0.00
S00000012,2015,W12 15/06/2015,0.00,0.00,0.00,0.00,0.00,0.00,650.00,0.00,0.00
My PowerShell script looks like this:
Import-Csv INPUT_FILE.csv -Header service_id,year,period,cash_exp,cash_inc,cash_def,act_exp,act_inc,act_def,comm_exp,comm_inc,comm_def |
Group-Object -Property "service_id" |
Foreach-Object {
$path = $_.Name + ".csv";
$_.group | Export-Csv -Path $path -NoTypeInformation
}
Output files:
S00000009.csv:
"service_id","year","period","cash_exp","cash_inc","cash_def","act_exp","act_inc","act_def","comm_exp","comm_inc","comm_def"
"S00000009","2016","M04 01/07/2016","0.00","0.00","0.00","0.00","0.00","0.00","750.00","0.00","0.00"
"S00000009","2016","M05 01/08/2016","0.00","0.00","0.00","0.00","0.00","0.00","600.00","0.00","0.00"
"S00000009","2016","M06 01/09/2016","0.00","0.00","0.00","0.00","0.00","0.00","600.00","0.00","0.00"
S00000010.csv:
"service_id","year","period","cash_exp","cash_inc","cash_def","act_exp","act_inc","act_def","comm_exp","comm_inc","comm_def"
"S00000010","2015","W28 05/10/2015","2275.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00"
"S00000010","2015","W41 04/01/2016","0.00","0.00","0.00","0.00","0.00","0.00","568.75","0.00","0.00"
"S00000010","2015","W42 11/01/2016","0.00","0.00","0.00","0.00","0.00","0.00","568.75","0.00","0.00"
It is generating the new files using the header value in column 1 (service_id).
There are 2 problems.
The output CSV file contains a header row which I don't need.
The columns are enclosed with double quotes which I don't need.

First of all the .csv file needs headers and the quote marks as a csv file structure. But if you don't want them then you can go on with a text file or...
$temp = Import-Csv INPUT_FILE.csv -Header service_id,year,period,cash_exp,cash_inc,cash_def,act_exp,act_inc,act_def,comm_exp,comm_inc,comm_def | Group-Object -Property "service_id" |
Foreach-Object {
$path=$_.name+".csv"
$temp0 = $_.group | ConvertTo-Csv -NoTypeInformation | Select-Object -Skip 1
$temp1 = $temp0.replace("""","")
$temp1 > $path
}
But this output is not a "real" csv file.
Hope that helps.

For your particular scenario you could probably use a simpler approach. Read the input file as a plain text file, group the lines by splitting off the first field, then write the groups to output files named after the groups:
Get-Content 'INPUT_FILE.csv' |
Group-Object { $_.Split(',')[0] } |
ForEach-Object { $_.Group | Set-Content ($_.Name + '.csv') }

Another solution,
using no named headers but simply numbers (as they aren't wanted in output anyway)
avoiding unneccessary temporary files.
removing only field delimiting double quotes.
Import-Csv INPUT_FILE.csv -Header (1..12) |
Group-Object -Property "1" | Foreach-Object {
($_.Group | ConvertTo-Csv -NoType | Select-Object -Skip 1).Trim('"') -replace '","',',' |
Set-Content -Path ("{0}.csv" -f $_.Name)
}

Spliting a csv file based on the value of one column

I have a csv file that I have to split based on the value of a column.
I'm using the following script to do so:
Import-Csv test.csv | Group-Object -Property "Nr dep" |
Foreach-Object {$path=$_.name+".csv" ; $_.group |
Export-Csv -Path E:\PowerShell\script\$path -NoTypeInformation}
The file is split to files based on the Nr dep value column but with quotes and it works only with comma delimited csv files.
I tried to use -replace but still no result (maybe I'am writing it in a bad manner)
Import-Csv test.csv | Group-Object -Property "Nr dep" |
Foreach-Object {$path=$_.name+".csv" ; ($_.group |
ConvertTo-Csv -NoTypeInformation) -replace '"', "" | Out-File E:\PowerShell\script\$path -Force}
1) How can I make the delimitation a semi comma instead of a comma
2) how can I get rid of the quotes
3) is it possible to have an .xlsx outfile instead of a .csv file

1) You can specify the delimiter by using the -Delimiter parameter:
Import-Csv test.csv -Delimiter ';'
2) By using -replace '"' which you already do.
3) You will need a framework // application for that.

Export-Csv emits Length and not values

I want to read a CSV file and output a CSV file with only one (1) field. I have tried to create a concise example.
PS C:\src\powershell> Get-Content .\t.csv
field1,field2,field3
1,2,3
4,55,6
7,888,9
PS C:\src\powershell> Import-Csv -Path .\t.csv | `
>> ForEach-Object {
>> $_.field2 `
>> } | `
>> Export-Csv -Path .\x.csv -NoTypeInformation
>>
The problem is that the Length of field2 is written to the exported CSV file. I want the field header to be "field2" and the values to be the value from the original CSV file. Also, I only want quotes where they are required; not everywhere.
I have read Export-CSV exports length but not name and Export to CSV only returning string length. But these do not seem to address producing an actual CSV file with a header and one field value.
PS C:\src\powershell> get-content .\x.csv
"Length"
"1"
"2"
"3"

CSV object uses note properties in each row to store its fields so we'll need to filter each row object and leave just the field(s) we want using Select-Object cmdlet (alias: select), which processes the entire CSV object at once:
Import-Csv 1.csv | select field2 | Export-Csv 2.csv -NoTypeInformation
Note, there's no need to escape the end of line if it ends with |, {, (, or ,.
It's possible to specify several fields: select field2, field3.
To strip unneeded doublequotes, general multi-field case:
Import-Csv 1.csv |
select field2 |
%{
$_.PSObject.Properties | %{ $_.value = $_.value -replace '"', [char]1 }
$_
} |
ConvertTo-Csv -NoTypeInformation |
%{ $_ -replace '"(\S*?)"', '$1' -replace '\x01', '""' } |
Out-File 2.csv -Encoding ascii
Simplified one-field case:
Import-Csv 1.csv |
select field2 |
%{
$_.field2 = $_.field2 -replace '"', [char]1
$_
} |
ConvertTo-Csv -NoTypeInformation |
%{ $_ -replace '"(\S*?)"', '$1' -replace '\x01', '""' } |
Out-File 2.csv -Encoding ascii
A tricky case of embedded quotes inside a field was solved by temporary replacing them with a control character code 01 (there are just a few that can be used in a typical non-broken text file: 09/tab, 0A/line feed, 0D/carriage return).

As per WOxxOm's response, Select-Object is best way to select only field from an input and pipe to output.
Regarding the quote marks, this is a known (and frustrating) issue with PowerShell. Specifying , as the delimiter did not help.
I have gotten round it by using ConvertTo-Csv and Foreach-Object replacements. THe replacements will need to be more complex if your data contains quote marks.
Import-Csv .\1.csv |
Select-Object field2 |
ConvertTo-Csv -NoTypeInformation |
ForEach-Object {$_ -replace '"',''} |
Out-File .\2.csv