powershell script to change column position in text file - powershell

Do you know how to use powershell to move column DocumentNo to last column ?
| D|Ref.Doc. |Row|DocumentNo |CoCd|Pstng Date
| W|5007534739| 1|65713191 |STCD|01/17/2016
| W|5007534739| 1|65713191 |STCD|01/17/2016
| W|5007534739| 1|65713191 |STCD|01/17/2016
Expected output
| D|Ref.Doc. |Row|CoCd|Pstng Date|DocumentNo
| W|5007534739| 1|STCD|01/17/2016|65713191
| W|5007534739| 1|STCD|01/17/2016|65713191
| W|5007534739| 1|STCD|01/17/2016|65713191
Here the command that I've tried
(get-content $file -ReadCount 0) |
foreach {
'{0}|{1}|{2}|{3}|{5}|{6}|{4}' -f $_.split('|')
} | Set-Content $file2
The code works properly, but in case DocumentNo containt a pipe as data, how to handle it ?

A delimeter is no good if it's used inside the data itself as it breaks the csv. I see that the data is fixed width (though the length and index may change from file to file), so I would take a different apporach.
Find the location for the DocumentNo-column. You can use regex for this (https://regex101.com/r/pH2oL9/1). I use [Regex]::Match() because it returns index (start position) and length (number of characters) of a match (the column).
Create a regex that finds the content at the same position and length on every line. It makes groups for "before", "content", "after" by counting "any character" until the start position I've decided, then the length of the column, then "until end of line". I use subexpressions $() in the regex to insert the index and length from step 2 since they might not be the same for every file.
Use -replace with the generated regex to modify the text on each line (since $text is an array). -replace finds the groups, and by using $1$3$2 I can say which order I want the groups inserted in the result. (https://regex101.com/r/vE6vO9/1)
Solution:
#Sample text
$text = Get-Content .\Test.txt
#Analyze header (find DocumentNo placement in fixed-width file) and create regex
$regex = [regex]::Match($text[0], '\|DocumentNo\s+') | ForEach-Object { "^(.{$($_.Index)})(.{$($_.Length)})(.*)$" }
#Modify text
$text -replace $regex, '$1$3$2' | Set-Content .\TestOut.txt
TestOut.txt
| D|Ref.Doc. |Row|CoCd|Pstng Date|DocumentNo
| W|5007534739| 1|STCD|01/17/2016|65713191
| W|5007534739| 1|STCD|01/17/2016|65713191
| W|5007534739| 1|STCD|01/17/2016|65713191
You might want to trim the trailing whitespaces. Use Trim() for this:
$text -replace $regex, '$1$3$2' | ForEach-Object { $_.Trim() } | Set-Content .\TestOut.txt

You could do the following:
$delimiter = "|"
$data = Get-Content "c:\tmp\test.csv";
$newCsv= ($data|Foreach-object { ($_ -split $delimiter)[#(0..3;5;6;4)] -join $delimiter})
# Set the new-ordered column content to the new-file
$newCsv|Set-Content C:\tmp\test2.csv
If you want all of this into one line you can do the following:
Get-Content "c:\tmp\test.csv"|Foreach-object { ($_ -split '|')[#(0..3;5;6;4)] -join '|'}|Set-Content C:\tmp\test2.csv
Note that you cannot use set-content if you have get-content as a part of the pipeline, because the file will be open and cannot replace it's content as a part of the stream.
You'd have to use either the first approach (read the content first, then pass it to the stream) or set the content into a different file.

Related

Import 1 column of csv, replace string, then join each row into one row with values surrounded in quotes

Im using powershell and I have a csv file that has a ton of data but the only field I am concerned with is the first one.
I need to:
Extract the first column of data with the header: Ticket_ID
Remove a trailing string from the values in this column (site:)
Join all of the values in the column into a single, space delimited row, but ensure each value retains the double quotes wrapping each of them.
Prepend this row with a long string and write it to a file. (I havent tried to tackle this part yet because I cant get 1-3 to work fully.)
I have tried several permutations of this code:
$TT = (Import-Csv '.\sourcefile.csv').______Ticket_ID______ -join (" ")
$TT = $TT -replace 'site:',''
$TT | Set-Content -Path .\JulyTickets.csv
Example of my csv:
"______Ticket_ID______","____________Title____________","___________CreateDate___________"
"site:TICKET1","SOMETITLE","01/01/1901"
"site:TICKET2","SOMETITLE","01/01/1901"
"site:TICKET3","SOMETITLE","01/01/1901"
"site:TICKET4","SOMETITLE","01/01/1901"
"site:TICKET5","SOMETITLE","01/01/1901"
My frustrations:
If I use Export-CSV the resulting csv only contains the character length of the values.
If I use Set-Content, I can successfully extract the first column, and remove the leading string (site:), join all of the rows into a single row, and write this out to a file but it does not retain the double quotes around each of the values in the csv ("site:TICKET1"). And the output looks like this:
TICKET1 TICKET2 TICKET3 TICKET4 TICKET5
I'm not sure if I'm doing things out of order here, or if I'm missing something but I'm unable to get both the values inline, and individually surrounded in quotes.
Required Output:
this is a long string that needs to be prepended to the data "TICKET1" "TICKET2" "TICKET3" "TICKET4" "TICKET5"
This should do it. The need for pipes and PowerShell's Calculated Properties expression in the end is to overcome powershell removing double quotes
$string = 'this is a long string that needs to be prepended to the data ' + (Get-Content -Path \.sourcefile.csv | Convertfrom-Csv -Delimiter "," | select ______Ticket_ID______ -ExpandProperty ______Ticket_ID______ | ForEach-Object {$_ -replace 'site:', '';} | select #{ Name = 'Ticket'; Expression = {$([char]34) + $_ + $([char]34)}} | Select Ticket -ExpandProperty Ticket);
write-host $string
returns
this is a long string that needs to be prepended to the data "TICKET1" "TICKET2" "TICKET3" "TICKET4" "TICKET5"
Try this out, I'm using the -Header argument since the only needed column is the first one.
$csv = #'
"______Ticket_ID______","____________Title____________","___________CreateDate___________"
"site:TICKET1","SOMETITLE","01/01/1901"
"site:TICKET2","SOMETITLE","01/01/1901"
"site:TICKET3","SOMETITLE","01/01/1901"
"site:TICKET4","SOMETITLE","01/01/1901"
"site:TICKET5","SOMETITLE","01/01/1901"
'# |
ConvertFrom-Csv -Header ID |
Select-Object -Skip 1 |
ForEach-Object {
'"{0}"' -f $_.ID.TrimStart('site:')
}
$toPrepend = 'this is a long string that needs to be prepended to the data'
"$toPrepend $($csv -join ' ')"
# Above yields => this is a long string that needs to be prepended to the data "TICKET1" "TICKET2" "TICKET3" "TICKET4" "TICKET5"
Note, I'm only using ConvertFrom-Csv as an example here but in your case, it should look like this:
$TT = Import-Csv '.\sourcefile.csv' -Header ID | Select-Object -Skip 1 | ForEach-Object {
'"{0}"' -f $_.ID.TrimStart('site:')
}
$toPrepend = 'this is a long string that needs to be prepended to the data'
"$toPrepend $($TT -join ' ')" | Out-File JulyTickets.csv # => Not sure why CSV here, since this is no longer a CSV

How to add quotes and commas in PowerShell?

I have a CSV file where I only need 1 Column Called "SerialNumber" I need to combine the text lines, remove any blank space, add each line in quotes and separate by comma.
So far I have multiple miles of code that work, but it adds quotes at the end and doesn't add quotes in the beginning.
$SerialList = import-csv .\log.csv | select -ExpandProperty Serialnumber | Out-File -FilePath .\Process.txt
(gc process.txt) | ? {$_.trim() -ne "" } | set-content process.txt
gc .\process.txt | %{$_ -replace '$','","'} | out-file process1.csv
Get-Content .\process1.csv| foreach {
$out = $out + $_
}
$out| Out-File .\file2.txt
Output:
SerialNumber
1234
1234
4567
4567
Expected Output:
"1234","1234","4567","4567"
Try the following (PSv3+):
(Import-Csv .\log.csv).SerialNumber -replace '^.*$', '"$&"' -join "," > .\file2.txt
(Import-Csv .\log.csv).SerialNumber imports the CSV file and .SerialNumber uses member-access enumeration to extract the SerialNumber column values as an array.
-replace '^.*$', '"$&"' encloses each array element in "...".
Regex ^.*$ matches each array element in full.
Replacement expression "$&" replaces the element with what was matched ($&) enclosed in " chars. - for background, see this answer
-join "," joins the resulting array elements with , as the separator.

Powershell Remove spaces in the header only of a csv

First line of csv looks like this spaces are at after Path as well
author ,Revision ,Date ,SVNFolder ,Rev,Status,Path
I am trying to remove spaces only and rest of the content will be the same .
author,Revision,Date,SVNFolder,Rev,Status,Path
I tried below
Import-CSV .\script.csv | ForEach-Object {$_.Trimend()}
expanding on the comment with an example since it looks like you may be new:
$text = get-content .\script.csv
$text[0] = $text[0] -replace " ", ""
$csv = $text | ConvertFrom-CSV
Note: The solutions below avoid loading the entire CSV file into memory.
First, get the header row and fix it by removing all whitespace from it:
$header = (Get-Content -TotalCount 1 .\script.csv) -replace '\s+'
If you want to rewrite the CSV file to fix its header problem:
# Write the corrected header and the remaining lines to the output file.
# Note: I'm outputting to a *new* file, to be safe.
# If the file fits into memory as a whole, you can enclose
# Get-Content ... | Select-Object ... in (...) and write back to the
# input file, but note that there's a small risk of data loss, if
# writing back gets interrupted.
& { $header; Get-Content .\script.csv | Select-Object -Skip 1 } |
Set-content -Encoding utf8 .\fixed.csv
Note: I've chosen -Encoding utf8 as the example output character encoding; adjust as needed; note that the default is ASCII(!), which can result in data loss.
If you just want to import the CSV using the fixed headers:
& { $header; Get-Content .\script.csv | Select-Object -Skip 1 } | ConvertFrom-Csv
As for what you tried:
Import-Csv uses the column names in the header as property names of the custom objects it constructs from the input rows.
This property names are locked in at the time of reading the file, and cannot be changed later - unless you explicitly construct new custom objects from the old ones with the property names trimmed.
Import-Csv ... | ForEach-Object {$_.Trimend()}
Since Import-Csv outputs [pscustomobject] instances, reflected one by one in $_ in the ForEach-Object block, your code tries call .TrimEnd() directly on them, which will fail (because it is only [string] instances that have such a method).
Aside from that, as stated, your goal is to trim the property names of these objects, and that cannot be done without constructing new objects.
Read the whole file into an array:
$a = Get-Content test.txt
Replace the spaces in the first array element ([0]) with empty strings:
$a[0] = $a[0] -replace " ", ""
Write over the original file: (Don't forget backups!)
$a | Set-Content test.txt
$inFilePath = "C:\temp\headerwithspaces.csv"
$content = Get-Content $inFilePath
$csvColumnNames = ($content | Select-Object -First 1) -Replace '\s',''
$csvColumnNames = $csvColumnNames -Replace '\s',''
$remainingFile = ($content | Select-Object -Skip 1)

How to extract lines between index 1 and index 2 to another csv variable?

Instead of outputting to a file with Set-Content like in How to remove First and Last Line in Powershell
$csv = Import-Csv in.csv -header Date,Time,O,H,L,C,V |
Select * -ExcludeProperty time |
Foreach {$_.date = [datetime]::ParseExact($_.date,"yyyy.MM.dd",$null).tostring("yyMMdd");$_.v=1;$_} |
ConvertTo-Csv -NoTypeInformation
for ($i = 1; $i -lt ($csv.Length - 1); $i++ {
$csv[$i] -replace '"' | Set-Content out.csv -encoding ascii
}
I just want to put these lines in $csv2 var instead of out.csv.
Set-Content does not work with var. How to do so (I don't have Powershell 5 ) ?
The issue with the code is that every time you call Set-Content it will rewrite the file and replace any content the file already have.
Consider adding the -append switch to Set-Content. This will add to the file instead of overwriting it. Remember to also make sure the file is empty befor you begin Writing to it.
I would also consider using a more simple way of getting the "mid" content of the file. Check the following sample. It might not cover all of your requirements, but is a simple way of getting Everything in an array except, the first and last element using the Range operator.
# First setup the test data
$filecontent = #"
Line 1 skip please
Line 2 include
Line 3 include
Line 4 include
Line 5 include
Line 6 include
Line 7 include
Line 8 include
Line 9 include
Line 10 skip please
"#
$filecontent | Set-Content in.csv
$content = Get-Content in.csv
$content[-($content.Length-1)..-2] | Set-Content out.csv
You are almost there. Instead of piping to Set-Content, just assign it to your variable. (BTW your original code was missing a closing parentheses on the for loop, which I have corrected here as well.)
$csv = Import-Csv in.csv -header Date,Time,O,H,L,C,V |
Select * -ExcludeProperty time |
Foreach {$_.date = [datetime]::ParseExact($_.date,"yyyy.MM.dd",$null).tostring("yyMMdd");$_.v=1;$_} |
ConvertTo-Csv -NoTypeInformation
$csv2 = for ($i = 1; $i -lt ($csv.Length - 1); $i++) {
$csv[$i] -replace '"'
}

How to change column position in powershell?

Is there any easy way how to change column position? I'm looking for a way how to move column 1 from the beginning to the and of each row and also I would like to add zero column as a second last column. Please see txt file example below.
Thank you for any suggestions.
File sample
TEXT1,02/10/2015,55.930,57.005,55.600,56.890,1890
TEXT2,02/10/2015,51.060,52.620,50.850,52.510,4935
TEXT3,02/10/2015,50.014,50.74,55.55,52.55,5551
Output:
02/10/2015,55.930,57.005,55.600,56.890,1890,0,TEXT1
02/10/2015,51.060,52.620,50.850,52.510,4935,0,TEXT2
02/10/2015,50.014,50.74,55.55,52.55,5551,0,TEXT3
Another option:
#Prepare test file
(#'
TEXT1,02/10/2015,55.930,57.005,55.600,56.890,1890
TEXT2,02/10/2015,51.060,52.620,50.850,52.510,4935
TEXT3,02/10/2015,50.014,50.74,55.55,52.55,5551
'#).split("`n") |
foreach {$_.trim()} |
sc testfile.txt
#Script starts here
$file = 'testfile.txt'
(get-content $file -ReadCount 0) |
foreach {
'{1},{2},{3},{4},{5},{6},0,{0}' -f $_.split(',')
} | Set-Content $file
#End of script
#show results
get-content $file
02/10/2015,55.930,57.005,55.600,56.890,1890,0,TEXT1
02/10/2015,51.060,52.620,50.850,52.510,4935,0,TEXT2
02/10/2015,50.014,50.74,55.55,52.55,5551,0,TEXT3
Sure, split on commas, spit the results back minus the first result joined by commas, add a 0, and then add the first result to the end and join the whole thing with commas. Something like:
$Input = #"
TEXT1,02/10/2015,55.930,57.005,55.600,56.890,1890
TEXT2,02/10/2015,51.060,52.620,50.850,52.510,4935
TEXT3,02/10/2015,50.014,50.74,55.55,52.55,5551
"# -split "`n"|ForEach{$_.trim()}
$Input|ForEach{
$split = $_.split(',')
($Split[1..($split.count-1)]-join ','),0,$split[0] -join ','
}
I created file test.txt to contain your sample data. I Assigned each field a name, "one","two","three" etc so that i could select them by name, then just selected and exported back to csv in the order you wanted.
First, add the zero to the end, it will end up as second last.
gc .\test.txt | %{ "$_,0" } | Out-File test1.txt
Then, rearrange order.
Import-Csv .\test.txt -Header "one","two","three","four","five","six","seven","eight" | Select-Object -Property two,three,four,five,six,seven,eight,one | Export-Csv test2.txt -NoTypeInformation
This will take the output file and get rid of quotes and header line if you would rather not have them.
gc .\test2.txt | %{ $_.replace('"','')} | Select-Object -Skip 1 | out-file test3.txt