Transpose specific rows into columns using PowerShell - powershell

Question

So I have a CSV file with rows that I want to transpose (some of them) into columns using PowerShell.
The example is as follows:
ALPHA
CD
CL
CM
-5
0.1
-0.2
0.05
0
0.4
0.4
-0.08
5
0.5
0.8
-0.1
What I want is something like this:
Alpha CD CL CM
-5 0.1 -0.2 0.05
0 0.4 0.4 -0.08
5 0.5 0.8 -0.1
For reference I got these values from a .dat data file output with over 400 rows full of information. I reformatted it into a CSV file using out-file, and I skipped all the rows I don't need.
The information was split into rows but not columns, meaning ALPHA CD CL CM were all in one cell with spaces in between, so I used the split command as shown below to break them into rows.
$ text .split () | where { $ _ }
Now I want to transpose SOME of them back into columns.
The problem is it's not fixed amounts, meaning it's not always four rows into four columns, sometimes I would get five rows that I want to turn into five columns, and THEN turn every four rows into four columns AFTER that.
Sorry if I'm rambling but it's something like this:
Row 1 > Column 1 Row 1
Row 2 > Column 2 Row 1
Row 3 > Column 3 Row 1
Row 4 > Column 4 Row 1
Row 5 > Column 5 Row 1
Row 6 > Column 1 Row 2
Row 7 > Column 2 Row 2
Row 8 > Column 3 Row 2
Row 9 > Column 4 Row 2
Row 10 > Column 5 Row 2
Row 11 > Column 1 Row 3
Row 12 > Column 2 Row 3
Row 13 > Column 3 Row 3
Row 14 > Column 4 Row 3
Row 15 > Column 1 Row 4
Please notice how it went from five columns to four columns now.
If it can be done easier in other methods other than PowerShell where I can use PowerShell to run them, i.e. a batch file that calls PowerShell, that would be good by me as I need to automate a very long process, and this is one of the later process steps.
PS: The data are NOT comma separated cleanly. The used program DATCOM outputs a data file where it looks neat and structured in text format, but when you export CSV it destroys it, so it has to be done using:
out-file name csv
PPS: There is no clear delimiter/cutoff point, and there are no repeating numbers or anything else that can be used as a hint. I have to do it by row number, which I know due to dealing with DATCOM before.
I explained more above, but I tried using split commands. It dropped them all into rows. So if there is a way that can just do a literal text to columns delimit using spaces (exactly like in Excel) that would be perfect, and even better than breaking them into rows then transposing to columns. However, it has to be EXACTLY like Excel. The problem is there are 4-8 "spaces" between each value, so if I try to
import-csv -delim " "
on the file I get something like Alpha H1 H2 H3 CD H4 H5 H6 H7 H8 CL and everything else gets destroyed, whereas if I actually open Excel, highlight cells, text to columns > delimited > check "spaces" the results are perfect.
Here are the files: https://easyupload.io/m/6q70ei
for006.dat is the data file generated by DATCOM.
Output1 is what I want done as described above (row to column).
Output2 is what I hope I can do later, i.e. delete a column and a row to make it cleaner, this is my ideal final output.

Answer 1

Mmm... I am afraid your description is pretty confusing, so I forgot it and focused in your files...
The Batch file below read the for006.dat file and generate your "ideal final output" Output2.xlsx file in .csv form.
#echo off
setlocal EnableDelayedExpansion
set "skip="
set "lines="
for /F "delims=:" %%a in ('findstr /N /L /C:" ALPHA" for006.dat') do (
if not defined skip (
set /A "skip=%%a-1"
) else if not defined lines (
set /A "lines=%%a-skip-1"
)
)
< for006.dat (
for /L %%a in (1,1,%skip%) do set /P "="
for /L %%a in (1,1,%lines%) do (
set /P "line="
set "line=!line:~2!"
if defined line call :reformat
)
) > Output2.csv
goto :EOF
:reformat
set "newLine=%line: = %"
if "%newLine%" == "%line%" goto continue
set "line=%newLine%"
goto reformat
:continue
if "%line:~0,1%" == " " set "line=%line:~1%"
if "%line:~-1%" == " " set "line=%line:~0,-1%"
echo "%line: =","%"
This is Output2.csv:
"ALPHA","CD","CL","CM","CN","CA","XCP","CLA","CMA","CYB","CNB","CLB"
"-6.0","0.013","-0.175","0.2807","-0.176","-0.006","-1.599","3.100E+00","-3.580E+00","-5.643E-02","-3.080E-03","-8.679E-02"
"-3.0","0.011","-0.011","0.0926","-0.012","0.010","-7.977","3.172E+00","-3.626E+00","-8.989E-02"
"0.0","0.013","0.157","-0.0990","0.157","0.013","-0.631","3.286E+00","-3.740E+00","-9.305E-02"
"3.0","0.019","0.333","-0.2991","0.334","0.001","-0.897","3.426E+00","-3.901E+00","-9.635E-02"
"6.0","0.029","0.516","-0.5075","0.516","-0.025","-0.984","3.529E+00","-4.084E+00","-9.979E-02"
"7.5","0.036","0.609","-0.6158","0.608","-0.044","-1.013","3.472E+00","-4.002E+00","-1.015E-01"
"9.0","0.043","0.698","-0.7171","0.696","-0.067","-1.031","3.218E+00","-3.679E+00","-1.032E-01"
"10.0","0.047","0.752","-0.7791","0.748","-0.084","-1.041","2.895E+00","-3.489E+00","-1.042E-01"
"11.0","0.051","0.799","-0.8388","0.794","-0.102","-1.057","2.572E+00","-3.345E+00","-1.051E-01"
"12.0","0.055","0.841","-0.8958","0.835","-0.121","-1.073","2.320E+00","-3.178E+00","-1.059E-01"
"13.0","0.059","0.880","-0.9498","0.870","-0.140","-1.091","2.041E+00","-2.983E+00","-1.066E-01"
"14.0","0.063","0.913","-0.9999","0.901","-0.160","-1.110","1.738E+00","-2.772E+00","-1.072E-01"
"15.0","0.066","0.940","-1.0465","0.925","-0.180","-1.131","1.356E+00","-2.567E+00","-1.077E-01"
"16.0","0.067","0.960","NA","0.941","-0.201","NA","1.798E-02","NA","-1.081E-01"
"18.0","0.055","0.883","NA","0.857","-0.220","NA","-4.434E+00","NA","-1.066E-01"
You can also generate the .csv output file with no quotes by just removing the quotes from the last echo command

Answer 2

Try following :
$columns = 4
$data =
"#ALPHA
CD
CL
CM
-5
0.1
-0.2
0.05
0
0.4
0.4
-0.08
5
0.5
0.8
-0.1#"
$data | Format-Table
$headers = [System.Collections.ArrayList]::new()
$table = [System.Collections.ArrayList]::new()
$rows = [System.IO.StringReader]::new($data)
for($i = 0; $i -lt $columns; $i++)
{
$headers.Add($rows.ReadLine())
}
$rowCount = 0
Write-Host $headers
While(($line = $rows.ReadLine()) -ne $null)
{
if($rowCount % $columns -eq 0)
{
$newRow = New-Object -TypeName psobject
$table.Add($newRow)
}
$newRow | Add-Member -NotePropertyName $headers[$rowCount % $columns] -NotePropertyValue $line
$rowCount++
}
$table | Format-Table

Answer 3

You can use PowerShell's Begin/Process/End lifecycle to "buffer" input data until you have enough for a "row", then output that and start collecting for the next row:
# define width of each row as well as the column separator
$columnCount = 5
$delimiter = "`t"
# read in the file contents, "cell-by-cell"
$rawCSVData = Get-Content path\to\input\file.txt |ForEach-Object -Begin {
# set up a buffer to hold 1 row at a time
$index = 0
$buffer = [psobject[]]::new($columnCount)
} -Process {
# add input to buffer, and optionally output
$buffer[$index++] = $_
if($index -eq $columnCount){
# output row, reset column index
$buffer -join $delimiter
$index = 0
}
} -End {
# Output any partial last row
if($index){
$buffer -join $delimiter
}
}
This will produce a list of strings that can either be written to disk or parsed using regular CSV-parsing tools in PowerShell:
$rawCSVData |Set-Content path\to\output.csv
# or
$rawCSVData |ConvertFrom-Csv -Delimiter $delimiter

Answer 4

Once you know how many rows form the headers and the data, you can convert the file into
an array of objects by using ConvertFrom-Csv.
When done, it is easy to create a new csv from this as below:
# in this example the first 4 lines are the columns, the rest is data
# other files may need a different number of columns
$columns = 4
$data = #(Get-Content -Path 'X:\Somewhere\data.txt')
$count = 0
$result = while ($count -lt ($data.Count - ($columns - 1))) {
$data[$count..($count + $columns - 1)] -join "`t" # join the lines with a TAB
$count += $columns
}
$result = $result | ConvertFrom-Csv -Delimiter "`t"
# output on screen
$result | Format-Table -AutoSize
# write to new csv file
$result | Export-Csv -Path 'X:\Somewhere\data_new.csv' -NoTypeInformation
Output on screen:
ALPHA CD CL CM
----- -- -- --
-5 0.1 -0.2 0.05
0 0.4 0.4 -0.08
5 0.5 0.8 -0.1

Answer 5

Two (custom) functions that might help you to scrape your data from the for006.dat:
SelectString -From -To (see: #15136 Add -From and -To parameters to Select-String)
ConvertFrom-SourceTable
Get-Content .\for006.dat |
SelectString -From '(?=0 ALPHA CD CL.*)' -To '^0.+' |
ForEach-Object { $_.SubString(1) } |
ConvertFrom-SourceTable |
ConvertTo-Csv
Results:
"ALPHA","CD","CL","CM","CN","CA","XCP","CLA","CMA","CYB","CNB","CLB"
"-6","0.013","-0.175","0.2807","-0.176","-0.006","-1.599","3.100E+00","-3.580E+00","-5.643E-02","-3.080E-03","-8.679E-02"
"-3","0.011","-0.011","0.0926","-0.012","0.01","-7.977","3.172E+00","-3.626E+00","","","-8.989E-02"
"0","0.013","0.157","-0.0990","0.157","0.013","-0.631","3.286E+00","-3.740E+00","","","-9.305E-02"
"3","0.019","0.333","-0.2991","0.334","0.001","-0.897","3.426E+00","-3.901E+00","","","-9.635E-02"
"6","0.029","0.516","-0.5075","0.516","-0.025","-0.984","3.529E+00","-4.084E+00","","","-9.979E-02"
"7.5","0.036","0.609","-0.6158","0.608","-0.044","-1.013","3.472E+00","-4.002E+00","","","-1.015E-01"
"9","0.043","0.698","-0.7171","0.696","-0.067","-1.031","3.218E+00","-3.679E+00","","","-1.032E-01"
"10","0.047","0.752","-0.7791","0.748","-0.084","-1.041","2.895E+00","-3.489E+00","","","-1.042E-01"
"11","0.051","0.799","-0.8388","0.794","-0.102","-1.057","2.572E+00","-3.345E+00","","","-1.051E-01"
"12","0.055","0.841","-0.8958","0.835","-0.121","-1.073","2.320E+00","-3.178E+00","","","-1.059E-01"
"13","0.059","0.88","-0.9498","0.87","-0.140","-1.091","2.041E+00","-2.983E+00","","","-1.066E-01"
"14","0.063","0.913","-0.9999","0.901","-0.160","-1.110","1.738E+00","-2.772E+00","","","-1.072E-01"
"15","0.066","0.94","-1.0465","0.925","-0.180","-1.131","1.356E+00","-2.567E+00","","","-1.077E-01"
"16","0.067","0.96","NA","0.941","-0.201","NA","1.798E-02","NA","","","-1.081E-01"
"18","0.055","0.883","NA","0.857","-0.220","NA","-4.434E+00","NA","","","-1.066E-01"